Research & Think
Research Keywords, Genres & Fields
The themes covered in this lab span a wide range of areas related to human senses and sensibility. Below are the main keywords and their meanings.
The Five Senses and Cross-Sensory Interaction
We normally perceive the world through the five senses: vision, hearing, touch, smell, and taste. This lab focuses particularly on phenomena where multiple senses influence each other.
- Crossmodal Effects --- A general term for various perceptual phenomena caused by interactions between the five senses. It refers to how information from one sensory modality (e.g., vision) affects the perception of another modality (e.g., touch).
- Multimodal --- Presenting or utilizing multiple senses simultaneously. For example, movies are a multimodal experience combining vision and hearing. Brain imaging research has confirmed that presenting multiple senses simultaneously produces stronger brain activity (superadditive activation) than the sum of individual presentations.
Crossmodal Effects in Everyday Life
Crossmodal effects are actually present everywhere in daily life:
- Shaved ice --- The syrups actually taste nearly the same, but due to different food coloring, we perceive distinct flavors like "strawberry" or "melon" (vision -> taste)
- Train illusion (Vection) --- When a train on the adjacent platform starts moving, you feel as if your own train is moving (vision -> somatosensory)
- Bouba-Kiki effect --- People tend to associate round shapes with the sound "bouba" and spiky shapes with "kiki." This is observed universally across cultures and languages (vision -> hearing)
Visual-Auditory Crossmodal Effects
| Effect | Description |
|---|---|
| Synesthesia | A phenomenon where a sensory stimulus automatically triggers another sense, such as seeing colors when hearing sounds or perceiving colors on letters. It occurs in approximately 1 in 200 to 25,000 people |
| McGurk effect | When visual lip movements and actual audio are mismatched, the listener perceives a different phoneme influenced by the visual information |
| Ventriloquist effect | When a puppet moves its mouth, the voice appears to come from the puppet's mouth. Sound localization is drawn toward the visual information |
Visual-Tactile Crossmodal Effects
| Effect | Description |
|---|---|
| Size-weight illusion (Charpentier effect) | Even at the same weight, smaller-looking objects feel heavier |
| Rubber hand illusion | When a visible rubber hand and a hidden real hand are simultaneously stroked with a brush, the person begins to feel the rubber hand as their own |
| Pseudo-haptics | Simply slowing down a mouse cursor creates a sensation of "resistance" or "heaviness" in the operation, even though no physical force is applied |
Other Crossmodal Effects
- Vision x Smell --- When white wine is dyed red, even wine experts evaluate it as having a red wine aroma
- Hearing x Taste --- When the crunching sound of potato chips is picked up by a microphone and played back louder, the "crispness" increases; when played back with a slight delay, the "crunchiness" increases
Applied Examples of Crossmodal Effects
| Example | Field | Description |
|---|---|---|
| Meta Cookie | Food | A system that changes the perceived flavor of the same cookie by altering AR (visual) and scent (olfactory) information |
| Chewing Jockey | Food | A system that changes perceived food texture by linking sound effects to chewing actions |
| BrainPort | Assistive Tech | A device that converts camera images into electrical stimulation patterns on the tongue, enabling visually impaired people to "see with their tongue" |
| VEST | Assistive Tech | A vest-type device that converts sound into vibration patterns, enabling hearing-impaired people to "feel sound with their body" |
Affective Information and Impressions
- Affective information (Kansei) --- Subjective impressions and feelings that people receive from objects or environments, such as "cute," "luxurious," or "calming." Difficult to quantify but extremely important in design and art.
- Impression --- The sensory and emotional response when encountering a subject. Examples include "rough," "comfortable," etc.
- Preference --- The intuitive feeling that something is "good," "fitting," or "appropriate."
Enactivism
- Enactivism --- The idea that "perception is not the passive reception of information, but emerges only through one's own actions." For example, you cannot know the weight of a cup without lifting it. In other words, action gives rise to perception, and perception gives rise to the next action --- this cyclical process is the essence of perception. In interactive art, this provides the theoretical foundation for the idea that the meaning of a work only emerges when the experiencer "touches" or "moves" it.
Academic Fields
- Affective Informatics (Kansei Engineering) --- A discipline that treats human sensibility (impressions, preferences, aesthetic sense, etc.) as data and applies it to engineering and design. It scientifically explores questions like "Why does this color combination feel pleasant?"
- Perceptual Psychology --- A branch of psychology that studies how humans perceive and recognize information from the external world. It addresses questions like "Why does the same sound seem different depending on the situation?"
Art Genres
- Sound Art --- Art that uses sound as its primary material and means of expression. Unlike music, it often focuses on the relationship between space/environment and sound.
- Kinetic Art --- Artworks with physically moving elements. They express physical movement using motors, wind, gravity, etc.
- Device Art --- Art that uses technology and devices (apparatus) themselves as a means of expression. It often allows experiencers to touch and operate the work, not just observe it. A concept originating in Japan.
Interaction-Related
- HCI (Human-Computer Interaction) --- A field that studies the interaction between humans and computers. It ranges from designing usable interfaces to creating new experiences.
- Tangible User Interface --- An interface that allows digital information to be manipulated by directly handling physical objects (blocks, knobs, cards, etc.). Instead of clicking buttons on a screen, users move actual physical objects.
- Signifier --- A design cue that communicates "how to operate" to the user. The protrusion of a button naturally communicates "press," and a door handle communicates "pull." In artworks, it is important to design cues that intuitively tell experiencers what to do next.
Sound-Related Fields
- Soundscape --- The sonic landscape that characterizes a particular place or environment. It refers to the totality of sounds that shape the atmosphere of a place --- birdsong, traffic noise, rain sounds, etc.
- Digital Musical Instrument (DMI) --- Musical instruments using digital technology. Unlike traditional instruments, the relationship between input (performance actions) and output (sound) can be freely designed.