What You See is What You Hear: the McGurk Effect

Instructions:

1. Watch the video and listen to the sounds iLs co-founder Randall Redfield is making.  Note the sounds.
2. Next, play the video a second time, but close your eyes and listen.  What do you hear? Is it different from when your eyes were open?
3. Read below to find out why your ears seem to be deceiving you!

Explanation:
The key to this multisensory illusion is a mismatch between what you see and what you hear. Randall is repeatedly mouthing the syllable “va,” but the video is dubbed with a sound track of him saying the syllable “ba.” You probably don’t often stop to think about it, but when you make the sound “va,” your top teeth connect with your bottom lip (v is a labio-dental consonant).  To say, “ba,” your lips come together at the beginning of the sound (b is a bi-labial consonant). They look pretty different. The way your brain deals with the conflict of seeing “va” and hearing “ba” is to override what you hear with what you see. And so you hear “va” (or “fa” which looks the same as “va” when you say it).

As you noticed when you listened with your eyes closed, inhibiting the visual input allows your brain to accept that the auditory input is indeed “ba” the whole time. But open them again and you can only hear “va” (or “fa”). The crazy thing is that the effect works even when you know what is happening – and it persists for as long as you experiment with it. Play the video over and over randomly opening and closing your eyes: the illusion is almost impossible to prevent!

The McGurk Effect (named for Harry McGurk who, with John McDonald, described it in a 1976 paper, “Hearing Lips and Seeing Voices”) helps us to understand how our senses are integrated and what happens when they conflict. Depending on which sensory modality is providing more salient information, our brain will combine or override what the other modality is experiencing. It shows how what we hear can be strongly influenced by what we see and proves how highly interdependent our senses are.

How is iLs involved?
It is essential for the senses to work together in processing most information. iLs helps the senses integrate better by combining activities that stimulate listening together with brain networks involved in movement, balance, and cognition. This neuroplastic strengthening of neural circuits supports a better perception of the world around us.