The McGurk Effect, Hearing With Your Eyes
You’ve undoubtedly seen badly dubbed movies where you try and read the actors’ lips to work out what they’re saying. Suddenly, you find yourself “listening with your eyes”. In other words, you’re deciphering sounds through a different channel to the auditory one. This phenomenon, when there’s a mismatch between sound and visual signals is known as the McGurk effect.
Although this kind of experience sometimes leads you to make some mistakes, it’s also an extremely important strategy of the brain. For example, when you’re at a party with very loud music, trying to have a conversation with someone, you look at their mouth to try and “read” their words.
This demonstrates something important. It’s the fact that we don’t use each of our senses separately. In fact, in reality, all perceptual experiences happen as a result of the interaction of more than one sense. Sight and hearing work together here, so although sometimes small errors may occur, they’re our best allies…
People with Alzheimer’s disease frequently experience disturbances of the McGurk effect. They can’t identify sounds and words visually.
What is the McGurk effect?
Much of your communication processes occur in situations where you can hear the other person clearly. However, in recent times in the wake of the rise in video calls, you tend to experience the McGurk effect more frequently.
For example, sometimes, due to bad connections, you don’t hear the other person very well. Then, you change from your auditory channel to your visual one to try and read their lips. However, this isn’t easy. In fact, sometimes, even with the person’s face and mouth as a reference point, you make mistakes.
Cognitive psychologists, Harry McGurk and John MacDonald studied this phenomenon in the 1970s. In fact, it was in their article, Hearing Lips and Seeing Voices in 1976, that they first coined the term the McGurk effect.
They did it after checking something very specific in a famous experiment. It’s something that had been proven for decades and that had never failed. The fact that if someone moves their lips pronouncing the syllable “ga”, but what the listener hears is the syllable “ba”, their brain will perceive it as “da”. In other words, sometimes what the eyes read isn’t the same as what the ears perceive.
Why does the McGurk effect occur?
You already know that the McGurk effect happens when you try to “read” with your eyes and what you see (movement of the lips) doesn’t match the actual sound. The brain region where this phenomenon occurs is in the superior temporal sulcus. When trying to decode two sensory modalities (sight and sound) it’s common for errors to appear.
The Department of Neurosurgery at the Houston Medical College (Texas) conducted a study in which they pointed out that when you have a face-to-face conversation, your brain listens through the ear and also deciphers words with your eyes. You do this unconsciously.
Attending to the movement of the lips allows you to reinforce communication in case of difficulties, such as environmental noise. However, there are sounds and syllables that are often misinterpreted. Small inferences and errors occur.
Some people suffer this experience to a greater degree
As we mentioned earlier, people with Alzheimer’s disease tend to be severely affected by the McGurk effect. Indeed, if at some point they have to read lips to understand a message, they won’t be able to decipher it.
In 2007, Delbeuck, Collette, and Van der Linden conducted research that suggested the cause of this might be the deterioration of brain connectivity.
Similarly, it’s been observed that children with expressive language disorder don’t pay attention to the articulation of words. In fact, they don’t use the visual channel to imitate or understand what’s expressed to them. This hinders the development of the language itself.
The visual system helps you to discriminate sounds
The McGurk effect appears when you read other people’s lips and confuse some syllables or words. It’s something completely normal as well as interesting. It demonstrates the fact that as humans, we always use our visual system during communication. It’s not something that only people with hearing impairments do.
Your brain needs to make sure that the communication process is carried out efficiently. To do this doesn’t only involve listening, it also concerns sight.
Looking at the mouth of your interlocutor facilitates understanding in any context. This explains the importance of face-to-face conversations. In fact, reading with the eyes is one more human skill we should be proud of.
All cited sources were thoroughly reviewed by our team to ensure their quality, reliability, currency, and validity. The bibliography of this article was considered reliable and of academic or scientific accuracy.
- Alsius, A., Navarra, J., & Soto-Faraco, S. (2007). Attention to touch weakens audiovisual speech integration. Experimental Brain Research, 183(3), 399-404.
- Baynes, K., Funnell, M. G., & Fowler, C. A. (1994). Hemispheric contributions to the integration of visual and auditory information in speech perception. Perception & Psychophysics, 55(6), 633-641.
- Delbeuck, X., Collette, F., & Van der Linden, M. (2007). Is Alzheimer’s disease a disconnection syndrome?: Evidence from a crossmodal audio-visual illusory experiment. Neuropsychologia, 45(14), 3315-3323.
- Tiippana K. (2014). What is the McGurk effect?. Frontiers in psychology, 5, 725. https://doi.org/10.3389/fpsyg.2014.00725
- McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748.
- Nath, A. R., & Beauchamp, M. S. (2012). A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage, 59(1), 781-787.
