Have you heard of the McGurk effect?
The McGurk effect was first realized, discussed and named in 1976. It's an effect that shows the importance of speech in how we interpret sound, especially very nuanced sounds. The McGurk effect can be demonstrated in a lab when a spoken sound is played for someone and at the same time they are presented with a video of lips moving that form another sound. The end result is that person interprets that he or she heard the sound that the mouth was making - visual cues override what the person has actually heard.
For example, take two very similar syllables - "ba" and "va." If a person in the lab hears a recorded voice saying "ba," but sees a video of a voice saying "va," that person will report that he or she has heard the word va, even though that is untrue. However, if the person is recorded mouthing the syllable "tha," this will be too obvious a mismatch and the person will report correctly that he or she has heard "ba."
The McGurk effect reveals that speech reading is effective for those with hearing loss because visual cues are very important. For those with minor hearing loss, speech reading can be a very valuable way to maximize the hearing they still do have. Also, this reveals more about why watching the mouth is so important in intense language learning.
However, until now, the mechanisms of the McGurk effect were unknown. But bioengineers from the University of Utah, in a study published in PLOS ONE in September 2013, discovered the mechanisms in the brain that allow for the McGurk effect, as lead author Elliot Smith explains:
"For the first time, we were able to link the auditory signal in the brain to what a person said they heard when what they actually heard was something different. We found vision is influencing the hearing part of the brain to change your perception of reality – and you can't turn off the illusion," Smith said. "People think there is this tight coupling between physical phenomena in the world around us and what we experience subjectively, and that is not the case."
These findings might also reveal why it's often difficult for older adults with hearing loss, who often have vision problems as well, to hear better with well-fitted, high-quality hearing aids and other listening devices. The researchers suggest that assistive listening devices and speech-recognition software might also benefit from a camera, in addition to a microphone, to enhance users' hearing experience.
According to bioengineer Bradley Greger, our brains are doing something interesting:
"We've shown neural signals in the brain that should be driven by sound are being overridden by visual cues that say, 'Hear this!'" he said. "Your brain is essentially ignoring the physics of sound in the ear and following what's happening through your vision."
This study might also have implications for other research on what drives language processing and what happens when it goes wrong, such as with dyslexia.