The same effect that makes ventriloquism work could help people with hearing difficulties, according to Duke researchers.
In order to make their puppets speak, ventriloquists rely on a well-known link between sight and sound. The audience hears a joke, sees the puppet move and thinks the puppet must be talking.
Duke researchers recently set out to investigate how this trick works. Jennifer Groh, a professor at Duke’s Center for Cognitive Neuroscience, led a study to determine how the brain links separate audio and visual stimuli. The findings contradict conventional wisdom that timing is the most important factor in this connection by showing that the brain uses visual cues to take a guess as to where the sound is coming from. Not only does it further explain ventriloquism, it has implications for people with hearing impairments.
“The question we’re looking at is, ‘How does vision help us learn?'” Groh said. “That’s why we have a ventriloquist illusion, because generally vision helps us to do better at localizing sound."
The study—which Groh conducted along with Daniel Pages, a graduate student in psychology and neuroscience—involved exposing participants to a light and a sound coming from different places. One group experienced the light and sound at the same time, while the other group was shown the light slightly after the sound played.
They found that the brain is better at associating visual cues that come after a sound rather than ones that come at the same time. People who saw the light after they heard a sound are more likely to look toward the light than people who experienced them simultaneously. This indicates that the brain “guesses and checks” after hearing something to associate a sight with the sound, Pages said.
“It was surprising that it was so difficult for the brain to calibrate [the different stimuli] with just simultaneity,” he said.
He added that they expected to see a larger effect from the simultaneous stimuli than they did, because the conventional wisdom has been that timing plays a key role in associating different sensory inputs.
The results of this study have implications for people with mild or moderate hearing impairments. Knowing that the human brain has to search for the source of a sound might be useful in designing therapies.
“You want to encourage people [with hearing difficulties] to actively look for the sound,” Groh said. “That kind of active behavior could help them to some degree.”
This study could help in developing devices to aid people who are either vision or hearing impaired because it explores the link between those two senses, said Norbert Kopco, a psychology and neuroscience research associate, who has worked with Groh on previous studies in the same field.
“There is huge potential for the development of prosthetic devices that help people whose vision or hearing is impaired," Kopco said. "To the extent that one of the senses can supply the information that the other sense cannot, understanding how the visual and auditory information interacts is critical for the development of such medical devices."
The study also explored the similarities between human and monkey brains by including two monkeys as participants. Groh explained that humans and monkeys use similar mechanisms for vision and hearing, so conducting this study on monkeys allowed her to examine whether the mechanisms for associating visual and auditory stimuli are consistent between the two species.
Get The Chronicle straight to your inbox
Signup for our weekly newsletter. Cancel at any time.