Vancouver, British Columbia—A new technology lets people control a speech synthesizer with gestures, allowing them to speak or sing with their hands. Along with opening new realms of musical expression, research with the speech-generating system may deepen understanding of how the brain drives spoken language and song.
Sidney Fels, a professor of electrical and computer engineering at the University of British Columbia, led the team of researchers that created the device. He talked about it in a session here at the annual meeting of the American Association for the Advancement of Science.
The new technology is mimicking a complicated physiological process. Speech uses 200 muscles between the abdomen and the nose, with the lungs driving air movement and the larynx generating sound.
The speech-generating system (a video demonstration is available for download here) uses two gloves and a foot pedal to control the synthesizer. When the right glove opens it creates vowels, just as the tongue does with its movements in the mouth. Certain gestures create certain consonants, and the left glove makes “stops,” such as the sound for “b.” The foot pedal controls volume, the equivalent of lung pressure. Raising or lowering the right hand also raises or lowers pitch.
With about 100 hours of practice, Mr. Fels says, someone learning how to use the device “can get reasonably good at intelligible speech.” One of the first subjects who used the device said that he had a different personality when he was “talking with the gloves.”
The device makes it possible for singers to have duets with themselves, and there have been seven performances of music written for what is becoming a new instrument. Judging by videos of the performances, it may be a few years yet before the new technology wins any Grammy awards.
The researchers have created a simplified version of the speech technology that can be used on a tablet computer, but the more complex version requires a backpack full of equipment. “You wouldn’t want to go to a restaurant with it to order sushi,” says Mr. Fels.
