ISCA Archive AVSP 2008
ISCA Archive AVSP 2008

Building a portable gesture-to-audio/visual speech system

Sidney Fels, Robert Pritchard, Eric Vatikiotis-Bateson

We have constructed an easy-to-use portable, wearable gesture-to-speech system based on the Glove-TalkII and GRASSP gesture-controlled speech systems and a vizeme based face-synthesizer. Our new portable system is called a Digital Ventriloquized Actor (DIVA) and refines the use of the formant speech synthesizer. Using a DIVA, a user can speak using hand gestures mapped to both synthetic sound and face using a mapping function that preserves gesture trajectories. By making DIVAs portable and self-contained, speakers can communicate with others in the community and perform in new music/theatre stage productions. DIVA performers also allow us to study the relationship between visible gestures and speech/song production.