ISCA Archive AVSP 1998
ISCA Archive AVSP 1998

Visual Speech Synthesis Based on Parameter Generation From HMM: Speech-Driven and Text-And-Speech-Driven Approaches

Masatsune Tamura, Takashi Masuko, Takao Kobayashi, Keiichi Tokuda

This paper describes a technique for synthesizing synchronized lip movements from auditory input speech signal. The technique is based on an algorithm for parameter generation from HMM with dynamic features, which has been successfully applied to text-to-speech synthesis. Audio-visual speech unit HMMs, namely, syllable HMMs are trained with parameter vector sequences that represent both auditory and visual speech features. Input speech is recognized using the syllable HMMs and converted into a transcription and a state sequence. A sentence HMM is constructed by concatenating the syllable HMMs corresponding to the transcription for the input speech. Then an optimum visual speech parameter sequence is generated from the sentence HMM in ML sense. Since the generated parameter sequence reflects statistical information of both static and dynamic features of several phonemes before and after the current phonemes, synthetic lip motion becomes smooth and realistic. We show experimental results which demonstrate the effectiveness of the proposed technique.


Cite as: Tamura, M., Masuko, T., Kobayashi, T., Tokuda, K. (1998) Visual Speech Synthesis Based on Parameter Generation From HMM: Speech-Driven and Text-And-Speech-Driven Approaches. Proc. Auditory-Visual Speech Processing, 221-224

@inproceedings{tamura98_avsp,
  author={Masatsune Tamura and Takashi Masuko and Takao Kobayashi and Keiichi Tokuda},
  title={{Visual Speech Synthesis Based on Parameter Generation From HMM: Speech-Driven and Text-And-Speech-Driven Approaches}},
  year=1998,
  booktitle={Proc. Auditory-Visual Speech Processing},
  pages={221--224}
}