Speaker adaptation and the evaluation of speaker similarity in the EMIME speech-to-speech translation project

Wester, Mirjam; Dines, John; Gibson, Matthew; Liang, Hui; Wu, Yi-Jian; Saheer, Lakshmi; King, Simon; Oura, Keiichiro; Garner, Philip N.; Byrne, William; Guan, Yong; Hirsimäki, Teemu; Karhila, Reima; Kurimo, Mikko; Shannon, Matt; Shiota, Sayaka; Tian, Jilei; Tokuda, Keiichi; Yamagishi, Junichi

Speaker adaptation and the evaluation of speaker similarity in the EMIME speech-to-speech translation project

Mirjam Wester, John Dines, Matthew Gibson, Hui Liang, Yi-Jian Wu, Lakshmi Saheer, Simon King, Keiichiro Oura, Philip N. Garner, William Byrne, Yong Guan, Teemu Hirsimäki, Reima Karhila, Mikko Kurimo, Matt Shannon, Sayaka Shiota, Jilei Tian, Keiichi Tokuda, Junichi Yamagishi

This paper provides an overview of speaker adaptation research carried out in the EMIME speech-to-speech translation (S2ST) project. We focus on how speaker adaptation transforms can be learned from speech in one language and applied to the acoustic models of another language. The adaptation is transferred across languages and/or from recognition models to synthesis models. The various approaches investigated can all be viewed as a process in which a mapping is defined in terms of either acoustic model states or linguistic units. The mapping is used to transfer either speech data or adaptation transforms between the two models. Because the success of speaker adaptation in text-to-speech synthesis is measured by judging speaker similarity, we also discuss issues concerning evaluation of speaker similarity in an S2ST scenario.

Index Terms: speech-to-speech translation