ISCA Archive SMM 2023
ISCA Archive SMM 2023

Comparison of acoustic-to-articulatory and brain-to-articulatory mapping during speech production using ultrasound tongue imaging and EEG

Tamás Gábor Csapó, Frigyes Viktor Arthur, Péter Nagy, Ádám Boncz

With the investigation of speech-related biosignals we can enhance traditional speech synthesis which might be useful for future brain-computer interfaces. In a recent previous research, from the brain signal measured with EEG, we predicted directly measured articulation, i.e., ultrasound images of the tongue, with a fully connected deep neural network. The results showed that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e., the network can differentiate articulated speech and neutral (resting state) tongue position. In the current study, we extend this with a focus on acoustic-to-articulatory inversion (AAI), and estimate articulatory movement from the speech signal. After that, we analyze the similarities between AAI-estimated articulation and EEG-estimated articulation. We compare the original articulatory data with DNN-predicted ultrasound and show that EEG input is only suitable to distinguish neutral tongue position and articulated speech, whereas melspectrogram-to-ultrasound can also predict articulatory trajectories of the tongue.


doi: 10.21437/SMM.2023-4

Cite as: Csapó, T.G., Arthur, F.V., Nagy, P., Boncz, Á. (2023) Comparison of acoustic-to-articulatory and brain-to-articulatory mapping during speech production using ultrasound tongue imaging and EEG. Proc. SMM23, Workshop on Speech, Music and Mind 2023, 16-20, doi: 10.21437/SMM.2023-4

@inproceedings{csapo23_smm,
  author={Tamás Gábor Csapó and Frigyes Viktor Arthur and Péter Nagy and Ádám Boncz},
  title={{Comparison of acoustic-to-articulatory and brain-to-articulatory mapping during speech production using ultrasound tongue imaging and EEG}},
  year=2023,
  booktitle={Proc. SMM23, Workshop on Speech, Music and Mind 2023},
  pages={16--20},
  doi={10.21437/SMM.2023-4}
}