ISCA Archive AVSP 1998
ISCA Archive AVSP 1998

Is it Possible to Evaluate the Contribution of Visual Information to the Process of Speech Comprehension?

Loredana Cerrato, Federico Albano Leoni, Mauro Falcone

We report the results of a series of comprehension tests run with the aim of investigating the contribution of visual information to the process of comprehension of conversational speech. Using as sample material a short conversation held by two male speakers, edited from an Italian TV soap opera, we run 3 comprehension tests: 1. submission of the multi-modal speech signal (auditory + visual); 2. submission of the sample only in the auditory modality (i.e. without the integration of visual cues); 3. submission of the sample only in the visual modality (without the integration of auditory cues). It is clear from our results that the visual cues help the subjects to understand the main topic of conversation and to remember some of the details of the conversation. Moreover they seem to play an important role for the interpretation of the emotional state of the speakers. In some cases visual cues appear to be misleading.


Cite as: Cerrato, L., Albano Leoni, F., Falcone, M. (1998) Is it Possible to Evaluate the Contribution of Visual Information to the Process of Speech Comprehension? Proc. Auditory-Visual Speech Processing, 141-146

@inproceedings{cerrato98_avsp,
  author={Loredana Cerrato and Federico {Albano Leoni} and Mauro Falcone},
  title={{Is it Possible to Evaluate the Contribution of Visual Information to the Process of Speech Comprehension?}},
  year=1998,
  booktitle={Proc. Auditory-Visual Speech Processing},
  pages={141--146}
}