ISCA Archive Interspeech 2014
ISCA Archive Interspeech 2014

Multimodal understanding for person recognition in video broadcasts

Frederic Bechet, Meriem Bendris, Delphine Charlet, Géraldine Damnati, Benoit Favre, Mickael Rouvier, Remi Auguste, Benjamin Bigot, Richard Dufour, Corinne Fredouille, Georges Linarès, Jean Martinet, Gregory Senay, Pierre Tirilly

This paper describes a multi-modal person recognition system for video broadcast developed for participating in the Defi-Repere challenge. The main track of this challenge targets the identification of all persons occurring in a video either in the audio modality (speakers) or the image modality (faces). This system is developed by the PERCOL team involving 4 research labs in France and was ranked first at the 2014 Defi-Repere challenge. The main scientific issue addressed by this challenge is the combination of audio and video information extraction processes for improving the extraction performance in both modalities. In this paper, we present the strategy followed by the PERCOL team for speaker identification based on enriching the speaker diarization with features related to the “understanding” of the video scenes: text overlay transcription and analysis, automatic situation identification (TV set, report), the amount of people visible, TV set disposition and even the camera when available. Experiments on the REPERE corpus show interesting results on the speaker identification system enriched by the scene understanding features and the usefulness of the speaker to identify faces.

doi: 10.21437/Interspeech.2014-146

Cite as: Bechet, F., Bendris, M., Charlet, D., Damnati, G., Favre, B., Rouvier, M., Auguste, R., Bigot, B., Dufour, R., Fredouille, C., Linarès, G., Martinet, J., Senay, G., Tirilly, P. (2014) Multimodal understanding for person recognition in video broadcasts. Proc. Interspeech 2014, 607-611, doi: 10.21437/Interspeech.2014-146

  author={Frederic Bechet and Meriem Bendris and Delphine Charlet and Géraldine Damnati and Benoit Favre and Mickael Rouvier and Remi Auguste and Benjamin Bigot and Richard Dufour and Corinne Fredouille and Georges Linarès and Jean Martinet and Gregory Senay and Pierre Tirilly},
  title={{Multimodal understanding for person recognition in video broadcasts}},
  booktitle={Proc. Interspeech 2014},