ISCA Archive Interspeech 2024
ISCA Archive Interspeech 2024

Understanding “understanding”: presenting a richly annotated multimodal corpus of dyadic interaction

Leonie Schade, Nico Dallmann, Olcay Tük, Stefan Lazarov, Petra Wagner

This paper presents the MUNDEX corpus (MUltimodal UNDerstanding of EXplanations) together with past and current investigations using its data. The corpus is constructed to observe the dynamics of co-constructed communication and the understanding of explanations on multiple modalities in dyadic interactions. These modalities are annotated on several levels, including orthographic transcriptions, acoustic information, annotations of head movement, gaze, manual gestures, and further non-verbal behaviour as well as discourse annotations. Present and past projects are also concerned with adding further to these annotations. The interlocutors’ level of understanding is currently investigated in regard to several verbal and non-verbal behaviour markers of both the explaining and listening side.