ISCA Archive Interspeech 2023
ISCA Archive Interspeech 2023

Verbal and nonverbal feedback signals in response to increasing levels of miscommunication

Maeva Garnier, Eric Le Ferrand, Fabien Ringeval

This study aims to explore in detail how listeners respond to communication disruptions in a task-oriented dialogue. We conducted an experiment with participants playing a map task with a partner via a video conferencing system that showed seemingly random breakdowns. In fact, the breakdowns were scripted to induce increasing levels of miscommunication. After an initial interactive session, a second non-interactive session was recorded with one-sided communication from the task leader. Among the fifty or so verbal and nonverbal feedback signals observed, twelve were produced by more than half of the participants. A detailed analysis of their use in different situations, their timing and their co-occurrence, supported that they may have different functions: some appear to be personal reactions of uncertainty, misunderstanding, or inability to complete the task, whereas others were clear repair initiators or turn-taking signals deliberately addressed to the interlocutor.