ISCA Archive SpeechProsody 2024
ISCA Archive SpeechProsody 2024

Visual Cues of Emotion Expression: Perceptual Evaluation and Automated System Analysis

Mario Augusto Souza Fontes, Sandra Madureira, Juliana Andreassa

The communicative relevance of visual cues has been highlighted in bimodal studies on speech perception of affective states and pragmatic meanings. These studies usually combine manual analysis of visual data based on FACS Action Unities and subjective perception tests. Automated analysis of facial expressions is also based on FACS and associates facial movements with affective expressions. The objective of this paper is analyzing facial expressions of four basic emotions by comparing the results of a perceptual test with those of an automated analysis based on the same stimuli. The research corpus comprises thirty utterances that were presented as video stimuli to thirty-four judges in a perceptual test as well as analyzed by means of an automated facial expression recognition software. The results of both evaluations are multidimensionally analyzed and theoretically discussed in relation to visual cue salience, valence and arousal features, and associations of Action Unities with affective states.