ISCA Archive Interspeech 2024
ISCA Archive Interspeech 2024

OCEAN-AI: open multimodal framework for personality traits assessment and HR-processes automatization

Elena Ryumina, Dmitry Ryumin, Alexey Karpov

Human personality traits (PT) reflect individual differences in patterns of thinking, feeling, and behaving. Knowledge on PT may be useful in many applied tasks in our everyday live. In this paper, we present a first open-source multimodal framework called OCEAN-AI for PT assessment (PTA) and HR-processes automatization. Our framework performs PTA analyzing three modalities, including audio, video, and text, and includes three processing modules. All the modules extract heterogeneous (deep neural and hand-crafted) features and use them for a com- plex analysis of human’s behavior. The final fourth module aggregates these six feature sets by a Siamese neural network with a gated attention mechanism. Our framework was tested on two free-available corpora, including First Impressions v2 and our MuPTA, and achieved the best results. Applying our framework, a user can automate solutions of some practical applied tasks, such as ranking potential candidates by professional responsibilities, forming efficient work teams and so on.