Human interaction is a complex process involving modalities such as speech, gestures, motion, and brain activities emitting a wide range of biosignals, which can be captured by a broad panoply of sensors. The processing and interpretation of these biosignals offer an inside perspective on human physical and mental activities and thus complement the traditional way of observing human interaction from the outside. As recent years have seen major advances in sensor technologies integrated into ubiquitous devices, and in machine learning methods to process and learn from the resulting data, the time is right to use of the full range of biosignals to gain further insights into the process of human-machine interaction.

In my talk I will present ongoing research at the Cognitive Systems Lab (CSL), where we explore interaction-related biosignals with the goal of advancing machine-mediated human communication and human-machine interaction. Several applications will be described such as Silent Speech Interfaces that rely on articulatory muscle movement captured by electromyography to recognize and synthesize silently produced speech, as well as Brain Computer Interfaces that use brain activity captured by electrocorticography to recognize speech (brain-to-text) and directly convert electrocortical signals into audible speech (brain-to-speech). I will also describe the recording, processing and automatic structuring of human everyday activities based on multimodal high-dimensional biosignals within the framework of EASE, a collaborative research center on cognition-enabled robotics. This work aims to establish an open-source biosignals corpus for investigations on how humans plan and execute interactions with the aim of facilitating robotic mastery of everyday activities.