ISCA Archive AVSEC 2024
ISCA Archive AVSEC 2024

Next-Generation Speech Recognition Using Radar Sensing

Michaela B. Reay, Balal Saleemi, Qammer H. Abbasi, Hira Hameed, Kia K. Dashtipour, Amir Hussain, Muhammad Ali Imran

Automated Lip-reading has become a major research challenge, aimed at recognizing speech from lip movements. Most current technologies rely on visual and audio signals as input data. However, video recording systems face issues such as occlusion, ambient lighting, and privacy concerns. Furthermore, these vision-based technologies are ineffective for multi-modal hearing aids in situations where face masks are common, such as during the COVID-19 pandemic. In light of these drawbacks, this paper introduces an RF-based lip-reading framework that addresses the limitations of camera-based systems and functions even with face masks. Utilizing radar technology for RF sensing, a dataset comprising ten publicly available Harvard Sentences is collected from three subjects, including both males and females. This data is used to train deep learning models, achieving a high classification accuracy of 79.12\% with the ResNet18 model on radar data.