ISCA Archive Interspeech 2022
ISCA Archive Interspeech 2022

Learning Under Label Noise for Robust Spoken Language Understanding systems

Anoop Kumar, Pankaj Kumar Sharma, Aravind Illa, Sriram Venkatapathy, Subhrangshu Nandi, Pritam Varma, Anurag Dwarakanath, Aram Galstyan

Most real-world datasets contain inherent label noise, which typically leads to memorization and overfitting when training over-parameterized deep neural networks (DNNs) on such data. While memorization in DNNs has been studied extensively in computer vision literature, the impact of noisy labels and var- ious mitigation strategies in spoken language understanding tasks is largely under-explored. In this paper, we perform a systematic study on the effectiveness of five noise-mitigation methods in spoken language text classification tasks. First, we experiment on three publicly available datasets by synthetically injecting noise into the labels, and evaluate the effectiveness of various methods at different noise intensity. We then evaluate those methods on a real-word data coming from the large-scale industrial Spoken Language Understanding system. Our results show that most methods are effective mitigating the impact of the noise, with two of those methods showing consistently bet- ter results. For the industrial Spoken Language Understand- ing systems, the best performing methods recover 65% (1.99% accuracy recovery out of 3.07%) of the underfitting caused by noise overall. In one class it recovers 97% underfitting (3.74% out of 3.86%).