ISCA Archive Interspeech 2022
ISCA Archive Interspeech 2022

Regularizing Transformer-based Acoustic Models by Penalizing Attention Weights

Munhak Lee, Joon-Hyuk Chang, Sang-Eon Lee, Ju-Seok Seong, Chanhee Park, Haeyoung Kwon

The application of deep learning has significantly advanced the performance of automatic speech recognition (ASR) systems. Various components make up an ASR system, such as the acoustic model (AM), language model, and lexicon. Generally, the AM has benefited the most from deep learning. Numerous types of neural network-based AMs have been studied, but the structure that has received the most attention in recent years is the Transformer. In this study, we demonstrate that the Transformer model is more vulnerable to input sparsity compared to the convolutional neural network (CNN) and analyze the cause of performance degradation through structural characteristics of the Transformer. Moreover, we also propose a novel regularization method that makes the transformer model robust against input sparsity. The proposed sparsity regularization method directly regulates attention weights using silence label information in forced-alignment and has the advantage of not requiring additional module training and excessive computation. We tested the proposed method on five benchmarks and observed an average relative error rate reduction (RERR) of 4.7%.


doi: 10.21437/Interspeech.2022-362

Cite as: Lee, M., Chang, J.-H., Lee, S.-E., Seong, J.-S., Park, C., Kwon, H. (2022) Regularizing Transformer-based Acoustic Models by Penalizing Attention Weights. Proc. Interspeech 2022, 56-60, doi: 10.21437/Interspeech.2022-362

@inproceedings{lee22b_interspeech,
  author={Munhak Lee and Joon-Hyuk Chang and Sang-Eon Lee and Ju-Seok Seong and Chanhee Park and Haeyoung Kwon},
  title={{Regularizing Transformer-based Acoustic Models by Penalizing Attention Weights}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={56--60},
  doi={10.21437/Interspeech.2022-362},
  issn={2958-1796}
}