ISCA Archive Interspeech 2014
ISCA Archive Interspeech 2014

Beyond cross-entropy: towards better frame-level objective functions for deep neural network training in automatic speech recognition

Zhen Huang, Jinyu Li, Chao Weng, Chin-Hui Lee

We propose two approaches for improving the objective function for the deep neural network (DNN) frame-level training in large vocabulary continuous speech recognition (LVCSR). The DNNs used in LVCSR are often constructed with an output layer with softmax activation and the cross-entropy objective function is always employed in the frame-leveling training of DNNs. The pairing of softmax activation and cross-entropy objective function contributes much in the success of DNN. The first approach developed in this paper improves the cross-entropy objective function by boosting the importance of the frames for which the DNN model has low target predictions (low target posterior probabilities) and the second one considers jointly minimizing the cross-entropy and maximizing the log posterior ratio between the target senone (tied-triphone states) and the most competing one. Experiments on Switchboard task demonstrate that the two proposed methods can provide 3.1% and 1.5% relative word error rate (WER) reduction , respectively, against the already very strong conventional cross-entropy trained DNN system.


doi: 10.21437/Interspeech.2014-306

Cite as: Huang, Z., Li, J., Weng, C., Lee, C.-H. (2014) Beyond cross-entropy: towards better frame-level objective functions for deep neural network training in automatic speech recognition. Proc. Interspeech 2014, 1214-1218, doi: 10.21437/Interspeech.2014-306

@inproceedings{huang14c_interspeech,
  author={Zhen Huang and Jinyu Li and Chao Weng and Chin-Hui Lee},
  title={{Beyond cross-entropy: towards better frame-level objective functions for deep neural network training in automatic speech recognition}},
  year=2014,
  booktitle={Proc. Interspeech 2014},
  pages={1214--1218},
  doi={10.21437/Interspeech.2014-306},
  issn={2308-457X}
}