ISCA Archive Interspeech 2024
ISCA Archive Interspeech 2024

CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions

Mario Zusag, Laurin Wagner, Bernhad Thallinger

We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder’s cross-attention scores. We fine- tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verba- tim speech transcription, word segmentation, and the timed de- tection of filler events, and can further mitigate transcription hallucinations. The code is available open source.