ISCA Archive ISCSLP 2000
ISCA Archive ISCSLP 2000

N-Gram Language Model Compression Using Scalar Quantization and Incremental Coding

Shuo Di, Lei Zhang, Zheng Chen, Eric Chang, Kai-Fu Lee

This paper describes a novel approach of compressing large trigram language models, which uses scalar quantization to compress log probabilities and back-off coefficients, and incremental coding to compress entry pointers. Experiments show that the new approach achieves roughly 2.5 times of compression ratio compared to the well-known tree-bucket format while keeps the perplexity and accessing speed almost unchanged. The high compression ratio enables our method to be used in various SLM-based applications such as Pinyin input method and dictation on handheld devices with little available memory.