The paper presented a novel method to speed up the likelihood computation of the speech recognition system based continuous Hidden Markov Model (CHMM). The block-diagonal covariance matrices were applied in the method and the technique to construct an optimal block-diagonal matrix was introduced. The experimental results demonstrated that the block-diagonal covariance matrices could achieve a large improvement in recognition speed without significant decrease of recognition rate compared with baseline system.