Much recent research has demonstrated that the correlation between a language model’s perplexity and its effect on the word error rate of a speech recognition system is not as strong as was once thought. This represents a major problem for those in-volved in developing language models. This paper describes the development of new measures of language model quality. These measures retain the ease of computation and task inde-pendence that are perplexity’s strengths, yet are considerably better correlated with word error rate. This paper also shows that mixture-based language models are improved by applying interpolation weights which are optimised with respect to these new measures, rather than a maximum likelihood criterion.