In this paper, a unified approach for integrating speech and language information in spoken language processing is proposed. This unified approach uses probabilistic score functions to characterize different levels of knowledge in a uniform way. By jointly considering the knowledge from different levels, from acoustics to semantics, the processing capability of speech and language processing are both enhanced with the information provided by the other modules. Besides, since the perplexity of this task is greatly reduced by this approach, the searching efficiency is also improved. After applying this proposed formulation to a Chinese Phonetic Typewriter task, great improvement in sentence accuracy, compared with the word lattice method, is observed.