Spoken queries submitted to question answering systems usually consist of query contents (e.g. about newspaper articles) and frozen patterns (e.g. WH-words), which can be modeled with N-gram models and grammar-based models, respectively. We propose a method to integrate those different types of models into a single N-gram model. We represent the two types of language models in a single word network. However, common smoothing methods, which are effective for N-gram models, decrease grammatical constraints for frozen patterns. For this problem, we propose a selective back-off smoothing method, which controls a degree to which smoothing is applied depending the network fragment. Additionally, resulting models are compatible with the conventional back-off N-gram models, and thus existing N-gram decoders can easily be used. We show the effectiveness of our method by way of experiments.