In this paper we present an approach that incorporates structural information into language models without really parsing the utterance. This approach brings together the advantages of a n-gram language model { speed, robustness and the ability to integrate with the speech recognizer with the need to model syntactic constraints, under a uniform representation. We also show that our approach produces better language models than language models based on part-of-speech tags.