This paper proposes a new speech recognition scheme using three linguistic constraints. Multi-class composite bigram models are used in the first and second passes to reflect word-neighboring characteristics as an extension of conventional word n-gram models. Trigram models with constituent boundary markers and word pattern models are both used in the third pass to utilize phrasal constraints and headword co-occurrences, respectively. These two models are made using a training text corpus with phrase structures given by an example-based Transfer-Driven Machine Translation (TDMT) parser. Speech recognition experiments show that the new recognition scheme reduces word errors 9.50% from the conventional scheme by using word-neighboring characteristics, that is only the multi-class composite bigram models.