This paper proposes a Japanese continuous speech recognition mechanism, in which full sentence level context-free grammar (CFG) and a kind of semantic constraint called the "dependency relationship between phrases" are used during speech recognition in integrated way. A dependency relationship in Japanese is a modification relationship between two phrases; it includes adverbial modification relationships such as a case-frame phrase to a predicate phrase, or adnominal modification relationships such as a noun phrase to a noun phrase. To improve efficiency using this kind of relationship in speech recognition, rigorous semantic analysis is not performed, rather, a simple "matching with examples" approach is adopted. Experiment has been carried out and results are compared with a case in which only CFG constraints are used. It shows that speech recognition accuracy is improved and overhead is small enough.