This paper describes a sentence speech recognition system based on phoneme-based hidden Markov models (HMMs) and two grammatical constraints: a syntactic grammar of phrase structure and a semantic dependency grammar of sentence structure. A joint score, combining acoustic likelihood and linguistic certainty factors derived from phoneme based HMMs and two grammatical constraints, is maximized to obtain the optimal sentence recognition. A semantic analysis algorithm globally optimizes the joint score. This algorithm is based on two key techniques: most likely multi-phrase candidate-detection using the Viterbi algorithm, and breadth-first search for dependency parsing. Where the perplexity of the phrase syntax is 40, this system increases phrase recognition performance in the sentences by approximately 14%, showing the effectiveness of semantic dependency analysis.