The recognition method of user’s feedback during the system’s utterance is proposed and its application to the spoken dialogue system is discussed. In human conversation, we can know the dialogue partner’s internal state by receiving such feedbacks. Our research topics are (1) developing the prosodic information based feedback recognizer and (2) appropriately controlling the system’s utterance timing along with the user’s feedbacks. The implemented recognizer can distinguish between back-channel and ask-back word-independently with prosodic information based features and statistical recognition method. Experiments of the spoken dialogue system with this function reveals when it should generate the next utterance after receiving the user’s feedback.