Sensitivity to a user’s emotional state offers promise in improving the state of the art in spoken dialog systems. In this work, we attempt to detect the speaker’s states of confusion and surprise using prosodic features from his/her utterances. We have collected a corpus of utterances in realistic settings using an experimental methodology aimed at eliciting confusion and surprise from users. Classification experiments have yielded up to a 27.2% improvement over baseline performance using F0 and power features. We achieved the greatest success at classification of emotions that were most successfully elicited.