Using linguistically positive and negative words uttered either pleasantly or unpleasantly by four speakers, we examined the brain regions that mediate speech communication through event-related functional magnetic resonance imaging (fMRI) analyses. Subjects were adult listeners who evaluated either speakers’ mind, their own mind, or (as a control condition) the number of letters for spoken stimuli, which were randomly presented through earphones. In both the self and speaker-mind judgment tasks, the dorsal medial prefrontal cortex (dMPFC), that has been implicated in theory of mind or self-referential processing, is significantly activated, in addition to the classical cortical regions involved in processing linguistic semantics and emotional prosody of speech. These results suggest that the mental state attribution accomplished by the dorsal medial prefrontal cortex plays an important role to understand our own and speaker’s mind in speech communication.