Using spoken phrases with positive or negative linguistic meanings uttered by a woman with two emotions, warmhearted and coldhearted, we analyzed gender differences in the brain activity based on a functional MRI measurement when subjects judged linguistic or emotional meanings of the phrases. Significant interaction effects of language and emotion were observed on the acoustic characteristics of utterances, such as F0 range, and also on the perceptual behavior evaluated by response time and judgment correctness. When compared to the female subjects, the male subjects showed significantly stronger activation in only the right frontomedian cortex, which can be hypothesized to construct and maintain the theory of mind to understand speaker’s hidden but true intensions. These results suggest that emotion modulates linguistic processes not only in speech production but also in speech perception, and such modulations may differ between the genders at least in perceptual processes.