The voice is a powerful medium for the communication of emotion. However, in conversation listeners typically attend to the content of a message rather than to the way it is communicated. Moreover, given the serial nature of speech processing, they focus on one talker at a time. Nevertheless, research on vocal processing indicates that listeners tend to be sensitive to emotional expressions from unattended sources. For example, unattended syllable sequences that rarely and unpredictably change in speaker prosody were found to elicit a larger mismatch negativity in the event-related potential (ERP) if syllables suddenly become emotional as compared to neutral. The present functional magnetic resonance imaging (fMRI) study aimed at elucidating the brain structures that mediate this effect. The observed evidence suggests that enhanced responses to unattended emotional as compared to neutral speaker prosody are mediated by activity in the middle frontal gyrus, the posterior superior temporal cortex, the anterior insula and the amygdala. Interestingly, however, listeners differ in their sensitivity to unattended and unexpected changes in speaker tone of voice. These differences appear to be linked to biological (e.g., estrogen) and cultural factors (e.g., social orientation) and affect how these individuals pursue and engage in social interactions.