To clarify how the brain understands the speaker’s mind for verbal acts, fMRI images obtained from 24 subjects and behavioral data obtained from 339 subjects were analyzed when they judged the linguistic meanings or emotional manners of spoken phrases. The target phrases had linguistically positive or negative meanings and were uttered warmheartedly or coldheartedly by a woman speaker. The results of the fMRI analyses suggest that neural resources responsible for the speakers’ mind reading are distributed over the superior temporal sulci, inferior frontal regions, medial frontal regions and posterior cerebellum. The correct judgment of the speaker intentions significantly increased with age for the phrases with inconsistent linguistic and emotional valences. Female children showed faster development than male children. The neural mechanism to interpret speaker’s real intensions from spoken phrases develops slowly during the school age.