We often smile (and frown) while we talk. Listeners to such affective speech have to integrate the affective and the linguistic cues in the speech signal. Following up on earlier work, we investigated whether and how affective phonetics (i.e. vocal expressions such as smiling) and affective semantics (sentence-level meaning) interact during spoken language comprehension of sentences and how perspective modifies these interactions. We explored this by presenting phonetically and semantically manipulated spoken Dutch sentences to listeners while collecting behavioral and neural (ERP) measurements.