Semantics and prosody are two cues for the perception of spoken emotion. In situations where cues conflict, older adults (OA) have difficulty inhibiting one channel and focusing on the other. OA with hearing loss may face more challenges. In this study, we examined the effects of aging and hearing loss on multi-channel emotion processing through a prosody-semantics Stroop task in three groups of participants, i.e., younger adults (YA) and OA with and without hearing loss. It was found that OA with hearing loss showed the most degraded performance in processing conflicting information. When information was incongruent in two channels, they judged emotions less accurately than the other two groups. Moreover, OA with hearing loss was the only group to show channel dominance, in that they performed lower accuracy in the prosodic channel. These findings suggest that hearing loss affects spoken emotional perception in conflict situations, independent of age-related changes.