The cognition test results of audio (A) and audio-visual (AV) effects on nine English emotions in a short sentence were compared to the physio-acoustic features of sound used for the cognition tests. Two groups of Japanese learners of English (JL2) and one group of English speakers (EL1) participated in these A and AV cognition tests. In the physioacoustic analyses we used F0 and intensity contours and calculated the area of sentential patterns and three forms of distance: area-, average, and pattern-distance for each emotion. Similar patterns of the F0 and intensity contours might have been caused by the cognitive confusions among emotions; the relationships between the cognition tests and physio-acoustic analyses confirmed that there was not a strong correlation between them, intensity seeming to be more correlated to the cognition test results for A for both JL2 and EL1 than F0. EL1’s correlation was higher than that of JL2.