In Speech Emotion Recognition (SER), significant progress has been made. Despite cutting-edge developments, faultless human-computer interaction remains a distant goal since established SoTA models cannot perceive the speaker's emotional state flawlessly. On the contrary, several studies in SER uncovered the possibility of language and culture-specific differences in this domain. Emotion recognition in speech can vary from person to person based on age, gender, language, and accent, amongst others. In this study, we explore and investigate how assorted accents of the English language influence SER. We employ four different English accents: American, British, Canadian, and Bengali English. Then we extracted a subset of best-performing accent-neutral features by incorporating filter and wrapper-based feature selection methods. Our investigations reveal that pitch, intensity, and MFCC-related features more effectively recognize emotions regardless of accent.