In daily life, we effortlessly switch attention between different sound sources while filtering out distractions. Prior research has shown that the target speaker can be decoded from EEG in such settings, however, how the switching of attention influences decoding has not been measured. We investigated EEG-based target speaker decoding with exogenous attention switches in a multi-talker environment, where twenty participants alternated listening to two target speakers taking turns while ignoring two background speakers. Speech envelope reconstruction from EEG showed higher accuracy for the attended speaker than distractors, with single-trial classification at 76%. Our findings showed a brief increase in target speaker reconstruction accuracy after attention switches, suggesting heightened alertness when shifting attention to a new speaker. These findings enhance the understanding of dynamic auditory attention and support the development of attention-based hearing applications.