Electroencephalography (EEG)-based speech decoding enables the development of non-invasive speech brain-computer interfaces (BCIs) for restoring communication of individuals with speech impairments. Previous work achieves much better performance in decoding spoken and intended speech from EEG signals, with imagined speech decoding lagging far behind. This paper proposes a novel framework to train a unified multi-mode decoding model for EEG signals of imagined, intended and spoken speech modes using a dynamic masking mechanism. Our multi-mode model achieves significantly better four-vowel decoding accuracies than baselines (34.95% vs. 29.18% for imagined speech). Training a single-mode model with a subset of EEG channels selected according to a multi-mode model as inputs provides superior performance than training a single-mode model from all channels. The accuracy improvements and channel selection capability demonstrate the effectiveness of the proposed joint modeling framework.