The cortical oscillatory framework proposes that neural oscillations at different frequency bands track linguistic structure in speech. Some scholars argue that neural tracking can be dissociated to some extent from the encoding of cues in the speech signal, whereas others have emphasized neural tracking at different timescales, matching stimuli specific properties. The modulations of the amplitude envelope, which mostly capture prosodic information, have been shown to play a key role in the process, suggesting that prosody is central to the neural tracking of speech. The aim of this study was to isolate prosodic processing from morphosyntactic and semantic processing to investigate neural tracking of prosodic structure. Using delexicalized speech (with the original syllables replaced by a single syllable, e.g., /mi/), we show that neurophysiological signals in the delta and theta bands track the rates of the utterance (~0.4-0.7Hz), the intonational phrase (IP, ~0.9-1.2Hz), the prosodic word/stress (~1.5-2.5Hz), and the syllable (~4-6Hz). Moreover, the evoked power spectrum of EEG responses distinguished between stimuli with a one-IP-utterance (larger power at the utterance rate) and stimuli with the utterance comprising two IPs (larger power at the IP rate). These results indicate that prosody alone is sufficient to drive neural tracking at different timescales.