Functional imaging studies, both fMRI and PET, have allowed us to elaborate the neural systems important in speech perception. In particular, these techniques have allowed us to integrate models of human speech perception in the framework of primate auditory cortex, where different functional and anatomical streams of processing can be identified. This neuroanatomical approach also has the potential of relating plasticity in speech processing to the known plasticity of auditory cortex. In this talk I will outline candidate neural systems involved in speech perception, and use this as a framework to address three kinds of plasticity in speech perception: after perceptual learning, after cochlear implantation, and after aphasic stroke. These different aspects of plasticity are associated with different aspects of neural change. Specifically, I will address the roles of streams of processing in speech plasticity, top down prefrontal modulation following cochlear implantation and right hemisphere changes after aphasic stroke.