In this paper, we present a novel architecture for a Speaker Recognition system over the telephone. The proposed system introduces acoustic information into a HMM-based recognizer. This is achieved by using a phonetic classifier during the training phase. Three broad phonetic classes: voiced frames, unvoiced frames and transitions, are defined. We design speaker templates by the parallel connection of the outputs of the single state HMM´s and by the combination of the single state HMM's into a four state HMM after estimation of the transition probabilities. The results show that this architecture performs better than others without phonetic classification.