We present a method for phoneme recognition using an expert system combining spectrogram reading knowledge and neural networks, and we report its performance. The proposed expert system consists of two parts: (1) phoneme segmentation based on spectrogram reading knowledge used by human experts, and (2) phoneme identification using neural networks applied to the phoneme boundaries determined in phoneme segmentation. Highly accurate phoneme segmentation can be achieved by using human-like contextual spectrogram reading knowledge. Moreover, high performance phoneme identification can be achieved by applying neural networks to the accurate phoneme segmentation result. The system was tested on Japanese consonants, with 90.8% of the phonemes correctly segmented and 92.4% of the phonemes correctly identified within the correct segment. 83.9% of the phonemes were correctly recognized both in segmentation and identification.