This study examined sources of talker differences in the McGurk effect, by questioning which of auditory speech or visual speech is more determining the size of the McGurk effect. Cross-talker dubbing was done between faces and voices of utterances (/ba/ and /ga/) pronounced by two talkers: one compelling talker (CT) to the McGurk effect, and one less compelling talker (LT), as measured in our previous studies. The two talkers and 18 subjects were native speakers of Japanese. There were three presentation conditions: Audio-only, video-only, and audiovisual. The results of unimodal conditions showed that CT was easier to speechread than LT, but that CT was more difficult to listen to than LT. The results of audiovisual condition showed that, although both the visual and auditory talker affect the size of the McGurk effect, the audio component is more responsible than the video component for the talker differences in the McGurk effect.