This paper presents an experimental automatic interpretation system named INTERTALKER, which recognizes speaker-independent naturally spoken Japanese and English, translates between the two languages, and converts the result into spoken output. In addition, the system also translates the input into French and Spanish at the same time. The system is composed of speech recognition, multi-lingual translation using the pivot method, and rule-based speech synthesis using the pitch controlled residual wave excitation method. Speaker-independent continuous speech recognition is achieved by using demi-syllable speech units. Speech recognition and language translation are tightly integrated using a conceptual representation, a language independent expression of a sentence. The system is robust in its ability to compensate for possible errors in speech recognition, as well as to be tolerant of the grammatical incorrectness and ambiguities in spoken language. Also, because necessary keywords in the network are directly mapped into a conceptual representation, variations in input sentences can easily be accommodated by adding them to the grammar network. The sentence recognition accuracy for the ticket reservation task of 500 word vocabulary was 83%, and its translation accuracy was 93%.