This paper reports an overview of the recent enhancement of ATR's spoken language translation system that can translate Japanese speech to English. First, a speaker adaptation technique is introduced in the speech recognition module so that the system can be available for many users. Second, a phrase category predictor that uses inter-phrase context-free-grammar rules is adopted in the speech recognition module instead of a dependency analysis so that the system can deal with a large vocabulary size of nearly 1,000 words while keeping high recognition accuracy. Third, a new interface between the speech recognizer and the spoken language translator is proposed. Finally, the results of experiments are reported and current performance and future direction of our study are discussed.