In this paper, an overview of the XMU phrase-based statistical machine translation system for the 2006 IWSLT Speech Translation Evaluation was given. In this year's evaluation, we participated in the open data track for ASR lattice and Cleaned Transcripts for the Chinese-English translation direction. The system ranked 7th among the 12 participating systems in the Chinese-English spontaneous speech ASR output task, 11th among the 15 participating systems in the Chinese-English read speech ASR output task and 8th among the 15 participating systems in the Cleaned Transcripts task.