We give an overview of the RWTH phrase-based statistical machine translation system that was used in the evaluation campaign of the International Workshop on Spoken Language Translation (IWSLT) 2006. The system was ranked first with respect to the BLEU measure in all language pairs it was used. Using a two-pass aproach, we first generate the N best translation candidates. The second pass consists of rescoring and reranking these candidates. We will give a description of the search algorithm as well as of the models used in each pass. We will also describe our method for dealing with punctuation restoration, in order to overcome the difficulties of spoken language translation. This work also includes a brief description of the system combination done by the partners participating in the European TC-Star project.