We have developed a task-oriented speech dialogue system based on spontaneous speech understanding and response generation (TOSBURG) for unspecified users. The system consists of a noise-robust keyword-spotter, a semantic keyword lattice parser, a user-initiative dialogue manager and a multimodal response generator. After noise immunity keyword-spotting has been performed, the spotted keyword candidates are analyzed by a new keyword lattice parser to extract the semantic content of input speech. Using the dialogue history and situation, the dialogue manager understands input speech based on the semantic contents, and generates a confirmation message to the user about ambiguous points to help overcome difficulties due to the imperfection of speech understanding. The real-time dialogue system has been constructed for a fast food ordering task using two general purpose workstations and four DSP accelerators.