ISCA Archive Eurospeech 1997
ISCA Archive Eurospeech 1997

Generic template for the evaluation of dialogue management systems

Gavin E. Churcher, Eric S. Atwell, Clive Souter

We present a generic template for spoken dialogue systems integrating speech recognition and synthesis with 'higher-level' natural language dialogue modelling components. The generic model is abstracted from a number of real application systems targetted at very different domains. Our research aim in developing this generic template is to investigate a new approach to the evaluation of Dialogue Management Systems. Rather than attempting to measure accuracy/speed of output, we propose principles for the evaluation of the underlying theoretical linguistic model of Dialogue Management in a given system, in terms of how well it fits our generic template for Dialogue Management Systems. This is a measure of 'genericness' or 'application-independence' of a given system, which can be used to moderate accuracy/speed scores in comparisons of very unlike DMSs serving different domains. This relates to (but is orthogonal to) Dialogue Management Systems evaluation in terms of naturalness and like measurable metrics; it follows more closely emerging qualitative evaluation techniques for NL grammatical parsing schemes.


doi: 10.21437/Eurospeech.1997-592

Cite as: Churcher, G.E., Atwell, E.S., Souter, C. (1997) Generic template for the evaluation of dialogue management systems. Proc. 5th European Conference on Speech Communication and Technology (Eurospeech 1997), 2247-2250, doi: 10.21437/Eurospeech.1997-592

@inproceedings{churcher97_eurospeech,
  author={Gavin E. Churcher and Eric S. Atwell and Clive Souter},
  title={{Generic template for the evaluation of dialogue management systems}},
  year=1997,
  booktitle={Proc. 5th European Conference on Speech Communication and Technology (Eurospeech 1997)},
  pages={2247--2250},
  doi={10.21437/Eurospeech.1997-592},
  issn={1018-4074}
}