This paper discusses the problem of endophoric/deictic references in multimodal dialogues involving speech and pointing gestures. The dialogue system in focus is a building information system implemented using a general workbench architecture "Chameleon". Chameleon and the building information system have been developed within the IntelliMedia 2000+ project initiated 1996 by the Institute of Electronic Systems, Aalborg University [1]. Further development and experiments were carried out in the EU Esprit-project 24 493 "Language and Image Data Fusion using Stochastic Models and Spatial Context Modeling" together with LIMSI and Bertin (coordinator), France [2]. We focus on two problems (1) cross-media references and (2) crossuser/system references.