The goals of our work are twofold: gain insight into how humans interact with complex data and visualizations thereof in order to make discoveries; and use our findings to develop a dialogue system for exploring data visualizations. In our work to date, we have collected a multimodal dialogue corpus and developed a pipeline that creates visualizations from spoken utterances. In this paper, we focus on the multimodal and contextual information that can give the system additional insight into the user’s intention, both through gestures and utterances nearby the actual request. To this end, we have annotated our corpus for context and both speech and gesture information; we developed a classifier for identifying rich themes found in the annotated context, in order to refine the interpretation of the request; and we have developed gesture detection using Kinect.