The quality of classification systems is usually measured in correct classification performance: compare the predicted classes of test items with their predefined classes (ground truth). This paper discusses a number of requirements and considerations that are important for a proper design of a ground truth database in the context of music classification. Two examples will show how the recommendations can be translated into concrete decisions to be taken and instructions to be followed during the design process. This paper will not provide an universal methodology for setting up a ground truth for any music classification system; it rather intends to give handles that help in building a reliable quality reference for such systems.