Automatic personality detection from language has applications in diverse domains. Most previous efforts to automatically detect personality have aimed to predict either self-reported personality measures or personality labels provided by observers. It is unclear which kind of personality labels are preferable for personality recognition tasks or whether they are correlated with each other. In this work we aim to understand how self-reported and observer-reported measures of personality relate to each other. We conduct a study of personality ratings by collecting personality judgments from external raters using a corpus of spontaneous speech from speakers with self-reported personality scores. We then proceed to build and compare predictive models of self-reported personality scores and observer-reported personality. Finally, we explore the effect of modality on observer-reported personality judgments, comparing judgments of audio stimuli with judgment of transcribed speech.