Excessive sleepiness is a major public and personal health burden that would benefit from being measured in ecological and passive setups. Speech recording is implemented in all smartphones and is thus a relevant tool to do so. To evaluate the feasibility of detecting sleepiness from speech by the human hearing, two previous perceptual studies on 90 samples from the SLEEP corpus have been conducted (Huckvale et al. 2020, Martin et al. 2023), which yielded contrasting results. A way to investigate the origin of this disagreement would have been to study on which speech characteristics the listeners have based their estimation. However, none of these studies have collected such information. In this study, we identify these characteristics by extracting speech features from the recordings, and training simple and explainable machine learning models to reproduce the annotation of each listener. Then, we measure the contribution of each feature to the decision of each model, and identify the most important ones. We then perform hierarchical clustering to draw listeners' profiles, depending on the features they rely on to identify sleepiness.