Stress can have a negative and costly impact on people's lives. Mitigating stress before it becomes a problem requires early, noninvasive identification and a deeper understanding of the signals of stress. To test automatic stress detection a new dataset was created with subjects completing the Stroop task under unstressed and stressed conditions. This paper examines to what degree speech features respond to stress and if so, what features are most informative. Features were extracted from recorded speech data and trained with several classification algorithms. We explored binary classification of stressed vs. unstressed across gender and per gender, with the best results on a held-out test set improving over the majority class baseline (MCB) by 16% across genders and with 20% and 21% for the female and male subsets respectively. Overall maximum intensity emerged as the most informative feature when comparing across classification conditions. In addition, we explored leave-one subject-out classification, resulting in a 15% improvement on average considering both genders when using random forests.