Remote, automated cognitive impairment (CI) diagnosis has the potential to facilitate care for the elderly. Speech is easily collected over the phone and already some common cognitive tests are administered remotely, resulting in regular audio data collections. Speech-based CI diagnosis leveraging existing audio is therefore an attractive approach for remote elderly cognitive health monitoring. In this paper, we demonstrate the predictive power of several speech features derived from remotely collected audio used for common clinical cognitive testing. Specifically, using phoneme-based measures, pseudo-syllable rate, pitch variance, and articulatory coordination derived from formant cross-correlation measures, we investigate the capability of speech features, estimated from paragraph-recall and animal fluency test speech, to predict clinical CI assessment. Using a database consisting of audio from elderly subjects collected over a 4 year period, we develop support vector machine classification models of the CI clinical assessments. The best performing models result in an average equal error rate (EER) of 13.5%.