Age estimation from speech is a problem that has wide applications in call centers, virtual assistants and IoT devices. The estimated age is used for various system decisions related to personalization, parental control, and anomaly detection. Performance of speech based automatic age estimation systems is generally stated in the literature using the Mean Absolute Error (MAE) and Pearson Correlation Coefficient (PC). In real-world applications of these systems, MAE and PC provide little insight into the confidence of a point estimate. An MAE of 5 years on a test set provides only the average error across all estimates in the test set and hence does not provide any information about the confidence of each individual estimate. A confidence score of predicted age is essential to know the trustworthiness of the predictions made by the system. This paper formulates age estimation from speech as a label distribution learning problem to come up with a measure of the confidence related to the point estimate being within a desired range from the ground-truth. It further uses it to analyze the age estimation system under conditions of varying speech quality. We show that the proposed measure of confidence is better than a fixed error-margin.