This work proposes a novel framework for automatically scoring children's oral narrative language abilities. We use audio recordings from 3rd-8th graders of the Atlanta, Georgia area as they take a portion of the Test of Narrative Language. We design a system which extracts linguistic features and fine-tuned BERT-based self-supervised learning representation from state-of-the-art ASR transcripts. We predict manual test scores from the extracted features. This framework significantly outperforms a deterministic method based on the assessment's scoring rubric. Last, we evaluate the system performance across student's reading level, dialect, and diagnosed learning/language disabilities to establish fairness across diverse demographics of students. Using this system, we achieve approximately 98% classification accuracy of student scores. We are also able to identify key areas of improvement for this type of system across demographic areas and reading ability.