Personalized reading instruction requires continuing, up-to-date assessment through word and text reading tests. Current tests administered by teachers who have to assess for each read word whether it is correct or incorrect are time-consuming, mentally demanding, and error-prone. Automatic assessment through ASR technology would be welcome, but needs to be investigated. In this study, we analysed speech recordings of six different first graders (7-8 years old) reading 144 words which were assessed by 51 teachers, three phoneticians, a Kaldi-based ASR model and an end-to-end ASR model. We investigated to what extent the teachers agree, how to optimally interpret the ASR output, and how readings with phonetic transcriptions that deviate from the prompt are assessed by the teachers and the ASR models. We discuss the results in relation to previous studies and outline avenues for future research.