Speech intelligibility (SI) is essential in communication and second language learning. In this study, non-native SI was measured through Visual Analogue Scale (VAS) scores and Orthographic Transcriptions (OTs) of read aloud sentences. Seven measures automatically derived from the OTs at word and subword levels were studied. The reliability of the intelligibility measures and the correlations between VAS scores and OT-based measures were also explored. Despite the different speaker language backgrounds, the recruited raters exhibited high scoring reliability. The correlations between VAS scores and OT-based measures were weak, corroborating previous assumptions that they refer to two related but distinct notions, comprehensibility (VAS) and intelligibility (OT). OT-based measures are reliable and valid indicators of SI. The results are discussed in relation to previous studies and avenues for future research are proposed.