The current study examined vowel production data from a high-variability phonetic training project (HVPT) focusing on the English vowel contrast /æ/-/ᴧ/ to compare the sensitivity of various acoustic measures of vowel overlap and distinctiveness (Pillai scores and Euclidean and Mahalanobis distances) in capturing L2 learners' training gains in (a) qualitatively distinguishing between /æ/ and /ᴧ/ in production, and (b) reducing qualitative differences with respect to native speakers' productions of /æ/ and /ᴧ/. A subset of these data was then perceptually judged by a group of native listeners for pronunciation accuracy. The results showed that all acoustic distance measures consistently captured phonetic training gains in measures of contrast distinctiveness and nativelikeness, correlating strongly with one another, and predicting perceptual distance measures. However, measures also differed substantially in effect sizes for training gains (pre- vs- post-test distance scores), training group differences (lexical vs. non-lexical training) and vowel (/æ/ vs. /ᴧ/). Acoustic distance measures were found to predict perceptual distance scores based on native English listeners' ratings of pronunciation accuracy. These findings suggest that when assessing gains in the pronunciation of L2 vowel contrasts it is important to measure both acoustic distances between vowels and acoustic distances between learners' and native speakers' productions.