In this paper, a new decoding method for unsupervised acoustic model adaptation is presented. In unsupervised adaptation framework, the effectiveness of adaptation process is greatly affected by the mis-recognized labels. Therefore, selection of the adaptation data guided by the confidence measures is effective in unsupervised adaptation. We propose phoneme error minimization framework for exact phoneme labels and use of phoneme-level confidence measures for improved unsupervised adaptation. Experimental results showed that the proposed method could reduce the mis-recognized labels in the adaptation process, and consequently improved the adaptation accuracy. Furthermore, it was confirmed that the proposed method is effective in an iterative unsupervised adaptation framework.