We have already proposed a tree-structured speaker clustering method and its application to supervised speaker adaptation. This adaptation method is based on the selection of a speaker cluster from among multiple reference speaker clusters. Since the adaptation method employs cluster selection rather than parameter training, it can adapt quickly using only a small amount of training data. In this paper, we extend this method for application to unsupervised speaker adaptation and speaker-independent speech recognition. The results show that the adaptation method using short calibration speech (less than 5 sec) outperforms a speaker-independent recognition system.