To realize good speaker adaptation for context dependent HMM using small-size training data, reasonable adaptation of unseen models have to be realized using the relation of appeared models and the training data. In the paper, a new speaker adaptation method for context dependent HMM using two spatial constraints is proposed: 1) spatial relation of the phoneme context hierarchical models, and 2) spatial relation between speaker specific models and speaker independent models. Several implementations based on the idea are proposed and are evaluated under 520 word speech recognition. 25 words are used for adaptation par speaker. The best result improved 30% error rate showing the effectiveness of the proposed method.