This paper describes a method of speaker adaptation for continuous mixture density HMMs (CDHMMs). Speaker adaptation in CDHMMs is regarded as a kind of retraining problem where a small amount of training data is available. The "Vector Field Smoothing method (VFS)" is used to deal with the problem of retraining with insufficient training data. "VFS" is applied simultaneously to inter-speaker and speaking-style adaptation. In this paper, the standard speaker is a male and the unknown speakers for adaptation are both one male and one female. When 11 sentences are uttered for adaptation phrase-by-phrase instead of word-by-word, the 23 phoneme recognition rate is 87.4% (none adaptation: 47.3%). The phrase recognition rate for HMM-LR is 85.1% (none adaptation: 21.5%).