A novel type of Gaussian mixture model for text-independent speaker verification, Hierarchical Gaussian Mixture Model (HGMM) is proposed in this paper. HGMM aims at maximizing the efficiency of MAP training on the Universal Background Model (UBM). Based on the hierarchical structure, the parameters of one Gaussian component can also be adapted by the observation vectors of neighboring Gaussian components. HGMM can also be considered as a generalized GMM which replaces the Gaussian component in the GMM models with a local GMM. The hierarchical Gaussian mixture description of the local observation space is better than one single Gaussian distribution. Experiment on NIST 99 Evaluation corpus shows that the HGMM achieves an 18% relative reduction in EER compared with the conventional GMM.