Analysis and modeling of speaker variability is important to help understand in-depth inter-speaker variances and to enhance current speech/speaker recognition system. In this paper we introduce adapted Gaussian mixture model (GMM) based speaker representation for the task. Two powerful multivariate statistical analysis methods, principal component analysis (PCA) and independent component analysis (ICA), are used to extract the sources of dominant speaker variability. In addition, analysis of variance (ANOVA) is adopted to evaluate the dominance of a factor in a certain principal/independent component. Further, the generalization ability of our method is investigated by experiments.