In this paper, the statistical method of Factor Analysis (FA) is studied on Gaussian Mixture Model (GMM) based speaker identification (SI) system to model the data covariance which is usually neglected due to the training data sparseness. Because the variance of GMM can represents speaker variability, it is very important in SI systems. By FA modeled the data covariance, a relative gain of 39.6% over GMM baseline can be seen at the same amount of training data. Parameter tying is important when data is sparse and is helpful to balance precision and generalization of models. Various tying strategies are studied in this paper too. With the best result, a relative gain of 48.9% gain can be seen. We also present some interpretations of the factors tentatively.