Total variability model (TVM) was recently proposed for the com-pression of speech utterances to low dimensional vectors (i.e., the so-call identity vector or i-vector). Compared to the variable-length nature of the speech utterances, the i-vectors have fixed length and therefore could be used with simple classifier for text-independent speaker verification task. This paper proposes the local variability model (LVM) the central idea of which is to capture the local vari-ability associated with individual Gaussians in the acoustic space that are absent in the i-vector representation. We analyze the latent structure of both the total and local variability model and show that parameter tying across mixtures leads to powerful methods for information extraction. Experimental results on NIST SRE’08 and SRE’10 datasets show that the proposed LVM is effective for speaker verification.