In a real environment, it is essential to adapt acoustic models to variations in background noises in order to realize robust speech recognition. In this paper, we construct an extended acoustic model by combining a mismatch model with a clean acoustic model trained using only clean speech data. We assume the mismatch model conforms to a Gaussian distribution with time-varying population parameters. The proposed method adapts on-line the extended acoustic model to the unknown noises by estimating the time-varying population parameters using a Gaussian Mixture Model (GMM) and Gain-Adapted Hidden Markov Model (GA-HMM) decomposition method. We performed recognition experiments under noisy conditions using the AURORA2 database in order to confirm the effectiveness of the proposed method.