In this paper, we propose a novel model based adaptation for deep neural networks based on a linear least square method. Our proposed algorithm can perform unsupervised adaptation even if the auto transcripts may have 60-70% of word error rate. We evaluate our algorithm on low resource languages, from the IARPA BABEL program, such as Assamese, Bengali, Haitian Creole, Lao and Zulu. Our experiments focus on unsupervised speaker, dialect and environment adaptation and we show that it can improve both speech recognition and keyword search performance.