We emulate continual learning observed in real life, where new training data, which represent new application domain, are used for gradual improvement of an Automatic Speech Recognizer (ASR) trained on old domains. The data on which the original classifier was trained is no longer required and we observe no loss of performance on the original domain. Further, on previously unseen domain, our technique appears to yield slight advantage over offline multi-condition training. The proposed learning technique is consistent with our previously studied ad hoc stream attention based multi-stream ASR.