In real-world applications, a preferred personalized automatic speech recognition (ASR) model should exhibit robust recognition capabilities that include both personalization and generalization. Speaker adaptation is a commonly used approach towards personalized ASR. However, most speaker adaptation methods only focus on improving the performance of the target speaker but neglect or even sacrifice generalization. In this paper, we propose the Personality-memory Gated Adaptation (PGA) approach to adapt models on target speakers while preserving generalized recognition performance. Specifically, we incorporate parallel adapters into the encoder to capture the target speaker’s vocal characteristics. The hidden outputs of the adapters and original encoder layers are fused via a scalar gate, which is derived from the similarity between input samples and personality memory units, i.e., personality embeddings from the target speaker. In this manner, the vocal characteristics of the target speaker are memorized by adapters while the ASR backbone preserves generalization. Experimental results show that, compared to the unadapted ASR backbone, the proposed PGA achieves 21.62% relative character error rate (CER) reduction on the target domain but a negligible CER increase on the source domain with limited adaptation data and training steps.