Deep context biasing improves rare word recognition in automatic speech recognition (ASR) but often struggles with larger biasing lists. Performance on Out-of-Vocabulary (OOV) words remains limited as the ASR model must learn to generate unseen token sequences. These limitations become more pronounced in contact center applications with prevalent business-specific terminologies. Additionally, using an existing ASR model is challenging, as deep biasing requires full joint training of the ASR model and the biasing module. To address these issues, we introduce 'spot and merge' (SAM), a novel Low-Rank Adapter (LoRA) based system that spots bias words in the cross-attention weights of the biasing module and merges them with ASR output. Unlike existing methods, our approach maintains strong performance even with larger biasing lists, achieving a 1.0% absolute word error rate (WER) reduction on LibriSpeech. It also demonstrates robust OOV recognition on an in-house contact center dataset.