While modern CNN-based speaker recognition systems have achieved impressive accuracy, their computational and memory demands continue to pose challenges for on-device deployments, such as in smart home devices and consumer electronics. We propose an iterative channel pruning approach that is Hard set-aware, ensuring that channels essential for discriminating near-boundary cases are preserved. Our framework employs an angular medoid algorithm to dynamically partition samples into a hard set and a normal set. Channel importance is computed by integrating gradient-based measures, SE attention, and embedding perturbation, with the contributions of these metrics experimentally optimized. Coupled with iterative fine-tuning and pruning ratio adjustments, our method efficiently reduces model parameters while maintaining robust performance on critical hard set samples, thereby mitigating the severe degradation typically observed in conventional one-shot pruning approaches.