The widespread adoption of cloud-based speaker verification raises privacy concerns, as unauthorized access to speaker voice can be exploited for fraudulent purposes. In response, privacy preserving speaker verification (PPSV) schemes have emerged to safeguard speaker information and prevent unauthorized access. However, existing PPSV methods often compromise accuracy in their efforts to protect privacy. To enhance both privacy and accuracy, we propose an end-to-end framework for speaker template encryption that simultaneously optimizes verification and encryption tasks. Our approach enhances the security and accuracy of speaker verification by introducing quintuple loss and sharpness-aware minimization, respectively, eliminating texture details that could be exploited by attackers. We distill our methodologies into a PPSV method, Secure-RawNet. Experimental results demonstrate that Secure-RawNet achieves high accuracy and robust privacy protection.