Classical speaker verification metrics used to evaluate voice anonymization systems, such as the equal error rate (EER), fail to properly quantify the residual re-identification risk. This paper introduces a new evaluation framework based on two metrics, Linkability and Singling Out, derived from the legal definitions in the Article 29 Working Party's Opinion 05/2014 on Anonymization Techniques endorsed by the European Data Protection Board (EDPB). Our framework translates these legal concepts into quantitative metrics for speech data. The proposed framework has been legally validated by the French Data Protection Authority. Experiments across various attack scenarios reveal that, while the EER remains stable, Linkability and Singling Out vary much more. This demonstrates that the residual privacy risk after anonymization is far more variable than indicated by the EER, underscoring the need for evaluation metrics that align with legal criteria.