In any-to-any singing voice conversion (SVC), singing content can be encoded using either token-based or embedding-based approaches. Token-based methods often struggle with accurate content reconstruction, while embedding-based methods face significant timbre leakage. To address this trade-off, we propose a novel self-supervised learning (SSL)-based content representation method. By randomly selecting a subset of channels during training to serve as the new embedding and fixing them for subsequent SVC training, our approach achieves superior content modeling compared to token-based methods while mitigating timbre leakage typically observed in embedding-based approaches. We validate the effectiveness and generalizability of our method across SSL-based embeddings, SSL-based soft embeddings, and ContentVec.