Self-supervised learning has become the dominant paradigm for speech representation learning, yet its impact on speaker diarization has not been systematically studied at scale. In this work, we present the first large-scale study of how key SSL design choices affect end-to-end diarization. Using WavLM models trained on subsets of the Yodas2 corpus and diarization benchmarks, we analyze four factors: (i) prediction targets for SSL pretraining, (ii) pretraining data scale, (iii) model capacity and pruning, and (iv) segmentation architecture and chunk duration. Our experiments show that combining representations from multiple SSL layers using a learned weighted average yields the best performance. Scaling training data and model capacity further improves results, though at increased computational cost. Finally, we show that Conformer-based segmentation models outperform LSTM baselines. These results provide practical guidelines for building SSL models for downstream diarization tasks.