We propose a spatially weighted contrastive loss (SWeC loss) for sound source localization in real-world scenarios using multi-channel speech data. In multi-channel localization, phase differences between microphone channels provide critical cues for estimating the azimuth angle of incoming speech. To effectively extract azimuth information, we leverage contrastive learning and introduce a novel loss function that incorporates spatial relationships between azimuth classes. Specifically, our loss assigns weights to negative pairs based on their angular distance, penalizing high similarity between embeddings corresponding to distant angles. Furthermore, we propose a contrastive data generation method tailored to multi-channel localization, enhancing the effectiveness of contrastive learning. Experimental results demonstrate that the proposed loss function and data generation strategy significantly improve localization performance.