There has been increasing attention drawn to modelling inter-rater ambiguity in Continuous Emotion Recognition (CER) systems using probability distributions for arousal and valence. However, the relationship between modelling label ambiguity and robustness to noise, and more broadly, the impact of real-world noise on CER systems remains insufficiently explored. In this study, we argue that incorporating inter-rater ambiguity during training can regularize the noise response, leading to noise robustness. To this end, we propose a novel loss function that incorporates inter-rater ambiguity into model training. Experiments conducted on the RECOLA dataset demonstrate that our proposed method achieves a maximum Concordance Correlation Coefficient (CCC) improvement of 0.117 and 0.077 for mean and standard deviation predictions, respectively, across all noise conditions. We further integrate traditional noisy augmentation strategies with our proposed method and observe promising results.