Anomalous Sound Detection (ASD) requires modeling a compact and discriminative normal sound distribution. Recently, angular margin loss with multiple sub-centers has been shown to be effective for ASD by extending sub-centers to capture intra-class diversity and maximizing their orthogonality to enhance model discriminability. However, existing methods do not consider that the orthogonality of intra-class sub-centers needs to be optimized based on the inherent data structure to avoid over-extension of the representation space due to over-orthogonality. To address this issue, we propose a Dual Orthogonality Sub-Center Loss (DOSCL) that enforces strict orthogonality of inter-class sub-centers to improve anomaly discrimination while applying relaxed constraints on intra-class sub-centers to capture the data structure. Experiments on the DCASE2023 Challenge Task2 dataset show that DOSCL achieves 1.74% AUC and 0.62% pAUC improvements over a strong baseline, validating its effectiveness.