In this paper, we present our CRMI-DKU system description for the Short-duration Speaker Verification Challenge (SdSVC) 2021. We introduce the whole pipeline of our cross-lingual speaker verification system, including data preprocessing, training strategy, utterance-level speaker embedding extractor, domain-adaptation, and score calibration. We also propose methods to learn language-invariant features and perform domain adaptation to reduce the cross-lingual mismatch. In addition, we explore a semi-supervised method to utilize the unlabeled training data. The final submitted score level fusion system achieves 0.0476 minDCF and 0.98% EER on the evaluation set.