Replay-attack is a serious issue for automatic speaker verification (ASV) and recently lots of countermeasures have been proposed to protect ASV from spoofing attacks. Traditional countermeasures are a binary-classification system which was observed to have limited generalization on unseen attacks. Oneclass learning methods which have been widely used in anomaly detection is a promising method to enhance the robustness of replay detection system. In this paper, we propose a deep one-class learning scheme to model the genuine speeches in a compact embedding space. To reduce the variance of genuine embedding space, we design an architecture unit, called residual variability block, which can be flexibly integrated into usual convolutional neural networks. Comprehensive experiments show that the proposed deep one-class learning scheme is effective for replay attack detection under cross-database scenarios. Besides, in our internal collected dataset, such a scheme shows better robustness under mismatched conditions between the enrollment and test phase.