In speech anti-spoofing, artefacts used to detect spoofed speech are often located in specific sub-bands. Previous works often use Convolution Neural Networks (CNNs) as backbone which are good at capturing local features. However, if artefacts simultaneously exist in different sub-bands, CNNs cannot model this kind of information. Thus, we propose to use Feature Pyramid Conformer to solve this issue. Conformer can capture both local and global features. We aggregate the outputs of each Conformer block with Feature Pyramid Module. Through addition and lateral connection, the aggregation can be better integrated. Besides, to improve generalization of detecting unknown attacks, we propose to adopt Elastic penalty Margin Softmax. It can enhance intra-class compactness and inter-class discrepancy flexibly. Without data augmentaion, our system achieve an Equal Error Rate (EER) of 1.65% on the evaluation set of ASVspooof 2019 logical access, outperforming most existing systems.