To achieve 3D audio rendering, high-quality Head-Related Transfer Functions (HRTFs) are essential. As measuring HRTFs is time-consuming and tedious, spatial interpolation is often adopted to generate high-resolution HRTFs from low-resolution ones. In this paper, we propose a Dual-Graph Attention Network (D-GAT) for HRTF upsampling. Specifically, we first design a branch of GAT to learn the relationship among HRTFs from adjacent points. In addition, we introduce another branch of GAT to find a mapping from physical features (including the absolute target position and the anthropometric characteristics) to HRTFs. By combining such two GATs in a parallel architecture, the D-GAT is built. Furthermore, a dynamic edge weighting mechanism is adopted in the D-GAT, which allows the model to learn geometry relationships among vertices more flexibly. Experimental results demonstrate the efficacy of the proposed D-GAT in accurately predicting HRTFs, yielding state-of-the-art performance.