Recent works have demonstrated that the integration of group-level personality and vocal behaviors can provide enhanced prediction power on task performance for small group interactions. In this work, we propose that the impact of member personality for task performance prediction in groups should be explicitly modeled from both intra and inter-group perspectives. Specifically, we propose a Graph Interlocutor Acoustic Network (G-IAN) architecture that jointly learns the relationship between vocal behaviors and personality attributes with intra-group attention and inter-group graph convolutional layer. We evaluate our proposed G-IAN on two group interaction databases and achieve 78.4% and 72.2% group performance classification accuracy, which outperforms the baseline model that models vocal behavior only by 14% absolute. Further, our analysis shows that Agreeableness and Conscientiousness demonstrate a clear positive impact in our model that leverages the inter-group personality structure for enhanced task performance prediction.