Accurate estimation of room impulse responses (RIRs) is crucial for applications like augmented reality and sound field modeling. Current methods either neglect the spatial relationships between the source and receiver or rely on computationally intensive volumetric grids or panoramic images to estimate RIRs. To address these challenges, we introduce TA-RIR, a topology-aware neural network that uses spatial coordinates of sources and receivers, along with reverberant speech, to learn compact embeddings encoding room geometry and acoustics. The topology-aware encoder captures structural relationships between spatial and acoustic features, integrated through a propagation-informed decoder to synthesize RIRs. Experimental results show that TA-RIR generates high-fidelity RIRs, accurately preserving target acoustic parameters such as reverberation time, while significantly reducing computational complexity compared to methods requiring detailed 3D models or room acoustic properties.