Deep neural network-based speech enhancement (SE) algorithms often utilize an encoder - dual-path processing module - decoder architecture. The encoder, consisting of pointwise convolutions and a dilated densenet (DDN), maps time-frequency bins to a higher-dimensional space to better capture signal and noise characteristics but significantly adds to the computational complexity. In this work, we propose a computationally efficient modification by replacing the DDN block within the encoder with bispectral features, which effectively capture frequency correlations and phase information. We demonstrate the effectiveness of the method by incorporating it in two SE models, CMGAN and MP-SENet, and evaluating on two datasets, DNS-2020 and VoiceBank+DEMAND. Our experiments show that the proposed modification achieves similar or better performance while reducing the multiply-accumulate operations by 23.87% in CMGAN and 17.45% in MP-SENet, demonstrating its computational efficiency.