We have already proposed novel robust parameter estimation algorithms of time-varying complex AR (TV-CAR) model for analytic speech signal, which are based on GLS (Generalized Least Square) and its modification of ELS (Extended Least Square). The ELS method is more sophisticated method that can be derived by matrix inversion theorem. We have shown that the methods can achieve robust speech spectrum estimation against additive white Gaussian. However, these methods suffer from spectral gaps between adjacent analysis frames. This paper proposes improved ELS-based TV-CAR speech analysis using forward and backward linear prediction. The experiment with natural speech demonstrates that the improved method can estimate more smooth time-varying speech spectrum.