In this paper, we propose a novel approach to statistical model-based voice activity detection (VAD) that incorporates a second-order conditional maximum a posteriori (MAP) criterion. As a technical improvement for the first-order conditional MAP criterion in, we consider both the current observation and the voice activity decision in the previous two frames to take full consideration of the inter-frame correlation of voice activity. The soft decision scheme is incorporated to result in time-varying thresholds for further performance improvement. Experimental results show that the proposed algorithm outperforms the conventional CMAP-based VAD technique under various experimental conditions.