A new technique is proposed to estimate the robust continuous observation densities of hidden Markov model (HMM) for improving the performance of speaker-independent (SI) automatic speech recognition system. First, a scheme of generalized common vector (GCV), which originated from the common vector approach (CVA), is proposed. The objective of this scheme is to extract a robust speech feature over different speakers. That is, we attempt to obtain a common feature to represent an invariant characteristic over many speakers. Then, based on this scheme, we construct a GCV-based HMM (GCVHMM). An element to extract GCV is integrated into HMM. A re-estimation algorithm for the parameters of GCVHMM is also derived.