This paper proposes an algorithm for speech parameter generation from continuous mixture HMMs which include dynamic features, i.e., delta and delta-delta parameters of speech. We show that the parameter generation from HMMs using the dynamic features results in searching for the optimal state sequence and solving a set of linear equations for each possible state sequence. To solve the problem, we derive a fast algorithm on the analogy of the RLS algorithm for adaptive filtering. We show that the generated speech parameter vectors reflect not only the means of static and dynamic feature vectors but also the co-variances of those. An example presenting effectiveness of the proposed algorithm in speech synthesis is given.