We present a directed Markov random field (MRF) model that combines � -gram models, probabilistic context free grammars (PC FGs) and probabilistic latent semantic analysis (PLSA) for the purpose of statistical language modeling. The composite directed MRF model has potentially exponential number of loops and becomes context sensitive grammar, nevertheless we are able to estimate its parameters in cubic time using an efficient modified EM method, the generalized inside-outside algorithm, which extends inside-outside algorithm to incorporate the effects of the � -gram and PLSA language models.