A recurring problem in speech and language modelling is the estimation of context-specific probability distributions from sparse data. Robust estimates can be obtained by interpolating the probability estimates obtained from context-specific statistics with more general ones. An interpolation technique is described here which, is based on a least-squares weighting formula but with deleted estimation incorporated to optimise its parameters. Perplexity results are given for various statistical language models incorporating this interpolation technique.