Efficient language model adaptation through MDI estimation

Published 5 September 1999

Marcello Federico

DOI10.21437/eurospeech.1999-401x

Citations52

TL;DR

The proposed method for n-gram language model adaptation based on the principle of minimum discrimination information has been evaluated on an Italian 60K-word broadcast news task and is extended to interpolated language models.

Abstract

This paper presents a method for n-gram language model adaptation based on the principle of minimum discrimination information. A background language model is adapted to fit constraints on its marginal distributions that are derived from new observed data. This work gives a different derivation of the model by Kneser et al. (1997) and extends its application to interpolated language models. The proposed method has been evaluated on an Italian 60K-word broadcast news task.

Keywords

Computer Science

IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities from sparse data for the language model component of a speech recognizer

1,645 Citations1987Slava M. Katz

The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data, and compares favorably to other proposed methods.

The Annals of Mathematical StatisticsGeneralized Iterative Scaling for Log-Linear Models

1,203 Citations1972J. N. Darroch, D. Ratcliff

View PDF

Computer Speech & LanguageOn structuring probabilistic dependences in stochastic language modelling

516 Citations1994Hermann Ney, U. Essen +1 more

The problem of stochastic language modelling is studied from the viewpoint of introducing suitable structures into the conditional probability distributions, and nonlinear interpolation as an alternative to linear interpolation; equivalence classes for word histories and single words; cache memory and word associations are considered.

Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)Adaptive Statistical Language Modeling: A Maximum Entropy Approach

276 Citations2018Roni Rosenfeld

This thesis views language as an information source which emits a stream of symbols from a finite alphabet (the vocabulary), and applies the principle of Maximum Entropy to identify and exploit sources of information in the language stream, so as to minimize its perceived entropy.

IEEE International Conference on Acoustics Speech and Signal ProcessingTrigger-based language models: a maximum entropy approach

175 Citations1993Raymond Y.K. Lau, Roni Rosenfeld +1 more

The model described here was trained on five million words of Wall Street Journal text and produced a language model whose perplexity was 12% lower than that of a conventional trigram, as measured on independent data.

Language model adaptation using dynamic marginals

91 Citations1997Reinhard Kneser, Jörg Peters +1 more

A new method is presented to quickly adapt a given language model to local text characteristics by choosing the adaptive models as close as possible to the background estimates while constraining them to respect the locally estimated unigram probabilities.

Adaptive language modeling using minimum discriminant estimation

29 Citations1992S. Della Pietra, V. Della Pietra +2 more

The authors present an algorithm to adapt a n-gram language model to a document as it is dictated that results in a perplexity of 208 instead of 290 for the static trigram model on a document of 321 words.