Large Language Models in Machine Translation
Published 22 June 2007
Thorsten Brants, Ashok C. Popat, Peng Xu, Franz Josef Och, Jay B. Dean
Citations550
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Systems, methods, and computer program products for machine translation are provided for backoff score determination as a function of a backoff factor and a relative frequency of a corresponding backoff n-gram in the corpus.
Abstract
This paper reports on the benefits of largescale statistical language modeling in machine translation. A distributed infrastructure is proposed which we use to train on up to 2 trillion tokens, resulting in language models having up to 300 billion n-grams. It is capable of providing smoothed probabilities for fast, single-pass decoding. We introduce a new smoothing method, dubbed Stupid Backoff, that is inexpensive to train on large data sets and approaches the quality of Kneser-Ney Smoothing as the amount of training data increases. 1
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
The mathematics of statistical machine translation: parameter estimation
4,125 Citations1993Peter F. Brown, Vincent J. Della Pietra +2 more
It is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus, given a set of pairs of sentences that are translations of one another.
Computer Speech & LanguageAn empirical study of smoothing techniques for language modeling
2,077 Citations1999Stanley F. Chen, Joshua Goodman
IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities from sparse data for the language model component of a speech recognizer
1,645 Citations1987Slava M. Katz
The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data, and compares favorably to other proposed methods.
Improved backing-off for M-gram language modeling
1,490 Citations2002Reinhard Kneser, Hermann Ney
This paper proposes to use distributions which are especially optimized for the task of back-off, which are quite different from the probability distributions that are usually used for backing-off.
Statistical Significance Tests for Machine Translation Evaluation.
1,465 Citations2004Philipp Koehn
Computational LinguisticsThe Alignment Template Approach to Statistical Machine Translation
933 Citations2004Franz Josef Och, Hermann Ney
A phrase-based statistical machine translation approach the alignment template approach is described, which allows for general many-to-many relations between words and is easier to extend than classical statistical machinetranslation systems.
Computer Speech & LanguageA bit of progress in language modeling
463 Citations2001Joshua Goodman
A combination of all techniques together to a Katz smoothed trigram model with no count cutoffs achieves perplexity reductions between 38 and 50% (1 bit of entropy), depending on training data size, as well as a word error rate reduction of 8.9%.
IEEE Signal Processing MagazineDynamic programming search for continuous speech recognition
173 Citations1999Hermann Ney, S. Ortmanns
A unifying view of the dynamic programming approach to the search problem from the statistical point-of-view is given and how the search space results from the acoustic and language models required by the statistical approach are shown.
IEEE International Conference on Acoustics Speech and Signal ProcessingThe estimation of powerful language models from small and large corpora
81 Citations1993Paul Placeway, Richard Schwartz +2 more
The authors consider the estimation of powerful statistical language models using a technique that scales from very small to very large amounts of domain-dependent data, and considers the problem of robustness of a model based on a small training corpus by grouping words into obvious semantic classes.
Vocabulary-independent search in spontaneous speech
69 Citations2004Frank Seide, Peng Yu +2 more
This work presents a vocabulary-independent system to index and to search rapidly spontaneous speech, and introduces a new method of phonetic word-fragment lattice generation, which uses longer-span language knowledge than a phoneme recognizer.
Distributed language modeling for<i>N</i>-best list re-ranking
54 Citations2006Ying Zhang, Almut Silja Hildebrand +1 more
A novel distributed language model based on the client/server paradigm for N-best list re-ranking that allows for using an arbitrarily large corpus in a very efficient way and provides a natural platform for relevance weighting and selection.
Large-Scale Distributed Language Modeling
31 Citations2007Ahmad Emami, Kishore Papineni +1 more
A novel distributed language model that has no constraints on the n-gram order and no practical constraints on vocabulary size is presented, which is scalable and allows for an arbitrarily large corpus to be queried for statistical estimates.
Estimating Language Models Using Hadoop and Hbase
11 Citations2008Xiaoyang Yu
The experiments evaluate and compare different table structures on training 100 million words for unigram, bigram and trigram models, and the results suggest a table based on half ngram structure is a good choice for distributed language model.