A maximum entropy approach to natural language processing
Computational LinguisticsPublished 1 March 1996
Adam Berger, Vincent J. Della Pietra, Stephen A. Della Pietra
Citations3,120
SJR quartileQ1
SJR score1.15
SNIP4.16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A maximum-likelihood approach for automatically constructing maximum entropy models is presented and how to implement this approach efficiently is described, using as examples several problems in natural language processing.
Abstract
The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing.
Keywords
Computer SciencePhysics and Astronomy
Journal of the Royal Statistical Society Series B (Statistical Methodology)Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
49,657 Citations1977A. P. Dempster, N. M. Laird +1 more
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
The mathematics of statistical machine translation: parameter estimation
4,125 Citations1993Peter F. Brown, Vincent J. Della Pietra +2 more
It is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus, given a set of pairs of sentences that are translations of one another.
Class-based n -gram models of natural language
2,893 Citations1992Peter F. Brown, P.V. deSouza +3 more
This work addresses the problem of predicting a word from previous words in a sample of text and discusses n-gram models based on classes of words, finding that these models are able to extract classes that have the flavor of either syntactically based groupings or semanticallybased groupings, depending on the nature of the underlying statistics.
A statistical approach to machine translation
1,699 Citations1990Peter F. Brown, John Cocke +6 more
The application of the statistical approach to translation from French to English and preliminary results are described and the results are given.
The Annals of Probability$I$-Divergence Geometry of Probability Distributions and Minimization Problems
1,648 Citations1975Imre Csiszár
The Annals of Mathematical StatisticsGeneralized Iterative Scaling for Log-Linear Models
1,203 Citations1972J. N. Darroch, D. Ratcliff
Journal of The Electrochemical SocietyMathematics of Physics and Modern Engineering
360 Citations1958I. S. Sokolnikoff, R. M. Redheffer +1 more
IEEE Transactions on Acoustics Speech and Signal ProcessingA tree-based statistical language model for natural language speech recognition
253 Citations1989L.R. Bahl, Peter F. Brown +2 more
Algorithms are presented for automatically constructing a binary decision tree designed to estimate the probability that a given word will be the next word uttered, which is compared to an equivalent trigram model and shown to be superior.
Towards history-based grammars
145 Citations1993Ezra Black, Fred Jelinek +4 more
HBG incorporates lexical, syntactic, semantic, and structural information from the parse tree into the disambiguation process in a novel way and significantly outperforms P-CFG, increasing the parsing accuracy rate from 60% to 75%, a 37% reduction in error.
The Candide system for machine translation
140 Citations1994Adam Berger, Peter F. Brown +7 more
An overview of Candide, a system for automatic translation of French text to English text, which uses methods of information theory and statistics to develop a probability model of the translation process to generate English translations of previously unseen French sentences.
A statistical approach to sense disambiguation in machine translation
105 Citations1991Peter F. Brown, Stephen A. Della Pietra +2 more
A statistical technique for assigning senses to words is described and an instance of a word is assigned a sense by asking a question about the context in which the word appears.
Information and ControlA note on approximations to discrete probability distributions
95 Citations1959David T. Brown
An iterative method is presented which gives an optimum approximation to the Joint probability distribution of a set of binary variables given the joint probability distributions of any subsets of the variables (any set of component distributions).
An information theoretic approach to the automatic determination of phonemic baseforms
93 Citations2005Jan Lucassen, R. L. Mercer
A procedure, based on information theoretic principles, for discovering a set of spelling-to-sound rules automatically and a simple phoneme recognizer, which is able to predict the correct phoneme 97.1% of the time.
Towards history-based grammars
91 Citations1992Ezra Black, Fred Jelinek +4 more
HBG incorporates lexical, syntactic, semantic, and structural information from the parse tree into the disambiguation process in a novel way, and significantly outperforms P-CFG, increasing the parsing accuracy rate from 60% to 75%, a 37% reduction in error.
Tagging text with a probabilistic model
79 Citations1991Bernard Mérialdo
Experiments show that the best training is obtained by using as much tagged text as is available, and a maximum likelihood training may improve the accuracy of the tagging.
The Annals of StatisticsA Geometric Interpretation of Darroch and Ratcliff's Generalized Iterative Scaling
77 Citations1989Imre Csiszár
Notes on Present Status and Future Prospects
77 Citations1991E. T. Jaynes
arXiv (Cornell University)Inducing Features of Random Fields
46 Citations1995S. Della Pietra, V. Della Pietra +1 more
Decision SciencesAn Information‐Theoretic Approach*
44 Citations1971Jerome B. Kernan, George H. Haines
The search aspect of decision making is examined by construing the information environment in the form of a maze and measuring “transmitted” information according to the Shannon-Weaver logarithmic metric to assess the power of subjects' search heuristics.
Lecture notes in computer scienceInference and estimation of a long-range trigram model
40 Citations1994S. Della Pietra, V. Della Pietra +4 more
A novel technique for indexing the model parameters that allows us to avoid all sorting in the M-step of the training algorithm results in significant savings in computation time, and is applicable to the training of a general probabilistic link grammar.
Continuous speech recognition with automatically selected acoustic prototypes obtained by either bootstrapping or clustering
31 Citations2005Arthur Nádas, R. L. Mercer +6 more
Two methods for automatically obtaining a set of acoustic prototypes for use by a centisecond labeling acoustic processor are described, one of which is based on bootstrapping, the other on clustering.