A hierarchical Dirichlet language model
Natural Language EngineeringPublished 1 September 1995
David Mackay, Linda C. Bauman Peto
Citations276
SJR quartileQ1
SJR score0.64
SNIP1.66
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A hierarchical probabilistic model whose predictions are similar to those of the popular language modelling procedure known as ‘smoothing’ is discussed, and the methods prove to be about equally accurate, with the hierarchical model using fewer computational resources.
Abstract
Abstract We discuss a hierarchical probabilistic model whose predictions are similar to those of the popular language modelling procedure known as ‘smoothing’. A number of interesting differences from smoothing emerge. The insights gained from a probabilistic view of this problem point towards new directions for language modelling. The ideas of this paper are also applicable to other problems such as the modelling of triphomes in speech, and DNA and protein sequences in molecular biology. The new algorithm is compared with smoothing on a two million word corpus. The methods prove to be about equally accurate, with the hierarchical model using fewer computational resources.
Keywords
Computer Science
Numerical recipes in C
15,253 Citations1994William H. Press, Saul A. Teukolsky +2 more
The Diskette v 2.06, 3.5''[1.44M] for IBM PC, PS/2 and compatibles [DOS] Reference Record created on 2004-09-07, modified on 2016-08-08.
The mathematics of statistical machine translation: parameter estimation
4,125 Citations1993Peter F. Brown, Vincent J. Della Pietra +2 more
It is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus, given a set of pairs of sentences that are translations of one another.
The Annals of StatisticsMixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems
2,030 Citations1974C. Antoniak
This paper gives a formal definition for Dirichlet process mixtures and develops several theorems about their properties, the most important of which is a closure property for such mixtures.
IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities from sparse data for the language model component of a speech recognizer
1,645 Citations1987Slava M. Katz
The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data, and compares favorably to other proposed methods.
American Journal of PhysicsProbability, Frequency and Reasonable Expectation
1,293 Citations1946R. T. Cox
IEEE Transactions on Pattern Analysis and Machine IntelligenceA Maximum Likelihood Approach to Continuous Speech Recognition
1,220 Citations1983L.R. Bahl, Frederick Jelinek +1 more
This paper describes a number of statistical models for use in speech recognition, with special attention to determining the parameters for such models from sparse data, and describes two decoding methods appropriate for constrained artificial languages and one appropriate for more realistic decoding tasks.
A program for aligning sentences in bilingual corpora
956 Citations1993William A. Gale, Kenneth Church
This paper will describe a method and a program for aligning sentences based on a simple statistical model of character lengths, which uses the fact that longer sentences in one language tend to be translated into longer sentence in the other language, and that shorter sentences tend to been translated into shorter sentences.
Statistics and ComputingLearning classification trees
395 Citations1992Wray Buntine
This paper introduces Bayesian techniques for splitting, smoothing, and tree averaging, which are similar to Quinlan's information gain, while smoothing and averaging replace pruning.
The COCOON platform (University of Paris)An estimate of an upper bound for the entropy of English
354 Citations1992Peter F. Brown, Vincent J. Della Pietra +3 more
An estimate of an upper bound of 1.75 bits for the entropy of characters in printed English is presented by constructing a word trigram model and then computing the cross-entropy between this model and a balanced sample of English text.
Developments in Maximum Entropy Data Analysis
256 Citations1989Stephen F. Gull
The derivation of “Classic” MaxEnt is completed, showing that it leads to a natural choice for the regularising parameter α, that supersedes the traditional practice of setting x2=N, and the new condition is that the dimensionless measure of structure -2αS should be equal to the number of good singular values contained in the data.
Classic Maximum Entropy
231 Citations1989John Skilling
This paper presents a fully Bayesian derivation of maximum entropy image reconstruction, formalised as the axioms ofmaximum entropy, which shows that the prior probability distribution for any positive, additive distribution must be monotonic in the entropy.
Pattern Recognition LettersPattern Recognition in Practice VI
192 Citations1999E. S. Gelsema, Jifke F. Veenland
This special issue contains most of the papers presented at the conference ``Pattern Recognition in Practice VI'' held on Vlieland, The Netherlands, from 2 to 4 June 1999, to stimulate interaction between pattern recognition professionals and scientists from other disciplines who bring pattern recognition techniques into practice.
Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated EquipmentBayesian neural networks and density networks
178 Citations1995David Mackay
This paper reviews the Bayesian approach to learning in neural networks, then introduces a new adaptive model, the density network, a neural network for which target outputs are provided, but the inputs are unspecified.
Bayesian Mixture Modeling
147 Citations1992Radford M. Neal
It is shown that Bayesian inference from data modeled by a mixture distribution can feasibly be performed via Monte Carlo simulation, and the true Bayesian predictive distribution is exhibited, implicitly integrating over the entire underlying parameter space.
IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities in the language model of the IBM speech recognition system
89 Citations1984Arthur Nádas
The predictive power of the model thus fitted is compared by means of its experimental perplexity to the model as fitted by the Jelinek-Mercer deleted estimator and by the Turing-Good formulas for probabilities of unseen or rarely seen events.
Hyperparameters: Optimize, or Integrate Out?
89 Citations1996David Mackay
Two approximate methods for computational implementation of Bayesian hierarchical models which include unknown hyperparameters such as regularization constants are examined, and comparisons are made with the ideal hierarchical Bayesian solution.
Bayesian classification with correlation and inheritance
52 Citations1991Robin Hanson, John Stutz +1 more
The task of inferring a set of classes and class descriptions most likely to explain a given data set can be placed on a firm theoretical foundation using Bayesian statistics, and using various mathematical and algorithmic approximations the Auto Class system searches for the most probable classifications.
Elsevier eBooksMean field networks that learn to discriminate temporally distorted strings
34 Citations1991Christopher K. I. Williams, Geoffrey E. Hinton
This type of mean field network (MFN) with tied weights that is capable of approximating the recognizer for a hidden markov model (HMM) is presented as a way of allowing more powerful representations without abandoning the automatic parameter estimation procedures.
arXiv (Cornell University)A Comparison of Two Smoothing Methods for Word Bigram Models
3 Citations1994Linda C. Bauman Peto
Two smoothing methods for word bigram models estimated from text corpora were trained on a two-million-word corpus taken from the Canadian _Hansard_ and compared on the basis of the experimental perplexity that they assigned to a shared test corpus.