Quick Training of Probabilistic Neural Nets by Importance Sampling
Published 3 January 2003
Yoshua Bengio, Jean-Sébastien Senécal
Citations149
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Inspired by the contrastive divergence model, sampling-based methods which require network passes only for the observed “positive example” and a few sampled negative example words are proposed and evaluated.
Abstract
Our previous work on statistical language modeling introduced the use of probabilistic feedforward neural networks to help dealing with the curse of dimensionality. Training this model by maximum likelihood however requires for each example to perform as many network passes as there are words in the vocabulary. Inspired by the contrastive divergence model, we propose and evaluate sampling-based methods which require network passes only for the observed "positive example" and a few sampled negative example words. A very significant speed-up is obtained with an adaptive importance sampling.
Keywords
Computer Science
Foundations of statistical natural language processing
9,996 Citations1999Christopher D. Manning, Hinrich Schütze
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
Computational LinguisticsA maximum entropy approach to natural language processing
3,120 Citations1996Adam Berger, Vincent J. Della Pietra +1 more
A maximum-likelihood approach for automatically constructing maximum entropy models is presented and how to implement this approach efficiently is described, using as examples several problems in natural language processing.
Learning in Graphical Models
2,416 Citations1998Michael I. Jordan
Computational LinguisticsHead-Driven Statistical Models for Natural Language Parsing
1,860 Citations2003Michael Collins
Three statistical models for natural language parsing are described, leading to approaches in which a parse tree is represented as the sequence of decisions corresponding to a head-centered, top-down derivation of the tree.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities from sparse data for the language model component of a speech recognizer
1,645 Citations1987Slava M. Katz
The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data, and compares favorably to other proposed methods.
Statistical ScienceCombining Probability Distributions: A Critique and an Annotated Bibliography
1,000 Citations1986Christian Genest, James V. Zidek
A taxonomy of solutions is presented which serves as the framework for a survey of recent theoretical developments in the area and a number of current research directions are mentioned and an extensive, current annotated bibliography is included.
Learning distributed representations of concepts.
821 Citations1989Geoffrey E. Hinton
Computer Speech & LanguageStructured language modeling
275 Citations2000Ciprian Chelba, Frederick Jelinek
An attempt at using the syntactic structure in natural language for improved language models for speech recognition using an original probabilistic parameterization of a shift-reduce parser.
Exploiting Tractable Substructures in Intractable Networks
221 Citations1995Lawrence K. Saul, Michael I. Jordan
A refined mean field approximation for inference and learning in probabilistic neural networks is developed, and it is shown how to incorporate weak higher order interactions into a first-order hidden Markov model.
IEEE Transactions on Neural NetworksTaking on the curse of dimensionality in joint distributions using neural networks
120 Citations2000Samy Bengio, Yoshua Bengio
Neural ComputationBias/Variance Decompositions for Likelihood-Based Estimators
102 Citations1998Tom Heskes
A similar simple decomposition is derived, valid for any kind of error measure that, when using the appropriate probability model, can be derived from a Kullback-Leibler divergence or log-likelihood.
Text prediction for translators
29 Citations2002Guy Lapalme, George Foster
A novel tool intended to give a translator interactive access to the most powerful translation technology available: a machine translation system is proposed, to use the target text being produced as the medium of interaction with the computer.