Deep Boltzmann machines
Published 15 April 2009
Ruslan Salakhutdinov, Geoffrey E. Hinton
Citations1,778
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.
Abstract
We present a new learning algorithm for Boltz-mann machines that contain many layers of hid-den variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and data-independent expectations are approximated us-ing persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden lay-ers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training ” phase that allows variational in-ference to be initialized with a single bottom-up pass. We present results on the MNIST and NORB datasets showing that deep Boltzmann machines learn good generative models and per-form well on handwritten digit and visual object recognition tasks. 1
Keywords
Computer Science
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
The Annals of Mathematical StatisticsA Stochastic Approximation Method
9,581 Citations1951Herbert Robbins, Sutton Monro
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
2,187 Citations1998Radford M. Neal, Geoffrey E. Hinton
An incremental variant of the EM algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step is shown empirically to give faster convergence in a mixture estimation problem.
MIT Press eBooksInformation processing in dynamical systems: foundations of harmony theory
1,666 Citations1986Paul Smolensky
The work reported in this chapter rests on the conviction that a methodology that has a crucial role to play in the development of cognitive science is mathematical analysis.
Learning methods for generic object recognition with invariance to pose and lighting
1,332 Citations2004Yann LeCun, Fu Jie Huang +1 more
A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second and proved impractical, while convolutional nets yielded 16/7% error.
Statistics and ComputingAnnealed importance sampling
1,225 Citations2001Radford M. Neal
It is shown how one can use the Markov chain transitions for such an annealing sequence to define an importance sampler, which can be seen as a generalization of a recently-proposed variant of sequential importance sampling.
Autoencoders, Minimum Description Length and Helmholtz Free Energy
1,101 Citations1993Geoffrey E. Hinton, Richard S. Zemel
It is shown that the recognition weights of an autoencoder can be used to compute an approximation to the Boltzmann distribution and that this approximation gives an upper bound on the description length.
Scaling learning algorithms towards AI
927 Citations2007Yoshua Bengio, Yann LeCun
It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.
Training restricted Boltzmann machines using approximations to the likelihood gradient
857 Citations2008Tijmen Tieleman
A new algorithm for training Restricted Boltzmann Machines is introduced, which is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data.
Machine LearningTraining Invariant Support Vector Machines
582 Citations2002Dennis DeCoste, Bernhard Schölkopf
Artificial IntelligenceConnectionist learning of belief networks
532 Citations1992Radford M. Neal
The “Gibbs sampling” simulation procedure for “sigmoid” and “noisy-OR” varieties of probabilistic belief networks can support maximum-likelihood learning from empirical data through local gradient ascent.
OPTIMAL PERCEPTUAL INFERENCE
461 Citations1983Geoffrey E. Hinton, J. Sejnowski
A particular nondeterministic operator is given, based on statistical mechanics, for updating the truth values of hypothcses, and a learning rule is described which allows a parallel system to converge on a set ofweights that optimizes its perccptt~al inferences.
Exponential Family Harmoniums with an Application to Information Retrieval
436 Citations2004Max Welling, Michal Rosen‐Zvi +1 more
An alternative two-layer model based on exponential family distributions and the semantics of undirected models is proposed, which performs well on document retrieval tasks and provides an elegant solution to searching with keywords.
On the quantitative analysis of deep belief networks
432 Citations2008Ruslan Salakhutdinov, Iain Murray
It is shown that Annealed Importance Sampling (AIS) can be used to efficiently estimate the partition function of an RBM, and a novel AIS scheme for comparing RBM's with different architectures is presented.
Stochastics and stochastics reportsOn the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates
142 Citations1999Laurent Younès
This work analyses the convergence of stochastic algorithms with Markovian noise when the ergodicity of the Markov chain governing the noise rapidly decreases as the control parameter tends to infinity and provides sufficient condition which ensure convergence.
Implicit Mixtures of Restricted Boltzmann Machines
76 Citations2008Vinod Nair, Geoffrey E. Hinton
Results for the MNIST and NORB datasets are presented showing that the implicit mixture of RBMs learns clusters that reflect the class structure in the data.
Learning and Evaluating Boltzmann Machines
58 Citations2008Ruslan Salakhutdinov
An annealed importance sampling (AIS) procedure for estimating partition functions of rest ricted Boltzmann machines (RBM’s), semi-restricted Boltz Mann machines (SRBM”s), an d Boltzman machines (BM) is developed and empirical results indicate that the AIS procedu re provides much better estimates of the partition function than some of the popular variational-based methods.