Training restricted Boltzmann machines using approximations to the likelihood gradient
Published 1 January 2008
Tijmen Tieleman
Citations857
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new algorithm for training Restricted Boltzmann Machines is introduced, which is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data.
Abstract
A new algorithm for training Restricted Boltzmann Machines is introduced. The algorithm, named Persistent Contrastive Divergence, is different from the standard Contrastive Divergence algorithms in that it aims to draw samples from almost exactly the model distribution. It is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data. The Persistent Contrastive Divergence algorithm outperforms the other algorithms, and is equally fast and simple.
Keywords
Computer Science
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
The Annals of Mathematical StatisticsA Stochastic Approximation Method
9,581 Citations1951Herbert Robbins, Sutton Monro
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks
4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
Journal of the Royal Statistical Society Series B (Statistical Methodology)On the Statistical Analysis of Dirty Pictures
4,228 Citations1986Julian Besag
A simple, iterative method of reconstruction is proposed, which does not depend on these large-scale characteristics of the random field, and is illustrated by computer simulations in which the original scene is not directly related to the assumed random field.
now publishers, Inc. eBooksGraphical Models, Exponential Families, and Variational Inference
3,154 Citations2007Martin J. Wainwright, Michael I. Jordan
Restricted Boltzmann machines for collaborative filtering
1,882 Citations2007Ruslan Salakhutdinov, Andriy Mnih +1 more
This paper shows how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM's), can be used to model tabular data, such as user's ratings of movies, and demonstrates that RBM's can be successfully applied to the Netflix data set.
MIT Press eBooksInformation processing in dynamical systems: foundations of harmony theory
1,666 Citations1986Paul Smolensky
The work reported in this chapter rests on the conviction that a methodology that has a crucial role to play in the development of cognitive science is mathematical analysis.
An empirical evaluation of deep architectures on problems with many factors of variation
1,030 Citations2007Hugo Larochelle, Dumitru Erhan +3 more
A series of experiments indicate that these models with deep architectures show promise in solving harder learning problems that exhibit many factors of variation.
On Contrastive Divergence Learning.
595 Citations2005Miguel Á. Carreira-Perpiñán, Geoffrey E. Hinton
The properties of CD learning are studied and it is shown that it provides biased estimates in general, but that the bias is typically very small.
Artificial IntelligenceConnectionist learning of belief networks
532 Citations1992Radford M. Neal
The “Gibbs sampling” simulation procedure for “sigmoid” and “noisy-OR” varieties of probabilistic belief networks can support maximum-likelihood learning from empirical data through local gradient ascent.
Exponential Family Harmoniums with an Application to Information Retrieval
436 Citations2004Max Welling, Michal Rosen‐Zvi +1 more
An alternative two-layer model based on exponential family distributions and the semantics of undirected models is proposed, which performs well on document retrieval tasks and provides an elegant solution to searching with keywords.
On the quantitative analysis of deep belief networks
432 Citations2008Ruslan Salakhutdinov, Iain Murray
It is shown that Annealed Importance Sampling (AIS) can be used to efficiently estimate the partition function of an RBM, and a novel AIS scheme for comparing RBM's with different architectures is presented.
Combining Top-Down and Bottom-Up Segmentation
274 Citations2005Eran Borenstein, E. Sharon +1 more
This work shows how to combine bottom-up and top-up approaches into a single figure-ground segmentation process that provides accurate delineation of object boundaries that cannot be achieved by either the top-down or bottom- up approach alone.
Neural ComputationJustifying and Generalizing Contrastive Divergence
211 Citations2008Yoshua Bengio, Olivier Delalleau
An expansion of the log likelihood in undirected graphical models such as the restricted Boltzmann machine (RBM), where each term in the expansion is associated with a sample in a Gibbs chain alternating between two random variables, shows that its residual term converges to zero, justifying the use of a truncation of only a short Gibbs chain.
Stochastics and stochastics reportsOn the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates
142 Citations1999Laurent Younès
This work analyses the convergence of stochastic algorithms with Markovian noise when the ergodicity of the Markov chain governing the noise rapidly decreases as the control parameter tends to infinity and provides sufficient condition which ensure convergence.
The rate adapting poisson model for information retrieval and object recognition
100 Citations2006Peter Gehler, Alex Holub +1 more
An alternative undirected graphical model suitable for modelling count data, called the "Rate Adapting Poisson" (RAP) model, is shown to generate superior dimensionally reduced representations for subsequent retrieval or classification.