Modeling Documents with Deep Boltzmann Machines
arXiv (Cornell University)Published 26 September 2013Open access
Nitish Srivastava, Ruslan Salakhutdinov, Geoffrey E. Hinton
Citations59
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This parameter tying enables an efficient pretraining algorithm and a state initialization scheme that aids inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.
Keywords
Computer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
Communications of the ACMProbabilistic topic models
5,549 Citations2012David M. Blei
Surveying a suite of algorithms that offer a solution to managing large document archives suggests they are well-suited to handle large amounts of data.
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
Journal of the American Statistical AssociationHierarchical Dirichlet Processes
3,567 Citations2006Yee Whye Teh, Michael I. Jordan +2 more
This work considers problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups, and considers a hierarchical model, specifically one in which the base measure for the childDirichlet processes is itself distributed according to a Dirichlet process.
Deep Boltzmann machines
1,778 Citations2009Ruslan Salakhutdinov, Geoffrey E. Hinton
A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.
Statistics and ComputingAnnealed importance sampling
1,225 Citations2001Radford M. Neal
It is shown how one can use the Markov chain transitions for such an annealing sequence to define an importance sampler, which can be seen as a generalization of a recently-proposed variant of sequential importance sampling.
Training restricted Boltzmann machines using approximations to the likelihood gradient
857 Citations2008Tijmen Tieleman
A new algorithm for training Restricted Boltzmann Machines is introduced, which is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data.
Journal of the ACMThe nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies
667 Citations2010David M. Blei, Thomas L. Griffiths +1 more
An application to information retrieval in which documents are modeled as paths down a random tree, and the preferential attachment dynamics of the nCRP leads to clustering of documents according to sharing of topics at multiple levels of abstraction.
arXiv (Cornell University)Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression
298 Citations2012David Mimno, Andrew McCallum
Online Inference of Topics with Latent Dirichlet Allocation
235 Citations2009Kevin Robert Canini, Lei Shi +1 more
Two related RaoBlackwellized online inference algorithms for the latent Dirichlet allocation (LDA) model – incremental Gibbs samplers and particle filters – are introduced and their runtime and performance are compared to that of existing algorithms.
A Neural Autoregressive Topic Model
188 Citations2012Hugo Larochelle, Stanislas Lauly
A new model for learning meaningful representations of text documents from an unlabeled collection of documents that takes inspiration from the conditional mean-field recursive equations of the Replicated Softmax to define a neural network architecture that estimates the probability of observing a new word in a given document given the previously observed words.
Collapsed Variational Inference for HDP
145 Citations2007Yee Whye Teh, Kenichi Kurihara +1 more
This work obtains the first variational algorithm to deal with the hierarchical Dirichlet process and with hyperparameters ofDirichlet variables, and shows a significant improvement in accuracy.
Stochastics and stochastics reportsOn the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates
142 Citations1999Laurent Younès
This work analyses the convergence of stochastic algorithms with Markovian noise when the ergodicity of the Markov chain governing the noise rapidly decreases as the control parameter tends to infinity and provides sufficient condition which ensure convergence.
Neural Information Processing SystemsA Better Way to Pretrain Deep Boltzmann Machines
106 Citations2012Geoffrey E. Hinton, Ruslan Salakhutdinov
Variational Inference for the Nested Chinese Restaurant Process
52 Citations2009Chong Wang, David M. Blei
To employ variational methods, a tree-based stick-breaking construction of the nCRP mixture model is derived, and a novel variational algorithm is developed that efficiently explores a posterior over a large set of combinatorial structures.