Truncation-free stochastic variational inference for Bayesian nonparametric models
Published 3 December 2012
Chong Wang, David M. Blei
Citations34
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work presents a truncation-free stochastic variational inference algorithm for Bayesian nonparametric models that adapts model complexity on the fly and performs better than previous stochastically variational inferred inference algorithms.
Abstract
We present a truncation-free stochastic variational inference algorithm for Bayesian nonparametric models. While traditional variational inference algorithms require truncations for the model or the variational distribution, our method adapts model complexity on the fly. We studied our method with Dirichlet process mixture models and hierarchical Dirichlet process topic models on two large data sets. Our method performs better than previous stochastic variational inference algorithms. 1
Keywords
Computer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
Journal of Electronic ImagingPattern Recognition and Machine Learning
21,976 Citations2007Christopher Bishop
Probability Distributions, linear models for Regression, Linear Models for Classification, Neural Networks, Graphical Models, Mixture Models and EM, Sampling Methods, Continuous Latent Variables, Sequential Data are studied.
Cambridge University Press eBooksData Analysis Using Regression and Multilevel/Hierarchical Models
13,924 Citations2006Andrew Gelman, Jennifer Hill
Data Analysis Using Regression and Multilevel/Hierarchical Models is a comprehensive manual for the applied researcher who wants to perform data analysis using linear and nonlinear regression and multilevel models.
The Annals of Mathematical StatisticsA Stochastic Approximation Method
9,581 Citations1951Herbert Robbins, Sutton Monro
Wiley series in probability and statisticsFinite Mixture Models
7,410 Citations2018Geoffrey J. McLachlan, David Peel +1 more
The aim of this article is to provide an up-to-date account of the theory and methodological developments underlying the applications of finite mixture models.
Proceedings of the National Academy of SciencesFinding scientific topics
6,017 Citations2004Thomas L. Griffiths, Mark Steyvers
A generative model for documents is described, introduced by Blei, Ng, and Jordan, and a Markov chain Monte Carlo algorithm is presented for inference in this model, which is used to analyze abstracts from PNAS by using Bayesian model selection to establish the number of topics.
Machine LearningAn Introduction to Variational Methods for Graphical Models
3,775 Citations1999Michael I. Jordan, Zoubin Ghahramani +2 more
This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields), and describes a general framework for generating variational transformations based on convex duality.
Journal of the American Statistical AssociationHierarchical Dirichlet Processes
3,567 Citations2006Yee Whye Teh, Michael I. Jordan +2 more
This work considers problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups, and considers a hierarchical model, specifically one in which the base measure for the childDirichlet processes is itself distributed according to a Dirichlet process.
now publishers, Inc. eBooksGraphical Models, Exponential Families, and Variational Inference
3,154 Citations2007Martin J. Wainwright, Michael I. Jordan
Neural ComputationNatural Gradient Works Efficiently in Learning
2,700 Citations1998Шун-ичи Амари
The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters.
Machine LearningAn Introduction to MCMC for Machine Learning
2,418 Citations2003Christophe Andrieu, Nando de Freitas +2 more
This purpose of this introductory paper is to introduce the Monte Carlo method with emphasis on probabilistic machine learning and review the main building blocks of modern Markov chain Monte Carlo simulation.
Journal of the American Statistical AssociationBayesian Density Estimation and Inference Using Mixtures
2,241 Citations1995Michael Escobar, Mike West
These models provide natural settings for density estimation and are exemplified by special cases where data are modeled as a sample from mixtures of normal distributions, allowing for direct inference on a variety of practical issues.
Journal of Computational and Graphical StatisticsMarkov Chain Sampling Methods for Dirichlet Process Mixture Models
2,212 Citations2000Radford M. Neal
The Annals of StatisticsMixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems
2,030 Citations1974C. Antoniak
This paper gives a formal definition for Dirichlet process mixtures and develops several theorems about their properties, the most important of which is a closure property for such mixtures.
Proceedings of the AAAI Conference on Artificial IntelligenceToward an Architecture for Never-Ending Language Learning
1,991 Citations2010J. Andrew Carlson, Justin Betteridge +4 more
Reading Tea Leaves: How Humans Interpret Topic Models
1,874 Citations2009Jonathan Chang, Sean Gerrish +3 more
New quantitative methods for measuring semantic meaning in inferred topics are presented, showing that they capture aspects of the model that are undetected by previous measures of model quality based on held-out likelihood.
A Constructive Definition of Dirichlet Priors
1,611 Citations1991Jayaram Sethuraman
Journal of Machine Learning ResearchStochastic variational inference
1,479 Citations2013Matthew D. Hoffman, David M. Blei +2 more
Stochastic variational inference lets us apply complex Bayesian models to massive data sets, and it is shown that the Bayesian nonparametric topic model outperforms its parametric counterpart.
arXiv (Cornell University)Expectation Propagation for approximate Bayesian inference
1,457 Citations2013Thomas P. Minka
Bayesian AnalysisVariational inference for Dirichlet process mixtures
1,446 Citations2006David M. Blei, Michael I. Jordan
A variational inference algorithm forDP mixtures is presented and experiments that compare the algorithm to Gibbs sampling algorithms for DP mixtures of Gaussians and present an application to a large-scale image analysis problem are presented.
The Annals of ProbabilityThe two-parameter Poisson-Dirichlet distribution derived from a stable subordinator
1,255 Citations1997Jim Pitman, Marc Yor
Journal of the ACMThe nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies
667 Citations2010David M. Blei, Thomas L. Griffiths +1 more
An application to information retrieval in which documents are modeled as paths down a random tree, and the preferential attachment dynamics of the nCRP leads to clustering of documents according to sharing of topics at multiple levels of abstraction.
Infinite latent feature models and the Indian buffet process
552 Citations2005Zoubin Ghahramani, Thomas L. Griffiths
A probability distribution over equivalence classes of binary matrices with a finite number of rows and an unbounded number of columns is defined, suitable for use as a prior in probabilistic models that represent objects using a potentially infinite array of features.
A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation
516 Citations2007Yee Whye Teh, David Newman +1 more
Divergence measures and message passing
460 Citations2005Thomas P. Minka
This paper presents a unifying view of messagepassing algorithms, as methods to approximate a complex Bayesian network by a simpler network with minimum information divergence.
arXiv (Cornell University)On Smoothing and Inference for Topic Models
452 Citations2012Arthur Asuncion, Max Welling +2 more
Proceedings of the VLDB EndowmentAn architecture for parallel topic models
431 Citations2010Alexander J. Smola, Shravan Narayanamurthy
This paper describes a high performance sampling architecture for inference of latent topic models on a cluster of workstations and shows that this architecture is entirely general and that it can be extended easily to more sophisticated latent variable models such as n-grams and hierarchies.
Efficient methods for topic model inference on streaming document collections
425 Citations2009Limin Yao, David Mimno +1 more
Empirical results indicate that SparseLDA can be approximately 20 times faster than traditional LDA and provide twice the speedup of previously published fast sampling methods, while also using substantially less memory.
Distributed Algorithms for Topic Models
395 Citations2009David Newman, Arthur Asuncion +2 more
Neural ComputationOnline Model Selection Based on the Variational Bayes
382 Citations2001Masa-aki Sato
By combining sequential model selection procedures, the online VB method provides a fully online learning method with a model selection mechanism and was able to adapt the model structure to dynamic environments.
Online Variational Inference for the Hierarchical Dirichlet Process
330 Citations2011Chong Wang, John Paisley +1 more
This work proposes an online variational inference algorithm for the HDP, an algorithm that is easily applicable to massive and streaming data, and lets us analyze much larger data sets.
Scalable inference in latent variable models
261 Citations2012Amr Ahmed, Moahmed Aly +3 more
A scalable parallel framework for efficient inference in latent variable models over streaming web-scale data by introducing a novel delta-based aggregation system with a bandwidth-efficient communication protocol, schedule-aware out-of-core storage, and approximate forward sampling to rapidly incorporate new data.
Stick-breaking Construction for the Indian Buffet Process
202 Citations2007Yee Whye Teh, Dilan Görür +1 more
This paper derives a stick-breaking representation for the Indian buffet process and develops slice samplers for the IBP that are efficient, easy to implement and more generally applicable than the currently available Gibbs sampler.
Collapsed variational Dirichlet process mixture models
172 Citations2007Kenichi Kurihara, Max Welling +1 more
A number of variational Bayesian approximations to the Dirichlet process (DP) mixture model are studied and a novel collapsed VB approximation where mixture weights are marginalized out is considered.
Collapsed Variational Inference for HDP
145 Citations2007Yee Whye Teh, Kenichi Kurihara +1 more
This work obtains the first variational algorithm to deal with the hierarchical Dirichlet process and with hyperparameters ofDirichlet variables, and shows a significant improvement in accuracy.
Partout
131 Citations2014Luis Galárraga, Katja Hose +1 more
This paper proposes an effective approach for fragmenting RDF data sets based on a query log and allocating the fragments to hosts in a cluster of machines and produces efficient query execution plans for ad-hoc SPARQL queries.
Mr. LDA
126 Citations2012Ke Zhai, Jordan Boyd‐Graber +2 more
This paper introduces a novel and flexible large scale topic modeling package in MapReduce (Mr. LDA), which uses variational inference, which easily fits into a distributed environment and is easily extensible.
The MIT Press eBooksAccelerated Variational Dirichlet Process Mixtures
118 Citations2007Kenichi Kurihara, Max Welling +1 more
arXiv (Cornell University)Sparse Stochastic Inference for Latent Dirichlet allocation
109 Citations2012David Mimno, Matt Hoffman +1 more
Sparse stochastic inference for latent Dirichlet allocation
68 Citations2012Matt Hoffman, David M. Blei +1 more
A hybrid algorithm for Bayesian topic models that combines the efficiency of sparse Gibbs sampling with the scalability of online stochastic inference is presented that reduces the bias of variational inference and generalizes to many Bayesian hidden-variable models.
Variational Inference for the Nested Chinese Restaurant Process
52 Citations2009Chong Wang, David M. Blei
To employ variational methods, a tree-based stick-breaking construction of the nCRP mixture model is derived, and a novel variational algorithm is developed that efficiently explores a posterior over a large set of combinatorial structures.
The MIT Press eBooksFrom Naive Mean Field Theory to the TAP Equations
35 Citations2001Manfred Opper, Ole Winther
Practical collapsed variational bayes inference for hierarchical dirichlet process
26 Citations2012Issei Sato, Kenichi Kurihara +1 more
A novel collapsed variational Bayes (CVB) inference for the hierarchical Dirichlet process (HDP) is proposed, which is simple to implement, does not require variance counts to be maintained,does not need to set hyper-parameters, and has good predictive performance.
arXiv (Cornell University)Rethinking Collapsed Variational Bayes Inference for LDA
16 Citations2012Issei Sato, Hiroshi Nakagawa