Nested Hierarchical Dirichlet Processes
IEEE Transactions on Pattern Analysis and Machine IntelligencePublished 18 April 2014Open access
John Paisley, Chong Wang, David M. Blei, Michael I. Jordan
Citations216
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A stochastic variational inference algorithm is derived for the model, which enables efficient inference for massive collections of text documents and alleviates the rigid, single-path formulation of the nCRP.
Abstract
We develop a nested hierarchical Dirichlet process (nHDP) for hierarchical topic modeling. The nHDP generalizes the nested Chinese restaurant process (nCRP) to allow each word to follow its own path to a topic node according to a per-document distribution over the paths on a shared tree. This alleviates the rigid, single-path formulation assumed by the nCRP, allowing documents to easily express complex thematic borrowings. We derive a stochastic variational inference algorithm for the model, which enables efficient inference for massive collections of text documents. We demonstrate our algorithm on 1.8 million documents from The New York Times and 2.7 million documents from Wikipedia.
Keywords
Computer ScienceMathematics
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
The Annals of StatisticsA Bayesian Analysis of Some Nonparametric Problems
4,812 Citations1973Thomas S. Ferguson
Machine LearningAn Introduction to Variational Methods for Graphical Models
3,775 Citations1999Michael I. Jordan, Zoubin Ghahramani +2 more
This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields), and describes a general framework for generating variational transformations based on convex duality.
Journal of the American Statistical AssociationHierarchical Dirichlet Processes
3,567 Citations2006Yee Whye Teh, Michael I. Jordan +2 more
This work considers problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups, and considers a hierarchical model, specifically one in which the base measure for the childDirichlet processes is itself distributed according to a Dirichlet process.
Neural ComputationNatural Gradient Works Efficiently in Learning
2,700 Citations1998Шун-ичи Амари
The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters.
A Constructive Definition of Dirichlet Priors
1,611 Citations1991Jayaram Sethuraman
PubMedMixed Membership Stochastic Blockmodels.
1,526 Citations2008Edoardo M. Airoldi, David M. Blei +2 more
This paper describes a latent variable model of such data called the mixed membership stochastic blockmodel, which extends blockmodels for relational data to ones which capture mixed membership latent relational structure, thus providing an object-specific low-dimensional representation.
Journal of Machine Learning ResearchStochastic variational inference
1,479 Citations2013Matthew D. Hoffman, David M. Blei +2 more
Stochastic variational inference lets us apply complex Bayesian models to massive data sets, and it is shown that the Bayesian nonparametric topic model outperforms its parametric counterpart.
Bayesian AnalysisVariational inference for Dirichlet process mixtures
1,446 Citations2006David M. Blei, Michael I. Jordan
A variational inference algorithm forDP mixtures is presented and experiments that compare the algorithm to Gibbs sampling algorithms for DP mixtures of Gaussians and present an application to a large-scale image analysis problem are presented.
The Annals of StatisticsFerguson Distributions Via Polya Urn Schemes
1,346 Citations1973David Blackwell, James B. MacQueen
Online Learning for Latent Dirichlet Allocation
1,294 Citations2010Matthew D. Hoffman, Francis Bach +1 more
An online variational Bayes (VB) algorithm for Latent Dirichlet Allocation (LDA) based on online stochastic optimization with a natural gradient step is developed, which shows converges to a local optimum of the VB objective function.
Neural Information Processing SystemsHierarchical Topic Models and the Nested Chinese Restaurant Process
938 Citations2003Thomas L. Griffiths, Michael I. Jordan +2 more
A Bayesian approach is taken to generate an appropriate prior via a distribution on partitions that allows arbitrarily large branching factors and readily accommodates growing data collections.
arXiv (Cornell University)Mixed membership stochastic blockmodels
786 Citations2007Edoardo M. Airoldi, David M. Blei +2 more
Journal of the ACMThe nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies
667 Citations2010David M. Blei, Thomas L. Griffiths +1 more
An application to information retrieval in which documents are modeled as paths down a random tree, and the preferential attachment dynamics of the nCRP leads to clustering of documents according to sharing of topics at multiple levels of abstraction.
Journal of Machine Learning ResearchVariational Message Passing
644 Citations2005John Winn, Chris Bishop
Variational Message Passing is introduced, a general purpose algorithm for applying variational inference to Bayesian Networks and can be applied to very general class of conjugate-exponential models because it uses a factorised variational approximation.
An Introduction to Variational Methods for Graphical Models
563 Citations1998Michael I. Jordan, Zoubin Ghahramani +2 more
Neural ComputationOnline Model Selection Based on the Variational Bayes
382 Citations2001Masa-aki Sato
By combining sequential model selection procedures, the online VB method provides a fully online learning method with a model selection mechanism and was able to adapt the model structure to dynamic environments.
Online Variational Inference for the Hierarchical Dirichlet Process
330 Citations2011Chong Wang, John Paisley +1 more
This work proposes an online variational inference algorithm for the HDP, an algorithm that is easily applicable to massive and streaming data, and lets us analyze much larger data sets.
Journal of the American Statistical AssociationThe Nested Dirichlet Process
305 Citations2008Abel Rodríguez, David B. Dunson +1 more
Project Euclid (Cornell University)A sticky HDP-HMM with application to speaker\n diarization
220 Citations2011Fox, Emily B., Sudderth, Erik B. +2 more
An augmented HDP-HMM is described that provides effective control over the switching rate and makes it possible to treat emission distributions nonparametrically, and a sampling algorithm is developed that employs a truncated approximation of the Dirichlet process to jointly resample the full state sequence.
Collapsed Variational Inference for HDP
145 Citations2007Yee Whye Teh, Kenichi Kurihara +1 more
This work obtains the first variational algorithm to deal with the hierarchical Dirichlet process and with hyperparameters ofDirichlet variables, and shows a significant improvement in accuracy.
Neural Information Processing SystemsTree-Structured Stick Breaking for Hierarchical Data
136 Citations2010Zoubin Ghahramani, Michael I. Jordan +1 more
This paper uses nested stick-breaking processes to allow for trees of unbounded width and depth, where data can live at any node and are infinitely exchangeable, and applies the method to hierarchical clustering of images and topic modeling of text data.
The dynamic hierarchical Dirichlet process
119 Citations2008Lu Ren, David B. Dunson +1 more
The dynamic hierarchical Dirichlet process (dHDP) is developed to model the time-evolving statistical properties of sequential data sets, and a relatively simple Markov Chain Monte Carlo sampler is developed.
The MIT Press eBooksAccelerated Variational Dirichlet Process Mixtures
118 Citations2007Kenichi Kurihara, Max Welling +1 more
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)An Adaptive Learning Rate for Stochastic Variational Inference
76 Citations2018Rajesh Ranganath, Chong Wang +2 more
This work develops an adaptive learning rate for stochastic variational inference, which requires no tuning and is easily implemented with computations already made in the algorithm.
Modeling topic hierarchies with the recursive chinese restaurant process
63 Citations2012Joon Hee Kim, Dongwoo Kim +2 more
This work introduces the recursive Chinese restaurant process (rCRP) and a nonparametric topic model with rCRP as a prior for discovering a hierarchical topic structure with unbounded depth and width and suggests two metrics that quantify the characteristics of a topic hierarchy to compare the discovered topic hierarchies of r CRP and nCRP.
Variational Inference for the Nested Chinese Restaurant Process
52 Citations2009Chong Wang, David M. Blei
To employ variational methods, a tree-based stick-breaking construction of the nCRP mixture model is derived, and a novel variational algorithm is developed that efficiently explores a posterior over a large set of combinatorial structures.
Bayesian AnalysisThe Discrete Infinite Logistic Normal Distribution
48 Citations2012John Paisley, Chong Wang +1 more
A stochastic variational inference algorithm for DILN is developed and compared with similar algorithms for HDP and latent Dirichlet allocation on a collection of 350; 000 articles from Nature.
Neural Information Processing SystemsTruncation-free Online Variational Inference for Bayesian Nonparametric Models
44 Citations2012Chong Wang, David M. Blei
This work presents a truncation-free online variational inference algorithm for Bayesian nonparametric models that adapts model complexity on the fly and shows better performance than previous online Variational inference algorithms.