The Discrete Infinite Logistic Normal Distribution
Bayesian AnalysisPublished 27 November 2012Open access
John Paisley, Chong Wang, David M. Blei
Citations48
SJR quartileQ1
SJR score1.77
SNIP2.94
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A stochastic variational inference algorithm for DILN is developed and compared with similar algorithms for HDP and latent Dirichlet allocation on a collection of 350; 000 articles from Nature.
Abstract
We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN generalizes the hierarchical Dirichlet process (HDP) to model correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables and study its statistical properties. We derive a variational inference algorithm for approximate posterior inference. We apply DILN to topic modeling of documents and study its empirical performance on four corpora, comparing performance with the HDP and the correlated topic model (CTM). To compute with large-scale data, we develop a stochastic variational inference algorithm for DILN and compare with similar algorithms for HDP and latent Dirichlet allocation (LDA) on a collection of 350,000 articles from Nature.
Keywords
Computer ScienceMathematics
GeneticsInference of Population Structure Using Multilocus Genotype Data
34,260 Citations2000Jonathan K. Pritchard, Matthew Stephens +1 more
A model-based clustering method for using multilocus genotype data to infer population structure and assign individuals to populations that can be applied to most of the commonly used genetic markers, provided that they are not closely linked.
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
BiometrikaMonte Carlo sampling methods using Markov chains and their applications
15,200 Citations1970W. Keith Hastings
The MIT Press eBooksGaussian Processes for Machine Learning
10,409 Citations2005Carl Edward Rasmussen, Christopher K. I. Williams
The book provides a long-needed, systematic and unified treatment of theoretical and practical aspects of GPs in machine learning, targeted at researchers and students in machine learning and applied statistics.
The Annals of Mathematical StatisticsA Stochastic Approximation Method
9,581 Citations1951Herbert Robbins, Sutton Monro
Proceedings of the National Academy of SciencesFinding scientific topics
6,017 Citations2004Thomas L. Griffiths, Mark Steyvers
A generative model for documents is described, introduced by Blei, Ng, and Jordan, and a Markov chain Monte Carlo algorithm is presented for inference in this model, which is used to analyze abstracts from PNAS by using Bayesian model selection to establish the number of topics.
The Annals of StatisticsA Bayesian Analysis of Some Nonparametric Problems
4,812 Citations1973Thomas S. Ferguson
Springer texts in statisticsMonte Carlo Statistical Methods
4,050 Citations2004Christian P. Robert, George Casella
Machine LearningAn Introduction to Variational Methods for Graphical Models
3,775 Citations1999Michael I. Jordan, Zoubin Ghahramani +2 more
This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields), and describes a general framework for generating variational transformations based on convex duality.
Journal of the American Statistical AssociationHierarchical Dirichlet Processes
3,567 Citations2006Yee Whye Teh, Michael I. Jordan +2 more
This work considers problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups, and considers a hierarchical model, specifically one in which the base measure for the childDirichlet processes is itself distributed according to a Dirichlet process.
Journal of the Royal Statistical Society Series B (Statistical Methodology)The Statistical Analysis of Compositional Data
3,305 Citations1982J. Aitchison
now publishers, Inc. eBooksGraphical Models, Exponential Families, and Variational Inference
3,154 Citations2007Martin J. Wainwright, Michael I. Jordan
Journal of the American Statistical AssociationBayesian Density Estimation and Inference Using Mixtures
2,241 Citations1995Michael Escobar, Mike West
These models provide natural settings for density estimation and are exemplified by special cases where data are modeled as a sample from mixtures of normal distributions, allowing for direct inference on a variety of practical issues.
A Constructive Definition of Dirichlet Priors
1,611 Citations1991Jayaram Sethuraman
Journal of the American Statistical AssociationGibbs Sampling Methods for Stick-Breaking Priors
1,556 Citations2001Hemant Ishwaran, Lancelot F. James
Two general types of Gibbs samplers that can be used to fit posteriors of Bayesian hierarchical models based on stick-breaking priors are presented and the blocked Gibbs sampler, based on an entirely different approach that works by directly sampling values from the posterior of the random measure.
PubMedMixed Membership Stochastic Blockmodels.
1,526 Citations2008Edoardo M. Airoldi, David M. Blei +2 more
This paper describes a latent variable model of such data called the mixed membership stochastic blockmodel, which extends blockmodels for relational data to ones which capture mixed membership latent relational structure, thus providing an object-specific low-dimensional representation.
Bayesian AnalysisVariational inference for Dirichlet process mixtures
1,446 Citations2006David M. Blei, Michael I. Jordan
A variational inference algorithm forDP mixtures is presented and experiments that compare the algorithm to Gibbs sampling algorithms for DP mixtures of Gaussians and present an application to a large-scale image analysis problem are presented.
The Annals of StatisticsFerguson Distributions Via Polya Urn Schemes
1,346 Citations1973David Blackwell, James B. MacQueen
Online Learning for Latent Dirichlet Allocation
1,294 Citations2010Matthew D. Hoffman, Francis Bach +1 more
An online variational Bayes (VB) algorithm for Latent Dirichlet Allocation (LDA) based on online stochastic optimization with a natural gradient step is developed, which shows converges to a local optimum of the VB objective function.
OPAL (Open@LaTrobe) (La Trobe University)A correlated topic model of Science
1,137 Citations2018David M. Blei, John Lafferty
The Annals of StatisticsOn a Class of Bayesian Nonparametric Estimates: I. Density Estimates
738 Citations1984Albert Y. Lo
Journal of Machine Learning ResearchVariational Message Passing
644 Citations2005John Winn, Chris Bishop
Variational Message Passing is introduced, a general purpose algorithm for applying variational inference to Bayesian Networks and can be applied to very general class of conjugate-exponential models because it uses a factorised variational approximation.
An Introduction to Variational Methods for Graphical Models
563 Citations1998Michael I. Jordan, Zoubin Ghahramani +2 more
arXiv (Cornell University)On Smoothing and Inference for Topic Models
452 Citations2012Arthur Asuncion, Max Welling +2 more
Proceedings of the National Academy of SciencesMixed-membership models of scientific publications
445 Citations2004Elena A. Erosheva, Stephen E. Fienberg +1 more
This work explores an internal soft-classification structure of articles based only on semantic decompositions of abstracts and bibliographies of PNAS articles, and compares it with the formal discipline classifications.
BiometrikaSome matrix-variate distribution theory: Notational considerations and a Bayesian application
421 Citations1981A. P. Dawid
Neural ComputationOnline Model Selection Based on the Variational Bayes
382 Citations2001Masa-aki Sato
By combining sequential model selection procedures, the online VB method provides a fully online learning method with a model selection mechanism and was able to adapt the model structure to dynamic environments.
Journal of the American Statistical AssociationBayesian Nonparametric Spatial Modeling With Dirichlet Process Mixing
361 Citations2005Alan E. Gelfand, Athanasios Kottas +1 more
This work develops a spatial Dirichlet process model for spatial data and discusses its properties, and introduces mixing by convolving this process with a pure error process to produce a random spatial process that is neither Gaussian nor stationary.
Journal of the American Statistical AssociationOrder-Based Dependent Dirichlet Processes
359 Citations2006Jim E. Griffin, Mark F. J. Steel
This article allows the nonparametric distribution to depend on covariates through ordering the random variables building the weights in the stick-breaking representation and derives the correlation between distributions at different covariate values.
Statistics and ComputingSlice sampling mixture models
351 Citations2009Maria Kalli, Jim E. Griffin +1 more
A more efficient version of the slice sampler for Dirichlet process mixture models described by Walker allows for the fitting of infinite mixture models with a wide-range of prior specifications and considers priors defined through infinite sequences of independent positive random variables.
Online Variational Inference for the Hierarchical Dirichlet Process
330 Citations2011Chong Wang, John Paisley +1 more
This work proposes an online variational inference algorithm for the HDP, an algorithm that is easily applicable to massive and streaming data, and lets us analyze much larger data sets.
BiometrikaKernel stick-breaking processes
284 Citations2008David B. Dunson, Ji‐Hye Park
A retrospective Markov chain Monte Carlo algorithm is developed for posterior computation, and the methods are illustrated using a simulated example and an epidemiological application.
Canadian Journal of StatisticsExact and approximate sum representations for the Dirichlet process
275 Citations2002Hemant Ishwaran, Mahmoud Zarepour
Journal of the Royal Statistical Society Series B (Statistical Methodology)Bayesian Density Regression
263 Citations2007David B. Dunson, Natesh S. Pillai +1 more
The paper considers Bayesian methods for density regression, allowing a random probability distribution to change flexibly with multiple predictors, and proposes a kernel‐based weighting scheme that incorporates weights that are dependent on the distance between subjects’ predictor values.
BiometrikaGeneralized Spatial Dirichlet Process Models
177 Citations2007Jin‐Ao Duan, Michele Guindani +1 more
Journal of the Royal Statistical Society Series B (Statistical Methodology)A Method for Combining Inference Across Related Nonparametric Bayesian Models
177 Citations2004Peter Müller, Fernando A. Quintana +1 more
A Markov chain Monte Carlo scheme is developed to allow efficient implementation of full posterior inference in the given model and includes a regression at the level of the nonparametric model.
The Infinite PCFG Using Hierarchical Dirichlet Processes
164 Citations2007Percy Liang, Slav Petrov +2 more
This work presents a nonparametric Bayesian model of tree structures based on the hierarchical Dirichlet process (HDP) and develops an efficient variational inference procedure that can be applied to full-scale parsing applications.
The Annals of Applied StatisticsDescribing disability through individual-level mixture models for multivariate binary data
162 Citations2007Elena A. Erosheva, Stephen E. Fienberg +1 more
This work considers an extract of functional disability data from the National Long Term Care Survey (NLTCS) and attempts to develop disability profiles using variations of the Grade of Membership (GoM) model, proving the equivalence between individual-level and population-level mixture models.
Collapsed Variational Inference for HDP
145 Citations2007Yee Whye Teh, Kenichi Kurihara +1 more
This work obtains the first variational algorithm to deal with the hierarchical Dirichlet process and with hyperparameters ofDirichlet variables, and shows a significant improvement in accuracy.
The MIT Press eBooksAccelerated Variational Dirichlet Process Mixtures
118 Citations2007Kenichi Kurihara, Max Welling +1 more
Journal of the American Statistical AssociationThe Logistic Normal Distribution for Bayesian, Nonparametric, Predictive Densities
105 Citations1988Peter Lenk
Journal of Computational and Graphical StatisticsPosterior Simulation of Normalized Random Measure Mixtures
62 Citations2010Jim E. Griffin, Stephen G. Walker
Preliminary simulation methods for mixture models whose mixing distribution has a Normalized Random Measure prior are described and are shown to be computationally competitive with previously developed samplers for Dirichlet process mixture models.
Spatial Normalized Gamma Processes
54 Citations2009Vinayak Rao, Yee Whye Teh
A simple and general framework to construct dependent DPs by marginalizing and normalizing a single gamma process over an extended space is proposed and an empirical study of convergence on a synthetic dataset is reported.
The Annals of Applied StatisticsCorrection: A correlated topic model of Science
51 Citations2007David M. Blei, John Lafferty
This paper presents a meta-analyses of the determinants of infectious disease in eight operation theatres of the immune system and shows clear patterns of decline in the number of vaccinated patients and their ages.
The Discrete Infinite Logistic Normal Distribution for Mixed-Membership Modeling
25 Citations2011John Paisley, Chong Wang +1 more
Sparse Variational Analysis of Large Longitudinal Data Sets
2 Citations2010Artin Armagan, David B. Dunson
A novel approximation to the posterior relying on variational methods is recommended to approximate the posterior of the parameters in a decomposition of the variance components, with priors chosen to obtain a sparse solution that allows selection of random effects.