Variational Approximations between Mean Field Theory and the Junction\n Tree Algorithm
arXiv (Cornell University)Published 16 January 2013Open access
Wim Wiegerinck
Citations80
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Recently, variational approximations such as the mean field approximation\nhave received much interest. We extend the standard mean field method by using\nan approximating distribution that factorises into cluster potentials. This\nincludes undirected graphs, directed acyclic graphs and junction trees. We\nderive generalized mean field equations to optimize the cluster potentials. We\nshow that the method bridges the gap between the standard mean field\napproximation and the exact junction tree algorithm. In addition, we address\nthe problem of how to choose the graphical structure of the approximating\ndistribution. From the generalised mean field equations we derive rules to\nsimplify the structure of the approximating distribution in advance without\naffecting the quality of the approximation. We also show how the method fits\ninto some other variational approximations that are currently popular.\n
Keywords
Computer Science
Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference
16,927 Citations1988Judea Pearl
The author provides a coherent explication of probability as a language for reasoning with partial belief and offers a unifying perspective on other AI approaches to uncertainty, such as the Dempster-Shafer formalism, truth maintenance systems, and nonmonotonic logic.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Local Computations with Probabilities on Graphical Structures and Their Application to Expert Systems
3,970 Citations1988Steffen L. Lauritzen, David J. Spiegelhalter
This work exploits a range of local representations for the joint probability distribution, combined with topological changes to the original network termed 'marrying' and 'filling-in', which allows efficient algorithms for transfer between representations, providing rapid absorption and propagation of evidence.
Machine LearningAn Introduction to Variational Methods for Graphical Models
3,775 Citations1999Michael I. Jordan, Zoubin Ghahramani +2 more
This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields), and describes a general framework for generating variational transformations based on convex duality.
Introduction to Bayesian Networks
2,585 Citations2019
Learning in Graphical Models
2,416 Citations1998Michael I. Jordan
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
2,187 Citations1998Radford M. Neal, Geoffrey E. Hinton
An incremental variant of the EM algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step is shown empirically to give faster convergence in a mixture estimation problem.
Artificial IntelligenceThe computational complexity of probabilistic inference using bayesian belief networks
1,935 Citations1990Gregory F. Cooper
It is shown that probabilistic inference using belief networks is NP-hard,Therefore, research should be directed away from the search for a general, efficient probabilism inference algorithm, and toward the design of efficient special-case, average- case, and approximation algorithms.
TechnometricsGraphical Models in Applied Multivariate Statistics
1,827 Citations1991Colin Goodall, Joe Whittaker
Machine LearningFactorial Hidden Markov Models
1,189 Citations1997Zoubin Ghahramani, Michael I. Jordan
A generalization of HMMs in which this state is factored into multiple state variables and is therefore represented in a distributed manner, and a structured approximation in which the the state variables are decoupled, yielding a tractable algorithm for learning the parameters of the model.
Artificial IntelligenceConnectionist learning of belief networks
532 Citations1992Radford M. Neal
The “Gibbs sampling” simulation procedure for “sigmoid” and “noisy-OR” varieties of probabilistic belief networks can support maximum-likelihood learning from empirical data through local gradient ascent.
Journal of Artificial Intelligence ResearchMean Field Theory for Sigmoid Belief Networks
339 Citations1996L.K. Saul, Tommi Jaakkola +1 more
The utility of a mean field theory for sigmoid belief networks based on ideas from statistical mechanics is demonstrated on a benchmark problem in statistical pattern recognition-the classification of handwritten digits.
Exploiting Tractable Substructures in Intractable Networks
221 Citations1995Lawrence K. Saul, Michael I. Jordan
A refined mean field approximation for inference and learning in probabilistic neural networks is developed, and it is shown how to incorporate weak higher order interactions into a first-order hidden Markov model.
Variational Probabilistic Inference and the QMR-DT Network
132 Citations1999Tommi S. Jaakkola, Michael I. Jordan
Improving the Mean Field Approximation Via the Use of Mixture Distributions
101 Citations1998Tommi Jaakkola, Michael I. Jordan
This work employs mixture models as posterior approximations, where each mixture component is a factorized distribution, and describes efficient methods for optimizing the Parameters in these models.
arXiv (Cornell University)A Variational Approximation for Bayesian Networks with Discrete and Continuous Latent Variables
94 Citations2013Kevin P. Murphy
DSpace@MIT (Massachusetts Institute of Technology)Variational methods for inference and estimation in graphical models
93 Citations1997Tommi Jaakkola, Michael I. Jordan
This thesis proposes a principled framework for approximating graphical models based on variational methods and develops variational techniques from the perspective that unifies and expands their applicability to graphical models.
Radboud Repository (Radboud University)Tractable Variational Structures for Approximating Graphical Models
39 Citations1998David Barber, Wim Wiegerinck
This work presents two classes of distributions, decimatable Boltzmann Machines and Tractable Belief Networks that go beyond the standard factorized approach, and gives generalised mean-field equations for both these directed and undirected approximations.
Network Computation in Neural SystemsModel-independent mean-field theory as a local method for approximate propagation of information
23 Citations1999Michael Haft, Reimar Hofmann +1 more
Perspectives in neural computingMean Field Theory based on Belief Networks for Approximate Inference
8 Citations1998Wim Wiegerinck, David Barber
This paper proposes to use a (tractable) belief network as an approximating distribution, and shows that belief networks fit very well into mean field theory, and no additional bounds are required.
New Generation ComputingApproximations of Bayesian networks through KL minimisation
7 Citations2000Wim Wiegerinck, Bert Kappen
This paper shows how redundant structures of the approximating model can be pruned in advance, which is guaranteed to converge to a local minimum of the Kullback-Leibler divergence.