On Convergence Properties of the EM Algorithm for Gaussian Mixtures
Neural ComputationPublished 1 January 1996
Lei Xu, Michael I. Jordan
Citations763
SJR quartileQ1
SJR score0.83
SNIP1.45
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The mathematical connection between the Expectation-Maximization (EM) algorithm and gradient-based approaches for maximum likelihood learning of finite gaussian mixtures is built up and an explicit expression for the matrix is provided.
Abstract
We build up the mathematical connection between the “Expectation-Maximization” (EM) algorithm and gradient-based approaches for maximum likelihood learning of finite gaussian mixtures. We show that the EM step in parameter space is obtained from the gradient via a projection matrix P, and we provide an explicit expression for the matrix. We then analyze the convergence of EM in terms of special properties of P and provide new results analyzing the effect that P has on the likelihood surface. Based on these mathematical results, we present a comparative discussion of the advantages and disadvantages of EM and other algorithms for the learning of gaussian mixture models.
Keywords
Computer Science
Journal of the Royal Statistical Society Series B (Statistical Methodology)Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
49,657 Citations1977A. P. Dempster, N. M. Laird +1 more
The Annals of StatisticsOn the Convergence Properties of the EM Algorithm
3,252 Citations1983Changbao Wu
SIAM ReviewMixture Densities, Maximum Likelihood and the EM Algorithm
2,686 Citations1984Richard A. Redner, Homer F. Walker
This work discusses the formulation and theoretical and practical properties of the EM algorithm, a specialization to the mixture density context of a general algorithm used to approximate maximum-likelihood estimates for incomplete data problems.
Neural ComputationHierarchical Mixtures of Experts and the EM Algorithm
2,639 Citations1994Michael I. Jordan, Robert A. Jacobs
Bell System Technical JournalAn Introduction to the Application of the Theory of Probabilistic Functions of a Markov Process to Automatic Speech Recognition
981 Citations1983S. Levinson, L. R. Rabiner +1 more
This paper presents several of the salient theoretical and practical issues associated with modeling a speech signal as a probabilistic function of a (hidden) Markov chain, and focuses on a particular class of Markov models, which are especially appropriate for isolated word recognition.
Pacific Journal of MathematicsGrowth transformations for functions on manifolds
363 Citations1968Leonard E. Baum, George R. Sell
Journal of the Royal Statistical Society Series B (Statistical Methodology)On the Convergence of the <i>EM</i> Algorithm
336 Citations1983Russell A. Boyles
Neural NetworksInformation geometry of the EM and em algorithms for neural networks
333 Citations1995Шун-ичи Амари
A unified information geometrical framework for studying stochastic models of neural networks, by focusing on the EM and em algorithms, and proves a condition that guarantees their equivalence.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Recursive Parameter Estimation Using Incomplete Data
323 Citations1984D. M. Titterington
Neural NetworksConvergence results for the EM approach to mixtures of experts architectures
303 Citations1995Michael I. Jordan, Lei Xu
It is shown that the EM algorithm can be regarded as a variable metric algorithm with its searching direction having a positive projection on the gradient of the log likelihood and an acceleration technique that yields a significant speedup in simulation experiments.
An Input Output HMM Architecture
260 Citations1994Yoshua Bengio, Paolo Frasconi
A recurrent architecture having a modular structure that has similarities to hidden Markov models, but supports recurrent networks processing style and allows to exploit the supervised learning paradigm while using maximum likelihood estimation is introduced.
A new view of the EM algorithm that justifies incremental and other variants
165 Citations1993Radford M. Neal
Soft competitive adaptation: neural network learning algorithms based on fitting statistical mixtures
124 Citations1991Steven J. Nowlan
An unsupervised algorithm which is an alternative to the classical winner-take-all competitive algorithms and a supervised modular architecture in which a number of simple "expert" networks compete to solve distinct pieces of a large task are considered.
Neural ComputationStatistical Physics, Mixtures of Distributions, and the EM Algorithm
100 Citations1994Alan Yuille, Paul Stolorz +1 more
There are strong relationships between approaches to optmization and learning based on statistical physics or mixtures of experts, and the EM algorithm can be interpreted as converging either to a local maximum of the mixtures model or to a saddle point solution to the statistical physics system.
Training Neural Networks with Deficient Data
92 Citations1993Volker Tresp, Subutai Ahmad +1 more
The general solution requires a weighted integration over the unknown or uncertain input although computationally cheaper closed-form solutions can be found for certain Gaussian Basis Function networks.
Classification using hierarchical mixtures of experts
84 Citations2002Steve R. Waterhouse, A.J. Robinson
This paper extends the hierarchical mixture of experts to classification and results are reported for three common classification benchmark tests: exclusive-OR, N-input parity and two spirals.
IEEE Transactions on Signal ProcessingMixture of experts regression modeling by deterministic annealing
60 Citations1997A.V. Rao, David J. Miller +2 more
A new learning algorithm for regression modeling based on deterministic annealing, which consistently and substantially outperformed the competing methods for training NRBF and HME regression functions over a variety of benchmark regression examples.
Neural ComputationTime-Series Segmentation Using Predictive Modular Neural Networks
42 Citations1997Athanasios Kehagias, Vassilios Petridis
A predictive modular neural network method is applied to the problem of unsupervised time-series segmentation and finds the sources generating the time series, assigns data to each source, and trains one predictor for each source.
Neural ComputationA Constructive, Incremental-Learning Network for Mixture Modeling and Classification
36 Citations1997James R. Williamson
This work shows a close relationship between GAM and the well-known expectation-maximization approach to mixture modeling and forms an EM classification algorithm on three classification benchmarks, thereby demonstrating the advantage of the ART match criterion for regulating learning and the ARTMAP match tracking operation for incorporating environmental feedback in supervised learning situations.
International Journal of Neural SystemsAdaptive Rival Penalized Competitive Learning and Combined Linear Predictor Model for Financial Forecast and Investment
13 Citations1997Yiu‐ming Cheung, Wai-Man Leung +1 more
Experiments have shown that Adaptive RPCL-CLP not only outperforms the other approaches with the smallest prediction error and training costs, but also brings in considerable high profits in the trading simulation of foreign exchange market.