Sparse probabilistic projections
Published 8 December 2008
Cédric Archambeau, Francis R. Bach
Citations118
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work presents a generative model for performing sparse probabilistic projections, which includes sparse principal component analysis and sparse canonical correlation analysis as special cases, and derives a variational Expectation-Maximisation algorithm for the estimation of the hyperparameters.
Abstract
We present a generative model for performing sparse probabilistic projections, which includes sparse principal component analysis and sparse canonical correlation analysis as special cases. Sparsity is enforced by means of automatic relevance determination or by imposing appropriate prior distributions, such as generalised hyperbolic distributions. We derive a variational Expectation-Maximisation algorithm for the estimation of the hyperparameters and show that our novel probabilistic approach compares favourably to existing techniques. We illustrate how the proposed method can be applied in the context of cryptoanalysis as a preprocessing tool for the construction of template attacks. 1
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso
51,790 Citations1996Robert Tibshirani
A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.
Journal of the American Statistical AssociationVariable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
9,196 Citations2001Jianqing Fan, Runze Li
In this article, penalized likelihood approaches are proposed to handle variable selection problems, and it is shown that the newly proposed estimators perform as well as the oracle procedure in variable selection; namely, they work as well if the correct submodel were known.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Probabilistic Principal Component Analysis
3,730 Citations1999Michael E. Tipping, Chris Bishop
It is demonstrated how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model that is closely related to factor analysis.
Journal of Computational and Graphical StatisticsSparse Principal Component Analysis
3,211 Citations2006Hui Zou, Trevor Hastie +1 more
This work introduces a new method called sparse principal component analysis (SPCA) using the lasso (elastic net) to produce modified principal components with sparse loadings and shows that PCA can be formulated as a regression-type optimization problem.
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
2,187 Citations1998Radford M. Neal, Geoffrey E. Hinton
An incremental variant of the EM algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step is shown empirically to give faster convergence in a mixture estimation problem.
SIAM ReviewA Direct Formulation for Sparse PCA Using Semidefinite Programming
667 Citations2007Alexandre d’Aspremont, Laurent El Ghaoui +2 more
Scientific Repository (Petra Christian University)The EM algorithm for mixtures of factor analyzers
585 Citations1996Zoubin Ghahramani, GE Hinton
This work presents an exact Expectation{Maximization algorithm for determining the parameters of this mixture of factor analyzers which concurrently performs clustering and dimensionality reduction, and can be thought of as a reduced dimension mixture of Gaussians.
A Probabilistic Interpretation of Canonical Correlation Analysis
435 Citations2005Francis Bach, Michael I. Jordan
A probabilistic interpretation of canonical correlation (CCA) analysis as a latent variable model for two Gaussian random vectors for Fisher linear discriminant analysis within the CCA framework is given.
The MIT Press eBooksAnalysis of Sparse Bayesian Learning
286 Citations2002Anita Faul, Michael E. Tipping
It is shown that conditioned on an individual hyper-parameter, the marginal likelihood has a unique maximum which is computable in closed form, and it is further shown that if a derived 'sparsity criterion' is satisfied, this maximum is exactly equivalent to 'pruning' the corresponding parameter from the model.
Lecture notes in computer scienceTemplate Attacks in Principal Subspaces
281 Citations2006Cédric Archambeau, Eric Peeters +2 more
This work proposes to perform template attacks in the principal subspace of the traces, a new type of attack that requires five time less encrypted messages than the best reported correlation attack against similar block cipher implementations.
Expectation-maximization for sparse and non-negative PCA
123 Citations2008Christian Sigg, Joachim M. Buhmann
This work studies the problem of finding the dominant eigenvector of the sample covariance matrix, under additional constraints on the vector: a cardinality constraint limits the number of non-zero elements, and non-negativity forces the elements to have equal sign.
Sparse Bayesian nonparametric regression
97 Citations2008François Caron, Arnaud Doucet
This work proposes some flexible Bayesian models which can yield sparse estimates of β and shows that as K → ∞ these models are closely related to a class of Lévy processes.
Scandinavian Journal of StatisticsAbsolute Moments of Generalized Hyperbolic Distributions and Approximate Scaling of Normal Inverse Gaussian Lévy Processes
95 Citations2005Ole E. Barndorff–Nielsen, Robert Stelzer
Warwick Research Archive Portal (University of Warwick)Bayesian adaptive lassos with non-convex penalization
68 Citations2007Jim E. Griffin, Philip J. Brown
The Bayesian interpretation of the Lasso is adopted as the maximum a posteriori (MAP) estimate of the regression coefficients, which have been given independent, double exponential prior distributions, and the properties of this approach are explored.
Calibration of multivariate generalized hyperbolic distributions using the em algorithm, with applications in risk management, portfolio optimization and portfolio credit risk
64 Citations2005Alec N. Kercheval, Wenbo Hu
This dissertation describes a way to stably calibrate GH distributions for a wider range of parameters than has previously been reported and develops a version of the EM algorithm for calibrating GH distributions, which enables for the first time certain GH distributions to be used in modeling contexts when previously they have been numerically intractable.
Local dependent components
56 Citations2007Arto Klami, Samuel Kaski
A mixture of probabilistic canonical correlation analyzers model for analyzing local correlations, or more generally mutual statistical dependencies, in cooccurring data pairs and decomposes the variation in the data into shared and data set-specific components.
arXiv (Cornell University)Sparse Canonical Correlation Analysis
18 Citations2009David R. Hardoon, John Shawe‐Taylor