A Variational Method for Learning Sparse and Overcomplete Representations

An expectation-maximization algorithm for learning sparse and overcomplete data representations is presented, which exploits a variational approximation to a range of heavy-tailed distributions whose limit is the Laplacian.

Abstract

An expectation-maximization algorithm for learning sparse and overcomplete data representations is presented. The proposed algorithm exploits a variational approximation to a range of heavy-tailed distributions whose limit is the Laplacian. A rigorous lower bound on the sparse prior distribution is derived, which enables the analytic marginalization of a lower bound on the data likelihood. This lower bound enables the development of an expectation-maximization algorithm for learning the overcomplete basis vectors and inferring the most probable basis coefficients.