A Bilinear Model for Sparse Coding

Published 1 January 2002

David B. Grimes, Rajesh P. N. Rao

Citations14

TL;DR

An algorithm for sparse coding based on a bilinear generative model of images that helps reduce redundancy in the image code and provides a basis for transformation-invariant vision is described.

Abstract

Recent algorithms for sparse coding and independent component analy-sis (ICA) have demonstrated how localized features can be learned from natural images. However, these approaches do not take image transfor-mations into account. As a result, they produce image codes that are redundant because the same feature is learned at multiple locations. We describe an algorithm for sparse coding based on a bilinear generative model of images. By explicitly modeling the interaction between im-age features and their transformations, the bilinear approach helps reduce redundancy in the image code and provides a basis for transformation-invariant vision. We present results demonstrating bilinear sparse coding of natural images. We also explore an extension of the model that can capture spatial relationships between the independent features of an ob-ject, thereby providing a new framework for parts-based object recogni-tion. 1

Keywords

Computer ScienceBiochemistry, Genetics and Molecular Biology

NatureEmergence of simple-cell receptive field properties by learning a sparse code for natural images

5,840 Citations1996Bruno A. Olshausen, David J. Field

It is shown that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex.

Nature NeurosciencePredictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects

5,736 Citations1999Rajesh P. N. Rao, Dana H. Ballard

Results suggest that rather than being exclusively feedforward phenomena, nonclassical surround effects in the visual cortex may also result from cortico-cortical feedback as a consequence of the visual system using an efficient hierarchical strategy for encoding natural images.

Vision ResearchSparse coding with an overcomplete basis set: A strategy employed by V1?

3,720 Citations1997Bruno A. Olshausen, David J. Field

These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli.

Psychological ReviewSome informational aspects of visual perception.

2,910 Citations1954Fred Attneave

Special types of lawfulness which may exist in space at a fixed time, and which seem particularly relevant to processes of visual perception are focused on.

Vision ResearchThe “independent components” of natural scenes are edge filters

2,269 Citations1997Anthony J. Bell, T. J. Sejnowski

It is shown that a new unsupervised learning algorithm based on information maximization, a nonlinear "infomax" network, when applied to an ensemble of natural scenes produces sets of visual filters that are localized and oriented.

Neural ComputationLearning Overcomplete Representations

1,142 Citations2000Michael S. Lewicki, Terrence J. Sejnowski

It is shown that overcomplete bases can yield a better approximation of the underlying statistical distribution of the data and can thus lead to greater coding efficiency and provide a method for Bayesian reconstruction of signals in the presence of noise and for blind source separation when there are more sources than mixtures.

Neural ComputationSeparating Style and Content with Bilinear Models

876 Citations2000Joshua B. Tenenbaum, William T. Freeman

A general framework for learning to solve two-factor tasks using bilinear models, which provide sufficiently expressive representations of factor interactions but can nonetheless be fit to data using efficient algorithms based on the singular value decomposition and expectation-maximization are presented.

Nature NeuroscienceNatural signal statistics and sensory gain control

874 Citations2001Odelia Schwartz, Eero P. Simoncelli

It is shown that this decomposition, with parameters optimized for the statistics of a generic ensemble of natural images or sounds, provides a good characterization of the nonlinear response properties of typical neurons in primary visual cortex or auditory nerve, respectively.

Philosophical Transactions of the Royal Society B Biological SciencesGenerative models for discovering sparse distributed representations

228 Citations1997Geoffrey E. Hinton, Zoubin Ghahramani

A hierarchical, generative model that can be viewed as a nonlinear generalization of factor analysis and can be implemented in a neural network that learns to extract sparse, distributed, hierarchical representations is described.

Learning Lie Groups for Invariant Visual Perception

73 Citations1998Rajesh P. N. Rao, Daniel Ruderman

A Bayesian method for learning invariances based on Lie group theory and experimental results suggest that the proposed method can learn Lie group operators for handling reasonably large 1-D translations and 2-D rotations.