Denoising autoencoder with modulated lateral connections learns invariant representations of natural images

arXiv (Cornell University)Published 22 December 2014Open access

Antti Rasmus, Tapani Raiko, Harri Valpola

DOI10.48550/arxiv.1412.7210

Citations14

TL;DR

Adding modulated lateral connections to the model improves the accuracy of the probability model for inputs, as measured by denoising performance, results in representations whose degree of invariance grows faster towards the higher layers, and supports the formation of diverse invariant poolings.

Abstract

Suitable lateral connections between encoder and decoder are shown to allow higher layers of a denoising autoencoder (dAE) to focus on invariant representations. In regular autoencoders, detailed information needs to be carried through the highest layers but lateral connections from encoder to decoder relieve this pressure. It is shown that abstract invariant features can be translated to detailed reconstructions when invariant features are allowed to modulate the strength of the lateral connection. Three dAE structures with modulated and additive lateral connections, and without lateral connections were compared in experiments using real-world images. The experiments verify that adding modulated lateral connections to the model 1) improves the accuracy of the probability model for inputs, as measured by denoising performance; 2) results in representations whose degree of invariance grows faster towards the higher layers; and 3) supports the formation of diverse invariant poolings.

Keywords

Computer ScienceBiochemistry, Genetics and Molecular Biology

Learning Multiple Layers of Features from Tiny Images

25,432 Citations2024Alex Krizhevsky

View PDF

Neural NetworksDeep learning in neural networks: An overview

18,033 Citations2014Jürgen Schmidhuber

This historical survey compactly summarizes relevant work, much of it from the previous millennium, review deep supervised learning, unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

View PDF

Extracting and composing robust features with denoising autoencoders

7,343 Citations2008Pascal Vincent, Hugo Larochelle +2 more

This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.

NatureEmergence of simple-cell receptive field properties by learning a sparse code for natural images

5,840 Citations1996Bruno A. Olshausen, David J. Field

It is shown that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex.

arXiv (Cornell University)ADADELTA: An Adaptive Learning Rate Method

5,531 Citations2012Matthew D. Zeiler

A novel per-dimension learning rate method for gradient descent called ADADELTA that dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent is presented.

View PDF

The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks

4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more

These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.

Nature NeuroscienceHierarchical models of object recognition in cortex

3,404 Citations1999Maximilian Riesenhuber, Tomaso Poggio

A new hierarchical model consistent with physiological data from inferotemporal cortex that accounts for this complex visual task and makes testable predictions is described.

Neural ComputationLearning Invariance from Transformation Sequences

683 Citations1991Péter Földiák

An application of the algorithm is presented in which the network learns invariance to shift in retinal position, which may be involved in the development of the characteristic shift invariance property of complex cells in the primary visual cortex and also in theDevelopment of more complicated invariance properties of neurons in higher visual areas.

arXiv (Cornell University)Discriminative Unsupervised Feature Learning with Convolutional Neural Networks

649 Citations2014Alexey Dosovitskiy, Jost Tobias Springenberg +2 more

This paper presents an approach for training a convolutional neural network using only unlabeled data and trains the network to discriminate between a set of surrogate classes, finding that this simple feature learning algorithm is surprisingly successful when applied to visual object recognition.

View PDF

Neural ComputationEmergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces

546 Citations2000Aapo Hyvärinen, Patrik O. Hoyer

It is shown that the same principle of independence maximization can explain the emergence of phase- and shift-invariant features, similar to those found in complex cells, by maximizing the independence between norms of projections on linear subspaces.