Towards Understanding the Invertibility of Convolutional Neural Networks
Published 28 July 2017Open access
Anna C. Gilbert, Yi Zhang, Kibok Lee, Yuting Zhang, Honglak Lee
Citations19
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A theoretical explanation is focused on and a mathematical model of sparse signal recovery that is consistent with CNNs with random weights is developed that gives an exact connection to a particular model of model-based compressive sensing and random-weight CNNs.
Abstract
Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is consistent with CNNs with random weights. We give an exact connection to a particular model of model-based compressive sensing (and its recovery algorithms) and random-weight CNNs. We show empirically that several learned networks are consistent with our mathematical analysis and then demonstrate that with such a simple theoretical framework, we can obtain reasonable reconstruction results on real images. We also discuss gaps between our model assumptions and the CNN trained for classification in practical scenarios.
Keywords
Computer ScienceEngineering
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
61,841 Citations2009Jia Deng, Wei Dong +4 more
A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
SIAM Journal on Imaging SciencesA Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
12,045 Citations2009Amir Beck, Marc Teboulle
A new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically.
Neural ComputationBackpropagation Applied to Handwritten Zip Code Recognition
11,878 Citations1989Yann LeCun, Bernhard E. Boser +5 more
This paper demonstrates how constraints from the task domain can be integrated into a backpropagation network through the architecture of the network, successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service.
IEEE Signal Processing MagazineDeep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
10,313 Citations2012Geoffrey E. Hinton, Li Deng +9 more
IEEE Transactions on Signal ProcessingMatching pursuits with time-frequency dictionaries
9,115 Citations1993Stéphane Mallat, Zhifeng Zhang
The authors introduce an algorithm, called matching pursuit, that decomposes any signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions, chosen in order to best match the signal structures.
NatureEmergence of simple-cell receptive field properties by learning a sparse code for natural images
5,840 Citations1996Bruno A. Olshausen, David J. Field
It is shown that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex.
IEEE Transactions on Image ProcessingImage Super-Resolution Via Sparse Representation
5,228 Citations2010Shuicheng Yan, John Wright +2 more
This paper presents a new approach to single-image superresolution, based upon sparse signal representation, which generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods.
arXiv (Cornell University)Caffe: Convolutional Architecture for Fast Feature Embedding
4,290 Citations2014Yangqing Jia, Evan Shelhamer +6 more
Comptes Rendus MathématiqueThe restricted isometry property and its implications for compressed sensing
3,779 Citations2008Emmanuel J. Candès
arXiv (Cornell University)Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
3,282 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
MatConvNet
2,539 Citations2015Andrea Vedaldi, Karel Lenc
MatConvNet exposes the building blocks of CNNs as easy-to-use MATLAB functions, providing routines for computing convolutions with filter banks, feature pooling, normalisation, and much more.
Applied and Computational Harmonic AnalysisIterative hard thresholding for compressed sensing
2,317 Citations2009Thomas Blumensath, Mike E. Davies
This paper presents a theoretical analysis of the iterative hard thresholding algorithm when applied to the compressed sensing recovery problem and shows that the algorithm has the following properties - it gives near-optimal error guarantees, and is robust to observation noise.
Building high-level features using large scale unsupervised learning
1,650 Citations2013Quoc V. Le
Contrary to what appears to be a widely-held intuition, the experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not.
arXiv (Cornell University)Deep Speech: Scaling up end-to-end speech recognition
1,510 Citations2014Awni Hannun, Carl Case +9 more
Deep Speech, a state-of-the-art speech recognition system developed using end-to-end deep learning, outperforms previously published results on the widely studied Switchboard Hub5'00, achieving 16.0% error on the full test set.
Sparse deep belief net model for visual area V2
881 Citations2007Honglak Lee, Chaitanya Ekanadham +1 more
An unsupervised learning model is presented that faithfully mimics certain properties of visual area V2 and the encoding of these more complex "corner" features matches well with the results from the Ito & Komatsu's study of biological V2 responses, suggesting that this sparse variant of deep belief networks holds promise for modeling more higher-order features.
Neural Information Processing SystemsSparse Feature Learning for Deep Belief Networks
715 Citations2007Marc’Aurelio Ranzato, Y-Lan Boureau +1 more
This work proposes a simple criterion to compare and select different unsupervised machines based on the trade-off between the reconstruction error and the information content of the representation, and describes a novel and efficient algorithm to learn sparse representations.
Cambridge University Press eBooksIntroduction to the non-asymptotic analysis of random matrices
577 Citations2012Roman Vershynin
This is a tutorial on some basic non-asymptotic methods and concepts in random matrix theory, particularly for the problem of estimating covariance matrices in statistics and for validating probabilistic constructions of measurementMatrices in compressed sensing.
On Random Weights and Unsupervised Feature Learning
336 Citations2011Andrew Saxe, Pang Wei Koh +4 more
The answer is that certain convolutional pooling architectures can be inherently frequency selective and translation invariant, even with random weights, and the viability of extremely fast architecture search is demonstrated by using random weights to evaluate candidate architectures, thereby sidestepping the time-consuming learning process.
arXiv (Cornell University)Stacked What-Where Auto-encoders
228 Citations2015Junbo Zhao, Michaël Mathieu +2 more
A novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training.
IEEE Transactions on Signal ProcessingDeep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
161 Citations2016Raja Giryes, Guillermo Sapiro +1 more
It is formally proved that these networks with random Gaussian weights perform a distance-preserving embedding of the data, with a special treatment for in-class and out-of-class data.
arXiv (Cornell University)A Powerful Generative Model Using Random Weights for the Deep Image Representation
49 Citations2016Kun He, Yan Wang +1 more
This work explores new and powerful generative models for three popular deep visualization tasks using untrained, random weight convolutional neural networks and provides a new and fascinating tool to study the representation of deep network architecture.
arXiv (Cornell University)Why are deep nets reversible: A simple theory, with implications for training
41 Citations2015Sanjeev Arora, Yingyu Liang +1 more
A very simple generative model for RELU deep nets, with the following characteristics: its correctness can be proven under a clean theoretical assumption: the edge weights in real-life deep nets behave like random numbers.