Scaling the Scattering Transform: Deep Hybrid Networks
Published 1 October 2017Open access
Edouard Oyallon, Eugene Belilovsky, Sergey Zagoruyko
Citations143
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The scattering network is used as a generic and fixed initialization of the first layers of a supervised hybrid deep network, and it is shown that early layers do not necessarily need to be learned, providing the best results to-date with pre-defined representations while being competitive with Deep CNNs.
Abstract
We use the scattering network as a generic and fixed ini-tialization of the\nfirst layers of a supervised hybrid deep network. We show that early layers do\nnot necessarily need to be learned, providing the best results to-date with\npre-defined representations while being competitive with Deep CNNs. Using a\nshallow cascade of 1 x 1 convolutions, which encodes scattering coefficients\nthat correspond to spatial windows of very small sizes, permits to obtain\nAlexNet accuracy on the imagenet ILSVRC2012. We demonstrate that this local\nencoding explicitly learns invariance w.r.t. rotations. Combining scattering\nnetworks with a modern ResNet, we achieve a single-crop top 5 error of 11.4% on\nimagenet ILSVRC2012, comparable to the Resnet-18 architecture, while utilizing\nonly 10 layers. We also find that hybrid architectures can yield excellent\nperformance in the small sample regime, exceeding their end-to-end\ncounterparts, through their ability to incorporate geometrical priors. We\ndemonstrate this on subsets of the CIFAR-10 dataset and on the STL-10 dataset.\n
Keywords
Computer ScienceEngineering
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Histograms of Oriented Gradients for Human Detection
31,896 Citations2005Navneet Dalal, Bill Triggs
It is shown experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection, and the influence of each stage of the computation on performance is studied.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
24,381 Citations2015Sergey Ioffe, Christian Szegedy
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Object recognition from local scale-invariant features
16,195 Citations1999David Lowe
Experimental results show that robust object recognition can be achieved in cluttered partially occluded images with a computation time of under 2 seconds.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
Lecture notes in computer scienceVisualizing and Understanding Convolutional Networks
15,444 Citations2014Matthew D. Zeiler, Rob Fergus
A novel visualization technique is introduced that gives insight into the function of intermediate feature layers and the operation of the classifier in large Convolutional Network models, used in a diagnostic role to find model architectures that outperform Krizhevsky et al on the ImageNet classification benchmark.
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
7,929 Citations2006Svetlana Lazebnik, C. Schmid +1 more
This paper presents a method for recognizing scene categories based on approximate global geometric correspondence that exceeds the state of the art on the Caltech-101 database and achieves high accuracy on a large database of fifteen natural scene categories.
arXiv (Cornell University)Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
6,981 Citations2015Alec Radford, Luke Metz +1 more
Wide Residual Networks
5,985 Citations2016Sergey Zagoruyko, Nikos Komodakis
This paper conducts a detailed experimental study on the architecture of ResNet blocks and proposes a novel architecture where the depth and width of residual networks are decreased and the resulting network structures are called wide residual networks (WRNs), which are far superior over their commonly used thin and very deep counterparts.
arXiv (Cornell University)Intriguing properties of neural networks
5,745 Citations2013Christian Szegedy, Wojciech Zaremba +5 more
It is found that there is no distinction between individual highlevel units and random linear combinations of high level units, according to various methods of unit analysis, and it is suggested that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks.
arXiv (Cornell University)How transferable are features in deep neural networks?
3,453 Citations2014Jason Yosinski, Jeff Clune +2 more
arXiv (Cornell University)Striving for Simplicity: The All Convolutional Net
2,603 Citations2014Jost Tobias Springenberg, Alexey Dosovitskiy +2 more
Convolutional networks and applications in vision
2,138 Citations2010Yann LeCun, Koray Kavukcuoglu +1 more
New unsupervised learning algorithms, and new non-linear stages that allow ConvNets to be trained with very few labeled samples are described, including one for visual object recognition and vision navigation for off-road mobile robots.
arXiv (Cornell University)Wide Residual Networks
1,921 Citations2016Sergey Zagoruyko, Nikos Komodakis
IEEE Transactions on Pattern Analysis and Machine IntelligenceInvariant Scattering Convolution Networks
1,584 Citations2013Joan Bruna, Stéphane Mallat
The mathematical analysis of wavelet scattering networks explains important properties of deep convolution networks for classification.
International Journal of Computer VisionImage Classification with the Fisher Vector: Theory and Practice
1,518 Citations2013Jorge Sánchez, Florent Perronnin +2 more
This work proposes to use the Fisher Kernel framework as an alternative patch encoding strategy: it describes patches by their deviation from an “universal” generative Gaussian mixture model, and reports experimental results showing that the FV framework is a state-of-the-art patch encoding technique.
Communications on Pure and Applied MathematicsGroup Invariant Scattering
1,035 Citations2012Stéphane Mallat
This paper constructs translation-invariant operators on L 2 .R d /, which are Lipschitz-continuous to the action of diffeomorphisms, and extendsScattering operators are extended on L2 .G/, where G is a compact Lie group, and are invariant under theaction of G.
Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis
983 Citations2011Quoc V. Le, Will Y. Zou +2 more
This paper presents an extension of the Independent Subspace Analysis algorithm to learn invariant spatio-temporal features from unlabeled video data and discovered that this method performs surprisingly well when combined with deep learning techniques such as stacking and convolution to learn hierarchical representations.
Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering SciencesUnderstanding deep convolutional networks
683 Citations2016Stéphane Mallat
Deep convolutional networks provide state-of-the-art classifications and regressions results over many high-dimensional problems and a mathematical framework is introduced to analyse their properties.
arXiv (Cornell University)Discriminative Unsupervised Feature Learning with Convolutional Neural Networks
649 Citations2014Alexey Dosovitskiy, Jost Tobias Springenberg +2 more
This paper presents an approach for training a convolutional neural network using only unlabeled data and trains the network to discriminate between a set of surrogate classes, finding that this simple feature learning algorithm is surprisingly successful when applied to visual object recognition.
Digital Access to Scholarship at Harvard (DASH) (Harvard University)Multi-Task Bayesian Optimization
451 Citations2013Kevin Swersky, Jasper Snoek +1 more
This paper proposes an adaptation of a recently developed acquisition function, entropy search, to the cost-sensitive, multi-task setting and demonstrates the utility of this new acquisition function by leveraging a small dataset to explore hyper-parameter settings for a large dataset.
Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination
423 Citations2013Laurent Sifre, Stéphane Mallat
An affine invariant representation is constructed with a cascade of invariants, which preserves information for classification and state-of-the-art classification results are obtained over texture databases with uncontrolled viewing conditions.
Springer tracts in advanced roboticsUnsupervised Feature Learning for RGB-D Based Object Recognition
364 Citations2013Liefeng Bo, Xiaofeng Ren +1 more
HMP uses sparse coding to learn hierarchical feature representations from raw RGB-D data in an unsupervised way and enables superior object recognition results using linear support vector machines.
arXiv (Cornell University)What makes ImageNet good for transfer learning?
307 Citations2016Minyoung Huh, Pulkit Agrawal +1 more
The overall findings suggest that most changes in the choice of pre-training data long thought to be critical do not significantly affect transfer performance.
Ask the locals: Multi-way local pooling for image recognition
283 Citations2011Y-Lan Boureau, Nicolas Le Roux +3 more
This work argues that a common trait found in much recent work in image recognition or retrieval is that it leverages locality in feature space on top of purely spatial locality, and proposes to apply this idea in its simplest form to an object recognition system based on the spatial pyramid framework to increase the performance of small dictionaries with very little added engineering.
High-dimensional signature compression for large-scale image classification
263 Citations2011Jorge Sánchez, Florent Perronnin
This work reports results on two large databases — ImageNet and a dataset of lM Flickr images — showing that it can reduce the storage of the authors' signatures by a factor 64 to 128 with little loss in accuracy and integrating the decompression in the classifier learning yields an efficient and scalable training algorithm.
arXiv (Cornell University)Stacked What-Where Auto-encoders
228 Citations2015Junbo Zhao, Michaël Mathieu +2 more
A novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training.
Multipath Sparse Coding Using Hierarchical Matching Pursuit
208 Citations2013Liefeng Bo, Xiaofeng Ren +1 more
Multipath Hierarchical Matching Pursuit (M-HMP), a novel feature learning architecture that combines a collection of hierarchical sparse features for image classification to capture multiple aspects of discriminative structures, is proposed.
International Journal of Computer VisionThe Structure of Locally Orderless Images
162 Citations1999Jan J. Koenderink, Andrea J. van Doorn
It is argued that locally orderless images are ubiquitous in perception and the visual arts and how to construct and use them in a variety of local and global image processing operations is described.
Deep roto-translation scattering for object classification
144 Citations2015Edouard Oyallon, Stéphane Mallat
A deep scattering convolution network, with complex wavelet filters over spatial and angular variables is introduced, which shows that refining image representations by using geometric priors is a promising direction to improve image classification and its understanding.
Selecting Receptive Fields in Deep Networks
142 Citations2024Adam Coates
This paper proposes a fast method to choose local receptive fields that group together those low-level features that are most similar to each other according to a pairwise similarity metric, and produces results showing how this method allows even simple unsupervised training algorithms to train successful multi-layered networks that achieve state-of-the-art results on CIFAR and STL datasets.
Fisher vectors meet Neural Networks: A hybrid classification architecture
129 Citations2015Florent Perronnin, Diane Larlus
A hybrid architecture that combines their strengths: the first unsupervised layers rely on the FV while the subsequent fully-connected supervised layers are trained with back-propagation, which significantly outperforms standard FV systems without incurring the high cost that comes with CNNs.
Realistic Modeling of Simple and Complex Cell Tuning in the HMAX Model, and Implications for Invariant Object Recognition in Cortex
124 Citations2004T. Serre, Maximilian Riesenhuber
It is shown that units in the early levels of HMAX can be tuned to produce realistic simple and complex cell-like tuning, and that the earlier findings on the invariance properties of model VTUs still hold in this more realistic version of the model.
Generalized Analytic Signals in Image Processing: Comparison, Theory and Applications
39 Citations2013Swanhild Bernstein, Jean-Luc Bouchot +2 more
This article is intended as a mathematical overview of the generalizations of analytic signals to higher-dimensional problems, as well as of their applications to and of their comparison on artificial and real-world image samples.
arXiv (Cornell University)Audio Texture Synthesis with Scattering Moments
33 Citations2013Joan Bruna, Stéphane Mallat
An audio texture synthesis algorithm based on scattering moments is introduced and examples show that scattering representation provide good synthesis of audio textures with much fewer coefficients than the state of the art.
Learning Stable Group Invariant Representations with Convolutional Networks
20 Citations2013Joan Bruna, Arthur Szlam +1 more
It is shown that the invariance properties built by deep convolutional networks can be cast as a form of stable group invariance, enabling more abstract, powerful invariant representations.
arXiv (Cornell University)Learning Stable Group Invariant Representations with Convolutional Networks
19 Citations2013Joan Bruna, Arthur Szlam +1 more
Building a Regular Decision Boundary with Deep Networks
15 Citations2017Edouard Oyallon
This work builds a generic architecture of Convolutional Neural Networks to discover empirical properties of neural networks and shows that the nonlinearity of a deep network does not need to be continuous, non expansive or point-wise, to achieve good performance.
arXiv (Cornell University)Deep unsupervised learning through spatial contrasting
14 Citations2016Elad Hoffer, Itay Hubara +1 more
A novel approach for unsupervised training of Convolutional networks that is based on contrasting between spatial regions within images is presented, which can be employed within conventional neural networks and trained using standard techniques such as SGD and back-propagation, thus complementing supervised methods.
arXiv (Cornell University)Inverse Problems with Invariant Multiscale Statistics
9 Citations2016Ivan Dokmanić, Joan Bruna +2 more
This work uses a non-linear multiscale scattering transform which discards the phase and thus exposes strong spectral correlations otherwise hidden beneath the phase fluctuations, and applies the algorithm to super-resolution and tomography and shows that it outperforms ad hoc convex regularizers and stably recovers the missing spectrum.
arXiv (Cornell University)Multiscale Hierarchical Convolutional Networks
7 Citations2017Jörn-Henrik Jacobsen, Edouard Oyallon +2 more
This work introduces an efficient implementation of Multiscale hierarchical convolutional networks where the dimensionality is progressively reduced by averaging intermediate layers along attribute indices.
arXiv (Cornell University)Inverse Problems with Invariant Multiscale Statistics
4 Citations2016Ivan Dokmanić, Joan Bruna +2 more