Resnet in Resnet: Generalizing Residual Architectures
arXiv (Cornell University)Published 25 March 2016Open access
Sasha Targ, Diogo Almeida, Kevin Lyman
Citations653
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Resnet in Resnet (RiR): a deep dual-stream architecture that generalizes ResNets and standard CNNs and is easily implemented with no computational overhead is introduced.
Abstract
Residual networks (ResNets) have recently achieved state-of-the-art on challenging computer vision tasks. We introduce Resnet in Resnet (RiR): a deep dual-stream architecture that generalizes ResNets and standard CNNs and is easily implemented with no computational overhead. RiR consistently improves performance over ResNets, outperforms architectures with similar amounts of augmentation on CIFAR-10, and establishes a new state-of-the-art on CIFAR-100.
Keywords
Computer Science
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
61,841 Citations2009Jia Deng, Wei Dong +4 more
A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
18,777 Citations2015Kaiming He, Xiangyu Zhang +2 more
This work proposes a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit and derives a robust initialization method that particularly considers the rectifier nonlinearities.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
Neural ComputationLearning to Forget: Continual Prediction with LSTM
5,419 Citations2000Felix A. Gers, Jürgen Schmidhuber +1 more
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
On the importance of initialization and momentum in deep learning
3,526 Citations2013Ilya Sutskever, James Martens +2 more
It is shown that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs to levels of performance that were previously achievable only with Hessian-Free optimization.
arXiv (Cornell University)Striving for Simplicity: The All Convolutional Net
2,603 Citations2014Jost Tobias Springenberg, Alexey Dosovitskiy +2 more
arXiv (Cornell University)Training Very Deep Networks
1,100 Citations2015Rupesh K. Srivastava, Klaus Greff +1 more
arXiv (Cornell University)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
1,014 Citations2013Andrew Saxe, James L. McClelland +1 more
IEEE Transactions on Neural Networks and Learning SystemsOn the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures
517 Citations2014Monica Bianchini, Franco Scarselli
A new measure based on topological concepts is introduced, aimed at evaluating the complexity of the function implemented by a neural network, used for classification purposes, and results seem to support the idea that deep networks actually implements functions of higher complexity, so that they are able, with the same number of resources, to address more difficult problems.
arXiv (Cornell University)Learning Longer Memory in Recurrent Neural Networks
198 Citations2014Tomáš Mikolov, Armand Joulin +3 more
arXiv (Cornell University)Grid Long Short-Term Memory
172 Citations2015Nal Kalchbrenner, Ivo Danihelka +1 more