Shake-Shake regularization
arXiv (Cornell University)Published 21 May 2017Open access
Xavier Gastaldi
Citations312
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination.
Abstract
The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem. The idea is to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination. Applied to 3-branch residual networks, shake-shake regularization improves on the best single shot published results on CIFAR-10 and CIFAR-100 by reaching test errors of 2.86% and 15.85%. Experiments on architectures without skip connections or Batch Normalization show encouraging results and open the door to a large set of applications. Code is available at https://github.com/xgastaldi/shake-shake
Keywords
Computer ScienceEngineering
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
International Journal of Computer VisionImageNet Large Scale Visual Recognition Challenge
40,207 Citations2015Olga Russakovsky, Jia Deng +10 more
The creation of this benchmark dataset and the advances in object recognition that have been possible as a result are described, and the state-of-the-art computer vision accuracy with human accuracy is compared.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
Lecture notes in computer scienceIdentity Mappings in Deep Residual Networks
10,094 Citations2016Kaiming He, Xiangyu Zhang +2 more
The propagation formulations behind the residual building blocks suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation.
Proceedings of the AAAI Conference on Artificial IntelligenceInception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
4,494 Citations2017Christian Szegedy, Sergey Ioffe +2 more
Clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly is given and several new streamlined architectures for both residual and non-residual Inception Networks are presented.
On the importance of initialization and momentum in deep learning
3,526 Citations2013Ilya Sutskever, James Martens +2 more
It is shown that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs to levels of performance that were previously achievable only with Hessian-Free optimization.
Adaptive Algorithms and Stochastic Approximations
2,044 Citations1990Albert Benveniste, Michel Métivier +1 more
The juxtaposition of these two expressions in the title reflects the ambition of the authors to produce a reference work, both for engineers who use adaptive algorithms and for probabilists or statisticians who would like to study stochastic approximations in terms of problems arising from real applications.
arXiv (Cornell University)Densely Connected Convolutional Networks
1,877 Citations2016Gao Huang, Zhuang Liu +2 more
arXiv (Cornell University)SGDR: Stochastic Gradient Descent with Warm Restarts
1,744 Citations2016Ilya Loshchilov, Frank Hutter
This paper proposes a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks and empirically studies its performance on the CIFAR-10 and CIFARS datasets.
arXiv (Cornell University)FractalNet: Ultra-Deep Neural Networks without Residuals
616 Citations2016Gustav Larsson, Michael Maire +1 more
In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks.
Neural ComputationThe Effects of Adding Noise During Backpropagation Training on a Generalization Performance
489 Citations1996Guozhong An
It is shown that input noise and weight noise encourage the neural-network output to be a smooth function of the input or its weights, respectively, and in the weak-noise limit, noise added to the output of the neural networks only changes the objective function by a constant, it cannot improve generalization.
arXiv (Cornell University)Aggregated Residual Transformations for Deep Neural Networks
383 Citations2016Saining Xie, Ross Girshick +3 more
arXiv (Cornell University)Deep Networks with Stochastic Depth
286 Citations2016Gao Huang, Yu Sun +3 more
arXiv (Cornell University)Adding Gradient Noise Improves Learning for Very Deep Networks
265 Citations2015Arvind Neelakantan, Luke Vilnis +5 more
This paper explores the low-overhead and easy-to-implement optimization technique of adding annealed Gaussian noise to the gradient, which it is found surprisingly effective when training these very deep architectures.
International Conference on Learning RepresentationsConvergent Learning: Do different neural networks learn the same representations?
89 Citations2016Yixuan Li, Jason Yosinski +3 more
This paper investigates the extent to which neural networks exhibit convergent learning, which is when the representations learned by multiple nets converge to a set of features which are either individually similar between networks or where subsets of features span similar low-dimensional spaces.
Proceedings of the AAAI Conference on Artificial IntelligenceShakeout: A New Regularized Deep Neural Network Training Scheme
48 Citations2016Guoliang Kang, Jun Li +1 more
This paper presents a new training scheme: Shakeout, which leads to a combination of L1 regularization and L2 regularization imposed on the weights, which has been proved effective by the Elastic Net models in practice.
arXiv (Cornell University)Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
40 Citations2016Masoud Abdi, Saeid Nahavandi
A new convolutional neural network architecture is proposed which builds upon the success of residual networks by explicitly exploiting the interpretation of very deep networks as an ensemble, and generates models that are wider, rather than deeper, which significantly improves accuracy.