Three Mechanisms of Weight Decay Regularization
arXiv (Cornell University)Published 29 October 2018Open access
Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Grosse
Citations55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.
Keywords
Computer Science
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
arXiv (Cornell University)Very Deep Convolutional Networks for Large-Scale Image Recognition
75,540 Citations2014Karen Simonyan, Andrew Zisserman
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
15,635 Citations2024Sergey Ioffe
arXiv (Cornell University)Wide Residual Networks
1,921 Citations2016Sergey Zagoruyko, Nikos Komodakis
A Simple Weight Decay Can Improve Generalization
1,296 Citations1991Anders Krogh, John Hertz
It is proven that a weight decay has two effects in a linear network, and it is shown how to extend these results to networks with hidden layers and non-linear units.
arXiv (Cornell University)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
1,014 Citations2013Andrew Saxe, James L. McClelland +1 more
Deep learning via Hessian-free optimization
714 Citations2010James Martens
A 2nd-order optimization method based on the "Hessian-free" approach is developed, and applied to training deep auto-encoders, and results superior to those reported by Hinton & Salakhutdinov (2006) are obtained.
arXiv (Cornell University)On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
576 Citations2016Nitish Shirish Keskar, Dheevatsa Mudigere +3 more
arXiv (Cornell University)Decoupled Weight Decay Regularization
494 Citations2017Ilya Loshchilov, Frank Hutter
arXiv (Cornell University)Optimizing Neural Networks with Kronecker-factored Approximate Curvature
292 Citations2015James Martens, Roger Grosse
arXiv (Cornell University)Adding Gradient Noise Improves Learning for Very Deep Networks
265 Citations2015Arvind Neelakantan, Luke Vilnis +5 more
This paper explores the low-overhead and easy-to-implement optimization technique of adding annealed Gaussian noise to the gradient, which it is found surprisingly effective when training these very deep architectures.
arXiv (Cornell University)Sensitivity and Generalization in Neural Networks: an Empirical Study
224 Citations2018Roman Novak, Yasaman Bahri +3 more
arXiv (Cornell University)L2 Regularization versus Batch and Weight Normalization
210 Citations2017Twan van Laarhoven
It is shown that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate, and this leads to a discussion on other ways to mitigate this issue.
arXiv (Cornell University)Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
90 Citations2017Tengyuan Liang, Tomaso Poggio +2 more
An analytical characterization of the new Fisher-Rao norm is discovered, through which it is shown that the new measure serves as an umbrella for several existing norm-based complexity measures and establishes norm-comparison inequalities.
Neural ComputationOn “Natural” Learning and Pruning in Multilayered Perceptrons
69 Citations2000Tom Heskes
This article derives natural gradients in a slightly different manner and discusses implications for batch-mode learning and pruning, linking them to existing algorithms such as Levenberg-Marquardt optimization and optimal brain surgeon.
arXiv (Cornell University)Norm matters: efficient and accurate normalization schemes in deep networks
52 Citations2018Elad Hoffer, Ron Banner +2 more
arXiv (Cornell University)Noisy Natural Gradient as Variational Inference
28 Citations2017Guodong Zhang, Shengyang Sun +2 more
arXiv (Cornell University)A Coordinate-Free Construction of Scalable Natural Gradient
8 Citations2018Kevin Luk, Roger Grosse
This work analyzes the invariance properties of the Kronecker-Factored Approximate Curvature algorithm by constructing the algorithm in a coordinate-free way, and explicitly construct a Riemannian metric under which the natural gradient matches the K-FAC update.