Training Very Deep Networks
arXiv (Cornell University)Published 22 July 2015Open access
Rupesh K. Srivastava, Klaus Greff, Jürgen Schmidhuber
Citations546
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new architecture designed to overcome the challenges of training very deep networks, inspired by Long Short-Term Memory recurrent networks, which allows unimpeded information flow across many layers on information highways.
Abstract
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
arXiv (Cornell University)Very Deep Convolutional Networks for Large-Scale Image Recognition
75,540 Citations2014Karen Simonyan, Andrew Zisserman
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Going deeper with convolutions
46,825 Citations2015Christian Szegedy, Wei Liu +7 more
A deep convolutional neural network architecture codenamed Inception is proposed that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
18,777 Citations2015Kaiming He, Xiangyu Zhang +2 more
This work proposes a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit and derives a robust initialization method that particularly considers the rectifier nonlinearities.
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Understanding the difficulty of training deep feedforward neural networks
12,692 Citations2010Xavier Glorot, Yoshua Bengio
The objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks, to better understand these recent relative successes and help design better algorithms in the future.
Caffe
11,194 Citations2014Yangqing Jia, Evan Shelhamer +6 more
Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures.
Multi-column deep neural networks for image classification
3,764 Citations2012Dan Cireşan, Ueli Meier +1 more
On the very competitive MNIST handwriting benchmark, this method is the first to achieve near-human performance and improves the state-of-the-art on a plethora of common image classification benchmarks.
On the importance of initialization and momentum in deep learning
3,526 Citations2013Ilya Sutskever, James Martens +2 more
It is shown that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs to levels of performance that were previously achievable only with Hessian-Free optimization.
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)LOL: An Investigation into Cybernetic Humor, or: Can Machines Laugh?
3,084 Citations2016Alex Graves, Gervasi, Vincenzo +1 more
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time.
Learning to forget: continual prediction with LSTM
2,383 Citations1999Felix A. Gers
International Joint Conference on Artificial IntelligenceFlexible, high performance convolutional neural networks for image classification
1,203 Citations2011Dan Cireşan, Ueli Meier +3 more
arXiv (Cornell University)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
1,014 Citations2013Andrew Saxe, James L. McClelland +1 more
Maxout Networks
607 Citations2013Ian Goodfellow, David Warde-Farley +3 more
A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique.
arXiv (Cornell University)On the Number of Linear Regions of Deep Neural Networks
570 Citations2014Guido Montúfar, Razvan Pascanu +2 more
IEEE Transactions on Neural Networks and Learning SystemsOn the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures
517 Citations2014Monica Bianchini, Franco Scarselli
A new measure based on topological concepts is introduced, aimed at evaluating the complexity of the function implemented by a neural network, used for classification purposes, and results seem to support the idea that deep networks actually implements functions of higher complexity, so that they are able, with the same number of resources, to address more difficult problems.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
491 Citations2014Yann Dauphin, Razvan Pascanu +4 more
DSpace@MIT (Massachusetts Institute of Technology)Computational limitations of small-depth circuits
461 Citations1987Johan Torkel Håstad
Neural ComputationLearning Complex, Extended Sequences Using the Principle of History Compression
407 Citations1992Jürgen Schmidhuber
A simple principle for reducing the descriptions of event sequences without loss of information is introduced and this insight leads to the construction of neural architectures that learn to divide and conquer by recursively decomposing sequences.
Computational ComplexityOn the power of small-depth threshold circuits
254 Citations1991Johan H�stad, Mikael Goldmann
It is proved that there are monotone functionsfk that can be computed in depthk and linear size ⋎, ⋏-circuits but require exponential size to compute by a depthk−1 monot one weighted threshold circuit.
Lecture notes in computer scienceTraining Deep and Recurrent Networks with Hessian-Free Optimization
206 Citations2012James Martens, Ilya Sutskever
This chapter describes the basic HF approach, and examines well-known performance-improving techniques such as preconditioning which have been beneficial for neural network training and others of a more heuristic nature which are harder to justify, but which have found to work well in practice.
arXiv (Cornell University)Spatially-sparse convolutional neural networks
197 Citations2014Benjamin Graham
A CNN for processing spatially-sparse inputs, motivated by the problem of online handwriting recognition, and applying a deep convolutional network using sparsity has resulted in a substantial reduction in test error on the CIFAR small picture datasets.
Deep Learning Made Easier by Linear Transformations in Perceptrons
178 Citations2012Tapani Raiko, Harri Valpola +1 more
The usefulness of the transformations are confirmed, which make basic stochastic gradient learning competitive with state-of-the-art learning algorithms in speed and that they seem also to help find solutions that generalize better.
arXiv (Cornell University)Grid Long Short-Term Memory
172 Citations2015Nal Kalchbrenner, Ivo Danihelka +1 more
Neural Information Processing SystemsCompete to Compute
134 Citations2013Rupesh K. Srivastava, Jonathan Masci +3 more
This paper applies the concept of local competition among neighboring neurons to gradient-based, backprop-trained artificial multilayer NNs, finding that NNs with competing linear units tend to outperform those with non-competing nonlinear units, and avoid catastrophic forgetting when training sets change over time.
arXiv (Cornell University)Feature Learning in Deep Neural Networks - Studies on Speech Recognition\n Tasks
84 Citations2013Dong Yu, Michael L. Seltzer +3 more
arXiv (Cornell University)Random Walk Initialization for Training Very Deep Feedforward Networks
70 Citations2014David Sussillo, L. F. Abbott
It is shown that the successive application of correctly scaled random matrices to an initial vector results in a random walk of the log of the norm of the resulting vectors, and the scaling that makes this walk unbiased is computed.
arXiv (Cornell University)On the Expressive Efficiency of Sum Product Networks
35 Citations2014James Martens, Venkatesh Medabalimi
A result is established which establishes the existence of a relatively simple distribution with fully tractable marginal densities which cannot be efficiently captured by D&C SPNs of any depth, but which can be efficiently capture by various other deep generative models.
arXiv (Cornell University)Binding via Reconstruction Clustering
23 Citations2015Klaus Greff, Rupesh K. Srivastava +1 more
This work proposes an unsupervised algorithm that uses denoising autoencoders to dynamically bind features together in multi-object inputs through an Expectation-Maximization-like clustering process and develops a probabilistic framework that explicitly models inputs as a composition of multiple objects.