Deeply-Supervised Nets
arXiv (Cornell University)Published 18 September 2014Open access
Chen‐Yu Lee, Saining Xie, Patrick W. Gallagher, Zhengyou Zhang, Zhuowen Tu
Citations1,034
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robustness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce "companion objective" to the individual hidden layers, in addition to the overall objective at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advantage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN).
Keywords
Computer Science
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks
4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
IEEE Transactions on Audio Speech and Language ProcessingContext-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition
3,052 Citations2011George E. Dahl, Dong Yu +2 more
A pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output that can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs.
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
2,446 Citations2009Honglak Lee, Roger Grosse +2 more
The convolutional deep belief network is presented, a hierarchical generative model which scales to realistic image sizes and is translation-invariant and supports efficient bottom-up and top-down probabilistic inference.
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Adaptive Algorithms and Stochastic Approximations
2,044 Citations1990Albert Benveniste, Michel Métivier +1 more
The juxtaposition of these two expressions in the title reflects the ambition of the authors to produce a reference work, both for engineers who use adaptive algorithms and for probabilists or statisticians who would like to study stochastic approximations in terms of problems arising from real applications.
International review of cytologyRegularization of Neural Networks using DropConnect
1,916 Citations2013Li Wan, Matthew D. Zeiler +3 more
This work introduces DropConnect, a generalization of Dropout, for regularizing large fully-connected layers within neural networks, and derives a bound on the generalization performance of both Dropout and DropConnect.
Machine LearningDistributed representations, simple recurrent networks, and grammatical structure
1,381 Citations1991Jeffrey L. Elman
arXiv (Cornell University)Network In Network
1,040 Citations2014Min Lin, Qiang Chen +1 more
With enhanced local modeling via the micro network, the proposed deep network structure NIN is able to utilize global average pooling over feature maps in the classification layer, which is easier to interpret and less prone to overfitting than traditional fully connected layers.
arXiv (Cornell University)Deep Learning using Linear Support Vector Machines
740 Citations2013Yichuan Tang
The results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.
arXiv (Cornell University)Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
576 Citations2013Matthew D. Zeiler, Rob Fergus
Maxout Networks
451 Citations2013Ian Goodfellow, David Warde-Farley +3 more
arXiv (Cornell University)Visualizing and Understanding Convolutional Networks
447 Citations2013Matthew D. Zeiler, Rob Fergus
arXiv (Cornell University)Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
411 Citations2012Ohad Shamir, Tong Zhang
Deep learning via semi-supervised embedding
387 Citations2008Jason Weston, Frédéric Ratle +1 more
It is shown how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture.
Large-scale Learning with SVM and Convolutional for Generic Object Categorization
312 Citations2006Fu Jie Huang, Yann LeCun
It is shown that architectures such as convolutional networks are good at learning invariant features, but not always optimal for classification, while Support Vector Machines are good for producing decision surfaces from wellbehaved feature vectors, but cannot learn complicated invariances.
Neural Information Processing SystemsTiled convolutional neural networks
309 Citations2010Jiquan Ngiam, Zhenghao Chen +4 more
This paper proposes tiled convolution neural networks (Tiled CNNs), which use a regular "tiled" pattern of tied weights that does not require that adjacent hidden units share identical weights, but instead requires only that hidden units k steps away from each other to have tied weights.
arXiv (Cornell University)Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima
185 Citations2013Po‐Ling Loh, Martin J. Wainwright
Discriminative Transfer Learning with Tree-based Priors
171 Citations2013Nitish Srivastava, Ruslan Salakhutdinov
This work proposes a method for improving classification performance for high capacity classifiers by discovering similar classes and transferring knowledge among them, which learns to organize the classes into a tree hierarchy, and proposes an algorithm for learning the underlying tree structure.
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
140 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
Journal of Machine Learning ResearchNonparametric guidance of autoencoder representations using label information
57 Citations2012Jasper Snoek, Ryan P. Adams +1 more
This work proposes a nonparametric approach that uses a Gaussian process to guide the representation of a discriminative function, and demonstrates the superiority of this guidance mechanism on four data sets, including a real-world application to rehabilitation research.