Flexible, high performance convolutional neural networks for image classification
International Joint Conference on Artificial IntelligencePublished 16 July 2011
Dan Cireşan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, Jürgen Schmidhuber
Citations1,203
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
We present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned in a supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple back-propagation perform better than more shallow ones. Learning is surprisingly rapid. NORB is completely trained within five epochs. Test error rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.
Keywords
Computer Science
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
The Journal of PhysiologyReceptive fields of single neurones in the cat's striate cortex
4,691 Citations1959David H. Hubel, T. N. Wiesel
The present investigation, made in acute preparations, includes a study of receptive fields of cells in the cat's striate cortex, which resembled retinal ganglion-cell receptive fields, but the shape and arrangement of excitatory and inhibitory areas differed strikingly from the concentric pattern found in retinalganglion cells.
Vision ResearchSparse coding with an overcomplete basis set: A strategy employed by V1?
3,720 Citations1997Bruno A. Olshausen, David J. Field
These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli.
Nature NeuroscienceHierarchical models of object recognition in cortex
3,404 Citations1999Maximilian Riesenhuber, Tomaso Poggio
A new hierarchical model consistent with physiological data from inferotemporal cortex that accounts for this complex visual task and makes testable predictions is described.
Best practices for convolutional neural networks applied to visual document analysis
2,815 Citations2005P. Simard, Don Steinkraus +1 more
A set of concrete bestpractices that document analysis researchers can use to get good results with neural networks, including a simple "do-it-yourself" implementation of convolution with a flexible architecture suitable for many visual document problems.
An analysis of single-layer networks in unsupervised feature learning
2,457 Citations2011Adam Coates, Andrew Y. Ng +1 more
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Lecture notes in computer scienceEvaluation of Pooling Operations in Convolutional Architectures for Object Recognition
1,518 Citations2010Dominik Scherer, Andreas Müller +1 more
The aim is to gain insight into different functions by directly comparing them on a fixed architecture for several common object recognition tasks, and empirical results show that a maximum pooling operation significantly outperforms subsampling operations.
Learning methods for generic object recognition with invariance to pose and lighting
1,332 Citations2004Yann LeCun, Fu Jie Huang +1 more
A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second and proved impractical, while convolutional nets yielded 16/7% error.
A Theoretical Analysis of Feature Pooling in Visual Recognition
1,148 Citations2010Y-Lan Boureau, Jean Ponce +1 more
It is shown that the reasons underlying the performance of various pooling methods are obscured by several confounding factors, such as the link between the sample cardinality in a spatial pool and the resolution at which low-level features have been extracted.
Neural ComputationDeep, Big, Simple Neural Nets for Handwritten Digit Recognition
1,064 Citations2010Dan Claudiu Cireşan, Ueli Meier +2 more
Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark.
Object Recognition with Features Inspired by Visual Cortex
828 Citations2005T. Serre, Lior Wolf +1 more
The performance of the approach constitutes a suggestive plausibility proof for a class of feedforward models of object recognition in cortex and exhibits excellent recognition performance and outperforms several state-of-the-art systems on a variety of image datasets including many different object categories.
High Performance Convolutional Neural Networks for Document Processing
431 Citations2006Kumar Chellapilla, Sidd Puri +1 more
Three novel approaches to speeding up CNNs are presented: a) unrolling convolution, b) using BLAS (basic linear algebra subroutines), and c) using GPUs (graphic processing units).
International Journal of Computer VisionObject Class Recognition and Localization Using Sparse Features with Limited Receptive Fields
377 Citations2008Jim Mutch, David Lowe
A committee of neural networks for traffic sign classification
375 Citations2011Dan Cireşan, Ueli Meier +2 more
This work describes the approach that won the preliminary phase of the German traffic sign recognition benchmark with a better-than-human recognition rate, and obtains an even better recognition rate by further training the nets.
3D Object Recognition with Deep Belief Nets
304 Citations2009Vinod Nair, Geoffrey E. Hinton
A new type of top-level model for Deep Belief Nets is introduced, a third-order Boltzmann machine, trained using a hybrid algorithm that combines both generative and discriminative gradients that substantially outperforms shallow models such as SVMs.
PLoS Computational BiologyA High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation
254 Citations2009Nicolas Pinto, David Doukhan +2 more
Performance and Scalability of GPU-Based Convolutional Neural Networks
242 Citations2010Daniel Strigl, Klaus Kofler +1 more
This paper presents the implementation of a framework for accelerating training and classification of arbitrary Convolutional Neural Networks (CNNs) on the GPU and describes the basic parts of a CNN and demonstrates the performance and scalability improvement that can be achieved by shifting the computation-intensive tasks of aCNN to the GPU.
Network Computation in Neural SystemsIndependent component analysis applied to feature extraction from colour and stereo images
234 Citations2000Patrik O. Hoyer, Aapo Hyvärinen
The similarities between these results and the observed properties of simple cells in the primary visual cortex are further evidence for the hypothesis that visual cortical neurons perform some type of redundancy reduction, which was one of the original motivations for ICA in the first place.
Lecture notes in computer scienceHierarchical Neural Networks for Image Interpretation
222 Citations2003Sven Behnke
The results show clear trends in the direction of improvement in the level of supervised learning in relation to the recognition of meter values and in the application of Matrix Codes.
Improved Local Coordinate Coding using Local Tangents
148 Citations2010Kai Yu, Tong Zhang
This paper further develops the idea of integrating geometry in machine learning by extending the original LCC method to include local tangent directions to lead to better approximation of high dimensional nonlinear functions when the underlying data manifold is locally relatively flat.
NeurocomputingNeocognitron for handwritten digit recognition
136 Citations2003Kunihiko Fukushima
This paper proposes an improved version of the neocognitron and tests its ability using a large database of handwritten digits (ETL1) to improve the recognition rate and removes accessory circuits that were appended to the previous versions.
Large-scale object recognition with CUDA-accelerated hierarchical neural networks
81 Citations2009Rafael Uetz, Sven Behnke
This work presents a hierarchical, locally-connected neural network model that is well-suited for large-scale, high-performance object recognition and creates a massively parallel implementation of the model which is executed on a state-of-the-art graphics card.
Better Digit Recognition with a Committee of Simple Neural Nets
69 Citations2011Ueli Meier, Dan Claudiu Cireşan +2 more
A new method to train the members of a committee of one-hidden-layer neural nets is presented, which obtains a recognition error rate on the MNIST digit recognition benchmark set of 0.39%, on par with state-of-the-art recognition rates of more complicated systems.
Neural ComputationSemilinear Predictability Minimization Produces Well-Known Feature Detectors
48 Citations1996Jürgen Schmidhuber, Martin Eldracher +1 more
This paper applies semilinear PM to static real world images and finds that without a teacher and without any significant preprocessing, the system automatically learns to generate distributed representations based on well-known feature detectors, thus extracting simple features related to those considered useful for image preprocessing and compression.