Compete to Compute
Neural Information Processing SystemsPublished 5 December 2013
Rupesh K. Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, Jürgen Schmidhuber
Citations134
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper applies the concept of local competition among neighboring neurons to gradient-based, backprop-trained artificial multilayer NNs, finding that NNs with competing linear units tend to outperform those with non-competing nonlinear units, and avoid catastrophic forgetting when training sets change over time.
Abstract
Local competition among neighboring neurons is common in biological neural networks (NNs). In this paper, we apply the concept to gradient-based, backprop-trained artificial multilayer NNs. NNs with competing linear units tend to outperform those with non-competing nonlinear units, and avoid catastrophic forgetting when training sets change over time.
Keywords
Computer Science
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
International Conference on Machine LearningRectified Linear Units Improve Restricted Boltzmann Machines
13,244 Citations2010Vinod Nair, Geoffrey E. Hinton
Restricted Boltzmann machines were developed using binary stochastic hidden units that learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.
Biological CyberneticsSelf-organized formation of topologically correct feature maps
9,519 Citations1982Teuvo Kohonen
In a simple network of adaptive physical elements which receives signals from a primary event space, the signal representations are automatically mapped onto a set of output responses in such a way that the responses acquire the same topological order as that of the primary events.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
Deep Sparse Rectifier Neural Networks
5,408 Citations2012Xavier Glorot, Antoine Bordes +1 more
This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-dierentiabil ity.
Multi-column deep neural networks for image classification
3,764 Citations2012Dan Cireşan, Ueli Meier +1 more
On the very competitive MNIST handwriting benchmark, this method is the first to achieve near-human performance and improves the state-of-the-art on a plethora of common image classification benchmarks.
The Psychology of learning and motivation/The psychology of learning and motivationCatastrophic Interference in Connectionist Networks: The Sequential Learning Problem
3,578 Citations1989Michael McCloskey, Neal J. Cohen
This chapter discusses the catastrophic interference in connectionist networks, and the simulation results demonstrate only that interference is catastrophic in some specific networks.
Nature NeuroscienceHierarchical models of object recognition in cortex
3,404 Citations1999Maximilian Riesenhuber, Tomaso Poggio
A new hierarchical model consistent with physiological data from inferotemporal cortex that accounts for this complex visual task and makes testable predictions is described.
The Cerebellum as a Neuronal Machine
3,234 Citations1967John C. Eccles, Masao Ito +1 more
Best practices for convolutional neural networks applied to visual document analysis
2,815 Citations2005P. Simard, Don Steinkraus +1 more
A set of concrete bestpractices that document analysis researchers can use to get good results with neural networks, including a simple "do-it-yourself" implementation of convolution with a flexible architecture suitable for many visual document problems.
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification
2,026 Citations2007John Blitzer, Mark Dredze +1 more
This work extends to sentiment classification the recently-proposed structural correspondence learning (SCL) algorithm, reducing the relative error due to adaptation between domains by an average of 30% over the original SCL algorithm and 46% over a supervised baseline.
Domain adaptation for large-scale sentiment classification: A deep learning approach
1,563 Citations2011Xavier Glorot, Antoine Bordes +1 more
A deep learning approach is proposed which learns to extract a meaningful representation for each review in an unsupervised fashion and clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products.
KybernetikSelf-organization of orientation sensitive cells in the striate cortex
1,481 Citations1973Chr. von der Malsburg
A nerve net model for the visual cortex of higher vertebrates is presented and a simple learning procedure is shown to be sufficient for the organization of some essential functional properties of single units.
ComputerThe ART of adaptive pattern recognition by a self-organizing neural network
1,290 Citations1988Gail A. Carpenter, Stephen Grossberg
Improving deep neural networks for LVCSR using rectified linear units and dropout
1,270 Citations2013George E. Dahl, Tara N. Sainath +1 more
Modelling deep neural networks with rectified linear unit (ReLU) non-linearities with minimal human hyper-parameter tuning on a 50-hour English Broadcast News task shows an 4.2% relative improvement over a DNN trained with sigmoid units, and a 14.4% relative improved over a strong GMM/HMM system.
The MIT Press eBooksEfficient Learning of Sparse Representations with an Energy-Based Model
1,079 Citations2007Marc’Aurelio Ranzato, Christopher S. Poultney +2 more
A novel unsupervised method for learning sparse, overcomplete features using a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector.
Boston studies in the philosophy of scienceContour Enhancement, Short Term Memory, and Constancies in Reverberating Neural Networks
650 Citations1982Stephen Grossberg
It is suggested that competition solves a sensitivity problem that confronts all cellular systems: the noise-saturation dilemma.
Maxout Networks
607 Citations2013Ian Goodfellow, David Warde-Farley +3 more
A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique.
Nature NeuroscienceAttention activates winner-take-all competition among visual filters
463 Citations1999D. Kathleen Lee, L. Itti +2 more
This model predicts that the effects of attention on visual cortical neurons include increased contrast gain as well as sharper tuning to orientation and spatial frequency.
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
428 Citations2013Matthew D. Zeiler, Rob Fergus
Neural ComputationOn the Computational Power of Winner-Take-All
369 Citations2000Wolfgang Maass
The theoretical analysis shows that winner- take-all is a surprisingly powerful computational module in comparison with threshold gates (also referred to as McCulloch-Pitts neurons) and sigmoidal gates, and proves an optimal quadratic lower bound for computing winner-takeall in any feedforward circuit consisting of threshold gates.
Biological CyberneticsPattern formation, contrast control, and oscillations in the short term memory of shunting on-center off-surround networks
277 Citations1975Samuel A. Ellias, Stephen Grossberg
The transformation of spatial patterns and their storage in short term memory by shunting neural networks are studied herein and various mechanisms are described for real-time regulation of the amount of contrast with which a pattern will be stored.
Computational Maps in the Visual Cortex
238 Citations2005Risto Miikkulainen, James A. Bednar +2 more
This paper describes the development of Maps and Connections and the construction of LISSOM, a Computational Map Model of V1, and the role of plasticity, Hierarchical Model in this development.
Neural NetworksComplex dynamics in winner-take-all neural nets with slow inhibition
94 Citations1992Bard Ermentrout
A layer of excitatory neurons with small asymmetric excitation connections and strong coupling to a single inhibitory interneuron is considered and if the inhibition is sufficiently slower than excitation the neurons completely synchronize to a global periodic solution.
Connection ScienceA Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks
88 Citations1989Jürgen Schmidhuber
This paper proposes a parallel on-line learning algorithms which performs local computations only, yet still is designed to deal with hidden units and with units whose past activations are ‘hidden in time’.
Neural ComputationModeling Selective Attention Using a Neuromorphic Analog VLSI Device
85 Citations2000Giacomo Indiveri
A hardware model of a selective attention mechanism implemented on a very large-scale integration (VLSI) chip, using analog neuromorphic circuits, that exploits a spike-based representation to receive, process, and transmit signals.
CUDAMat: a CUDA-based matrix class for Python
67 Citations2009Volodymyr Mnih
The feature set of CUDAMat is biased towards providing functionality useful for implementing standard machine learning algorithms, however, it is general enough to be useful in other elds.
Neural Computation with Winner-Take-All as the Only Nonlinear Operation
48 Citations1999Wolfgang Maass
Neural NetworksFirst experiments with PowerPlay
39 Citations2013Rupesh K. Srivastava, Bas R. Steunebrink +1 more
This work describes how the POWERPLAY-driven SLIM RNN automatically self-modularizes, frequently re-using code for previously invented skills, always trying to invent novel tasks that can be quickly validated because they do not require too many weight changes affecting too many previous tasks.
Spiking Inputs to a Winner-take-all Network
22 Citations2005Matthias Oster, Shih‐Chii Liu
This work shows how to configure the connectivity in the network so that the winner is selected after a pre-determined number of input spikes, and discusses spiking inputs with both regular frequencies and Poisson-distributed rates.