Maxout Networks
Published 18 February 2013
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio
Citations451
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout. We define a simple new model called maxout (so named because its output is the max of a set of inputs, and because it is a natural companion to dropout) designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. We empirically verify that the model successfully accomplishes both of these tasks. We use maxout and dropout to demonstrate state of the art classification performance on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.
Keywords
Computer Science
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
arXiv (Cornell University)Practical Bayesian Optimization of Machine Learning Algorithms
5,612 Citations2012Jasper Snoek, Hugo Larochelle +1 more
This work describes new algorithms that take into account the variable cost of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation and shows that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms.
Deep Sparse Rectifier Neural Networks
5,408 Citations2012Xavier Glorot, Antoine Bordes +1 more
This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-dierentiabil ity.
Reading digits in natural images with unsupervised feature learning
4,546 Citations2024Yuval Netzer
A new benchmark dataset for research use is introduced containing over 600,000 labeled digits cropped from Street View images, and variants of two recently proposed unsupervised feature learning methods are employed, finding that they are convincingly superior on benchmarks.
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Deep Boltzmann machines
1,778 Citations2009Ruslan Salakhutdinov, Geoffrey E. Hinton
A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.
Neural ComputationDeep, Big, Simple Neural Nets for Handwritten Digit Recognition
1,064 Citations2010Dan Claudiu Cireşan, Ueli Meier +2 more
Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark.
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
428 Citations2013Matthew D. Zeiler, Rob Fergus
Proceedings of the National Academy of SciencesA model of multiplicative neural responses in parietal cortex.
337 Citations1996Emilio Salinas, L. F. Abbott
It is shown that multiplicative responses can arise in a network model through population effects and suggest that parietal responses may be based on this architecture.
Lecture notes in computer scienceRank, Trace-Norm and Max-Norm
332 Citations2005Nathan Srebro, Adi Shraibman
This work studies the rank, trace-norm and max-norm as complexity measures of matrices, focusing on the problem of fitting a matrix with matrices having low complexity, and presents generalization error bounds for predicting unobserved entries that are based on these measures.
Improving Neural Networks with Dropout
270 Citations2013Nitish Srivastava
In this work, models that improve the performance of neural networks using dropout are described, often obtaining state-of-the-art results on benchmark datasets.
The Manifold Tangent Classifier
157 Citations2011Salah Rifai, Yann Dauphin +3 more
A representation learning algorithm can be stacked to yield a deep architecture and it is shown how it builds a topological atlas of charts, each chart being characterized by the principal singular vectors of the Jacobian of a representation mapping.
Deep convex net: a scalable architecture for speech pattern classification
156 Citations2011Li Deng, Dong Yu
Results on both MNIST and TIMIT tasks evaluated thus far demonstrate superior performance of DCN over the DBN (Deep Belief Network) counterpart that forms the basis of the DNN, reflected not only in training scalability and CPU-only computation, but more importantly in classification accuracy in both tasks.
Neural NetworksOn the piecewise analysis of networks of linear threshold neurons
129 Citations1998Richard H. R. Hahnloser
The study of oscillations of random networks suggests that all cyclic trajectories of linear threshold networks are a result of the existence of partitions with undamped linear oscillations.
IEEE Transactions on Circuits and Systems I Fundamental Theory and ApplicationsGeneral Constructive Representations for Continuous Piecewise-Linear Functions
38 Citations2004Shaobin Wang
This paper first transforms the lattice PWL representation into the difference of two convex functions, then proposes a constructive procedure to rewrite the latter as a canonical representation that consists of at most n-level nestings of absolute-value functions in n dimensions, hence giving a thorough solution to the problem mentioned above.
MPG.PuRe (Max Planck Society)Learnable Pooling Regions for Image Classification
20 Citations2013Mateusz Malinowski, Mario Fritz