Rectified Linear Units Improve Restricted Boltzmann Machines
International Conference on Machine LearningPublished 21 June 2010
Vinod Nair, Geoffrey E. Hinton
Citations13,244
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Restricted Boltzmann machines were developed using binary stochastic hidden units that learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.
Abstract
Restricted Boltzmann machines were developed using binary stochastic hidden units. These can be generalized by replacing each binary unit by an infinite number of copies that all have the same weights but have progressively more negative biases. The learning and inference rules for these Stepped Sigmoid Units are unchanged. They can be approximated efficiently by noisy, rectified linear units. Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset. Unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors.
Keywords
Computer Science
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
HAL (Le Centre pour la Communication Scientifique Directe)Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
4,503 Citations2008Gary B. Huang, Marwan Mattar +2 more
The database contains labeled face photographs spanning the range of conditions typically encountered in everyday life, and exhibits “natural” variability in factors such as pose, lighting, race, accessories, occlusions, and background.
Learning a Similarity Metric Discriminatively, with Application to Face Verification
3,870 Citations2005Sumit Chopra, Raia Hadsell +1 more
The idea is to learn a function that maps input patterns into a target space such that the L/sub 1/ norm in the target space approximates the "semantic" distance in the input space.
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Restricted Boltzmann machines for collaborative filtering
1,882 Citations2007Ruslan Salakhutdinov, Andriy Mnih +1 more
This paper shows how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM's), can be used to model tabular data, such as user's ratings of movies, and demonstrates that RBM's can be successfully applied to the Netflix data set.
Attribute and simile classifiers for face verification
1,382 Citations2009Neeraj Kumar, Alexander C. Berg +2 more
Two novel methods for face verification using binary classifiers trained to recognize the presence or absence of describable aspects of visual appearance and a new data set of real-world images of public figures acquired from the internet.
Learning methods for generic object recognition with invariance to pose and lighting
1,332 Citations2004Yann LeCun, Fu Jie Huang +1 more
A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second and proved impractical, while convolutional nets yielded 16/7% error.
An empirical evaluation of deep architectures on problems with many factors of variation
1,030 Citations2007Hugo Larochelle, Dumitru Erhan +3 more
A series of experiments indicate that these models with deep architectures show promise in solving harder learning problems that exhibit many factors of variation.
Scaling learning algorithms towards AI
927 Citations2007Yoshua Bengio, Yann LeCun
It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.
The MIT Press eBooksModeling Human Motion Using Binary Latent Variables
702 Citations2007Graham W. Taylor, Geoffrey E. Hinton +1 more
A non-linear generative model for human motion data that uses an undirected model with binary latent variables and real-valued "visible" variables that represent joint angles that makes on-line inference efficient and allows for a simple approximate learning procedure.
Neural Information Processing SystemsReplicated Softmax: an Undirected Topic Model
440 Citations2009Geoffrey E. Hinton, Ruslan Salakhutdinov
This work introduces a two-layer undirected graphical model, called a "Replicated Softmax", that can be used to model and automatically extract low-dimensional latent semantic representations from a large unstructured collection of documents.
Lecture notes in computer scienceSimilarity Scores Based on Background Samples
282 Citations2010Lior Wolf, Tal Hassner +1 more
“background samples”, that is, examples which do not belong to any of the classes being learned, may provide a significant performance boost to such face recognition systems, and is defined and evaluated as an extension to the recently proposed “One-Shot Similarity” (OSS) measure.
Unsupervised learning of distributions on binary vectors using two layer networks
274 Citations1991Yoav Freund, David Haussler
It is shown that arbitrary distributions of binary vectors can be approximated by the combination model and shown how the weight vectors in the model can be interpreted as high order correlation patterns among the input bits, and how the combination machine can be used as a mechanism for detecting these patterns.
Neural ComputationPermitted and Forbidden Sets in Symmetric Threshold-Linear Networks
211 Citations2003Richard H. R. Hahnloser, H. Sebastian Seung +1 more
By viewing permitted sets as memories stored in the synaptic connections, this work provides a formulation of long-term memory that is more general than the traditional perspective of fixed-point attractor networks.
Rate-coded Restricted Boltzmann Machines for Face Recognition
135 Citations2000Yee Whye Teh, Geoffrey E. Hinton
A neurally-inspired, unsupervised learning algorithm that builds a non-linear generative model for pairs of face images from the same individual and individuals are then recognized by finding the highest relative probability pair among all pairs that consist of a test image and an image whose identity is known.
Phone recognition using Restricted Boltzmann Machines
108 Citations2010Abdelrahman Mohamed, Geoffrey E. Hinton
Conditional Restricted Boltzmann Machines (CRBMs) have recently proved to be very effective for modeling motion capture sequences and this paper investigates the application of this more powerful type of generative model to acoustic modeling.
Implicit Mixtures of Restricted Boltzmann Machines
76 Citations2008Vinod Nair, Geoffrey E. Hinton
Results for the MNIST and NORB datasets are presented showing that the implicit mixture of RBMs learns clusters that reflect the class structure in the data.
A Hierarchical Community of Experts
26 Citations1998Geoffrey E. Hinton, Brian Sallans +1 more
It is shown that Gibbs sampling can be used to learn the parameters of the linear and binary units even when the sampling is so brief that the Markov chain is far from equilibrium.
Diffusion Networks, Products of Experts, and Factor Analysis
25 Citations2001Tim K. Marks, Javier R. Movellan
It is shown that when the unit activation functions are linear, this PoE architecture is equivalent to a factor analyzer, which suggests novel non-linear generalizations of factor analysis and independent component analysis that could be implemented using interactive neural circuitry.