Training Neural Networks with Implicit Variance
Lecture notes in computer sciencePublished 1 January 2013Open access
Justin Bayer, Christian Osendorfer, Sebastian Urban, Patrick van der Smagt
Citations5
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of likelihood.
Abstract
We present a novel method to train predictive Gaussian distributions p(z|x) for regression problems with neural networks. While most approaches either ignore or explicitly model the variance as another response variable, it is trained implicitly in our case. Establishing stochasticty by the injection of noise into the input and hidden units, the outputs are approximated with a Gaussian distribution by the forward propagation method introduced for fast dropout [1]. We have designed our method to respect that probabilistic interpretation of the output units in the loss function. The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of likelihood.
Keywords
Computer Science
NatureLearning representations by back-propagating errors
30,885 Citations1986David E. Rumelhart, Geoffrey E. Hinton +1 more
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
NeurocomputingAdvances in neural information processing systems 7
22,300 Citations1997Krzysztof J. Cios, Mark E. Shields
Journal of Electronic ImagingPattern Recognition and Machine Learning
21,976 Citations2007Christopher Bishop
Probability Distributions, linear models for Regression, Linear Models for Classification, Neural Networks, Graphical Models, Mixture Models and EM, Sampling Methods, Continuous Latent Variables, Sequential Data are studied.
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
The MIT Press eBooksGaussian Processes for Machine Learning
10,409 Citations2005Carl Edward Rasmussen, Christopher K. I. Williams
The book provides a long-needed, systematic and unified treatment of theoretical and practical aspects of GPs in machine learning, targeted at researchers and students in machine learning and applied statistics.
Random search for hyper-parameter optimization
7,937 Citations2012James Bergstra, Yoshua Bengio
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIENew extension of the Kalman filter to nonlinear systems
5,171 Citations1997Simon Julier, Jeffrey Uhlmann
It is argued that the ease of implementation and more accurate estimation features of the new filter recommend its use over the EKF in virtually all applications.
Multi-column deep neural networks for image classification
3,764 Citations2012Dan Cireşan, Ueli Meier +1 more
On the very competitive MNIST handwriting benchmark, this method is the first to achieve near-human performance and improves the state-of-the-art on a plethora of common image classification benchmarks.
On the importance of initialization and momentum in deep learning
3,526 Citations2013Ilya Sutskever, James Martens +2 more
It is shown that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs to levels of performance that were previously achievable only with Hessian-Free optimization.
IEEE Transactions on Audio Speech and Language ProcessingContext-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition
3,052 Citations2011George E. Dahl, Dong Yu +2 more
A pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output that can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs.
Lecture notes in computer scienceNeural Networks: Tricks of the Trade
2,146 Citations2012Montavon, Grégoire, Müller, Klaus-Robert +1 more
The second edition of the book augments the first edition with more tricks, which have resulted from 14 years of theory and experimentation by some of the world's most prominent neural network researchers.
Aston Publications Explorer (Aston University)Mixture density networks
1,269 Citations1994Chris Bishop
This paper introduces a new class of network models obtained by combining a conventional neural network with a mixture density model, called a Mixture Density Network, which can in principle represent arbitrary conditional probability distributions in the same way that aventional neural network can represent arbitrary functions.
An empirical evaluation of deep architectures on problems with many factors of variation
1,030 Citations2007Hugo Larochelle, Dumitru Erhan +3 more
A series of experiments indicate that these models with deep architectures show promise in solving harder learning problems that exhibit many factors of variation.
Lecture notes in computer scienceEfficient BackProp
867 Citations1998Yann LeCun, Léon Bottou +2 more
Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications, and this paper gives some of those tricks, and offers explanations of why they work.
Artificial IntelligenceConnectionist learning of belief networks
532 Citations1992Radford M. Neal
The “Gibbs sampling” simulation procedure for “sigmoid” and “noisy-OR” varieties of probabilistic belief networks can support maximum-likelihood learning from empirical data through local gradient ascent.
Neural ComputationIncremental Online Learning in High Dimensions
529 Citations2005Sethu Vijayakumar, Aaron D’Souza +1 more
Locally weighted projection regression is the first truly incremental spatially localized learning method that can successfully and efficiently operate in very high-dimensional spaces.
On rectified linear units for speech processing
418 Citations2013Matthew D. Zeiler, M. Ranzato +9 more
This work shows that it can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units.
Training recurrent neural networks
391 Citations2013Geoffrey E. Hinton, Ilya Sutskever
A new probabilistic sequence model that combines Restricted Boltzmann Machines and RNNs is described, more powerful than similar models while being less difficult to train, and a random parameter initialization scheme is described that allows gradient descent with momentum to train Rnns on problems with long-term dependencies.
Disordered Systems and Biological Organization
315 Citations1986Elie Bienenstock, Françoise Fogelman Soulié +1 more
The present introduction outlInes the relationships between the contributions presented at the NATO workshop on Disordered Systems and Biological Organization and discusses each paper in its particular scientific context.
International Conference on Machine LearningFast dropout training
273 Citations2013Sida Wang, Christopher D. Manning
The results show that [cyclo-Leu-His-Thr]∼LCP-3 has potent anti-inflammatory potency in RAW264.7 and HMC-1 cells under LPS (lipopolysaccharide) stimuli and mechanistic studies show that LCP-3 attenuates pro-inflammatory cytokine expression.
arXiv (Cornell University)Deep Generative Stochastic Networks Trainable by Backprop
238 Citations2013Yoshua Bengio, Eric Laufer +2 more
Theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders are provided and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood.
Neural Information Processing SystemsUsing Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes
188 Citations2007Geoffrey E. Hinton, Ruslan Salakhutdinov
This work shows how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process.
Heteroscedastic Gaussian process regression
175 Citations2005Quoc V. Le, Alex Smola +1 more
An algorithm to estimate simultaneously both mean and variance of a non parametric regression problem which can be solved via Newton's method is presented and is able to estimate variance locally unlike standard Gaussian Process regression or SVMs.
Learning Process in an Asymmetric Threshold Network
133 Citations1986Y. Le Cun
A learning procedure which requires the outside world to specify the state of every neuron during the learning session can hardly be considered as a general learning rule because in real-world conditions, only a partial information on the “ideal” network state for each task is available from the environment.
arXiv (Cornell University)Deep Generative Stochastic Networks Trainable by Backprop
108 Citations2013Yoshua Bengio, Éric Thibodeau-Laufer +2 more
Learning Stochastic Feedforward Neural Networks
100 Citations2013Yichuan Tang, Ruslan Salakhutdinov
A stochastic feedforward network with hidden layers composed of both deterministic and stochastics variables is proposed that achieves superior performance on synthetic and facial expressions datasets compared to conditional Restricted Boltzmann Machines and Mixture Density Networks.
arXiv (Cornell University)Estimating or Propagating Gradients Through Stochastic Neurons
81 Citations2013Yoshua Bengio
It is demonstrated that a simple biologically plausible formula gives rise to an an unbiased (but noisy) estimator of the gradient with respect to a binary stochastic neuron firing proba bility, and an approach to approximating this unbiased but high-variance estimator by learning to predict it using a biased estimator.
Deep architectures for articulatory inversion
69 Citations2012Benigno Uría, Iain Murray +2 more
A deep neural network and a deep trajectory mixture density network are implemented for the acousticarticulatory inversion mapping problem and it is found that in both cases, deep architectures produce more accurate predictions than shallow architectures.