Learning Stochastic Recurrent Networks
arXiv (Cornell University)Published 27 November 2014Open access
Justin Bayer, Christian Osendorfer
Citations198
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a reliable estimator of the marginal likelihood and iv) is a generalisation of deterministic recurrent neural networks. We evaluate the method on four polyphonic musical data sets and motion capture data.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Random search for hyper-parameter optimization
7,937 Citations2012James Bergstra, Yoshua Bengio
arXiv (Cornell University)ADADELTA: An Adaptive Learning Rate Method
5,531 Citations2012Matthew D. Zeiler
A novel per-dimension learning rate method for gradient descent called ADADELTA that dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent is presented.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
3,526 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
On the importance of initialization and momentum in deep learning
3,526 Citations2013Ilya Sutskever, James Martens +2 more
It is shown that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs to levels of performance that were previously achievable only with Hessian-Free optimization.
arXiv (Cornell University)Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
3,282 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)LOL: An Investigation into Cybernetic Humor, or: Can Machines Laugh?
3,084 Citations2016Alex Graves, Gervasi, Vincenzo +1 more
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time.
arXiv (Cornell University)Stochastic Backpropagation and Approximate Inference in Deep Generative Models
2,644 Citations2014Danilo Jimenez Rezende, Shakir Mohamed +1 more
The MIT Press eBooksModeling Human Motion Using Binary Latent Variables
702 Citations2007Graham W. Taylor, Geoffrey E. Hinton +1 more
A non-linear generative model for human motion data that uses an undirected model with binary latent variables and real-valued "visible" variables that represent joint angles that makes on-line inference efficient and allows for a simple approximate learning procedure.
Learning Recurrent Neural Networks with Hessian-Free Optimization
534 Citations2011James Martens, Ilya Sutskever
This work solves the long-outstanding problem of how to effectively train recurrent neural networks on complex and difficult sequence modeling problems which may contain long-term data dependencies and offers a new interpretation of the generalized Gauss-Newton matrix of Schraudolph which is used within the HF approach of Martens.
arXiv (Cornell University)Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
490 Citations2012Nicolas Boulanger-Lewandowski, Yoshua Bengio +1 more
Exponential Family Harmoniums with an Application to Information Retrieval
436 Citations2004Max Welling, Michal Rosen‐Zvi +1 more
An alternative two-layer model based on exponential family distributions and the semantics of undirected models is proposed, which performs well on document retrieval tasks and provides an elegant solution to searching with keywords.
Applied Mathematics LettersTuring computability with neural nets
385 Citations1991Hava T. Siegelmann, Eduardo D. Sontag
The existence of a finite neural network, made up of sigmoidal neurons, which simulates a universal Turing machine, composed of less than 10 5 synchronously evolving processors, interconnected linearly is shown.
The Recurrent Temporal Restricted Boltzmann Machine
373 Citations2008Ilya Sutskever, Geoffrey E. Hinton +1 more
The Recurrent TRBM is introduced, which is a very slight modification of the TRBM for which exact inference is very easy and exact gradient learning is almost tractable.
ACM Transactions on GraphicsStyle translation for human motion
299 Citations2005Eugene Hsu, Kari Pulli +1 more
arXiv (Cornell University)How to Construct Deep Recurrent Neural Networks
245 Citations2013Razvan Pascanu, Çaǧlar Gülçehre +2 more
Bern Open Repository and Information System (University of Bern)Unconstrained On-line Handwriting Recognition with Recurrent Neural Networks
219 Citations2007Alex Graves, Marcus Liwicki +3 more
A system capable of directly transcribing raw online handwriting data is described, consisting of an advanced recurrent neural network with an output layer designed for sequence labelling, combined with a probabilistic language model.
High-dimensional sequence transduction
96 Citations2013Nicolas Boulanger-Lewandowski, Yoshua Bengio +1 more
A probabilistic model based on a recurrent neural network that is able to learn realistic output distributions given the input is introduced and an efficient algorithm to search for the global mode of that distribution is devised.
arXiv (Cornell University)On Fast Dropout and its Applicability to Recurrent Networks
45 Citations2013Justin Bayer, Christian Osendorfer +4 more
arXiv (Cornell University)Speech Recognition with Deep Recurrent Neural Networks
10 Citations2013Alex Graves, Abdelrahman Mohamed +1 more
Lecture notes in computer scienceTraining Neural Networks with Implicit Variance
5 Citations2013Justin Bayer, Christian Osendorfer +2 more
The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of likelihood.