Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network
Physica D Nonlinear PhenomenaPublished 21 January 2020Open access
A. Sherstinsky
Citations5,418
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A Machine Learning practitioner seeking guidance for implementing the new augmented LSTM model in software for experimentation and research will find the insights and derivations in this treatise valuable as well.
Abstract
Because of their effectiveness in broad practical applications, LSTM networks\nhave received a wealth of coverage in scientific journals, technical blogs, and\nimplementation guides. However, in most articles, the inference formulas for\nthe LSTM network and its parent, RNN, are stated axiomatically, while the\ntraining formulas are omitted altogether. In addition, the technique of\n"unrolling" an RNN is routinely presented without justification throughout the\nliterature. The goal of this paper is to explain the essential RNN and LSTM\nfundamentals in a single document. Drawing from concepts in signal processing,\nwe formally derive the canonical RNN formulation from differential equations.\nWe then propose and prove a precise statement, which yields the RNN unrolling\ntechnique. We also review the difficulties with training the standard RNN and\naddress them by transforming the RNN into the "Vanilla LSTM" network through a\nseries of logical arguments. We provide all equations pertaining to the LSTM\nsystem together with detailed descriptions of its constituent entities. Albeit\nunconventional, our choice of notation and the method for presenting the LSTM\nsystem emphasizes ease of understanding. As part of the analysis, we identify\nnew opportunities to enrich the LSTM system and incorporate these extensions\ninto the Vanilla LSTM network, producing the most general LSTM variant to date.\nThe target reader has already been exposed to RNNs and LSTM networks through\nnumerous available resources and is open to an alternative pedagogical\napproach. A Machine Learning practitioner seeking guidance for implementing our\nnew augmented LSTM model in software for experimentation and research will find\nthe insights and derivations in this tutorial valuable as well.\n
Keywords
Computer SciencePhysics and Astronomy
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
The Journal of Chemical PhysicsEquation of State Calculations by Fast Computing Machines
37,074 Citations1953N. Metropolis, Arianna W. Rosenbluth +3 more
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
The MIT Press eBooksParallel Distributed Processing
15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
Discrete-Time Signal Processing
11,537 Citations1989Alan V. Oppenheim, Ronald W. Schafer
The definitive, authoritative text on DSP, written by prominent, DSP pioneers, it provides thorough treatment of the fundamental theorems and properties of discrete- time linear systems, filtering, sampling, and discrete-time Fourier Analysis.
Cognitive ScienceFinding Structure in Time
10,841 Citations1990Jeffrey L. Elman
A proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory and suggests a method for representing lexical categories and the type/token distinction is developed.
IEEE Transactions on Signal ProcessingBidirectional recurrent neural networks
10,022 Citations1997Mike Schuster, Kuldip K. Paliwal
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
Proceedings of the National Academy of SciencesNeurons with graded response have collective computational properties like those of two-state neurons.
6,955 Citations1984J. J. Hopfield
A model for a large network of "neurons" with a graded response (or sigmoid input-output relation) is studied and collective properties in very close correspondence with the earlier stochastic model based on McCulloch - Pitts neurons are studied.
IEEE Transactions on Neural Networks and Learning SystemsLSTM: A Search Space Odyssey
6,837 Citations2016Klaus Greff, Rupesh K. Srivastava +3 more
This paper presents the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling, and observes that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.
Neural NetworksFramewise phoneme classification with bidirectional LSTM and other neural network architectures
5,463 Citations2005Alex Graves, Jürgen Schmidhuber
The main findings are that bidirectional networks outperform unidirectional ones, and Long Short Term Memory (LSTM) is much faster and also more accurate than both standard Recurrent Neural Nets (RNNs) and time-windowed Multilayer Perceptrons (MLPs).
Physica D Nonlinear PhenomenaFundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network
5,418 Citations2020A. Sherstinsky
A Machine Learning practitioner seeking guidance for implementing the new augmented LSTM model in software for experimentation and research will find the insights and derivations in this treatise valuable as well.
IEEE Transactions on Circuits and SystemsCellular neural networks: theory
4,746 Citations1988Leon O. Chua, L. Yang
Neural ComputationA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
4,451 Citations1989Ronald J. Williams, David Zipser
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
3,778 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
Studies in computational intelligenceSupervised Sequence Labelling with Recurrent Neural Networks
3,084 Citations2012Alex Graves
A new type of output layer that allows recurrent networks to be trained directly for sequence labelling tasks where the alignment between the inputs and the labels is unknown, and an extension of the long short-term memory network architecture to multidimensional data, such as images and video sequences.
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)LOL: An Investigation into Cybernetic Humor, or: Can Machines Laugh?
3,084 Citations2016Alex Graves, Gervasi, Vincenzo +1 more
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time.
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
2,924 Citations2014Haşim Sak, Andrew Senior +1 more
The first distributed training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machines is introduced and it is shown that a two-layer deep LSTm RNN where each L STM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance.
Parallel & distributed processing
2,601 Citations2005Philipp Slusallek, Peter Shirley +3 more
IEEE Transactions on Circuits and SystemsCellular neural networks: applications
2,383 Citations1988Leon O. Chua, L. Yang
Examples of cellular neural networks which can be designed to recognize the key features of Chinese characters are presented and their applications to such areas as image processing and pattern recognition are demonstrated.
arXiv (Cornell University)Recurrent Neural Network Regularization
2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more
This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
arXiv (Cornell University)A Critical Review of Recurrent Neural Networks for Sequence Learning
2,085 Citations2015Zachary C. Lipton, John Berkowitz +1 more
The goal of this survey is to provide a selfcontained explication of the state of the art of recurrent neural networks together with a historical perspective and references to primary research.
LSTM neural networks for language modeling
1,990 Citations2012Martin Sundermeyer, Ralf Schlüter +1 more
This work analyzes the Long Short-Term Memory neural network architecture on an English and a large French language modeling task and gains considerable improvements in WER on top of a state-of-the-art speech recognition system.
Equation of state calculations by fast computing machines
1,823 Citations1953N. Metropolis, Arianna W. Rosenbluth +3 more
Neural NetworksNonlinear neural networks: Principles, mechanisms, and architectures
1,619 Citations1988Stephen Grossberg
An historical discussion is provided of the intellectual trends that caused nineteenth century interdisciplinary studies of physics and psychobiology by leading scientists such as Helmholtz, Maxwell, and Mach to splinter into separate twentieth-century scientific movements.
Neural NetworksOptimal unsupervised learning in a single-layer linear feedforward neural network
1,516 Citations1989Terence D. Sanger
An optimality principle is proposed which is based upon preserving maximal information in the output units and an algorithm for unsupervised learning based upon a Hebbian learning rule, which achieves the desired optimality is presented.
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
1,261 Citations2009
This chapter contains sections titled: Introduction Exponential Error Decay Dilemma: Avoiding Aradient Decay Prevents Long-Term Latching Remedies and Conclusion.
Advances in psychologySerial Order: A Parallel Distributed Processing Approach
1,230 Citations1997Michael I. Jordan
A theory of learned sequential behavior is presented, with a focus on coarticulatory phenomena in speech, implemented as a recurrent parallel distributed processing network that is trained via a generalized error-correcting algorithm.
Generating Text with Recurrent Neural Networks
1,166 Citations2011Ilya Sutskever, James Martens +1 more
The power of RNNs trained with the new Hessian-Free optimizer by applying them to character-level language modeling tasks is demonstrated, and a new RNN variant that uses multiplicative connections which allow the current input character to determine the transition matrix from one hidden state vector to the next is introduced.
IEEE Transactions on Audio and ElectroacousticsA computer program for designing optimum FIR linear phase digital filters
1,139 Citations1973James H. McClellan, T.W. Parks +1 more
A general-purpose computer program which is capable of designing a large Class of optimum (in the minimax sense) FIR linear phase digital filters and is shown to be exceedingly efficient.
Neural NetworksGeneralization of backpropagation with application to a recurrent gas market model
952 Citations1988Paul J. Werbos
This paper will derive a generalization of backpropagation to recurrent systems (which input their own output), such as hybrids of perceptron-style networks and Grossberg/Hopfield networks, and does not require the storage of intermediate iterations to deal with continuous recurrence.
Physical Review LettersGeneralization of back-propagation to recurrent neural networks
951 Citations1987Fernando J. Pineda
An adaptive neural network with asymmetric connections is introduced that bears a resemblance to the master/slave network of Lapedes and Farber but it is architecturally simpler.
Neural ComputationLearning State Space Trajectories in Recurrent Neural Networks
675 Citations1989Barak A. Pearlmutter
A procedure for finding E/wij, where E is an error functional of the temporal trajectory of the states of a continuous recurrent network and wij are the weights of that network, which seems particularly suited for temporally continuous domains.
Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.Framewise phoneme classification with bidirectional LSTM networks
601 Citations2006Alexander Graves, Jürgen Schmidhuber
It is found that bidirectional LSTM outperforms both RNNs and unidirectionalLSTM, and the significance of framewise phoneme classification to continuous speech recognition and the validity of usingbidirectional networks for online causal tasks is discussed.
arXiv (Cornell University)Recent Advances in Recurrent Neural Networks
576 Citations2017Hojjat Salehinejad, Julianne Baarbé +3 more
A survey on RNNs and several new advances for newcomers and professionals in the field are presented and the research challenges are introduced.
arXiv (Cornell University)Neural Ordinary Differential Equations
566 Citations2018Ricky T. Q. Chen, Yulia Rubanova +2 more
This work shows how to scalably backpropagate through any ODE solver, without access to its internal operations, which allows end-to-end training of ODEs within larger models.
Training recurrent neural networks
391 Citations2013Geoffrey E. Hinton, Ilya Sutskever
A new probabilistic sequence model that combines Restricted Boltzmann Machines and RNNs is described, more powerful than similar models while being less difficult to train, and a random parameter initialization scheme is described that allows gradient descent with momentum to train Rnns on problems with long-term dependencies.
Inverse ProblemsStable architectures for deep neural networks
368 Citations2017Eldad Haber, Lars Ruthotto
This paper relates the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and presents several strategies for stabilizing deep learning for very deep networks.
Infoscience (Ecole Polytechnique Fédérale de Lausanne)Long short-term memory in recurrent neural networks
235 Citations2001Felix A. Gers
These Ecole polytechnique federale de Lausanne EPFL, n° 2366 (2001)Faculte informatique et communicationsJury: Paolo Frasconi, Roger Hersch, Martin Rajman, Jurgen Schmidhuber Public defense.
Transactions of the Association for Computational LinguisticsDeep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
227 Citations2016Jie Zhou, Ying Cao +3 more
This work introduces a new type of linear connections, named fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks, and an interleaved bi-directional architecture for stacking the LSTM layers, and achieves state-of-the-art performance and outperforms the best conventional model by 0.7 BLEU points.
Proceedings of the AAAI Conference on Artificial IntelligenceReversible Architectures for Arbitrarily Deep Residual Neural Networks
207 Citations2018Bo Chang, Lili Meng +4 more
From this interpretation, a theoretical framework on stability and reversibility of deep neural networks is developed, and three reversible neural network architectures that can go arbitrarily deep in theory are derived.
A new concept using LSTM Neural Networks for dynamic system identification
205 Citations2017Yu Wang
A new concept of applying one of the most popular RNN approach - LSTM to identify and control dynamic system is to be investigated and both identification (or learning) dynamic system and design of controller based on identification are going to be discussed.
arXiv (Cornell University)Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
192 Citations2016Qianli Liao, Tomaso Poggio
A generalization of both RNN and ResNet architectures and the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex are proposed.
arXiv (Cornell University)HMSPC: A Hybrid Mechanistic-Stochastic Physical-Continuous Model for Battery Dynamics
155 Citations2019Yulia Rubanova
This work generalizes RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model they are called ODE-RNNs, which outperform their RNN-based counterparts on irregularly-sampled data.
arXiv (Cornell University)Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
153 Citations2017Yiping Lu, Aoxiao Zhong +2 more
arXiv (Cornell University)Smart Reply: Automated Response Suggestion for Email
141 Citations2016Anjuli Kannan, Karol Kurach +9 more
Maynooth University ePrints and eTheses Archive (Maynooth University)Dynamic recurrent neural networks
132 Citations2018Barak A. Pearlmutter
This work surveys learning algorithms for recurrent neural networks with hidden units and attempts to put the various techniques into a common framework, resulting in a unified presentation that leads to generalizations of various sorts.
Neural Information Processing SystemsGeneralization of Back propagation to Recurrent and Higher Order Neural Networks
123 Citations1987Fernando J. Pineda
A general method for deriving backpropagation algorithms for networks with recurrent and higher order networks and to a constrained dynamical system for training a content addressable memory.
Journal of Vibration and ControlOn the Use of Delay Equations in Engineering Applications
106 Citations2010Yuliya N. Kyrychko, S. J. Hogan
This paper is a review of applications of delay differential equations to different areas of engineering science, and introduces the main idea of real-time dynamic substructuring, or hybrid testing, which is an emerging area for modeling with the help of delay equations.
IEEE Transactions on Neural NetworksOn the efficiency of the orthogonal least squares training method for radial basis function networks
102 Citations1996A. Sherstinsky, Rosalind W. Picard
The efficiency of the orthogonal least squares method for training approximation networks is examined using the criterion of energy compaction and it is shown that the selection of basis vectors produced by the procedure is not the most compact when the approximation is performed using a nonorthogonal basis.
IRE Transactions on Communications SystemsTechniques for Designing Finite-Duration Impulse-Response Digital Filters
91 Citations1971L. R. Rabiner
The motivation behind three design techniques that have been proposed are reviewed here, and the resulting designs are compared with respect to filter characteristics, ease of design, and methods of realization.
MDPI (MDPI AG)Critical Behavior in Physics and Probabilistic Formal Languages
89 Citations2017Henry W. Lin
It is proved that Markov/hidden Markov processes generically exhibit exponential decay in their mutual information, which explains why natural languages are poorly approximated by Markov Processes.
arXiv (Cornell University)AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
86 Citations2019Bo Chang, Minmin Chen +2 more
This paper draws connections between recurrent networks and ordinary differential equations and proposes a special form of recurrent networks called AntisymmetricRNN, able to capture long-term dependencies thanks to the stability property of its underlying differential equation.
DSpace@MIT (Massachusetts Institute of Technology)A theory of nonlinear systems
80 Citations1956Amar G. Bose
arXiv (Cornell University)Reversible Architectures for Arbitrarily Deep Residual Neural Networks
70 Citations2017Bo Chang, Lili Meng +4 more
Neural Information Processing SystemsA Theory for Neural Networks with Time Delays
63 Citations1990Bert de Vries, José C. Prı́ncipe
This model, the gamma neural model, is as general as a convolution delay model with arbitrary weight kernels w(t) and it is shown that the gamma model can be formulated as a (partially prewired) additive model.
IEEE Signal Processing LettersOptimizing FIR approximation for discrete-time IIR filters
51 Citations2003Yutaka Yamamoto, Brian D. O. Anderson +2 more
A new method is proposed to approximate an IIR filter by an FIR filter, which directly yields an optimal approximation with respect to the H/sup /spl infin// error norm, and it is shown that this design problem can be reduced to a linear matrix inequality.
Journal of Mathematical Imaging and VisionDeep Neural Networks Motivated by Partial Differential Equations
40 Citations2019Lars Ruthotto, Eldad Haber
A new PDE interpretation of a class of deep convolutional neural networks (CNN) that are commonly used to learn from speech, image, and video data is established and three new ResNet architectures are derived that fall into two new classes: parabolic and hyperbolic CNNs.
On the training of recurrent neural networks
38 Citations2011Mounir Ben Nasr, Mohamed Chtourou
This paper proposes a new approach which combines unsupervised and supervised learning for training recurrent neural networks (RNNs) and results show that the proposed method performs well in comparison with the back propagation through time algorithm.
arXiv (Cornell University)Criticality in Formal Languages and Statistical Physics
30 Citations2016Henry W. Lin, Max Tegmark
arXiv (Cornell University)NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
26 Citations2018Marco Ciccone, Marco Gallieri +3 more
It is proved that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReLU units.
IEEE Transactions on Image ProcessingM-lattice: from morphogenesis to image processing
22 Citations1996A. Sherstinsky, Rosalind W. Picard
The M-lattice is introduced, a system that preserves the pattern-formation properties of reaction-diffusion and is bounded and enables computer or analog VLSI implementations to simulate a variety of partial and ordinary differential equations.
DSpace@MIT (Massachusetts Institute of Technology)M-lattice: a system for signal synthesis and processing based on reaction-diffusion
14 Citations1994Alexander Semyon Sherstinsky
A bounded and more exible non-linear system, the M-lattice, which preserves the natural pattern-formation properties of reaction-diiusion and establishes relationships between reaction- diiusion systems and paradigms in linear systems theory and certain types of artiicial "neurally-inspired" systems.
Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum
14 Citations2018Omer Levy, Kenton Lee +2 more
This work decoupling the LSTM’s gates from the embedded simple RNN, producing a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input.
IEEE Transactions on Circuits and Systems I Fundamental Theory and ApplicationsOn stability and equilibria of the M-lattice
2 Citations1998A. Sherstinsky, Rosalind W. Picard
It is proved that a subclass of the M-lattice is totally stable, and this result also applies to the original cellular neural network as a rigorous proof of its total stability.
eScholarship (California Digital Library)Beyond Deep Learning: Scalable Methods and Models for Learning
2 Citations2013Oriol Vinyals
A theoretical bound is derived showing a fundamental limitation of shallow architectures based on sparse coding, thus justifying the need for deeper architectures, while also empirically verifying these architectural choices on speech recognition.