Long Short-Term Memory Learns Context Free and Context Sensitive Languages
Artificial Neural Nets and Genetic AlgorithmsPublished 1 January 2001
Felix A. Gers, Jürgen Schmidhuber
Citations22
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
LSTM variants are also the first RNNs to learn a context sensitive language (\mbox{CSL}), namely $a^nb^n c^n$.
Abstract
Previous work on learning regular languages from exemplary training sequences showed that Long Short- Term Memory (LSTM) outperforms traditional recurrent neural networks (RNNs). Here we demonstrate LSTM's superior performance on context free language (CFL) benchmarks, and show that it works even better than previous hardwired or highly specialized architectures. To the best of our knowledge, LSTM variants are also the first RNNs to learn a context sensitive language (CSL), namely, a n b n c n .
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
NeurocomputingAdvances in neural information processing systems 7
22,300 Citations1997Krzysztof J. Cios, Mark E. Shields
Neural ComputationLearning to Forget: Continual Prediction with LSTM
5,419 Citations2000Felix A. Gers, Jürgen Schmidhuber +1 more
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
Recurrent nets that time and count
641 Citations2000Felix A. Gers, Jürgen Schmidhuber
Surprisingly, LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes separated by either 50 or 49 discrete time steps, without the help of any short training exemplars.
IEEE Transactions on Neural NetworksGradient calculations for dynamic recurrent neural networks: a survey
590 Citations1995Barak A. Pearlmutter
The author discusses advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones and presents some "tricks of the trade" for training, using, and simulating continuous time and recurrent neural networks.
Connection ScienceA Recurrent Neural Network that Learns to Count
240 Citations1999Paul Rodriguez, Janet Wiles +1 more
This research employs standard backpropagation training techniques for a recurrent neural network in the task of learning to predict the next character in a simple deterministic CFL (DCFL), and shows that an RNN can learn to recognize the structure of a simple DCFL.
arXiv (Cornell University)The neural network pushdown automation: model, stack and learning simulations
32 Citations1993Guo-Zheng Sun, C. Lee Giles +2 more
Recurrent Neural Networks Can Learn to Implement Symbol-Sensitive Counting
28 Citations1997Paul Rodriguez, Janet Wiles
This work shows that a RNN can learn a harder CFL, a simple palindrome, by organizing its resources into a symbol-sensitive counting solution, and provides a dynamical systems analysis which demonstrates how the network can not only count, but also copy and store counting information.
arXiv (Cornell University)The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations
15 Citations2017Guo-Zheng Sun, C. Lee Giles +2 more