An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories

Neural ComputationPublished 1 December 1990

Ronald J. Williams, Jing Peng

DOI10.1162/neco.1990.2.4.490

Source URL

Citations593

SJR quartileQ1

SJR score0.83

SNIP1.45

TL;DR

A novel variant of the familiar backpropagation-through-time approach to training recurrent networks is described, intended to be used on arbitrary recurrent networks that run continually without ever being reset to an initial state.

Abstract

A novel variant of the familiar backpropagation-through-time approach to training recurrent networks is described. This algorithm is intended to be used on arbitrary recurrent networks that run continually without ever being reset to an initial state, and it is specifically designed for computationally efficient computer implementation. This algorithm can be viewed as a cross between epochwise backpropagation through time, which is not appropriate for continually running networks, and the widely used on-line gradient approximation technique of truncated backpropagation through time.

Keywords

Computer SciencePhysics and Astronomy

Learning Internal Representations by Error Propagation

16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more

Neural ComputationA Learning Algorithm for Continually Running Fully Recurrent Neural Networks

4,451 Citations1989Ronald J. Williams, David Zipser

The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.

Physical Review LettersGeneralization of back-propagation to recurrent neural networks

951 Citations1987Fernando J. Pineda

An adaptive neural network with asymmetric connections is introduced that bears a resemblance to the master/slave network of Lapedes and Farber but it is architecturally simpler.

Neural ComputationLearning State Space Trajectories in Recurrent Neural Networks

675 Citations1989Barak A. Pearlmutter

A procedure for finding E/wij, where E is an error functional of the temporal trajectory of the states of a continuous recurrent network and wij are the weights of that network, which seems particularly suited for temporally continuous domains.

Neural ComputationFinite State Automata and Simple Recurrent Networks

462 Citations1989Axel Cleeremans, David Servan‐Schreiber +1 more

A network architecture introduced by Elman (1988) for predicting successive elements of a sequence and shows that long distance sequential contingencies can be encoded by the network even if only subtle statistical properties of embedded strings depend on the early information.

View PDF

Connection ScienceExperimental Analysis of the Real-time Recurrent Learning Algorithm

293 Citations1989Ronald J. Williams, David Zipser

A series of simulation experiments are used to investigate the power and properties of the real-time recurrent learning algorithm, a gradient-following learning algorithm for completely recurrent networks running in continually sampled time.

IEEE Transactions on Neural NetworksComparison of four neural net learning methods for dynamic system identification

184 Citations1992Sijing Qin, H.-T. Su +1 more

It is shown that the feedforward network (FFN) pattern learning rule is a first-order approximation of the FFN-batch learning rule, and is valid for nonlinear activation networks provided the learning rate is small.

A learning algorithm for analog, fully recurrent neural networks

69 Citations1989Gherrity

A learning algorithm for recurrent neural networks is derived that allows a network to learn specified trajectories in state space in response to various input sequences.

Neural ComputationA Subgrouping Strategy that Reduces Complexity and Speeds Up Learning in Recurrent Networks

44 Citations1989David Zipser

A technique is described here for reducing the amount of computation required by RTRL without changing the connectivity of the networks by dividing the original network into subnets for the purpose of error propagation while leaving them undivided for activity propagation.