Training Multilayer Perceptrons with the Extended Kalman Algorithm

Neural Information Processing SystemsPublished 1 January 1988

Sharad Singhal, Lance Wu

Citations319

TL;DR

It is shown that training multilayer perceptrons is an identification problem for a nonlinear dynamic system which can be solved using the Extended Kalman Algorithm.

Abstract

A large fraction of recent work in artificial neural nets uses multilayer perceptrons trained with the back-propagation algorithm described by Rumelhart et. al. This algorithm converges slowly for large or complex problems such as speech recognition, where thousands of iterations may be needed for convergence even with small data sets. In this paper, we show that training multilayer perceptrons is an identification problem for a nonlinear dynamic system which can be solved using the Extended Kalman Algorithm. Although computationally complex, the Kalman algorithm usually converges in a few iterations. We describe the algorithm and compare it with back-propagation using two-dimensional examples.

Keywords

Computer Science

Journal of Basic EngineeringA New Approach to Linear Filtering and Prediction Problems

30,999 Citations1960R. E. Kalman

CERN Document Server (European Organization for Nuclear Research)Applied Optimal Estimation

6,388 Citations1974A. Gelb

This is the first book on the optimal estimation that places its major emphasis on practical applications, treating the subject more from an engineering than a mathematical orientation, and the theory and practice of optimal estimation is presented.

Elsevier eBooksLearning Internal Representations by Error Propagation

3,013 Citations1988David E. Rumelhart, Geoffrey E. Hinton +1 more

This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion.

IEEE Transactions on Acoustics Speech and Signal ProcessingPhoneme recognition using time-delay neural networks

2,625 Citations1989Alexander Waibel, Toshiyuki Hanazawa +3 more

The authors present a time-delay neural network (TDNN) approach to phoneme recognition which is characterized by two important properties: using a three-layer arrangement of simple computing units, a hierarchy can be constructed that allows for the formation of arbitrary nonlinear decision surfaces, which the TDNN learns automatically using error backpropagation.

IEEE International Conference on Neural NetworksCapabilities of three-layered perceptrons

400 Citations1988Irie, Miyake

A theorem is proved that three-layered perceptrons with an infinite number of computing units can represent arbitrary mapping if the desired mapping and the input-output characteristics of the computing units satisfy some constraints.

Computer Speech & LanguageThe modified Kanerva model for automatic speech recognition

57 Citations1989Richard W. Prager, F. Fallside

This paper describes how the design for the modified Kanerva model is derived from Kanerva's original theory, and develops a method to deal with the time varying nature of the speech signal by recognizing static patterns together with a fixed quantity of contextual information.