Using the Karhunen-Loe've transformation in the back-propagation training algorithm

IEEE Transactions on Neural NetworksPublished 1 January 1991

Heidar Malki, A. Moghaddamjoo

Citations31

TL;DR

A novel training approach based on the back-propagation algorithm is introduced that reduces the number of computations, the learning rate is improved and the performance of this method is compared with the standard back- Propagation algorithms in segmenting a synthetic noisy image.

Abstract

A novel training approach based on the back-propagation algorithm is introduced. In the proposed approach, initially, a set of training vectors is obtained by applying the Karhunen-Loe've transform on the training patterns. The training is first started in the direction of the major eigenvectors of the correlation matrix of the training patterns and then continues by gradually including the remaining components, in their order of significance. With this approach, the number of computations is significantly reduced and the learning rate is improved. The performance of this method is compared with the standard back-propagation algorithm in segmenting a synthetic noisy image.

Keywords

Computer ScienceEngineering

Proceedings of the National Academy of SciencesNeural networks and physical systems with emergent collective computational abilities.

19,349 Citations1982J. J. Hopfield

A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.

The MIT Press eBooksParallel Distributed Processing

15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more

Neural NetworksIncreased rates of convergence through learning rate adaptation

1,795 Citations1988Robert A. Jacobs

A study of Steepest Descent and an analysis of why it can be slow to converge and four heuristics for achieving faster rates of convergence are proposed.

Biological CyberneticsAccelerating the convergence of the back-propagation method

1,101 Citations1988Thomas P. Vogl, J. K. Mangis +3 more

Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.

The Annals of Mathematical StatisticsAccelerated Stochastic Approximation

435 Citations1958Harry Kesten

View PDF

Defense Technical Information Center (DTIC)Goal Seeking Components for Adaptive Intelligence: An Initial Assessment.

28 Citations1981Andrew G. Barto, Richard S. Sutton

It is shown that components designed with attention to the temporal aspects of reinforcement learning can acquire knowledge about feedback pathways in which they are embedded and can use this knowledge to seek their preferred inputs, thus combining pattern recognition, search, and control functions.

IEEE Transactions on Circuits and SystemsAlternating projection neural networks

27 Citations1989Robert J. Marks, Sung‐Kwun Oh +1 more

A class of neural networks whose performance can be analyzed and geometrically visualized in a signal space environment and can be improved by the use of sigmoid-type nonlinearities and/or increasing the number of neurons in a hidden layer is considered.

Optical EngineeringNondispersive Propagation Skew In Iterative Neural Network And Optical Feedback Processors

4 Citations1989Seho Oh, Dong C. Park +2 more

It is shown that in the absence of temporal dispersion, certain iterative algorithms have stable steady-state solutions that are independent of clock skew.

International Conference on Acoustics, Speech, and Signal ProcessingA learning rule eliminating local minima in multilayer perceptrons

4 Citations2002Pietro Burrascano, P. Lucci

It has been verified that the proposed modification virtually eliminates nonconvergence problems of a moderate eta value is used and is shown to smooth the cost function to be minimized during the learning phase.