On-line backpropagation in two-layered neural networks

Journal of Physics A Mathematical and GeneralPublished 21 October 1995

Peter Riegler, Michael Biehl

Citations66

TL;DR

This work presents an exact analysis of learning a rule by on-line gradient descent in a two-layered neural network with adjustable hidden-to-output weights (backpropagation of error) with fixed weights in the second layer.

Abstract

We present an exact analysis of learning a rule by on--line gradient descent in a two--layered neural network with adjustable hidden--to--output weights (backpropagation of error). Results are compared with the training of networks having the same architecture but fixed weights in the second layer. PACS.: 07.05.Mh, 87.10, 02.50 The ability of neural networks to learn a rule from examples [1] has been studied successfully in a statistical mechanics context, see e.g. [2, 3, 4] for recent reviews. So far most of the analysis has been restricted to very simple networks like the single layer perceptron [1] or networks with one layer of hidden units and a fixed hidden--to--output relation, e.g. the so--called committee machine [3]. In the following we extend the recent investigation of learning by on--line gradient descent [8, 10] to two--layered networks with adjustable weights connecting the hidden units and the output. This topic is of crucial importance as systems with variable hidden--...

Keywords

Computer ScienceEngineering

Neural NetworksIntroduction to the theory of neural computation

6,435 Citations1994

NeurocomputingBackpropagation and stochastic gradient descent method

831 Citations1993Шун-ичи Амари

The wide applicability of the stochastic gradient descent method to various types of models and loss functions is reviewed, and in particular, it is applied to the pattern recognition problem, obtaining a new learning algorithm based on the information criterion.

IEEE Transactions on Electronic ComputersA Theory of Adaptive Pattern Classifiers

520 Citations1967Шун-ичи Амари

It is proved that, by the procedures proposed here, the weight vector converges to the optimal one even under nonseparable pattern distributions, and there is an important tradeoff between speed and accuracy of convergence.

Physical Review AStatistical mechanics of learning from examples

513 Citations1992H. Sebastian Seung, Haim Sompolinsky +1 more

It is shown that for smooth networks, i.e., those with continuously varying weights and smooth transfer functions, the generalization curve asymptotically obeys an inverse power law, while for nonsmooth networks other behaviors can appear, depending on the nature of the nonlinearities as well as the realizability of the rule.

Reviews of Modern PhysicsThe statistical mechanics of learning a rule

473 Citations1993Timothy L. H. Watkin, Albrecht Rau +1 more

View PDF

Physical Review LettersExact Solution for On-Line Learning in Multilayer Neural Networks

174 Citations1995David Saad, Sara A. Solla

An analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks is presented.

Journal of Physics A Mathematical and GeneralLearning by on-line gradient descent

153 Citations1995Michael Biehl, H Schwarze

On-line gradient-descent learning in multilayer networks analytically and numerically and for architectures with hidden layers and fixed hidden-to-output weights, such as the parity and the committee machine, is studied.

Journal of Physics A Mathematical and GeneralOn-line learning in the committee machine

40 Citations1995Mauro Copelli, Nestor Caticha

The dynamics of learning from examples in the K=3 non-overlapping committee machine with single presentation of examples is studied and the optimal algorithm, in the sense of mean generalization, is obtained from a variational analysis of the differential equations which describe the dynamics.