Optimal convergence of on-line backpropagation
IEEE Transactions on Neural NetworksPublished 1 January 1996
Marco Gori, Marco Maggini
Citations61
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper proves the companion of Rosenblatt's PC (perceptron convergence) theorem for feedforward networks (1960), stating that pattern mode backpropagation converges to an optimal solution for linearly separable patterns.
Abstract
Many researchers are quite skeptical about the actual behavior of neural network learning algorithms like backpropagation. One of the major problems is with the lack of clear theoretical results on optimal convergence, particularly for pattern mode algorithms. In this paper, we prove the companion of Rosenblatt's PC (perceptron convergence) theorem for feedforward networks (1960), stating that pattern mode backpropagation converges to an optimal solution for linearly separable patterns.
Keywords
Computer Science
The MIT Press eBooksParallel Distributed Processing
15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more
neural information processing systemsHandwritten Digit Recognition with a Back-Propagation Network
3,637 Citations1989Yann LeCun, Bernhard E. Boser +5 more
Minimal preprocessing of the data was required, but architecture of the network was highly constrained and specifically designed for the task, and has 1% error rate and about a 9% reject rate on zipcode digits provided by the U.S. Postal Service.
Neural NetworksNeural networks and principal component analysis: Learning from examples without local minima
1,360 Citations1989Pierre Baldi, Kurt Hornik
The main result is a complete description of the landscape attached to E in terms of principal component analysis, showing that E has a unique minimum corresponding to the projection onto the subspace generated by the first principal vectors of a covariance matrix associated with the training patterns.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOn the problem of local minima in backpropagation
656 Citations1992Marco Gori, A. Tesi
A theoretical framework for backpropagation (BP) is proposed and it is proven in particular that the convergence holds if the classes are linearly separable and that multilayered neural networks (MLNs) exceed perceptrons in generalization to new examples.
IEEE Transactions on Neural NetworksLearning without local minima in radial basis function networks
202 Citations1995Monica Bianchini, Paolo Frasconi +1 more
It is proved that the attached cost function is local minima free with respect to all the weights in the case of networks using radial basis functions (RBF), which provides some theoretical foundations for a massive application of RBF in pattern recognition.
IEEE Transactions on Neural NetworksAn accelerated learning algorithm for multilayer perceptron networks
115 Citations1994A.G. Parlos, B. Fernandez +3 more
An accelerated learning algorithm (ABP-adaptive back propagation) is proposed for the supervised training of multilayer perceptron networks with superior convergence speed for analog problems only, as compared to other competing methods, as well as reduced sensitivity to algorithm step size parameter variations.
IEEE Transactions on Neural NetworksCan backpropagation error surface not have local minima
109 Citations1992Xiao-Hu Yu
It is shown theoretically that for an arbitrary T-element training set with t(t=/<T) different inputs, the backpropagation error surface does not have suboptimal local minima if the network is capable of exactly implementing an arbitrary training set consisting of t different patterns.
Neural ComputationOn the Geometry of Feedforward Neural Network Error Surfaces
97 Citations1993An Mei Chen, Haw-minn Lu +1 more
The geometric structure of these equioutput weight space transformations is explored for the case of multilayer perceptron networks with tanh activation functions and it is shown that these transformations form an algebraic group isomorphic to a direct product of Weyl groups.
IEEE Transactions on Neural NetworksOn the problem of local minima in recurrent neural networks
54 Citations1994Monica Bianchini, Marco Gori +1 more
This paper analyses the problem of optimal learning in recurrent networks by proposing conditions that guarantee local minima free error surfaces and a formal relationship between recurrent and static feedforward networks is established.
Backpropagation separates when perceptrons do
34 Citations1989Sontag, Sussmann
Terminal attractor learning algorithms for back propagation neural networks
22 Citations1991Sheng-De Wang, Chia‐Hao Hsu
The proposed algorithms preserve the parallel and distributed features of neurocomputing, guarantee that the learning process can converge in finite time, and find the set of weights minimizing the error function in global, provided such a set ofweights exists.
IEEE Transactions on Neural NetworksOn a novel unsupervised competitive learning algorithm for scalar quantization
19 Citations1994Marc M. Van Hulle, Dominique Martinez
It is shown both mathematically and by simulations that BAR converges to equiprobable quantizations of univariate probability density functions and that, in this way, it outperforms other unsupervised competitive learning rules.