Deterministic Convergence of an Online Gradient Method for BP Neural Networks
IEEE Transactions on Neural NetworksPublished 1 May 2005
Wei Wu, Guofang Feng, Zheng Li, Ying Xu
Citations157
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper proves a convergence theorem for an online gradient method with variable step size for backward propagation neural networks with a hidden layer that has a deterministic and monotone nature.
Abstract
Online gradient methods are widely used for training feedforward neural networks. We prove in this paper a convergence theorem for an online gradient method with variable step size for backward propagation (BP) neural networks with a hidden layer. Unlike most of the convergence results that are of probabilistic and nonmonotone nature, the convergence result that we establish here has a deterministic and monotone nature.
Keywords
Computer SciencePhysics and Astronomy
Journal of the American Statistical AssociationProbability and Measure.
6,654 Citations1996PE, P. Billingsley
Neural NetworksApproximation capabilities of multilayer feedforward networks
6,086 Citations1991Kurt Hornik
It is shown that standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to L p (μ) performance criteria, for arbitrary finite input environment measures μ.
Neural ComputationFirst- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method
1,193 Citations1992Roberto Battiti
First- and second-order optimization methods for learning in feedforward neural networks are reviewed to illustrate the main characteristics of the different methods and their mutual relations.
Journal of the American Statistical AssociationStochastic Approximation Algorithms and Applications
1,025 Citations1998Anatolii A. Puhalskii, Harold J. Kushner +1 more
IEEE Transactions on Neural NetworksConvergence of learning algorithms with constant learning rates
141 Citations1991Chi-Bang Kuan, Kurt Hornik
It is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation, in the sense of weak convergence of random processes as epsilon tends to zero.
Neural ComputationOn the Convergence of the LMS Algorithm with Adaptive Learning Rate for Linear Feedforward Networks
141 Citations1991Zhi‐Quan Luo
It is shown that, by dynamically decreasing the learning rate during each training cycle, the sequence of matrices generated by the LMS algorithm will converge to the optimal weight matrix.
Journal of Sound and VibrationPROPER ORTHOGONAL DECOMPOSITION AND ITS APPLICATIONS – PART II: MODEL REDUCTION FOR MEMS DYNAMICAL ANALYSIS
113 Citations2002Yanhua Liang, Wenjuan Lin +4 more
A neural-network-based method that combines robust principal component analysis (PCA) neural network model with Galerkin procedure with a number of numerical advantages such as the stability and robustness to noise-injected data and the faster convergence of iterations in the training stages than the existing neural network technique.
Optimization methods & softwareSerial and parallel backpropagation convergence via nonmonotone perturbed minimization
101 Citations1994O. L. Mangasarian, M. V. Solodov
A general convergence theorem is proposed for a family of serial and parallel nonmonotone unconstrained minimization methods with perturbations to establish convergence of backpropagation (BP), the classical algorithm for training artificial neural networks.
Second-Order Methods for Neural Networks: Fast and Reliable Training Methods for Multi-Layer Perceptrons
101 Citations1997Adrian J. Shepherd
This chapter discusses second-order training methods for MLPs, which focus on reducing Storage and Computational Costs, and the development of the Hessian Matrix, which simplifies the training process.
IEEE Transactions on Neural NetworksImproving the error backpropagation algorithm with a modified error function
84 Citations1997Sang-Hoon Oh
A modified error function is proposed to improve the error backpropagation (EBP) algorithm of multilayer perceptrons (MLPs) which suffers from slow learning speed and prevents overspecialization of learning for training patterns.
Optimization methods & softwareAnalysis of an approximate gradient projection method with applications to the backpropagation algorithm
83 Citations1994Luo Zhi-quan, Paul Tseng
It is shown that if the gradients of the functions are bounded and Lipschitz continuous over a certain level set and the stepsizes are chosen to be proportional to a certain residual squared or to be square summable, then every cluster point of the iterates is a stationary point.
Optimization methods & softwareConvergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part 1
81 Citations1994Alexei A. Gaivoronski
Convergence properties of serial and parallel backpropagation algorithm for training of neural nets, as well as its modification with momentum term are studied, showing that they can be put into the general framework of the stochastic gradient methods.
IEEE Transactions on Neural NetworksOptimal convergence of on-line backpropagation
61 Citations1996Marco Gori, Marco Maggini
This paper proves the companion of Rosenblatt's PC (perceptron convergence) theorem for feedforward networks (1960), stating that pattern mode backpropagation converges to an optimal solution for linearly separable patterns.
Neural ComputationDiffusion Approximations for the Constant Learning Rate Backpropagation Algorithm and Resistance to Local Minima
59 Citations1994William Finnoff
The reasons why simple backpropagation is less likely to get stuck in local minima than the batch training process are indicated and demonstrated empirically on a number of examples.
Advances in Computational MathematicsTraining Multilayer Perceptrons Via Minimization of Sum of Ridge Functions
54 Citations2002Wei Wu, Guorui Feng +1 more
The problem of minimizing E(x)=∑i=1nfi(ξi⋅x), where ξi∈Rs, 1≤i≤n, and each fi(�elliⓂx) is a ridge function is considered.
Neural ComputationParameter Convergence and Learning Curves for Neural Networks
48 Citations1999Terrence L. Fine, Sayandev Mukherjee
This work revisits the oft-studied asymptotic behavior of the parameter or weight estimate returned by any member of a large family of neural network training algorithms and rigorously establishes conditions under which the parameter estimate converges strongly into the set of minima of the generalization error.
IEEE Transactions on Neural NetworksConvergent on-line algorithms for supervised learning in neural networks
45 Citations2000Luigi Grippo
It is shown that suitable training algorithms can be defined, in a way that the disagreement between the different copies of the network is asymptotically reduced, and convergence toward stationary points of the global error function can be guaranteed.
Journal of Computational and Applied MathematicsDeterministic convergence of an online gradient method for neural networks
43 Citations2002Wei Wu, Yuesheng Xu
A deterministic convergence of online gradient methods for the training of a class of nonlinear feedforward neural networks when the training examples are linearly independent is established.
Journal of Computational and Applied MathematicsConvergence of an online gradient method for feedforward neural networks with stochastic inputs
37 Citations2003Zhengxue Li, Wei Wu +1 more
A monotonicity and a weak convergence of deterministic nature are proved in the convergence of an online gradient method for feed-forward neural networks.
Journal of Optimization Theory and ApplicationsMinimization Algorithms Based on Supervisor and Searcher Cooperation
34 Citations2001Wenjun Liu, Y. H. Dai
This work proposes a class of algorithms within this framework for the design of new minimization algorithms with desirable characteristics, namely, supervisor-searcher cooperation, and examines a gradient algorithm in the class.
NeurocomputingSuccessive approximation training algorithm for feedforward neural networks
31 Citations2002Yanchun Liang, Dingchao Feng +3 more
A novel algorithm based on successive approximation training for feedforward neural networks is presented and it is shown that the proposed algorithm increases the rate of convergence and improves the generalization performance by avoiding local minima.
International Thomson Computer Press eBooksNeural networks : deterministic methods of analysis
16 Citations1996Steve Ellacott, Deb Bose
Neural ComputationOnline Learning from Finite Training Sets and Robustness to Input Bias
15 Citations1998Peter Sollich, David Barber
Online learning turns out to be much more robust to input bias and actually outperforms offline learning when such bias is present; for unbiased inputs, online and offline learning perform almost equally well.
Applied Mathematics LettersConvergence of online gradient methods for continuous perceptrons with linearly separable training patterns
10 Citations2003Wei Wu, Zhiqiong Shao
It is proved that the online gradient method for continuous perceptrons converges in finite steps when the training patterns are linearly separable.
Neural ComputationOnline Steepest Descent Yields Weights with Nonnormal Limiting Distribution
6 Citations1996Sayandev Mukherjee, Terrence L. Fine
It is proved, from an exact analysis of the one-dimensional case and constant learning rate, weak convergence to a distribution that is not gaussian in general.