Gauss-Newton approximation to Bayesian learning

Proceedings of International Conference on Neural Networks (ICNN'97)Published 22 November 2002

F. Dan Foresee, Martin Hagan

DOI10.1109/icnn.1997.614194

Citations1,391

TL;DR

The application of Bayesian regularization to the training of feedforward neural networks is described, using a Gauss-Newton approximation to the Hessian matrix to reduce the computational overhead.

Abstract

This paper describes the application of Bayesian regularization to the training of feedforward neural networks. A Gauss-Newton approximation to the Hessian matrix, which can be conveniently implemented within the framework of the Levenberg-Marquardt algorithm, is used to reduce the computational overhead. The resulting algorithm is demonstrated on a simple test problem and is then applied to three practical problems. The results demonstrate that the algorithm produces networks which have excellent generalization capabilities.

Keywords

Computer ScienceEngineering

IEEE Transactions on Neural NetworksTraining feedforward networks with the Marquardt algorithm

7,625 Citations1994Martin Hagan, Mohammad Bagher Menhaj

The Marquardt algorithm for nonlinear least squares is presented and is incorporated into the backpropagation algorithm for training feedforward neural networks and is found to be much more efficient than either of the other techniques when the network contains no more than a few hundred weights.

Belarusian State Pedagogical University repository (Belarusian State Pedagogical University)Neural network design

5,164 Citations1995Martin Hagan, Howard B. Demuth +1 more

This book, by the authors of the Neural Network Toolbox for MATLAB, provides a clear and detailed coverage of fundamental neural network architectures and learning rules, as well as methods for training them and their applications to practical problems.

Neural ComputationBayesian Interpolation

4,377 Citations1992David Mackay

The Bayesian approach to regularization and model-comparison is demonstrated by studying the inference problem of interpolating noisy data by examining the posterior probability distribution of regularizing constants and noise levels.

View PDF

Non-linear time series. A dynamical system approach

1,895 Citations1990Howell Tong

Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights

1,384 Citations1990Dat Nguyen, Bernard Widrow

The authors describe how a two-layer neural network can approximate any nonlinear function by forming a union of piecewise linear segments and a method is given for picking initial weights for the network to decrease training time.

IEEE Transactions on Knowledge and Data EngineeringGeneralization by neural networks

35 Citations1992Shashi Shekhar, M.B. Amin

A stochastic learning algorithm based on simulated annealing in weight space is presented and the authors verify the convergence properties and feasibility of the algorithm.

Neural NetworksExperience with selecting exemplars from clean data

33 Citations1996Mark Plutowski, Garrison W. Cottrell +1 more

This paper compares ΔISB to three other exemplar selection techniques on three time series prediction problems, the Mackey-Glass time series of dimension 2.1 and 3.5, the Rossler map, and a simplification of the technique called ‘maximum error’ performs nearly as well in most situations, although it is not as generally applicable.