Second-Order Methods for Neural Networks: Fast and Reliable Training Methods for Multi-Layer Perceptrons

This chapter discusses second-order training methods for MLPs, which focus on reducing Storage and Computational Costs, and the development of the Hessian Matrix, which simplifies the training process.

Abstract

1 Multi-Layer Perceptron Training.- 1.1 Introduction to MLPs.- 1.1.1 The MLP Architecture.- 1.1.2 MLP Training.- 1.2 Error Surfaces and Local Minima.- 1.2.1 Error Surface Fundamentals.- 1.2.2 MLP Error Surfaces.- 1.3 Backpropagation.- 1.3.1 An Introduction to Backpropagation.- 1.3.2 The Bold Driver Method.- 1.3.3 Backpropagation with Momentum.- 1.3.4 On-Line Backpropagation.- 1.3.5 The Delta-Bar-Delta Method.- 2 Classical Optimisation.- 2.1 Introduction to Classical Methods.- 2.1.1 The Linear Model and Steepest Descent.- 2.1.2 The Quadratic Model and Newton's Method.- 2.1.3 Line-Search Methods vs. Model-Trust Region Methods.- 2.2 General Numerical Considerations.- 2.2.1 Finite-Precision Arithmetic and Computational Errors.- 2.2.2 Positive Definiteness and the Model Hessian.- 2.2.3 Scaling and Preconditioning.- 3 Second-Order Optimisation Methods.- 3.1 Line-Search Strategies.- 3.1.1 Line Minimisation Fundamentals.- 3.1.2 Inaccurate Line Searches.- 3.1.3 Backtracking Line Searches.- 3.2 Model-Trust Region Strategies.- 3.2.1 A Simple Model-Trust Region Algorithm.- 3.2.2 Fletcher's Method.- 3.2.3 Modern Model-Trust Region Algorithms.- 3.2.4 Moller's 'Scaled' Model-Trust Region Strategy.- 3.3 Multivariate Methods for General Nonlinear Optimisation.- 3.3.1 Finite-Difference Newton's Method.- 3.3.2 Quasi-Newton Methods.- 3.3.3 The 'Memoryless' Quasi-Newton Method.- 3.3.4 Conjugate Gradient Methods.- 3.4 Special Methods for Nonlinear Least Squares.- 3.4.1 The Gauss-Newton Method.- 3.4.2 The Levenberg-Marquardt Method.- 3.5 Comparison of Methods.- 4 Second-Order Training Methods for MLPs.- 4.1 The Calculation of Second Derivatives.- 4.1.1 Exact Evaluation of the Hessian Matrix.- 4.1.2 Exact Evaluation of the Hessian Times a Vector.- 4.1.3 Exact Evaluation of the Jacobian Matrix.- 4.2 Reducing Storage and Computational Costs.- 4.2.1 Diagonal Approximations of the Hessian Matrix.- 4.2.2 Reduced Function and Gradient Evaluations.- 4.3 Second-Order On-Line Training.- 4.3.1 An Introduction to Second-Order On-Line Training Strategies.- 4.3.2 'Noise-Free' On-Line Training Schemes.- 4.4 Conclusion.- 5 An Experimental Comparison of MLP Training Methods.- 5.1 Benchmark Training Tasks.- 5.1.1 N-Parity.- 5.1.2 The sin(x) Problem.- 5.1.3 The sin(x)cos(2x) Problem.- 5.1.4 The tan(x) Problem.- 5.2 Initial Training Conditions.- 5.2.1 MLP Architectures.- 5.2.2 Training Algorithms.- 5.2.3 Termination Conditions.- 5.3 Experimental Results.- 5.3.1 'Global Reliability'.- 5.3.2 Training Speed Metrics.- 5.3.3 Training Speed Results.- 5.3.4 Conclusions.- 6 Global Optimisation.- 6.1 Introduction to Global Methods.- 6.1.1 Stochastic Methods.- 6.1.2 Deterministic Methods.- 6.2 Expanded Range Approximation (ERA).- 6.2.1 An Introduction to ERA.- 6.2.2 The ERA Method in Practice.- 6.3 The TRUST Method.- 6.3.1 The Tunnelling Function.- 6.3.2 The Tunnelling Phase.- 6.3.3 An Evaluation of the TRUST Method.