Comparison of Gradient Methods for the Solution of Nonlinear Parameter Estimation Problems

Modifications of the Gauss method (including Marquardt’s) performed best, followed by variable metric rank one and Davidon–Fletcher–Powell methods, in that order.

Abstract

The performance of several of the best known gradient methods is compared in the solution of some least squares, maximum likelihood, and Bayesian estimation problems. Modifications of the Gauss method (including Marquardt's) performed best, followed by variable metric rank one and Davidon–Fletcher–Powell methods, in that order. There appears to be no need to locate the optimum precisely in the one-dimensional searches. The matrix inversion method used with the Gauss algorithm must guarantee a positive definite inverse.