Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons

An adaptive method of directly obtaining the inverse of the Fisher information matrix is proposed and it generalizes the adaptive Gauss-Newton algorithms and provides a solid theoretical justification of them.

Abstract

The natural gradient learning method is known to have ideal performances for on-line training of multilayer perceptrons. It avoids plateaus, which give rise to slow convergence of the backpropagation method. It is Fisher efficient, whereas the conventional method is not. However, for implementing the method, it is necessary to calculate the Fisher information matrix and its inverse, which is practically very difficult. This article proposes an adaptive method of directly obtaining the inverse of the Fisher information matrix. It generalizes the adaptive Gauss-Newton algorithms and provides a solid theoretical justification of them. Simulations show that the proposed adaptive method works very well for realizing natural gradient learning.