Speeding up Backpropagation

This paper presents an improvement to the backpropagation algorithm, based on the use of an independent, adaptive learning rate parameter for each weight, and gives experimental results showing the improvement in learning speed.

Abstract

Backpropagation is probably the best known, and most widely used learning algorithm for neural networks. It is a gradient based optimization procedure, and it suffers from the common limitations of this kind of algorithms, namely in what concerns convergence speed. In this paper, we present an improvement to the backpropagation algorithm, based on the use of an independent, adaptive learning rate parameter for each weight. We discuss the reasons for using this modification to the basic algorithm, and we give experimental results showing the improvement in learning speed obtained with this method.