Optimization of the Backpropagation Algorithm for Training Multilayer Perceptrons

Learning rate adaptation for each training pattern 12 and nearly optimal learning rate adjust using line search 15 5.6.1 Polak–Ribiere method and line search 17 5.4 Evolutionarily adapted learning rate 12 5.5 Global learning rate adaptation 8 5.1 Fixed calculating of the learning rate.

Abstract

this report also have been published on ESANN '93 [Schiffmann et al., 1993]. The dataset used in this comparision is available by anonymous ftp (FTP server: ics.uci.edu, files: pub/machine-learningdatabases /thyroid-disease/ann). Backpropagation is one of the most popular training algorithms for multilayer perceptrons. Unfortunately it can be very slow for practical applications. Over the last years many improvement strategies have been developed to speed up backpropagation. It's very difficult to compare these different techniques, because most of them have been tested on very special data sets. The reported results are based on some kind of tiny and artificial training sets like XOR, encoder or decoder. It's very doubtful if this results hold for a much more complicate practical application. In these report an overview of many different speedup techniques is given. All of them are tested on a very hard practical classification task, which consists of a big medical data set. As you will see many of these optimized algorithms fail in learning the data set. 2 Application