When networks disagree: Ensemble methods for hybrid neural networks

World Scientific series in 20th century physicsPublished 1 September 1995

Michael Perrone, Leon N. Cooper

Citations764

TL;DR

Experimental results show that the ensemble method dramatically improves neural network performance on difficult real-world optical character recognition tasks.

Abstract

This paper presents a general theoretical framework for ensemble methods of constructing significantly improved regression estimates. Given a population of regression estimators, we construct a hybrid estimator which is as good or better in the MSE sense than any estimator in the population. We argue that the ensemble method presented has several properties: 1) It efficiently uses all the networks of a population - none of the networks need be discarded. 2) It efficiently uses all the available data for training without over-fitting. 3) It inherently performs regularization by smoothing in functional space which helps to avoid over-fitting. 4) It utilizes local minima to construct improved estimates whereas other neural network algorithms are hindered by local minima. 5) It is ideally suited for parallel computation. 6) It leads to a very useful and natural measure of the number of distinct estimators in a population. 7) The optimal parameters of the ensemble estimator are given in clo...

Keywords

Computer Science

Society for Industrial and Applied Mathematics eBooksThe Jackknife, the Bootstrap and Other Resampling Plans

7,903 Citations1982Bradley Efron

Journal of the American Statistical AssociationSpline Models for Observational Data.

5,025 Citations1991Hans‐Georg Müller, Grace Wahba

Neural ComputationAdaptive Mixtures of Local Experts

4,988 Citations1991Robert A. Jacobs, Michael I. Jordan +2 more

A new supervised learning procedure for systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases, which is demonstrated to be able to be solved by a very simple expert network.

IEEE Transactions on Pattern Analysis and Machine IntelligenceNeural network ensembles

4,269 Citations1990Lars Kai Hansen, Peter Salamon

It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks, which helps improve the performance and training of neural networks for classification.

View PDF

Choice Reviews OnlineApplied nonparametric regression

2,613 Citations1991

Applied Nonparametric Regression is the first book to bring together in one place the techniques for regression curve smoothing involving more than one variable and argues that all smoothing methods are based on a local averaging mechanism and can be seen as essentially equivalent to kernel smoothing.

IEEE Transactions on Systems Man and CyberneticsMethods of combining multiple classifiers and their applications to handwriting recognition

2,171 Citations1992Longchang Xu, Adam Krzyżak +1 more

On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly.

BiometrikaThe jackknife-a review

1,383 Citations1974Rupert G. Miller

Neural Information Processing SystemsBayesian Learning via Stochastic Dynamics

184 Citations1992Radford M. Neal

Bayesian methods avoid overfitting and poor generalization by averaging the outputs of many networks with weights sampled from the posterior distribution given the training data, by simulating a stochastic dynamical system that has the posterior as its stationary distribution.

Synergy of Clustering Multiple Back Propagation Networks

96 Citations1989William Lincoln, Josef Skrzypek

The properties of a cluster of multiple back-propagation (BP) networks are examined and compared to the performance of a single BP network, and it appears that cluster advantage follows from simple maxim "you can fool some of the single BP's in a cluster all of the time but you cannot fool all of them all ofThe time".

RecNorm: Simultaneous Normalisation and Classification applied to Speech Recognition

45 Citations1990John S. Bridle, Stephen Cox

A method of training this network to "tune in" the speaker parameters to a particular speaker based on a trick for converting a supervised network to an unsupervised mode is outlined, indicating an improvement over speaker-independent performance and, for unlabelled data, a performance close to that achieved on labelled data.