Combining Neural Network Regression Estimates with Regularized Linear Weights

An evaluation of the new approach, PCR*, based on principal components regression, reveals that it was the most robust combination method as the redundancy of the learned models increased, and redundancy could be handled without eliminating any of the learning models.

Abstract

When combining a set of learned models to form an improved estimator, the issue of redundancy or multicollinearity in the set of models must be addressed. A progression of existing approaches and their limitations with respect to the redundancy is discussed. A new approach, PCR*, based on principal components regression is proposed to address these limitations. An evaluation of the new approach on a collection of domains reveals that: 1) PCR* was the most robust combination method as the redundancy of the learned models increased, 2) redundancy could be handled without eliminating any of the learned models, and 3) the principal components of the learned models provided a continuum of "regularized" weights from which PCR* could choose. 1 Introduction Combining a set of learned models 1 to improve classification and regression estimates has been an area of much research in machine learning and neural networks ([Wolpert92, Merz95, PerroneCooper92, LeblancTibshirani93, Breiman92, Meir95...