Overfitting in Wrapper-Based Feature Subset Selection: The Harder You Try the Worse it Gets
Published 23 December 2007
John Loughrey, Pádraig Cunningham
Citations106
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work introduces a modified genetic algorithm called GAWES (Genetic Algorithm With Early Stopping) that reduces the level of overfitting and yields feature subsets that have a better generalization accuracy.
Abstract
In Wrapper based feature selection, the more states that are visited during the search phase of the algorithm the greater the likelihood of finding a feature subset that has a high internal accuracy while generalizing poorly. When this occurs, we say that the algorithm has overfitted to the training data. We outline a set of experiments to show this and we introduce a modified genetic algorithm to address this overfitting problem by stopping the search before overfitting occurs. This new algorithm called GAWES (Genetic Algorithm With Early Stopping) reduces the level of overfitting and yields feature subsets that have a better generalization accuracy.
Keywords
Computer Science
ScienceOptimization by Simulated Annealing
44,600 Citations1983Scott Kirkpatrick, C. D. Gelatt +1 more
A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems.
The MIT Press eBooksAn Introduction to Genetic Algorithms
11,203 Citations1996Melanie Mitchell
An Introduction to Genetic Algorithms focuses in depth on a small set of important and interesting topics -- particularly in machine learning, scientific modeling, and artificial life -- and reviews a broad span of research, including the work of Mitchell and her colleagues.
Artificial IntelligenceWrappers for feature subset selection
8,925 Citations1997Ron Kohavi, George H. John
The wrapper method searches for an optimal feature subset tailored to a particular algorithm and a domain and compares the wrapper approach to induction without feature subset selection and to Relief, a filter approach tofeature subset selection.
Programs for Machine Learning
5,794 Citations1994Steven L. Salzberg, Alberto M. Segre
In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.
CERN Document Server (European Organization for Nuclear Research)Fundamentals of neural networks: architectures, algorithms, and applications
3,046 Citations1994Laurene V. Fausett
IEEE Transactions on Pattern Analysis and Machine IntelligenceFeature selection: evaluation, application, and small sample performance
2,167 Citations1997Anil K. Jain, Douglas E. Zongker
This work studies the problem of choosing an optimal feature set for land use classification based on SAR satellite images using four different texture models and shows that pooling features derived from different texture Models, followed by a feature selection results in a substantial improvement in the classification accuracy.
Neural NetworksAutomatic early stopping using cross validation: quantifying the criteria
941 Citations1998Lutz Prechelt
To aid a more well-founded selection of the stopping criterion, 14 different automatic stopping criteria from three classes were evaluated empirically for their efficiency and effectiveness in 12 different classification and approximation tasks using multi-layer perceptrons with RPROP training.
Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping
869 Citations2000Rich Caruana, Steve Lawrence +1 more
It is shown that nets with excess capacity generalize well when trained with backprop and early stopping, and that conjugate gradient can yield worse generalization because it overfits regions of low non-linearity when learning to fit regions of high non- linearity.
Overfitting and neural networks: conjugate gradient and backpropagation
481 Citations2000Sandra Lawrence, C. Lee Giles
It is shown that overselection of the degrees of freedom for an MLP trained with backpropagation can improve the approximation in regions of underfitting, while not significantly overfitting in other regions, which can be a significant advantage over other models.
Lecture notes in statisticsA Comparative Evaluation of Sequential Feature Selection Algorithms
446 Citations1996David W. Aha, Richard L. Bankert
Positive empirical results are reported on variants of sequential feature selection that might be more appropriate for some performance tasks, and it is argued for their serious consideration in similar learning tasks.
Overfitting in making comparisons between variable selection methods
420 Citations2003Juha Reunanen
Feature Subset Selection Using a Genetic Algorithm
322 Citations1998Jihoon Yang, Vasant Honavar
Knowledge Discovery and Data MiningFeature subset selection using the wrapper method: overfltting and dynamic search space topology
262 Citations1995Ron Kohavi, Dan Sommerfield
Initial positive changes to health behaviours were observed post BC diagnosis, except healthy body weight, but maintenance of such changes over the long-term was poor, and BCSs may benefit from additional advice and support to make healthy lifestyle choices throughout survivorship.
Lecture notes in computer scienceDiversity versus Quality in Classification Ensembles Based on Feature Selection
206 Citations2000Pádraig Cunningham, John G. Carney
It is argued that feature subset-selection has emerged as a useful technique for creating diversity in ensembles and this diversity needs to be monitored in the creation of the ensemble, and an entropy measure of the outputs of the Ensemble Diversity Index is proposed.
Trinity's Access to Research Output (TARA) (Trinity College Dublin)Overfitting and Diversity in Classification Ensembles based on Feature Selection
32 Citations2000Pádraig Cunningham
This paper addresses Wrapper-like approaches to feature subset selection and the production of classifier ensembles based on members with different feature subsets with a tentative conclusion that diversity is not such a problem where a large number of features is involved but needs to be monitored for problems with smaller numbers of features.
Neural Information Processing SystemsKernel Regression and Backpropagation Training With Noise
25 Citations1991Petri Koistinen, Lasse Holmström