Model Selection and Assessment Using Cross-indexing

IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural NetworksPublished 1 August 2007

Juha Reunanen

DOI10.1109/ijcnn.2007.4371365

Source URL

Citations12

TL;DR

A generalization is introduced that covers the previously presented variations of cross-indexing A and B as special cases and reports the promising results of using the consequent method in three open competitions.

Abstract

Cross-indexing is a method for selecting the optimal model complexity and estimating the corresponding performance. It aims to reduce the optimistic selection bias that may emerge when too many models are compared to each other using cross-validation as the performance estimator. In this paper, a generalization is introduced that covers the previously presented variations (cross-indexing A and B) as special cases. Originally, cross-indexing was suggested for decreasing the selection bias in a feature selection setting. In order to apply it to a generic model selection problem that presents a large number of candidate model structures and an infinite number of potential hyperparameter values, the method needs to be modified. This paper also describes one way of doing such modifications, and reports the promising results of using the consequent method in three open competitions.

Keywords

Computer Science

Machine LearningRandom Forests

126,192 Citations2001Leo Breiman

Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.

View PDF

ScienceOptimization by Simulated Annealing

44,600 Citations1983Scott Kirkpatrick, C. D. Gelatt +1 more

A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems.

Cambridge University Press eBooksAn Introduction to Support Vector Machines and Other Kernel-based Learning Methods

13,883 Citations2000Nello Cristianini, John Shawe‐Taylor

A training algorithm for optimal margin classifiers

11,594 Citations1992Bernhard E. Boser, Isabelle Guyon +1 more

A training algorithm that maximizes the margin between the training patterns and the decision boundary is presented, applicable to a wide variety of the classification functions, including Perceptrons, polynomials, and Radial Basis Functions.

Proceedings of the National Academy of SciencesSelection bias in gene extraction on the basis of microarray gene-expression data

1,468 Citations2002Christophe Ambroise, Geoffrey J. McLachlan

It is demonstrated that when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes.

View PDF

Overfitting in making comparisons between variable selection methods

420 Citations2003Juha Reunanen

Machine LearningMultiple Comparisons in Induction Algorithms

243 Citations2000David Jensen, Paul R. Cohen

This work analyzes the statistical properties of MCPs and shows how failure to adjust for these properties leads to the pathologies of induction algorithms, including attribute selection errors, overfitting, and oversearching.

View PDF

Journal of Machine Learning ResearchOverfitting in making comparisons between variable selection methods

208 Citations2003ReunanenJuha

This paper addresses a common methodological flaw in the comparison of variable selection methods by addressing the problem of cross-validation performance estimates of the different variable subsets used with computationally intensive search algorithms.

The 2006 IEEE International Joint Conference on Neural Network ProceedingsPerformance Prediction Challenge

58 Citations2006Isabelle Guyon, Amir Reza Saffari Azar Alamdari +2 more

Over 100 participants have been trying to build the best possible classifier from training data and guess their generalization error on a large unlabeled test set and the challenge scores indicate that cross-validation yields good results both for model selection and performance prediction.

A Pitfall in Determining the Optimal Feature Subset Size

17 Citations2004Juha Reunanen