Adaptive Scaling for Feature Selection in SVMs

Published 1 January 2002

Yves Grandvalet, Stéphane Canu

Citations161

TL;DR

The resulting algorithm compares favorably to state-of-the-art feature selection procedures and demonstrates its effectiveness on a demanding facial expression recognition problem.

Abstract

This paper introduces an algorithm for the automatic relevance determi-nation of input variables in kernelized Support Vector Machines. Rele-vance is measured by scale factors defining the input space metric, and feature selection is performed by assigning zero weights to irrelevant variables. The metric is automatically tuned by the minimization of the standard SVM empirical risk, where scale factors are added to the usual set of parameters defining the classifier. Feature selection is achieved by constraints encouraging the sparsity of scale factors. The resulting algorithm compares favorably to state-of-the-art feature selection proce-dures and demonstrates its effectiveness on a demanding facial expres-sion recognition problem. 1

Keywords

Computer Science

The Nature of Statistical Learning Theory

39,279 Citations1995Vladimir Vapnik

The Elements of Statistical Learning: Data Mining, Inference, and Prediction

19,345 Citations2013Trevor Hastie, Robert Tibshirani +1 more

Lecture notes in statisticsBayesian Learning for Neural Networks

4,375 Citations1996Radford M. Neal

Bayesian Learning for Neural Networks shows that Bayesian methods allow complex neural network models to be used without fear of the "overfitting" that can occur with traditional neural network learning methods.

Machine LearningChoosing Multiple Parameters for Support Vector Machines

2,182 Citations2002Olivier Chapelle, Vladimir Vapnik +2 more

The problem of automatically tuning multiple parameters for pattern recognition Support Vector Machines (SVMs) is considered by minimizing some estimates of the generalization error of SVMs using a gradient descent algorithm over the set of parameters.

View PDF

The Annals of StatisticsHeuristics of instability and stabilization in model selection

1,149 Citations1996Leo Breiman

Feature Selection via Concave Minimization and Support Vector Machines

993 Citations1998Paul S. Bradley, O. L. Mangasarian

Numerical tests on 6 public data sets show that classi(cid:12)ers trained by the concave minimization approach and those trained by a support vector machine have comparable 10-fold cross-validation correctness.

Feature Selection for SVMs

977 Citations2000Jason Weston, Sayan Mukherjee +4 more

The resulting algorithms are shown to be superior to some standard feature selection algorithms on both toy data and real-life problems of face recognition, pedestrian detection and analyzing DNA microarray data.

Dynamically Adapting Kernels in Support Vector Machines

153 Citations1998Nello Cristianini, Colin Campbell +1 more

In this procedure model selection and learning are not separate, but kernels are dynamically adjusted during the learning process to find the kernel parameter which provides the best possible upper bound on the generalisation error.

arXiv (Cornell University)Feature Selection and Dualities in Maximum Entropy Discrimination

76 Citations2013Tony Jebara, Tommi Jaakkola

View PDF