Comparative analysis on margin based feature selection algorithms
International Journal of Machine Learning and CyberneticsPublished 11 April 2013
Wei Pan, Peijun Ma, Qinghua Hu, Xiaohong Su, Chaoqi Ma
Citations9
SJR quartileQ2
SJR score0.69
SNIP0.96
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work compares 14 margin based feature selections from the viewpoints of reduction capability, classification performance of reduced data and robustness, where four margin based loss functions and three search strategies are considered.
Abstract
Feature evaluation and selection is an important preprocessing step in classification and regression learning. As large quantity of irrelevant information is gathered, selecting the most informative features may help users to understand the task, and enhance the performance of the models. Margin has been widely accepted and used in evaluating feature quality these years. A collection of feature selection algorithms were developed using margin based loss functions and various search strategies. However, there is no comparative research conducted to study the effectiveness of these algorithms. In this work, we compare 14 margin based feature selections from the viewpoints of reduction capability, classification performance of reduced data and robustness, where four margin based loss functions and three search strategies are considered. Moreover, we also compare these techniques with two well-known margin based feature selection algorithms ReliefF and Simba. The derived conclusions give some guidelines for selecting features in practical applications.
Keywords
Computer Science
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
NeurocomputingAdvances in neural information processing systems 7
22,300 Citations1997Krzysztof J. Cios, Mark E. Shields
Data Mining and Knowledge DiscoveryA Tutorial on Support Vector Machines for Pattern Recognition
16,433 Citations1998Christopher J. C. Burges
There are several arguments which support the observed high accuracy of SVMs, which are reviewed and numerous examples and proofs of most of the key theorems are given.
NatureMolecular portraits of human breast tumours
16,304 Citations2000Charles M. Perou, Thérese Sørlie +16 more
Variation in gene expression patterns in a set of 65 surgical specimens of human breast tumours from 42 different individuals were characterized using complementary DNA microarrays representing 8,102 human genes, providing a distinctive molecular portrait of each tumour.
IEEE Transactions on Pattern Analysis and Machine IntelligenceFeature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy
10,470 Citations2005Hanchuan Peng, Fuhui Long +1 more
This work derives an equivalent form, called minimal-redundancy-maximal-relevance criterion (mRMR), for first-order incremental feature selection, and presents a two-stage feature selection algorithm by combining mRMR and other more sophisticated feature selectors (e.g., wrappers).
NatureDistinct types of diffuse large B-cell lymphoma identified by gene expression profiling
9,951 Citations2000Ash A. Alizadeh, Michael B. Eisen +29 more
It is shown that there is diversity in gene expression among the tumours of DLBCL patients, apparently reflecting the variation in tumour proliferation rate, host response and differentiation state of the tumour.
IEEE Transactions on Pattern Analysis and Machine IntelligenceRobust Face Recognition via Sparse Representation
9,444 Citations2009John Wright, A. Yang +3 more
This work considers the problem of automatically recognizing human faces from frontal views with varying expression and illumination, as well as occlusion and disguise, and proposes a general classification algorithm for (image-based) object recognition based on a sparse representation computed by C1-minimization.
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
The Annals of StatisticsAdditive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)
6,842 Citations2000Jerome H. Friedman, Trevor Hastie +1 more
This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.
International Conference on Neural Information ProcessingAdvances in kernel methods: support vector learning
5,815 Citations1999Bernhard Schölkopf, Christopher J. C. Burges +1 more
Support vector machines for dynamic reconstruction of a chaotic system, Klaus-Robert Muller et al pairwise classification and support vector machines, Ulrich Kressel.
Journal of Machine Learning ResearchDistance Metric Learning for Large Margin Nearest Neighbor Classification
4,253 Citations2009Kilian Q. Weinberger, Lawrence K. Saul
This paper shows how to learn a Mahalanobis distance metric for kNN classification from labeled examples in a globally integrated manner and finds that metrics trained in this way lead to significant improvements in kNN Classification.
Machine LearningOn the Optimality of the Simple Bayesian Classifier under Zero-One Loss
3,072 Citations1997Pedro Domingos, Michael J. Pazzani
The Bayesian classifier is shown to be optimal for learning conjunctions and disjunctions, even though they violate the independence assumption, and will often outperform more powerful classifiers for common training set sizes and numbers of attributes, even if its bias is a priori much less appropriate to the domain.
Lecture notes in computer scienceEstimating attributes: Analysis and extensions of RELIEF
2,950 Citations1994Igor Kononenko
In the context of machine learning from examples this paper deals with the problem of estimating the quality of attributes with and without dependencies among them and is analysed and extended to deal with noisy, incomplete, and multi-class data sets.
Nature MedicineClassification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks
2,648 Citations2001Javed Khan, Jun S. Wei +9 more
The ability of the trained ANN models to recognize SRBCTs is demonstrated, and the potential applications of these methods for tumor diagnosis and the identification of candidate targets for therapy are demonstrated.
Intelligent Data AnalysisFeature selection for classification
2,602 Citations1997Manoranjan Dash, Hua Liu
Nature MedicineGene-expression profiles predict survival of patients with lung adenocarcinoma
2,065 Citations2002David G. Beer, Sharon L. R. Kardia +15 more
The identification of a set of genes that predict survival in early-stage lung adenocarcinoma allows delineation of a high-risk group that may benefit from adjuvant therapy.
Pattern RecognitionA direct LDA algorithm for high-dimensional data — with application to face recognition
1,577 Citations2001Hua Yu, Jie Yang
However, for a task with very high dimensional data such as images, the traditional LDA algorithm encounters several difficulties, and before LDA can be used to reduce dimensionality, another procedure has to be first applied for dimensionality reduction.
Feature selection, <i>L</i><sub>1</sub> vs. <i>L</i><sub>2</sub> regularization, and rotational invariance
1,555 Citations2004Andrew Y. Ng
A lower-bound is given showing that any rotationally invariant algorithm---including logistic regression with L1 regularization, SVMs, and neural networks trained by backpropagation---has a worst case sample complexity that grows at least linearly in the number of irrelevant features.
Feature Selection for SVMs
977 Citations2000Jason Weston, Sayan Mukherjee +4 more
The resulting algorithms are shown to be superior to some standard feature selection algorithms on both toy data and real-life problems of face recognition, pedestrian detection and analyzing DNA microarray data.
IEEE Transactions on Pattern Analysis and Machine IntelligenceHybrid genetic algorithms for feature selection
925 Citations2004Il-Seok Oh, Jin-Seon Lee +1 more
Experiments revealed that the proposed hybrid GA is superior to both a simple GA and sequential search algorithms, and showed better convergence properties compared to the classical GAs.
Artificial IntelligenceConsistency-based search in feature selection
895 Citations2003Manoranjan Dash, Huan Liu
An empirical study is conducted to examine the pros and cons of these search methods, give some guidelines on choosing a search method, and compare the classifier error rates before and after feature selection.
Variable selection using svm based criteria
634 Citations2003Alain Rakotomamonjy
Computational Optimization and ApplicationsSSVM: A Smooth Support Vector Machine for Classification
516 Citations2001Yuh‐Jye Lee, O. L. Mangasarian
Smoothing methods are applied here to generate and solve an unconstrained smooth reformulation of the support vector machine for pattern classification using a completely arbitrary kernel, which converges globally and quadratically.
Margin based feature selection - theory and algorithms
416 Citations2004Ran Gilad-Bachrach, Amir Navot +1 more
This paper introduces a margin based feature selection criterion and applies it to measure the quality of sets of features and devise novel selection algorithms for multi-class classification problems and provide theoretical generalization bound.
Information RetrievalText Categorization Based on Regularized Linear Classification Methods
378 Citations2001Tong Zhang, Frank J. Oles
A number of known linear classification methods as well as some variants in the framework of regularized linear systems are compared to discuss the statistical and numerical properties of these algorithms, with a focus on text categorization.
Feature selection for clustering - a filter solution
378 Citations2003Manoranjan Dash, Keunho Choi +2 more
This paper proposes a 'filter' method that is independent of any clustering algorithm, based on the observation that data with clusters has a very different point-to-point distance histogram to that of data without clusters, and proposes an entropy measure that is low if data has distinct clusters and high if it does not.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOnline recognition of chinese characters: the state-of-the-art
351 Citations2004Cheng‐Lin Liu, Stefan Jaeger +1 more
This paper reviews the advances in online Chinese character recognition (OLCCR), with emphasis on the research works from the 1990s, in terms of pattern representation, character classification, learning/adaptation, and contextual processing.
IEEE Transactions on Pattern Analysis and Machine IntelligenceIterative RELIEF for Feature Weighting: Algorithms, Theories, and Applications
347 Citations2007Yijun Sun
Grafting: fast, incremental feature selection by gradient descent in function space
315 Citations2003Simon Perkins, Kevin Lacker +1 more
Grafting treats the selection of suitable features as an integral part of learning a predictor in a regularized learning framework, and operates in an incremental iterative fashion, gradually building up a feature set while training a predictor model using gradient descent.
Machine LearningAn Adaptive Version of the Boost by Majority Algorithm
269 Citations2001Yoav Freund
The paper describes two methods for finding approximate solutions to the differential equations and a method that results in a provably polynomial time algorithm based on the Newton-Raphson minimization procedure, which is much more efficient in practice but is not known to bePolynomial.
Stochastic gradient descent training for L1-regularized log-linear models with cumulative penalty
229 Citations2009Yoshimasa Tsuruoka, Jun’ichi Tsujii +1 more
Experimental results demonstrate that the proposed SGD method can produce compact and accurate models much more quickly than a state-of-the-art quasi-Newton method for L1-regularized loglinear models.
IEEE Transactions on Knowledge and Data EngineeringKernelized Fuzzy Rough Sets and Their Applications
211 Citations2011Qinghua Hu, Daren Yu +2 more
This study integrates kernel functions with fuzzy rough set models and proposes two types of kernelized fuzzy rough sets, and extends the measures existing in classical rough sets to evaluate the approximation quality and approximation abilities of the attributes.
IEEE Transactions on Fuzzy SystemsImproving Generalization of Fuzzy IF--THEN Rules by Maximizing Fuzzy Entropy
210 Citations2009Xizhao Wang, Chun-Ru Dong
A new rule-refinement scheme is proposed that is based on the maximization of fuzzy entropy on the training set, which is expected to have the advantages of improving the generalization capability of initial fuzzy IF-THEN rules and simultaneously overcoming the overfitting of refinement.
Margin Analysis of the LVQ Algorithm
203 Citations2002Koby Crammer, Ran Gilad-Bachrach +2 more
This paper presents margin based generalization bounds that suggest that prototypes based classifiers can be more accurate then the 1-NN rule and derived a training algorithm that selects a good set of prototypes using large margin principles.
IEEE Transactions on Knowledge and Data EngineeringMaximum Ambiguity-Based Sample Selection in Fuzzy Decision Tree Induction
172 Citations2011Xizhao Wang, Ling-Cai Dong +1 more
Experimental results show that generalization ability of the tree based on the selection mechanism is far more superior to that based on random selection mechanism and the adjustment of the fuzzy decision tree is minimized when adding selected samples to the training set.
On the Consistency of Feature Selection using Greedy Least Squares Regression
162 Citations2009Tong Zhang
Information SciencesInduction of multiple fuzzy decision trees based on rough set technique
159 Citations2008Xizhao Wang, Junhai Zhai +1 more
A numerical experiment on real data indicates that the proposed multiple tree induction is superior to the single tree induction based on the individual reduct or on the entire feature set for learning problems with many attributes.
Learning Sparse SVM for Feature Selection on Very High Dimensional Datasets
157 Citations2010Mingkui Tan, Li Wang +1 more
Comprehensive experimental results show that the proposed method can obtain better or competitive performance compared with existing SVM-based feature selection methods in term of sparsity and generalization performance, and can effectively handle large-scale and extremely high dimensional problems.
Journal of Process ControlFault detection and identification with a new feature selection based on mutual information
156 Citations2007Sylvain Verron, Téodor Tiplica +1 more
The application of the new fault diagnosis procedure on a benchmark problem, the Tennessee Eastman Process, shows better results than other well known published methods.
International Journal of Machine Learning and CyberneticsGenetic Algorithm-Neural Network (GANN): a study of neural network activation functions and depth of genetic algorithm search applied to feature selection
146 Citations2010Dong Tong, Robert Mintram
Results suggest that the hyperbolic tangent (tanh) activation function outperforms other common activation functions by extracting a smaller, but more significant feature set in the ANN architecture by extracting optimum feature selection capability.
Pattern RecognitionSparse ensembles using weighted combination methods based on linear programming
123 Citations2010Li Zhang, Weida Zhou
This paper proposes a framework of sparse ensembles and deals with new linear weighted combination methods for sparseEnsemble, which can be thought as implementing the structure risk minimization rule and naturally explains good performance of these methods.
NeurocomputingEffective feature selection scheme using mutual information
100 Citations2004Di Huang, Tommy W. S. Chow
The proposed feature selection scheme is able to effectively identify the salience features and is compared with the related study through applying to different classification problems in which the number of features ranged from less than 10 to over 12,600.
ON SURROGATE LOSS FUNCTIONS AND f-DIVERGENCES
98 Citations2008Xuanlong Nguyen, Martin J. Wainwright +1 more
An Efficient Greedy Method for Unsupervised Feature Selection
89 Citations2011Ahmed Farahat, Ali Ghodsi +1 more
A novel method for unsupervised feature selection, which efficiently selects features in a greedy manner and presents a novel algorithm for greedily minimizing the reconstruction error based on the features selected so far.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOn an Extended Fisher Criterion for Feature Selection
87 Citations1981Witold Malina
This correspondence considers the extraction of features as a task of linear transformation of an initial pattern space into a new space, optimal with respect to discriminating the data.
Gradient Feature Selection for Online Boosting
84 Citations2007Xiaoming Liu, Ting Yu
This paper proposes a gradient-based feature selection approach that iteratively updates each feature using the gradient descent, by minimizing the weighted least square error between the estimated feature response and the true label.
IEEE Transactions on Knowledge and Data EngineeringLarge Margin Feature Weighting Method via Linear Programming
77 Citations2009Bo Chen, Hongwei Liu +2 more
This paper considers feature selection method for multimodally distributed data, and presents a large margin feature weighting method for k-nearest neighbor (kNN) classifiers, which aims at separating different classes by large local margins and pulling closer together points from the same class.
International Journal of Machine Learning and CyberneticsNull space based feature selection method for gene expression data
74 Citations2011Alok Sharma, Seiya Imoto +2 more
A null space based feature selection method for gene expression data in terms of supervised classification that discards the redundant genes by applying the information of null space of scatter matrices is proposed.
Lecture notes in computer scienceAvoiding Boosting Overfitting by Removing Confusing Samples
72 Citations2007Alexander Vezhnevets, Olga Barinova
This paper shows that standard boosting algorithms are not appropriate in case of overlapping classes, and proposes an algorithm for removing confusing samples and experimentally study behavior of AdaBoost trained on the resulting data sets.
Neural Information Processing SystemsBoosting Algorithms for Maximizing the Soft Margin
68 Citations2007Gunnar Rätsch, Manfred K. Warmuth +1 more
This work presents a novel boosting algorithm, called SoftBoost, designed for sets of binary labeled examples that are not necessarily separable by convex combinations of base hypotheses, that produces a convex combination of hypotheses whose soft margin is within δ of its maximum.
The Annals of StatisticsOn surrogate loss functions and f-divergences
59 Citations2009XuanLong Nguyen, Martin J. Wainwright +1 more
This work presents conditions on loss functions such that empirical risk minimization yields Bayes consistency when both the discriminant function and the quantizer are estimated, and makes it possible to pick out the (strict) subset of surrogate loss functions that yield Bayes consistency for joint estimation of the discriminant function and the quantizer.
International Journal of Machine Learning and CyberneticsClassifying cognitive states of brain activity via one-class neural networks with feature selection by genetic algorithms
58 Citations2011Omer Boehm, David R. Hardoon +1 more
It is demonstrated how one-class recognition of cognitive brain functions across multiple subjects can be performed at the 90% level of accuracy via an appropriate choice of features which can be chosen automatically.
Pattern RecognitionFeature selection based on loss-margin of nearest neighbor classification
57 Citations2008Yun Li, Bao‐Liang Lu
A new feature selection criterion based on low-loss nearest neighbor classification and a novel feature selection algorithm that optimizes the margin of nearestNeighborhood classification through minimizing its loss function are proposed.
IEEE Transactions on Pattern Analysis and Machine IntelligenceSome Recent Results in Heuristic Search Theory
55 Citations1984Judea Pearl
The paper summarizes recent analytical investigations of the mathematical properties of heuristics and their influence on the performance of common search techniques and reports the optimality of A* and relations between the precision of the heuristic estimates and the average complexity of the search.
NeurocomputingNeighborhood based sample and feature selection for SVM classification learning
51 Citations2011Qiang He, Zongxia Xie +2 more
A set of experimental results show the model can select very few features and samples for training; in the mean time the classification performances are preserved or even improved.
Neural ComputationRobust Loss Functions for Boosting
46 Citations2007Takafumi Kanamori, Takashi Takenouchi +2 more
Numerical experiments illustrate that the proposed loss functions derived from the contamination models are useful for handling highly noisy data in comparison with other loss functions.
NeurocomputingLarge-margin nearest neighbor classifiers via sample weight learning
35 Citations2010Qinghua Hu, Pengfei Zhu +2 more
An approach to learning sample weights for enlarging margin by using a gradient descent algorithm to minimize margin based classification loss and consistently outperforms nearest neighbor classification and some other state-of-the-art methods.
NeurocomputingMargin maximization with feed-forward neural networks: a comparative study with SVM and AdaBoost
31 Citations2003Enrique Romero, Lluı́s Màrquez +1 more
A new learning model for FNN is proposed such that it tends to obtain the same solution as SVM, and the results obtained show that similar performance does not imply similar predictions, suggesting that di:erent models can be combined leading to better performance.
International Journal of Applied Earth Observation and GeoinformationMargin-based feature selection for hyperspectral data
31 Citations2009Mahesh Pal
It is suggested that margin-based feature selection algorithms provide a comparable performance to support vector machine-based recurring feature elimination approach.
The Annals of StatisticsAnalysis of boosting algorithms using the smooth margin function
30 Citations2007Cynthia Rudin, Robert E. Schapire +1 more
The smooth margin function is introduced, a differentiable approximation of the usual margin for boosting algorithms that is precisely bound the margin attained by AdaBoost when the edges of the weak classifiers fall within a specified range.
Margin distribution and learning algorithms
26 Citations2003Ashutosh Garg, Dan Roth
Computational Statistics & Data AnalysisStepwise feature selection using generalized logistic loss
17 Citations2008Changyi Park, Ja‐Yong Koo +2 more
A stepwise feature selection using the generalized logistic loss that is a smooth approximation of the usual hinge loss is proposed that can improve the quality of feature selection through standardization and retain similar predictive performance compared with the recursive feature elimination.