A discrete particle swarm optimization method for feature selection in binary classification problems
European Journal of Operational ResearchPublished 26 February 2010
Alper Ünler, Alper Murat
Citations434
SJR quartileQ1
SJR score2.24
SNIP2.62
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A modified discrete particle swarm optimization (PSO) algorithm is developed which dynamically accounts for the relevance and dependence of the features included the feature subset in an adaptive feature selection procedure.
Abstract
This paper investigates the feature subset selection problem for the binary classification problem using logistic regression model. We developed a modified discrete particle swarm optimization (PSO) algorithm for the feature subset selection problem. This approach embodies an adaptive feature selection procedure which dynamically accounts for the relevance and dependence of the features included the feature subset. We compare the proposed methodology with the tabu search and scatter search algorithms using publicly available datasets. The results show that the proposed discrete PSO algorithm is competitive in terms of both classification accuracy and computational performance.
Keywords
Computer Science
Particle swarm optimization
47,523 Citations2002James Kennedy, R.C. Eberhart
Statistical Comparisons of Classifiers over Multiple Data Sets
11,198 Citations2006Janez Demšar
A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
Artificial IntelligenceWrappers for feature subset selection
8,925 Citations1997Ron Kohavi, George H. John
The wrapper method searches for an optimal feature subset tailored to a particular algorithm and a domain and compares the wrapper approach to induction without feature subset selection and to Relief, a filter approach tofeature subset selection.
IEEE Transactions on Evolutionary ComputationThe particle swarm - explosion, stability, and convergence in a multidimensional complex space
8,768 Citations2002Maurice Clerc, James Kennedy
Journal of Machine Learning ResearchAn introduction to variable and feature selection
7,831 Citations2003GuyonIsabelle, ElisseeffAndré
An introduction to variable and feature selection
6,577 Citations2009Isabelle Guyon
A discrete binary version of the particle swarm algorithm
4,683 Citations2002James Kennedy, R.C. Eberhart
The paper reports a reworking of the particle swarm algorithm to operate on discrete binary variables, where trajectories are changes in the probability that a coordinate will take on a zero or one value.
Neural ComputationApproximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
3,654 Citations1998Thomas G. Dietterich
This article reviews five approximate statistical tests for determining whether one learning algorithm outperforms another on a particular learning task and measures the power (ability to detect algorithm differences when they do exist) of these tests.
Lecture notes in computer scienceParameter selection in particle swarm optimization
3,505 Citations1998Yuhui Shi, Russell C. Eberhart
This paper first analyzes the impact that inertia weight and maximum velocity have on the performance of the particle swarm optimizer, and then provides guidelines for selecting these two parameters.
Artificial IntelligenceSelection of relevant features and examples in machine learning
3,295 Citations1997Avrim Blum, Pat Langley
This survey reviews work in machine learning on methods for handling data sets containing large amounts of irrelevant information and describes the advances that have been made in both empirical and theoretical work in this area.
Pattern Recognition LettersFloating search methods in feature selection
3,070 Citations1994Pavel Pudil, Jana Novovičová +1 more
IEEE Transactions on Knowledge and Data EngineeringToward integrating feature selection algorithms for classification and clustering
2,680 Citations2005Huan Liu, Lei Yu
With the categorizing framework, the efforts toward-building an integrated system for intelligent feature selection are continued, and an illustrative example is presented to show how existing feature selection algorithms can be integrated into a meta algorithm that can take advantage of individual algorithms.
Intelligent Data AnalysisFeature selection for classification
2,602 Citations1997Manoranjan Dash, Hua Liu
Series in machine perception and artificial intelligenceIntroduction to Machine Learning
1,654 Citations2019
Introduction to Machine Learning is a comprehensive textbook on the subject, covering a broad array of topics not usually included in introductory machine learning texts, and discusses many methods from different fields, including statistics, pattern recognition, neural networks, artificial intelligence, signal processing, control, and data mining.
IEEE Intelligent Systems and their ApplicationsFeature subset selection using a genetic algorithm
1,363 Citations1998Jihoon Yang, Vasant Honavar
The authors' approach uses a genetic algorithm to select subsets of attributes or features to represent the patterns to be classified, achieving multicriteria optimization in terms of generalization accuracy and costs associated with the features.
IEEE Transactions on ComputersA Branch and Bound Algorithm for Feature Subset Selection
1,240 Citations1977Narendra, Fukunaga
A feature subset selection algorithm based on branch and bound techniques is developed to select the best subset of m features from an n-feature set with the computational effort of evaluating only 6000 subsets.
IEEE Transactions on Evolutionary ComputationDimensionality reduction using genetic algorithms
842 Citations2000Michael L. Raymer, William F. Punch +3 more
This work presents a new approach to feature extraction in which feature selection and extraction and classifier training are performed simultaneously using a genetic algorithm, and employs this technique in combination with the k nearest neighbor classification rule.
Pattern Recognition LettersFeature selection based on rough sets and particle swarm optimization
839 Citations2006Xiangyang Wang, Jie Yang +3 more
A new feature selection strategy based on rough sets and particle swarm optimization (PSO), which does not need complex operators such as crossover and mutation, and requires only primitive and simple mathematical operators, and is computationally inexpensive in terms of both memory and runtime.
IEEE Transactions on ComputersA Direct Method of Nonparametric Measurement Selection
831 Citations1971A.W. Whitney
A direct method of measurement selection is proposed to determine the best subset of d measurements out of a set of D total measurements, using a nonparametric estimate of the probability of error given a finite design sample set.
Data Mining and Knowledge DiscoveryOn Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach
830 Citations1997Steven L. Salzberg
Several phenomena that can, if ignored, invalidate an experimental comparison and the conclusions that follow apply not only to classification, but to computational experiments in almost any aspect of data mining.
Theoretical Computer ScienceOn the approximability of minimizing nonzero variables or unsatisfied relations in linear systems
759 Citations1998E. Amaldi, Viggo Kann
Strong bounds on the approximability of various variants of Min RVLS and min ULR are determined, including constrained ones where the variables are restricted to take binary values or where some relations are mandatory while others are optional.
Journal of the American Statistical AssociationFeature Extraction Construction and Selection: A Data Mining Perspective
722 Citations1999MTW, Huan Liu +1 more
This book can be used by researchers and graduate students in machine learning, data mining, and knowledge discovery, who wish to understand techniques of feature extraction, construction and selection for data pre-processing and to solve large size, real-world problems.
Academy of Management JournalVariations in Decision Makers' Use of Information Sources: The Impact of Quality and Accessibility of Information.
718 Citations1982Charles A. O’Reilly
Variations in the reported frequency of use of four information sources by decision makers was investigated, and the perceived quality of information available for decision making was related to this frequency.
Communications of the ACMMachine learning and data mining
675 Citations1999Tom M. Mitchell
The eld of data mining addresses the question of how best to use this historical data to discover general regularities and to improve future decisions.
IEEE Transactions on Information TheoryOn the effectiveness of receptors in recognition systems
655 Citations1963Thomas Marill, D. Green
Some of the theoretical problems encountered in trying to determine a more formal measure of the effectiveness of a set of tests are discussed; a measure which might be a practical substitute for the empirical evaluation.
Handbook of statistics39 Dimensionality and sample size considerations in pattern recognition practice
556 Citations1982A. Jain, B. Chandrasekaran
It is shown that as the number of samples increases, not only does the designer have more confidence in the performance of the classifier, but also more measurements can be incorporated in the design of the classify without the fear of peaking in its performance.
ePrints Soton (University of Southampton)Result Analysis of the NIPS 2003 Feature Selection Challenge
546 Citations2004Isabelle Guyon, Steve Gunn +2 more
The NIPS 2003 workshops included a feature selection competition organized by the authors, which took place over a period of 13 weeks and attracted 78 research groups and used a variety of methods for feature selection.
Overfitting in making comparisons between variable selection methods
420 Citations2003Juha Reunanen
Pattern RecognitionFeature selection using tabu search method
267 Citations2002Hongbin Zhang, Guangyu Sun
This paper uses tabu search to solve this feature selection problem and compares it with classic algorithms, such as sequential methods, branch and bound method, etc., and most other suboptimal methods proposed recently.
IEEE Transactions on Systems Man and CyberneticsOn the Possible Orderings in the Measurement Selection Problem
256 Citations1977Thomas M. Cover, Jan Van Campenhout
It is shown that for any ordering on the probability of error as a function of the subset of measurements (subject to an obvious set monotonicity condition), there exists a multivariate normal two-hypothesis problem N(K) versus N(¿¿,K) that exhibits this ordering.
Pattern Recognition LettersDifferent metaheuristic strategies to solve the feature selection problem
254 Citations2008Silvia Casado Yusta
This paper investigates feature subset selection for dimensionality reduction in machine learning and finds that GRASP and Tabu Search obtain significantly better results than the other methods.
IEEE Transactions on Pattern Analysis and Machine IntelligenceFast branch & bound algorithms for optimal feature selection
243 Citations2004Petr Somol, Pavel Pudil +1 more
A novel search principle for optimal feature subset selection using the branch & bound method using a simple mechanism for predicting criterion values is introduced and two implementations of the proposed prediction mechanism are proposed that are suitable for use with nonrecursive and recursive criterion forms.
A comparison of numerical optimizers for logistic regression
236 Citations2004Thomas P. Minka
This note compares eight different algorithms for computing the maximum a-posteriori parameter estimate and finds the fastest algorithms turn out to be conjugate gradient ascent and quasi-Newton algorithms, which far outstrip Iterative Scaling and its variants.
Decision Support SystemsA comparative assessment of classification methods
233 Citations2003Melody Y. Kiang
The objective of this research is to assess the relative performance of some well-known classification methods based on statistical and AI techniques and suggests that data characteristics considerably impact the classification performance of the methods.
European Journal of Operational ResearchOperations research and data mining
226 Citations2006Sigurður Ólafsson, Xiaonan Li +1 more
A survey of the intersection of operations research and data mining is provided to illustrate the range of interactions between the two fields, present some detailed examples of important research work, and provide comprehensive references to other important work in the area.
INFORMS journal on computingFeature Selection via Mathematical Programming
209 Citations1998Patricia Bradley, O. L. Mangasarian +1 more
Computational tests of three approaches to feature selection algorithm via concave minimization on publicly available real-world databases have been carried out and compared with an adaptation of the optimal brain damage method for reducing neural network complexity.
Journal of Machine Learning ResearchOverfitting in making comparisons between variable selection methods
208 Citations2003ReunanenJuha
This paper addresses a common methodological flaw in the comparison of variable selection methods by addressing the problem of cross-validation performance estimates of the different variable subsets used with computationally intensive search algorithms.
European Journal of Operational ResearchUsing simulated annealing to optimize the feature selection problem in marketing applications
193 Citations2004Ronen Meiri, Jacob Zahavi
The simulated annealing (SA) approach, which is one of the leading stochastic search methods, is employed for specifying a large-scale linear regression model and the results are compared to the results of the more common stepwise regression (SWR) approach for model specification.
Machine LearningIntroduction to machine learning
192 Citations2019Sandra Vieira, Walter Hugo Lopez Pinaya +1 more
European Journal of Operational ResearchSolving feature subset selection problem by a Parallel Scatter Search
187 Citations2004F. García López, Miguel García-Torres +3 more
These methods provide two sequential algorithms that are compared with a recent Genetic Algorithm and with a parallelization of the Scatter Search that presents better performance than the sequential algorithms.
Journal of Machine Learning ResearchParticle Swarm Model Selection
173 Citations2009Hugo Jair Escalante, Manuel Montes +1 more
Experimental results on benchmark data sets give evidence that the proposed approach to particle swarm optimization (PSO) for classification tasks is very effective, despite its simplicity, and results obtained in the framework of a model selection challenge show the competitiveness of the models selected with PSO, compared to modelsselected with other techniques that focus on a single algorithm and that use domain knowledge.
Communications of the ACMEvolving data into mining solutions for insights
151 Citations2002Usama M. Fayyad, Ramasamy Uthurusamy
The ability to capture and store data far outpaces the ability to process and exploit it, so the pace of innovation in this area needs to accelerate.
European Journal of Operational ResearchFramework for efficient feature selection in genetic algorithm based data mining
110 Citations2006Riyaz Sikora, Selwyn Piramuthu
This article presents a GA for doing both the tasks of mining and feature selection simultaneously by evolving a binary code along side the chromosome structure used for evolving the rules.
Computational Statistics & Data AnalysisApplications of optimization heuristics to estimation and modelling problems
95 Citations2004Peter Winker, Manfred Gilli
A brief introduction to the computational complexity of problems encountered in the fields of statistical modelling and econometrics as well as an overview and classification of the optimization heuristics used is provided.
Data Mining and Knowledge DiscoveryA Unifying View on Instance Selection
89 Citations2002Thomas Reinartz
This paper defines a broader perspective on focusing tasks, chooses instance selection as one particular focusing task, and outlines the specification of concrete evaluation criteria to measure success of instance selection approaches.
Journal of Intelligent Information SystemsFeature Subset Selection within a Simulated Annealing Data Mining Algorithm
86 Citations1997Justin C. W. Debuse, V. J. Rayward‐Smith
It is shown that a substantial reduction in the number of features may be achieved together with animation in the performance of the data mining system, and an entropic measure is developed, based upon the information gain approach used within ID3 and C4.5 to build trees.
BioinformaticsSupervised machine learning techniques for the classification of metabolic disorders in newborns
78 Citations2004Christian Baumgärtner, Christian Böhm +6 more
The results clearly prove that the mined data confirm the known and indicate some novel metabolic patterns which may contribute to a better understanding of newborn metabolism.
European Journal of Operational ResearchA variable selection method based on Tabu search for logistic regression models
75 Citations2008Joaquı́n Pacheco, Silvia Casado +1 more
A Tabu search method is proposed and analysed for selecting variables that are subsequently used in Logistic Regression Models and obtains significantly better results than the Stepwise, Backward or Forward methods used by classic statistical packages.
Comparison of population based metaheuristics for feature selection: Application to microarray data classification
69 Citations2008E.-G. Talbi, Laëtitia Jourdan +2 more
This work compares the use of a particle swarm optimization (PSO) and a genetic algorithm (GA) (both augmented with support vector machines SVM) for the classification of high dimensional microarray data and proves that PSOSVM is able to find interesting genes and to provide classification competitive performance.
A stochastic algorithm for feature selection in pattern recognition
60 Citations2007Sébastien Gadat, Laurent Younès +1 more
A new model addressing feature selection from a large dictionary of variables that can be computed from a signal or an image, using the probability as a state variable and optimizing a multi-task goodness of fit criterion for classifiers based on variable randomly chosen according to P is introduced.
Computers & Operations ResearchOptimization-based feature selection with adaptive instance sampling
56 Citations2005Jaekyung Yang, Sigurður Ólafsson
This paper focuses on the scalability of the nested partitions method and shows that its performance is vastly improved by incorporating random sampling of instances, and develops an adaptive variant of the algorithm that dynamically determines the required sample rate.
PubMedA genetic algorithm to select variables in logistic regression: example in the domain of myocardial infarction.
54 Citations1999Staal A. Vinterbo, Lucila Ohno‐Machado
A genetic algorithm was applied to search for the best variable combinations for modeling presence of myocardial infarction in a data set of patients with chest pain and the improvement in discriminatory ability yielded by the genetic algorithm variable selection method was statistically significant.
Computational Statistics & Data AnalysisAnalysis of new variable selection methods for discriminant analysis
41 Citations2006Joaquı́n Pacheco, Silvia Casado +2 more
Several methods to select variables that are subsequently used in discriminant analysis are proposed and analysed and it is found that they obtain significantly better results than the stepwise, backward or forward methods used by classic statistical packages.
INFORMS journal on computingIntelligent Partitioning for Feature Selection
37 Citations2005Sigurður Ólafsson, Jaekyung Yang
This paper develops a new optimization-based feature-selection framework for knowledge discovery in databases that has attractive theoretical properties such as proven convergence to an optimal set of relevant features and the ability for deriving rigorous statements regarding the quality of the set that is found.
IMA Journal of Management MathematicsUse of VNS and TS in classification: variable selection and determination of the linear discrimination function coefficients
16 Citations2007Joaquı́n Pacheco, Silvia Casado +1 more
TopOptimizing logistic regression coefficients for discrimination and calibration using estimation of distribution algorithms
13 Citations2008Vı́ctor Robles, Concha Bielza +3 more
This work presents a novel approach for fitting the logistic regression model based on estimation of distribution algorithms (EDAs), a tool for evolutionary computation from a double perspective: likelihood- based to calibrate the model and AUC-based to discriminate between the different classes.
A comparison of PSO and GA approaches for gene selection and classification of microarray data
13 Citations2007José García-Nieto, Enrique Alba +2 more
Find a subset of features F that maximizes a scoring function G such that F ′ = argmaxG⊂Γ{Θ(G)}, (1)
European Journal of Operational ResearchDEA based dimensionality reduction for classification problems satisfying strict non-satiety assumption
12 Citations2011Parag C. Pendharkar, Marvin D. Troutt
This study shows how data envelopment analysis (DEA) can be used to reduce vertical dimensionality of certain data mining databases and shows that the proposed heuristic approach outperforms the extended DEA-discriminant analysis MIP approach.