Feature Selection in Mixture-Based Clustering
Published 1 January 2002
M.H.C. Law, Anil K. Jain, Mário A. T. Figueiredo
Citations81
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Two approaches to feature selection in the context of Gaussian mixture-based clustering are proposed, one of which wraps mixture estimation in an outer layer that performs feature selection and the other extends Koller and Sahami's mutual-information-based feature relevance criterion to the unsupervised case.
Abstract
While there exist many approaches to clustering, the important issue of feature selection, that is, what attributes of the data are relevant, is rarely addressed. Feature selection for clustering is made difficult by the absence of class labels to guide the search. In this paper, we propose two approaches to deal with this problem. In the first one, instead of making hard selections, we estimate how salient each features is. An expectation-maximization (EM) algorithm is derived for this task. The second approach extends Koller and Sahami's mutual-information-based feature relevance criterion to the unsupervised case. Implementation is carried out by a backward search scheme. The resulting algorithm can be classified as a "wrapper", since it wraps mixture estimation in an outer layer that performs feature selection. Experimental results on synthetic and real data show that both methods have promising performance. 1
Keywords
Computer Science
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
Artificial IntelligenceWrappers for feature subset selection
8,925 Citations1997Ron Kohavi, George H. John
The wrapper method searches for an optimal feature subset tailored to a particular algorithm and a domain and compares the wrapper approach to induction without feature subset selection and to Relief, a filter approach tofeature subset selection.
IEEE Transactions on Pattern Analysis and Machine IntelligenceUnsupervised learning of finite mixture models
2,110 Citations2002Mário A. T. Figueiredo, Anil K. Jain
The novelty of the approach is that it does not use a model selection criterion to choose one among a set of preestimated candidate models; instead, it seamlessly integrate estimation and model selection in a single algorithm.
Queensland's institutional digital repository (The University of Queensland)Mixture models : inference and applications to clustering
2,074 Citations1988Geoffrey J. McLachlan, K. E. Basford
IEEE Transactions on Pattern Analysis and Machine IntelligenceUnsupervised feature selection using feature similarity
1,460 Citations2002Pabitra Mitra, C.A. Murthy +1 more
An unsupervised feature selection algorithm suitable for data sets, large in both dimension and size, based on measuring similarity between features whereby redundancy therein is removed, which does not need any search and is fast.
Toward optimal feature selection
1,444 Citations1996Daphne Koller, Mehran Sahami
An efficient algorithm for feature selection which computes an approximation to the optimal feature selection criterion is given, showing that the algorithm effectively handles datasets with a very large number of features.
Feature selection for high-dimensional genomic microarray data
629 Citations2001Eric P. Xing, Michael I. Jordan +1 more
This paper reports on the successful application of feature selection methods to a classification problem in molecular biology involving only 72 data points in a 7130 dimensional space and investigates regularization methods as an alternative to feature selection.
IEEE Transactions on Pattern Analysis and Machine IntelligenceA Problem of Dimensionality: A Simple Example
427 Citations1979G.V. Trunk
In pattern recognition problems it has been noted that beyond a certain point the inclusion of additional parameters (that have been estimated) leads to higher probabilities of error, so the probability of error approaches one-half as the dimensionality increases and parameters are estimated.
Machine LearningFeature Weighting in k-Means Clustering
371 Citations2003Dharmendra S. Modha, W. Scott Spangler
An abstract framework for integrating multiple feature spaces in the k-means clustering algorithm is presented and the effectiveness of feature weighting in clustering on several different application domains is demonstrated.
IEEE Transactions on Pattern Analysis and Machine IntelligenceInformation theoretic clustering
291 Citations2002Erhan Gökçay, José C. Prı́ncipe
A novel valley seeking clustering algorithm using an information theoretic measure to estimate the cost of partitioning the data set and it performs well, and was able to find nonlinear boundaries between clusters.
Feature selection in unsupervised learning via evolutionary search
290 Citations2000YeongSeog Kim, W. Nick Street +1 more
ELSA is used, an evolutionary local selection algorithm that maintains a diverse population of solutions that approximate the Pareto front in a multidimensional objectiv espace and shows promise in identifying the right features and the correct number of clusters.
Lecture notes in computer scienceFeature Selection for Clustering
257 Citations2000Manoranjan Dash, Huan Liu
Feature Subset Selection and Order Identification for Unsupervised Learning
236 Citations2000Jennifer Dy, Carla E. Brodley
This paper examines feature subset selection wrapped around expectation-maximization (EM) clustering with order identi cation (identifying the number of clusters in the data).
Statistics and ComputingMML clustering of multi-state, Poisson, von Mises circular and Gaussian distributions
166 Citations2000Chris S. Wallace, David L. Dowe
This work outlines how MML is used for statistical parameter estimation, and how the MML mixture modelling program, Snob, uses the message lengths from various parameter estimates to enable it to combine parameter estimation with selection of the number of components and estimation of the relative abundances of the components.
Journal of Computational and Graphical StatisticsA Component-Wise EM Algorithm for Mixtures
142 Citations2001Gilles Celeux, Stéphane Chrétien +2 more
This article proposes a component-wise EM for mixtures that uses, at each iteration, the smallest admissible missing data space by intrinsically decoupling the parameter updates, and proves that the mixing proportions will satisfy this constraint upon convergence.
Efficient Feature Selection in Conceptual Clustering
116 Citations1997Mark Devaney, Ashwin Ram
An implementation of a sequential feature selection algorithm based on an existing conceptual clustering system is described and an implementation which employs a technique for improving the efficiency of the search for an optimal description is presented.
IEEE Transactions on Pattern Analysis and Machine IntelligenceMinimum-entropy data partitioning using reversible jump Markov chain Monte Carlo
61 Citations2001Stephen Roberts, Chris Holmes +1 more
It is shown that an efficient partitioning may be given via a minimization of partition entropy and a reversible-jump sampling is introduced to explore the variable-dimension space of partition models.
Intelligent Data AnalysisDependency-based feature selection for clustering symbolic data
59 Citations2000Luis Talavera
This paper introduces an unsupervised feature selection method for symbolic clustering tasks based upon the assumption that, in the absence of class labels, those features that exhibit low dependencies with the rest of features are deemed as irrelevant.
Clustering very large databases using EM mixture models
57 Citations2002Patricia Bradley, Usama M. Fayyad +1 more
This work presents a decomposition of the EM algorithm requiring a small amount of memory by limiting iterations to small data subsets, and demonstrates that the scalable scheme outperforms similarly constrained EM approaches.
Generalized Model Selection for Unsupervised Learning in High Dimensions
44 Citations1999Shivakumar Vaithyanathan, Byron Dom
A Bayesian approach to model selection in unsupervised learning that determines both the feature set and the number of clusters is described and a scheme (based on marginal likelihood) and one based on cross-validated likelihood are evaluated.