Robust Feature Selection by Mutual Information Distributions
arXiv (Cornell University)Published 3 June 2002Open access
Marco Zaffalon, Marcus Hütter
Citations90
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirical value, one must consider sample-to-population inferential approaches. This paper deals with the distribution of mutual information, as obtained in a Bayesian framework by a second-order Dirichlet prior distribution. The exact analytical expression for the mean and an analytical approximation of the variance are reported. Asymptotic approximations of the distribution are proposed. The results are applied to the problem of selecting features for incremental learning and classification of the naive Bayes classifier. A fast, newly defined method is shown to outperform the traditional approach based on empirical mutual information on a number of real data sets. Finally, a theoretical development is reported that allows one to efficiently extend the above methods to incomplete samples in an easy and effective way.
Keywords
Computer ScienceMathematics
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
Journal of the American Statistical AssociationStatistical Analysis With Missing Data
17,489 Citations1989Maureen Lahiff, Roderick J. A. Little +1 more
Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference
16,927 Citations1988Judea Pearl
The author provides a coherent explication of probability as a language for reasoning with partial belief and offers a unifying perspective on other AI approaches to uncertainty, such as the Dempster-Shafer formalism, truth maintenance systems, and nonmonotonic logic.
Bayesian Data Analysis
13,688 Citations2003Andrew Gelman, John B. Carlin +2 more
Pattern classification and scene analysis
12,643 Citations1973Richard O. Duda, Peter E. Hart
Programs for Machine Learning
5,794 Citations1994Steven L. Salzberg, Alberto M. Segre
In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.
Artificial IntelligenceSelection of relevant features and examples in machine learning
3,295 Citations1997Avrim Blum, Pat Langley
This survey reviews work in machine learning on methods for handling data sets containing large amounts of irrelevant information and describes the advances that have been made in both empirical and theoretical work in this area.
Machine LearningOn the Optimality of the Simple Bayesian Classifier under Zero-One Loss
3,072 Citations1997Pedro Domingos, Michael J. Pazzani
The Bayesian classifier is shown to be optimal for learning conjunctions and disjunctions, even though they violate the independence assumption, and will often outperform more powerful classifiers for common training set sizes and numbers of attributes, even if its bias is a priori much less appropriate to the domain.
IEEE Transactions on Information TheoryApproximating discrete probability distributions with dependence trees
2,640 Citations1968Chee Lap Chow, C. Liu
It is shown that the procedure derived in this paper yields an approximation of a minimum difference in information when applied to empirical observations from an unknown distribution of tree dependence, and the procedure is the maximum-likelihood estimate of the distribution.
Intelligent Data AnalysisFeature selection for classification
2,602 Citations1997Manoranjan Dash, Hua Liu
Elsevier eBooksIrrelevant Features and the Subset Selection Problem
2,374 Citations1994George H. John, Ron Kohavi +1 more
A method for feature subset selection using cross-validation that is applicable to any induction algorithm is described, and experiments conducted with ID3 and C4.5 on artificial and real datasets are discussed.
Feature Selection for Knowledge Discovery and Data Mining
2,191 Citations1998Huan Liu, Hiroshi Motoda
Feature Selection for Knowledge Discovery and Data Mining offers an overview of the methods developed since the 1970's and provides a general framework in order to examine these methods and categorize them and suggests guidelines for how to use different methods under various circumstances.
Studies in computational intelligenceA Tutorial on Learning with Bayesian Networks
1,699 Citations2008David Heckerman
Methods for constructing Bayesian networks from prior knowledge are discussed and methods for using data to improve these models are summarized, including techniques for learning with incomplete data.
Toward optimal feature selection
1,444 Citations1996Daphne Koller, Mehran Sahami
An efficient algorithm for feature selection which computes an approximation to the optimal feature selection criterion is given, showing that the algorithm effectively handles datasets with a very large number of features.
Feature selection and feature extraction for text categorization
578 Citations1992David Lewis
The effect of selecting varying numbers and kinds of features for use in predicting category membership was investigated on the Reuters and MUC-3 text categorization data sets and the optimal feature set size for word-based indexing was found to be surprisingly low despite the large training sets.
arXiv (Cornell University)An evaluation of Naive Bayesian anti-spam filtering
527 Citations2000Ion Androutsopoulos, John Koutsias +3 more
It is reached that additional safety nets are needed for the Naive Bayesian anti-spam filter to be viable in practice.
IEEE Transactions on Knowledge and Data EngineeringA guide to the literature on learning probabilistic networks from data
477 Citations1996Wray Buntine
MLC++: a machine learning library in C++
197 Citations2002Ron Kohavi, George H. John +3 more
The problems M LC++ aims to solve, the design of MLC++, and the current functionality are discussed, including the attempt to extract commonalities of algorithms and decompose them for a unified view that is simple, coherent, and extensible.
Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topicsEstimating functions of probability distributions from a finite set of samples
195 Citations1995David H. Wolpert, David R. Wolf
This paper extends the analysis in other ways, for example by enlarging the class of potential priors beyond the uniform prior assumed in the first paper, and the use of the entropic and Dirichlet priors is considered.
Journal of the American Statistical AssociationPrincipal Information Theoretic Approaches
127 Citations2000Ehsan S. Soofi
ACM SIGKDD Explorations NewsletterKDD Cup 2001 report
110 Citations2002Jie Cheng, Christos Hatzis +5 more
This paper presents results and lessons from KDD Cup 2001, which involved three cutting-edge tasks related to drug design and genomics.
BiometricsTwo-Dimensional Contingency Tables with Both Completely and Partially Cross-Classified Data
102 Citations1974Tar Chen, Stephen E. Fienberg
The MIT Press eBooksDistribution of Mutual Information
59 Citations2002Marcus Hütter
The mutual information of two random variables i and j with joint probabilities {πij} is commonly used in learning Bayesian nets as well as in many other fields, and reliable and quickly computable approximations for p(I|n) are derived.
Soft ComputingThe posterior probability of Bayes nets with strong dependences
16 Citations1999Gernot D. Kleiter
This work determines the conditional posterior probabilities of structures given that the degree of dependence at each of their nodes exceeds a critical noise level in stochastic simulation of network structures.
International Symposium on Imprecise Probabilities and Their ApplicationsRobust discovery of tree-dependency structures
8 Citations2001Marco Zaffalon
Distribution of mutual information for robust feature selection
3 Citations2002Marcus Hütter, Marco Zaffalon
A fast, newly defined filter is shown to outperform the traditional approach based on empirical mutual information on a number of real data sets and allows the above methods to be extended to incomplete samples in an easy and effective way.