A feature selection framework for text filtering

Published 23 April 2004

Zhaohui Zheng, Rohini K. Srihari, Sargur N. Srihari

DOI10.1109/icdm.2003.1251013

Citations25

TL;DR

This feature selection framework not only unifies several standard feature selection methods, but also facilitates the proposal of a new method that optimally combines the positive and negative sets.

Abstract

We present a new framework for local feature selection in text filtering. In this framework, a feature set is constructed per category by first selecting a set of terms highly indicative of membership (positive set) and another set of terms highly indicative of nonmembership (negative set), and then combining these two sets. This feature selection framework not only unifies several standard feature selection methods, but also facilitates the proposal of a new method that optimally combines the positive and negative sets. The experimental comparison between the proposed method and standard methods was conducted on six feature selection metrics: chi-square, correlation coefficient, odds ratio, GSS coefficient and two proposed variants of odds ratio and GSS coefficient: OR-square and GSS-square respectively. The results show that the proposed feature selection method improves text filtering performance.

Keywords

Computer Science

ACM Computing SurveysMachine learning in automated text categorization

7,899 Citations2002Fabrizio Sebastiani

This survey discusses the main approaches to text categorization that fall within the machine learning paradigm and discusses in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.

View PDF

A Comparative Study on Feature Selection in Text Categorization

4,766 Citations1997Yiming Yang, Jan Pedersen

DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.

Information RetrievalAn Evaluation of Statistical Approaches to Text Categorization

1,946 Citations1999Yiming Yang

Analysis and empirical evidence suggest that the evaluation results on some versions of Reuters were significantly affected by the inclusion of a large portion of unlabelled documents, mading those results difficult to interpret and leading to considerable confusions in the literature.

Feature selection, perception learning, and a usability case study for text categorization

446 Citations1997Hwee Tou Ng, Wei Boon Goh +1 more

An automated learning approach to text categorization based on perception learning and a new feature selection metric, called correlation coefficient, is described and empirical results indicate that this approach outperforms the best published results on this % uters collection.

View PDF

Lecture notes in computer scienceExperiments on the Use of Feature Selection and Negative Evidence in Automated Text Categorization

209 Citations2000Luigi Galavotti, Fabrizio Sebastiani +1 more

This work proposes a novel variant, based on the exploitation of negative evidence, of the well-known k-NN method, and reports the results of systematic experimentation of these two methods performed on the standard REUTERS-21578 benchmark.

Optimally Combining Positive and Negative Features for Text Categorization

54 Citations2003Zhaohui Zheng, Rohini K. Srihari

The results show that the proposed approach improves text categorization performance, and is in contrast with the standard local feature selection approaches that either only select the terms most indicative of membership or implicitly but not optimally combine the termsmost indicative of members with non-membership.