A study on mutual information-based feature selectionfor text categorization

It is shown that the corrected MI method’s performance is similar to that of IG, and it is considerably better than PMI, and an MI method derived correctly from information theory is detail.

Abstract

Feature selection plays an important role in text categorization. Automatic feature selection methods such as document frequency thresholding (DF), gain (IG), (MI), and so on are commonly applied in text categorization. Many existing experiments show IG is one of the most effective methods, by contrast, MI has been demonstrated to have relatively poor performance. According to one existing MI method, the of a category c and a term t can be negative, which is in conflict with the definition of MI derived from theory where it is always non-negative. We show that the form of MI used in TC is not derived correctly from theory. There are two different MI based feature selection criteria which are referred to as MI in the TC literature. Actually, one of them should correctly be termed pointwise information (PMI). In this paper, we clarify the terminological confusion surrounding the notion of mutual information in TC, and detail an MI method derived correctly from theory. Experiments with the Reuters-21578 collection and OHSUMED collection show that the corrected MI method’s performance is similar to that of IG, and it is considerably better than PMI.