Feature Selection on Chinese Text Classification Using Character N-Grams

Lecture notes in computer sciencePublished 12 May 2008

Zhihua Wei, Duoqian Miao, Jean-Hugues Chauchat, Caiming Zhong

DOI10.1007/978-3-540-79721-0_68

Source URL

Citations20

SJR quartileQ2

SJR score0.35

SNIP0.55

TL;DR

It is found out that the feature selection methods based on n-gram frequency (absolute or relative) always give better results and produce denser matrices.

Abstract

In this paper, we perform Chinese text classification using n-gram text representation on TanCorp which is a new large corpus special for Chinese text classification more than 14,000 texts divided into 12 classes. We use different n-gram feature (1-, 2-grams or 1-, 2-, 3-grams) to represent documents. Different feature weights (absolute text frequency, relative text frequency, absolute n-gram frequency and relative n-gram frequency) are compared. The sparseness of "document by feature" matrices is analyzed in various cases. We use the C-SVC classifier which is the SVM algorithm designed for the multi-classification task. We perform our experiments in the TANAGRA platform. We found out that the feature selection methods based on n-gram frequency (absolute or relative) always give better results and produce denser matrices.

Keywords

Computer Science

ACM Computing SurveysMachine learning in automated text categorization

7,899 Citations2002Fabrizio Sebastiani

This survey discusses the main approaches to text categorization that fall within the machine learning paradigm and discusses in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.

View PDF

Journal of Machine Learning ResearchWorking Set Selection Using Second Order Information for Training Support Vector Machines

1,413 Citations2005Rong-En Fan, Pai-Hsuen Chen +1 more

A new technique for working set selection in SMO-type decomposition methods that uses second order information to achieve fast convergence andoretical properties such as linear convergence are established.

Learning to Classify Text Using Support Vector Machines

1,366 Citations2002Thorsten Joachims

This book gives a concise introduction to SVMs for pattern recognition, and it includes a detailed description of how to formulate text-classification tasks for machine learning.

Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms

934 Citations2002Thorsten Joachims

EGC eBooksTANAGRA : un logiciel gratuit pour l'enseignement et la recherche.

134 Citations2005Ricco Rakotomalala

A novel refinement approach for text categorization

106 Citations2005Songbo Tan, Xueqi Cheng +3 more

The results indicate that in each case, the refined classifiers achieve significant performance improvement over the base classifiers used, and the performance of the Refined Centroid Classifier implemented is comparable, if not better, to that of state-of-the-art support vector machine (SVM)-based classifier, but offers a much lower computational cost.

A CHINESE DOCUMENT CATEGORIZATION SYSTEM WITHOUT DICTIONARY SUPPORT AND SEGMENTATION PROCESSING

7 Citations2001Zhou Shui

A Chinese document categorization system without dictionary support and segmentation processing is developed, in which the N gram information instead of Chinese words is used so that the classifier can shake off the support of dictionaries and segmentations processing.