Exploiting Extremely Rare Features in Text Categorization
Lecture notes in computer sciencePublished 1 January 2006
Péter Schönhofen, András A. Benczúr
Citations7
SJR quartileQ2
SJR score0.35
SNIP0.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is found that while keeping the size of the training set constant, 5-25% of the test set can be classified essentially for free based on rare features without any loss of accuracy, even experiencing an improvement of 0.6-1.6%.
Abstract
One of the first steps of document classification, clustering and many other information retrieval tasks is to discard words occurring only a few times in the corpus, based on the assumption that they have little contribution to the bag of words representation. However, as we will show, rare n-grams and other similar features are able to indicate surprisingly well if two documents belong to the same category, and thus can aid classification. In our experiments over four corpora, we found that while keeping the size of the training set constant, 5-25% of the test set can be classified essentially for free based on rare features without any loss of accuracy, even experiencing an improvement of 0.6-1.6%.
Keywords
Computer Science
Journal of the Operational Research SocietyIntroduction to Algorithms
16,942 Citations1991V. J. Rayward‐Smith, Thomas H. Cormen +2 more
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
Communications of the ACMA vector space model for automatic indexing
7,434 Citations1975Gerard Salton, Anita M.-Y. Wong +1 more
An approach based on space density computations is used to choose an optimum indexing vocabulary for a collection of documents, demonstating the usefulness of the model.
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
Journal of DocumentationA STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
4,442 Citations1972Karen Spärck Jones
It is argued that terms should be weighted according to collection frequency, so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms.
Journal of Machine Learning ResearchRCV1: A New Benchmark Collection for Text Categorization Research
2,600 Citations2004David Lewis, Yiming Yang +2 more
This work describes the coding policy and quality control procedures used in producing the RCV1 data, the intended semantics of the hierarchical category taxonomies, and the corrections necessary to remove errorful data.
N-gram-based text categorization
1,503 Citations1994William B. Cavnar, John M. Trenkle
An N-gram-based approach to text categorization that is tolerant of textual errors is described, which worked very well for language classification and worked reasonably well for classifying articles from a number of different computer-oriented newsgroups according to subject.
IBM Journal of Research and DevelopmentA Statistical Approach to Mechanized Encoding and Searching of Literary Information
1,059 Citations1957H. P. Luhn
The problem of literature searching by machines still presents major difficulties and a statistical approach to this problem will be outlined and the various steps of a system based on this approach will be described.
Information Processing & ManagementRecent trends in hierarchic document clustering: A critical review
707 Citations1988Peter Willett
Algorithms that can be used to allow the implementation of hierarchic agglomerative clustering methods for document retrieval, and experimental evidence suggests that nearest neighbor clusters provide a reasonably efficient and effective means of including interdocument similarity information in document retrieval systems.
Discovering word senses from text
597 Citations2002Patrick Pantel, Dekang Lin
A clustering algorithm called CBC (Clustering By Committee) that automatically discovers word senses from text that initially discovers a set of tight clusters called committees that are well scattered in the similarity space.
Computer Speech & LanguageA bit of progress in language modeling
463 Citations2001Joshua Goodman
A combination of all techniques together to a Katz smoothed trigram model with no count cutoffs achieves perplexity reductions between 38 and 50% (1 bit of entropy), depending on training data size, as well as a word error rate reduction of 8.9%.
High-performing feature selection for text classification
304 Citations2002Monica Rogati, Yiming Yang
It is found that feature selection methods based on chi2 statistics consistently outperformed those based on other criteria (including information gain) for all four classifiers and both data collections, and that a further increase in performance was obtained by combining uncorrelated and high-performing feature selection Methods.
Lecture notes in computer scienceCombining Naive Bayes and n-Gram Language Models for Text Classification
170 Citations2003Fuchun Peng, Dale Schuurmans
This work augments the naive Bayes model with an n-gram language model to address two shortcomings of naive Baye text classifiers and shows that smoothing techniques from statistical language modeling can be used to recover better estimates than the Laplace smoothing technique usually used in naive Bayed classification.
Cluster-based text categorization
121 Citations1995Makoto Iwayama, Takenobu Tokunaga
The “efficiency, effectiveness, and noise tolerance of this search strategy were confirmed to be better than those of a full search, a category-based search, and a cluster- based search with nonprobabilistic clustering.
Computational LinguisticsExtracting the Lowest-Frequency Words: Pitfalls and Possibilities
37 Citations2000Marc Weeber, Rein Vos +1 more
An extraction system for the full word frequency range shows a recurrent, corpus-independent pattern in both recall and the number of significant words, and concludes that word-association-based extraction systems can be enhanced by also considering the lowest-frequency words.
Journal of the American Society for Information Science and TechnologyThe clustering power of low frequency words in academic Webs
19 Citations2005Liz Price, Mike Thelwall
Results for the Australian and New Zealand academic Web spaces indicate thatLow frequency words are useful for clustering academic Web sites along subject lines; removing low frequency words results in sites becoming, on average, less dissimilar to sites from other subjects.
Journal of the American Society for Information Science and TechnologyNon‐word identification or spell checking without a dictionary
12 Citations2003Donald C. Comeau, W. John Wilbur
An internal corpus based context quality rating α, frequency, and simple misspelling transformations are used to rank words from most likely to be misspellings to least likely, and examine the time history of low α words in MEDLINE® titles and abstracts.
IEEE/SP 13th Workshop on Statistical Signal Processing, 2005Inference for probabilistic unsupervised text clustering
8 Citations2005Loïs Rigouste, Olivier Cappé +1 more
This article investigates the use of a simple probabilistic model for unsupervised document clustering in large collections of texts, one of the most efficient method being based on vocabulary reduction and comparing those heuristics with other inference processes, such as Gibbs sampling.
Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004.Vocabulary spectral analysis as an exploratory tool for scientific Web intelligence
4 Citations2004Mike Thelwall
Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004.Vocabulary spectral analysis as an exploratory tool for scientific web intelligence
3 Citations2004Mike Thelwall
An exploratory technique, vocabulary spectral analysis, designed to build intuition to help design effective procedures for clustering is introduced and applied to New Zealand University Web sites, suggesting that subject-based academic Web site clusters are possible but requires extensive data filtering.