Distributional word clusters vs. words for text categorization
Published 1 March 2003
Ron Bekkerman, Ran El‐Yaniv, Naftali Tishby, Yoad Winter
Citations282
SJR quartileQ1
SJR score2.02
SNIP3.07
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
We study an approach to text categorization that combines distributional clustering of words and a Support Vector Machine (SVM) classifier. This word-cluster representation is computed using the recently introduced Information Bottleneck method, which generates a compact and efficient representation of documents. When combined with the classification power of the SVM, this method yields high performance in text categorization. This novel combination of SVM with word-cluster representation is compared with SVM-based categorization using the simpler bag-of-words (BOW) representation. The comparison is performed over three known datasets. On one of these datasets (the 20 Newsgroups) the method based on word clusters significantly outperforms the word-based representation in terms of categorization accuracy or representation efficiency. On the two other sets (Reuters-21578 and WebKB) the word-based representation slightly outperforms the word-cluster representation. We investigate the potential reasons for this behavior and relate it to structural differences between the datasets.
Keywords
Computer Science
Journal of the Royal Statistical Society Series B (Statistical Methodology)Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
49,657 Citations1977A. P. Dempster, N. M. Laird +1 more
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
Machine LearningSupport-Vector Networks
33,035 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
Data Mining and Knowledge DiscoveryA Tutorial on Support Vector Machines for Pattern Recognition
16,433 Citations1998Christopher J. C. Burges
There are several arguments which support the observed high accuracy of SVMs, which are reviewed and numerous examples and proofs of most of the key theorems are given.
Journal of the American Society for Information ScienceIndexing by latent semantic analysis
12,677 Citations1990Scott Deerwester, Susan Dumais +3 more
Pattern classification and scene analysis
12,643 Citations1973Richard O. Duda, Peter E. Hart
A training algorithm for optimal margin classifiers
11,594 Citations1992Bernhard E. Boser, Isabelle Guyon +1 more
A training algorithm that maximizes the margin between the training patterns and the decision boundary is presented, applicable to a wide variety of the classification functions, including Perceptrons, polynomials, and Radial Basis Functions.
Modern Information Retrieval
11,544 Citations1999Ricardo Baeza‐Yates, Berthier Ribeiro‐Neto
Foundations of statistical natural language processing
9,996 Citations1999Christopher D. Manning, Hinrich Schütze
The MIT Press eBooksLearning with Kernels
9,548 Citations2001Bernhard Schölkopf, Alexander J. Smola
Learning with Kernels provides an introduction to SVMs and related kernel methods that provide all of the concepts necessary to enable a reader equipped with some basic mathematical knowledge to enter the world of machine learning using theoretically well-founded yet easy-to-use kernel algorithms.
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
ACM Computing SurveysMachine learning in automated text categorization
7,899 Citations2002Fabrizio Sebastiani
This survey discusses the main approaches to text categorization that fall within the machine learning paradigm and discusses in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
Combining labeled and unlabeled data with co-training
5,604 Citations1998Avrim Blum, Tom M. Mitchell
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
ePrints Soton (University of Southampton)An Introduction to Support Vector Machines
4,120 Citations2000Nello Cristianini, John Shawe‐Taylor
Journal of Artificial Intelligence ResearchSolving Multiclass Learning Problems via Error-Correcting Output Codes
2,718 Citations1995Tom Dietterich, Ghulum Bakiri
It is demonstrated that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.
A re-examination of text categorization methods
2,660 Citations1999Yiming Yang, Xin Liu
The results show that SVM, kNN and LLSF signi cantly outperform NNet and NB when the number of positive training instances per category are small, and that all the methods perform comparably when the categories are over 300 instances.
Improved boosting algorithms using confidence-rated predictions
2,568 Citations1998Robert E. Schapire, Yoram Singer
Pattern Classification (2nd ed.)
2,513 Citations1999Richard O. Duda, Peter E. Hart +1 more
Machine LearningUnsupervised Learning by Probabilistic Latent Semantic Analysis
2,443 Citations2001Thomas Hofmann
This paper proposes to make use of a temperature controlled version of the Expectation Maximization algorithm for model fitting, which has shown excellent performance in practice, and results in a more principled approach with a solid foundation in statistical inference.
Machine LearningBoosTexter: A Boosting-based System for Text Categorization
2,194 Citations2000Robert E. Schapire, Yoram Singer
This work describes in detail an implementation, called BoosTexter, of the new boosting algorithms for text categorization tasks, and presents results comparing the performance of Boos Texter and a number of other text-categorization algorithms on a variety of tasks.
Inductive learning algorithms and representations for text categorization
1,465 Citations1998Susan Dumais, John Platt +2 more
A comparison of the effectiveness of five different automatic learning algorithms for text categorization in terms of learning speed, realtime classification speed, and classification accuracy is compared.
Toward optimal feature selection
1,444 Citations1996Daphne Koller, Mehran Sahami
An efficient algorithm for feature selection which computes an approximation to the optimal feature selection criterion is given, showing that the algorithm effectively handles datasets with a very large number of features.
A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization
1,264 Citations1997Thorsten Joachims
A Probabilistic analysis of the Rocchio relevance feedback algorithm, one of the most popular learning methods from information retrieval, is presented in a text categorization framework and suggests that the probabilistic algorithms are preferable to the heuristic Rocchio classifier.
Distributional clustering of English words
994 Citations1993Fernando Pereira, Naftali Tishby +1 more
The information bottleneck method
921 Citations2000Naftali Tishby, Fernando C. N. Pereira +1 more
The variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.
Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers
914 Citations2000Erin L. Allwein, Robert E. Schapire +1 more
A general method for combining the classifiers generated on the binary problems is proposed, and a general empirical multiclass loss bound is proved given the empirical loss of the individual binary learning algorithms.
Proceedings of the IEEEDeterministic annealing for clustering, compression, classification, regression, and related optimization problems
869 Citations1998Kenneth Rose
The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression.
Distributional clustering of words for text classification
683 Citations1998Lee D. Baker, Andrew Kachites McCallum
This paper describes the application of Distributional Clustering to document classi(cid:12)cation and shows that it can reduce the feature dimensionality by three orders of magnitude and lose only 2% accuracy, better than Latent Semantic In-dexing, class-based clustering, feature selection by mutual information, or Markov-blanket-based feature selection.
Learning to extract symbolic knowledge from the World Wide Web
675 Citations1998Mark Craven, Dan DiPasquo +5 more
The goal of the research described here is to automatically create a computer understandable world wide knowledge base whose content mirrors that of the World Wide Web, and several machine learning algorithms for this task are described.
Applied Physics Letters10.1162/153244302760200687
534 Citations2000
An approximation technique is introduced that is shown to deliver good approximations efficiently for large datasets and is shown to deliver good approximations efficiently for large datasets.
Bristol Research (University of Bristol)Text Classification using String Kernels
514 Citations2000Huma Lodhi, John Shawe‐Taylor +2 more
A novel kernel is introduced for comparing two text documents consisting of an inner product in the feature space consisting of all subsequences of length k, which can be efficiently evaluated by a dynamic programming technique.
Agglomerative Information Bottleneck
392 Citations1999Noam Slonim, Naftali Tishby
A novel distributional clustering algorithm that maximizes the mutual information per cluster between data and given categories and achieves compression by 3 orders of magnitudes loosing only 10% of the original mutual information.
Learning to classify text from labeled and unlabeled documents
330 Citations1998Kamal Nigam, Andrew McCallum +2 more
It is shown that the accuracy of text classifiers trained with a small number of labeled documents can be improved by augmenting this small training set with a large pool of unlabeled documents, and an algorithm is introduced based on the combination of Expectation-Maximization with a naive Bayes classifier.
Technische Universität Dortmund Eldorado (Technische Universität Dortmund)Estimating the generalization performance of a SVM efficiently
328 Citations1999Thorsten Joachims
Without any computation-intensive resampling, the new estimators developed here are computationally much more e cient than cross-validation or bootstrapping and address the special performancemeasures needed for evaluating text classi ers.
Unsupervised document classification using sequential information maximization
252 Citations2002Noam Slonim, Nir Friedman +1 more
A novel sequential clustering algorithm which is motivated by the Information Bottleneck method is presented, and it is found to be consistently superior to all the other clustering methods examined, typically by a significant margin.
A Statistical Learning Model of Text Classification for Support Vector Machines
238 Citations2001Thorsten Joachims
This model explains why and when SVMs perform well for text classification and connects the statistical properties of text-classification tasks with the generalization performance of a SVM in a quantitative way.
A statistical learning learning model of text classification for support vector machines
223 Citations2001Thorsten Joachims
IEEE Intelligent Systems and their ApplicationsMaximizing text-mining performance
211 Citations1999Sabine Weiß, Chid Apte +5 more
A learner-independent evaluation of the usefulness of statistical phrases for automated text categorization
180 Citations2001Maria Fernanda Caropreso, Stan Matwin +1 more
This work investigates the usefulness of n-grams for document indexing in text categorization (TC) independently of any specific learning algorithm, by applying feature selection to the pool of all k- grams, and checking how many n- Grams score high enough to be selected in the top σ k-rams.
Applied Physics Letters10.1162/153244302320884605
143 Citations2000
An empirical evaluation of round robin classification, implemented as a wrapper around the Ripper rule learning algorithm, on 20 multi-class datasets from the UCI database repository shows that the technique is very likely to improve Ripper's classification accuracy without having a high risk of decreasing it.
Machine LearningFeature Generation Using General Constructor Functions
142 Citations2002Shaul Markovitch, Dan Rosenstein
A generalized and flexible framework that is capable of generating features from any given set of constructor functions, and was applied to a variety of classification problems and was able to generate features that were strongly related to the underlying target concepts.
Econstor (Econstor)Estimating the Generalization Performance of a SVM Efficiently
125 Citations2001Thorsten Joachims
On feature distributional clustering for text categorization
124 Citations2001Ron Bekkerman, Ran El‐Yaniv +2 more
This work describes a text categorization approach that is based on a combination of feature distributional clusters with a support vector machine (SVM) classifier that yields high performance text classification that can outperform other recent methods in terms of categorization accuracy and representation efficiency.
Lecture notes in computer scienceExploiting Structural Information for Text Classification on the WWW
123 Citations1999Johannes Fürnkranz
Experimental evidence is presented that confirms the working hypothesis that it is often easier to classify a hypertext page using information provided on pages that point to it instead of using information that is provided on the page itself.
arXiv (Cornell University)Multivariate Information Bottleneck
118 Citations2013Nir Friedman, Ori Mosenzon +2 more
Extracting Relevant Structures with Side Information
97 Citations2002Gal Chechik, Naftali Tishby
This paper gives a general formulation of this problem and derives its formal, as well as algorithmic, solution and its operation is demonstrated in a synthetic example and in two real world problems in the context of text categorization and face images.
Improving multi-class text classification with Naive Bayes
87 Citations2001Jason D. M. Rennie
Lecture notes in computer scienceIterative Double Clustering for Unsupervised and Semi-supervised Learning
69 Citations2001Ran El‐Yaniv, Oren Souroujon
It is demonstrated that the IDC algorithm is especially advantageous when the data exhibits high attribute noise, and a natural extension of IDC for (semi-supervised) transductive learning where it is given both labeled and unlabeled examples is proposed.
Hypertext Categorization using Hyperlink Patterns and Meta Data
66 Citations2001Rayid Ghani, Seán Slattery +1 more
This investigation suggests that meta data which is often available, or can be acquired using Information Extraction techniques, can be extremely useful for improving classi cation accuracy.
Objective Classification of Galaxy Spectra Using the Information Bottleneck Method
60 Citations2000Slonim, N, Somerville, R S +2 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceAxiomatic approach to feature subset selection based on relevance
53 Citations1999Hui Wang, David Bell +1 more
This paper proposes two axioms for feature subset selection-sufficiency axiom and necessity axiom-based on which this link is formalized: the expected feature subset is the one which maximizes relevance and a heuristic algorithm to find the expected subset which has a polynomial time complexity.
Joining statistics with NLP for text categorization
44 Citations1992Paul S. Jacobs
The approach has been to use statistics in the knowledge acquisition component of a linguistic pattern-based categorization system, using statistical methods, for example, to associate words with industries and identify phrases that information about businesses or products.
Language sensitive text classification
31 Citations2000Roberto Basili, Alessandro Moschitti +1 more
An original classification model sensitive to document syntactic information and characterized by a novel inference method is described and the results show that a significant improvement can be derived using the proposed inference model.
Lecture notes in computer scienceSecond Order Features for Maximising Text Classification Performance
28 Citations2001Bhavani Raskutti, Herman Ferrá +1 more
The paper demonstrates that the addition of automatically selected word-pairs substantially increases the accuracy of text classification which is contrary to most previously reported research.