An information-theoretic perspective of tf–idf measures
Information Processing & ManagementPublished 10 December 2002
Akiko Aizawa
Citations1,376
SJR quartileQ1
SJR score2.06
SNIP2.91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The proposed PWI is expressed as a product of the occurrence probabilities of terms and their amounts of information, and corresponds well with the conventional term frequency-inverse document frequency measures that are commonly used in today's information retrieval systems.
Abstract
This paper presents a mathematical definition of the “probability-weighted amount of information” (PWI), a measure of specificity of terms in documents that is based on an information-theoretic view of retrieval events. The proposed PWI is expressed as a product of the occurrence probabilities of terms and their amounts of information, and corresponds well with the conventional term frequency–inverse document frequency measures that are commonly used in today’s information retrieval systems. The mathematical definition of the PWI is shown, together with some illustrative examples of the calculation.
Keywords
Computer Science
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
Modern Information Retrieval
11,544 Citations1999Ricardo Baeza‐Yates, Berthier Ribeiro‐Neto
Foundations of statistical natural language processing
9,996 Citations1999Christopher D. Manning, Hinrich Schütze
Information Processing & ManagementTerm-weighting approaches in automatic text retrieval
9,532 Citations1988Gerard Salton, Chris Buckley
This paper summarizes the insights gained in automatic term weighting, and provides baseline single term indexing models with which other more elaborate content analysis procedures can be compared.
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
Journal of DocumentationA STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
4,442 Citations1972Karen Spärck Jones
It is argued that terms should be weighted according to collection frequency, so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms.
Word association norms, mutual information, and lexicography
3,674 Citations1990Kenneth Church, Patrick Hanks
Birkhäuser Basel eBooksElements of Information Theory
2,908 Citations1992Konrad Jacobs
For information theory, messages are elements of a set, and when they are handled, their sole concern is to keep distinct messages distinct, i.e. to make the mappings resulting from the authors' manipulations injective (= one-to-one).
Accurate methods for the statistics of surprise and coincidence
2,688 Citations1993Ted Dunning
A re-examination of text categorization methods
2,660 Citations1999Yiming Yang, Xin Liu
The results show that SVM, kNN and LLSF signi cantly outperform NNet and NB when the number of positive training instances per category are small, and that all the methods perform comparably when the categories are over 300 instances.
Journal of the American Society for Information ScienceRelevance weighting of search terms
2,068 Citations1976Stephen Robertson, Karen Spärck Jones
This paper examines statistical techniques for exploiting relevance information to weight search terms using information about the distribution of index terms in documents in general and shows that specific weighted search methods are implied by a general probabilistic theory of retrieval.
Toward optimal feature selection
1,444 Citations1996Daphne Koller, Mehran Sahami
An efficient algorithm for feature selection which computes an approximation to the optimal feature selection criterion is given, showing that the algorithm effectively handles datasets with a very large number of features.
A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization
1,264 Citations1997Thorsten Joachims
A Probabilistic analysis of the Rocchio relevance feedback algorithm, one of the most popular learning methods from information retrieval, is presented in a text categorization framework and suggests that the probabilistic algorithms are preferable to the heuristic Rocchio classifier.
IBM Journal of Research and DevelopmentA Statistical Approach to Mechanized Encoding and Searching of Literary Information
1,059 Citations1957H. P. Luhn
The problem of literature searching by machines still presents major difficulties and a statistical approach to this problem will be outlined and the various steps of a system based on this approach will be described.
Hierarchically Classifying Documents Using Very Few Words
840 Citations1997Daphne Koller, Mehran Sahami
This work proposes an approach that utilizes the hierarchical topic structure to decompose the classification task into a set of simpler problems, one at each node in the classification tree, which can be solved accurately by focusing only on a very small set of features, those relevant to the task at hand.
Retrieving collocations from text: Xtract
799 Citations1993Frank Smadja
A set of techniques based on statistical methods for retrieving and identifying collocations from large textual corpora, based on some original filtering methods that allow the production of richer and higher-precision output are described.
Explorations in Automatic Thesaurus Discovery
757 Citations1994Gregory Grefenstette
The aim of this monograph is to provide a catalog of words and phrases used in ThesaurusGeneration, as well as some examples of other writers' work, which have been used in similar contexts.
Document clustering using word clusters via the information bottleneck method
476 Citations2000Noam Slonim, Naftali Tishby
A novel implementation of the recently introduced information bottleneck method for unsupervised document clustering that first finds word-clusters that capture most of the mutual information about to set of documents, and then finds document clusters that preserve the information about the word clusters.
Journal of DocumentationUSING PROBABILISTIC MODELS OF DOCUMENT RETRIEVAL WITHOUT RELEVANCE INFORMATION
436 Citations1979W. Bruce Croft, David J. Harper
This paper considers the situation where no relevance information is available, that is, at the start of the search, based on a probabilistic model, and proposes strategies for the initial search and an intermediate search.
Terminology International Journal of Theoretical and Applied Issues in Specialized CommunicationMethods of automatic term recognition
376 Citations1996Kyo Kageura, Bin Umino
This paper tries to give an overview of the principles and methods of automatic term recognition and two major trends are examined, i.e., studies in automatic recognition of significant elements for indexing mainly carried out in information-retrieval circles and current research in automaticterm recognition in the field of computational linguistics.
Journal of DocumentationON TERM SELECTION FOR QUERY EXPANSION
318 Citations1990Stephen Robertson
A quantitative argument is sketches a quantitative argument which suggests that the two purposes of relevance feedback system and retrieval purposes require different weighting formulae.
International Journal on Digital LibrariesA probabilistic justification for using tf×idf term weighting in information retrieval
183 Citations2000Djoerd Hiemstra
The paper shows that the new probabilistic interpretation of tf×idf term weighting might lead to better understanding of statistical ranking mechanisms, for example by explaining how they relate to coordination level ranking.
Information Processing & ManagementModels for retrieval with probabilistic indexing
177 Citations1989Norbert Fuhr
Three retrieval models for probabilistic indexing are described along with evaluation results for each, including the binary independence indexing (BII) model, which is a generalized version of the Maron and Kuhns indexing model.
Text, speech and language technologyInverse Document Frequency (IDF): A Measure of Deviations from Poisson
174 Citations1999Kenneth Church, William A. Gale
In inverse document frequency (IDF), a quantity borrowed from Information Retrieval, is used to distinguish words like somewhat and boycott, but boycott is a better keyword because its IDF is farther from what would be expected by chance (Poisson).
Lecture notes in computer scienceFeature subset selection in text-learning
171 Citations1998Dunja Mladenić
Experimental comparison given on real-world data collected from Web users shows that characteristics of the problem domain and machine learning algorithm should be considered when feature scoring measure is selected.
Information Processing & ManagementThe selection of good search terms
122 Citations1981C. J. van Rijsbergen, David J. Harper +1 more
This paper tackles the problem of how one might select further search terms, using relevance feedback, given the search terms in the query, by generating a number of different spanning trees from a variety of association measures.
Computational LinguisticsNatural Language Processing Using Very Large Corpora
112 Citations2000Susan Armstrong, Kenneth Church +4 more
This chapter discusses the Scope of Speech Acoustics, the Physics of Speech, and Segmental and Prosodic Cues, and Classification of Speech Data.
A theory of term weighting based on exploratory data analysis
88 Citations1998Warren R. Greiff
It is argued that exploratory data analysis can be a valuable tool for research whose goal is the development of an explanatory theory of information retrieval.
Machine TranslationA Technical Word- and Term-Translation Aid Using Noisy Parallel Corpora across Language Groups
76 Citations1997Pascale Fung, Kathleen McKeown
An algorithm for translating technical words and terms from noisy parallel corpora across language groups, which requires minimum preprocessing and is able to obtain technical-word translations without sentence-boundary identification or sentence alignment.
Information Retrieval: Uncertainty and Logics
70 Citations1998Fábio Crestani, Mounia Lalmas +1 more
The feature quantity
62 Citations2000Akiko Aizawa
This paper presents a formal description of the feature quantity, as well as some illustrative examples of applying such a quantity to different types of information retrieval tasks: representative term selection and text categorization.
Linguistic Techniques to Improve the Performance of Automatic Text Categorization.
54 Citations2001Akiko Aizawa
A method for incorporating natural language processing into existing text categorization procedures using a probabilistic language model and automatic extraction of terms based on POS tags automatically generated by a morphological analyzer is presented.
Information RetrievalExploiting the Similarity of Non-Matching Terms at Retrieval Time
42 Citations2000Fábio Crestani
A preliminary investigation is presented into a new class of retrieval models that attempt to solve the term mismatch problem by exploiting complete or partial knowledge of term similarity in the term space.
Journal of the American Society for Information ScienceAn information-theoretic measure of term specificity
40 Citations1992S. K. M. Wong, Yiyu Yao
An information-theoretic measure for term specificity is introduced and it is explicitly shown that the IDF weighting scheme can be derived from the proposed approach by assuming that the frequency of occurrence of each index term is uniform within the set of documents containing the term.
Communications of the ACMSemantic information retrieval
34 Citations1998Annelise Mark Pejtersen
The investigation of SIT concerns the axiomatisation of the logical principles for assigning probabilities or similar weighting functions to logical sentences and the relationship between information content of a sentence and its probability.
Semantic Information Retrieval
28 Citations1998Giambattista Amati, Keith van Rijsbergen
Journal of DocumentationDOCUMENTATION NOTE QUERY‐DOCUMENT SYMMETRY AND DUAL MODELS
13 Citations1994Stephen Robertson
The idea that there is some natural symmetry between queries and documents is explored, which leads to a conception of ‘dual’ models in information retrieval, which may invalidate this construction.
Journal of DocumentationTHE SHANNON MODEL OF IR SYSTEMS
12 Citations1972B. C. Brookes
The result obtained by Zunde and Slamecka, though plausible because it has some fortuitous semblance to the distributions of terms found in real systems, is in fact erroneous.
A Technical Word- and Term-Translation Aid Using Noisy Parallel Corpora across Language Groups
11 Citations1997Pascale Fung, Kathleen McKeown