A theory of term weighting based on exploratory data analysis
Published 1 August 1998Open access
Warren R. Greiff
Citations88
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is argued that exploratory data analysis can be a valuable tool for research whose goal is the development of an explanatory theory of information retrieval.
Abstract
Article Free Access Share on A theory of term weighting based on exploratory data analysis Author: Warren R. Greiff Computer Science Department, University of Massachusetts, Amherst Computer Science Department, University of Massachusetts, AmherstView Profile Authors Info & Claims SIGIR '98: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrievalAugust 1998 Pages 11–19https://doi.org/10.1145/290941.290948Online:01 August 1998Publication History 51citation1,123DownloadsMetricsTotal Citations51Total Downloads1,123Last 12 Months29Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF
Keywords
Computer Science
Journal of DocumentationA STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
4,442 Citations1972Karen Spärck Jones
It is argued that terms should be weighted according to collection frequency, so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms.
Journal of the American Society for Information ScienceRelevance weighting of search terms
2,068 Citations1976Stephen Robertson, Karen Spärck Jones
This paper examines statistical techniques for exploiting relevance information to weight search terms using information about the distribution of index terms in documents in general and shows that specific weighted search methods are implied by a general probabilistic theory of retrieval.
American Journal of PhysicsTransmission of Information: A Statistical Theory of Communications
866 Citations1961Robert M. Fano, David Hawkins
Database and Expert Systems ApplicationsThe INQUERY Retrieval System
526 Citations1992James P. Callan, W. Bruce Croft +1 more
A retrieval system (INQUERY) that is based on a probabilistic retrieval model and provides support for sophisticated indexing and complex query formulation is described.
Psychology Press eBooksUsing Statistics in Lexical Analysis
464 Citations2021Kenneth Church, William A. Gale +2 more
The computational tools available for studying machine-readable corpora are at present still rather primitive and use these corpora and the basic concordancing tool mentioned above to fill in detailed syntactic descriptions (prompting a move, towards more thorough descriptions of lexical syntax).
Journal of DocumentationUSING PROBABILISTIC MODELS OF DOCUMENT RETRIEVAL WITHOUT RELEVANCE INFORMATION
436 Citations1979W. Bruce Croft, David J. Harper
This paper considers the situation where no relevance information is available, that is, at the start of the search, based on a probabilistic model, and proposes strategies for the initial search and an intermediate search.
ACM Transactions on Information SystemsA probabilistic learning approach for document indexing
202 Citations1991Norbert Fuhr, Chris Buckley
A method for probabilistic document indexing using relevance feedback data that has been collected from a set of queries based on three new concepts, which allows the integration of new text analysis and knowledge-based methods in this approach as well as the consideration of document structures or different types of terms.
Probabilistic retrieval based on staged logistic regression
162 Citations1992William S. Cooper, Fredric C. Gey +1 more
This paper proposes a novel element of the proposed design of a probabilistic retrieval system design that the regression analysis be carried out in two or more levels or stages, and compensates automatically for systematic biases introduced by the statistical simplifying assumptions.
Journal of DocumentationAN EVALUATION OF FEEDBACK IN DOCUMENT RETRIEVAL USING CO‐OCCURRENCE DATA
160 Citations1978David J. Harper, C. J. van Rijsbergen
This paper reports experiments with a term weighting model incorporating relevance information in which it is assumed that index terms are distributed dependently and argues that if high recall searches are required, relevance feedback based on the modified dependence model may be superior to the widely used Boolean search.
On relevance weights with little relevance information
144 Citations1997Stephen Robertson, Steve Walker
This research highlights the need to understand more fully the role of emotion in the development of Syetema, as well as the role that language and social media have in this process.
ACM Transactions on Information SystemsOptimum polynomial retrieval functions based on the probability ranking principle
135 Citations1989Norbert Fuhr
This approach is not suited to log-linear probabilistic models and it needs large samples of relevance feedback data for its application, but it can handle very complex representations of documents and requests and it can be easily applied to multivalued relevance scales.
Text REtrieval ConferenceFull text retrieval based on probabilistic equations with coefficients fitted by logistic regression
86 Citations1993William S. Cooper, Aitao Chen +1 more
The experiments described here are part of research program whose objective is to develop a full text retrieval methodology that is statistically sound and powerful, yet reasonably simple.
Journal of the ACMTerm Weighting in Information Retrieval Using the Term Precision Model
57 Citations1982C. Yu, K.C. Lam +1 more
A mathematscal model is used in the present study to relate the term precision weights to the frequency of occurrence of the terms in a given document collecuon and to the number of relevant documents a user wishes to retrieve in response to a query.
Journal of the American Society for Information ScienceAn information-theoretic measure of term specificity
40 Citations1992S. K. M. Wong, Yiyu Yao
An information-theoretic measure for term specificity is introduced and it is explicitly shown that the IDF weighting scheme can be derived from the proposed approach by assuming that the frequency of occurrence of each index term is uniform within the set of documents containing the term.
Lecture notes in computer scienceWhat Is Quantum Information Retrieval?
39 Citations2011C.J. Keith van Rijsbergen
The theoretical foundations for quantum information retrieval derived from Quantum Theory are introduced and an explanation of how such a theoretical framework could be useful in IR, for example, by showing how logic, probability, and geometry can be represented in a consistent way in the underlying Hilbert Space.
Journal of the American Society for Information ScienceThe measurement of term importance in automatic indexing
39 Citations1981Gerard Salton, H. Wu +1 more
The frequency characteristics of terms in the documents of a collection have been used as indicators of term importance for content analysis and indexing purposes and recently automatic indexing theories have been devised that use not only the term frequency characteristics but also the relevance properties of the terms.
Information Processing & ManagementAutomatic indexing using term discrimination and term precision measurements
36 Citations1976Gerard Salton, Anita M.-Y. Wong +1 more
These two indexing systems are briefly described and experimental evidence is cited showing that a combination of both theories produces better retrieval performance than either one alone.
Probabilistic document indexing from relevance feedback data
30 Citations1989Norbert Fuhr, Chris Buckley
Based on the binary independence indexing model, three new concepts for probabilistic document indexing from relevance feedback data are applied:Abstraction from specific terms and documents, Flexibility of the representation, and Probabilistic learning or classification methods for the estimation of the indexing weights making better use of the available relevance information.
Two learning schemes in information retrieval
9 Citations1988C. Yu, Hidenobu Mizuno
Two methods are given to improve weighting schemes by using relevance information of a set of queries to estimate parameter values of two independence models in information retrieval — the binary independence model and the non-binary independence model.