An algorithm for the calculation of exact term discrimination values

Information Processing & ManagementPublished 1 January 1985

Peter Willett

DOI10.1016/0306-4573(85)90107-4

Source URL

Citations28

SJR quartileQ1

SJR score2.06

SNIP2.91

TL;DR

Evidence is presented to show that the relationship between term discrimination and term frequency is crucially dependent upon the type of inter-document similarity measure that is used for the calculation of the discrimination values.

Abstract

Term discrimination values have been suggested as an effective means for the selection and weighting of index terms in automatic document retrieval systems. This paper reports an algorithm for the calculation of term discrimination values that is sufficiently fast in operation to permit the use of exact values, rather than the approximate values studied in previous work. Evidence is presented to show that the relationship between term discrimination and term frequency is crucially dependent upon the type of inter-document similarity measure that is used for the calculation of the discrimination values.

Keywords

Computer Science

Program electronic library and information systemsAn algorithm for suffix stripping

8,136 Citations1980Martin Porter

An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL, and performs slightly better than a much more elaborate system with which it has been compared.

Communications of the ACMA vector space model for automatic indexing

7,434 Citations1975Gerard Salton, Anita M.-Y. Wong +1 more

An approach based on space density computations is used to choose an optimum indexing vocabulary for a collection of documents, demonstating the usefulness of the model.

View PDF

Journal of the American Society for Information ScienceA theory of term importance in automatic text analysis

388 Citations1975Gerard Salton, Chul‐Su Yang +1 more

Most existing automatic content analysis and indexing techniques are based on word frequency characteristics applied largely in an ad hoc manner, but terms exhibiting high occurence frequencies in individual documents are often useful for high recall performance, whereas terms with low frequency in the whole collection are useful forhigh precision.

Journal of the American Society for Information ScienceAutomatic ranked output from boolean searches in SIRE

93 Citations1977Terry Noreault, Matthew B. Koll +1 more

The study found that relevant documents were ranked significantly higher than nonrelevant documents in the set of documents retrieved in response to a Boolean query.

Journal of Information ScienceA review of the use of inverted files for best match searching in information retrieval systems

62 Citations1983Shirley A. Perry, Peter Willett

A critical evaluation is presented of a range of algorithms which have been described for the matching of documents with queries, and particular attention is paid to the computational efficiency of the various procedures.

Computers and the HumanitiesOn the role of words and phrases in automatic text analysis

15 Citations1976Gerard Salton, Anita M.-Y. Wong

A model, known as discrimination value analysis is introduced which assigns an appropriate role in the indexing operation to the terms, term phrases, and thesaurus classes.

Information Processing & ManagementThe computation of discrimination values

8 Citations1975Robert G. Crawford

Results have been reported showing the usefulness of discrimination value in automatic construction of dictionaries for information retrieval, and a relatively efficient algorithm is presented and an example is given.

ACM SIGIR ForumTerm frequency and term value

8 Citations1981Stephen Robertson