Document length normalization
Information Processing & ManagementPublished 1 September 1996Open access
Amit Singhal, Gerard Salton, Mandar Mitra, Chris Buckley
Citations202
SJR quartileQ1
SJR score2.06
SNIP2.91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A modified technique is presented that attempts to match the likelihood of retrieving a document of a certain length to thelihood of documents of that length being judged relevant, and it is shown that this technique yields significant improvements in retrieval effectiveness.
Abstract
In the TREC collection—a large full-text experimental text collection with widely varying document lengths—we observe that the likelihood of a document being judged relevant by a user increases with the document length. We show that a retrieval strategy, such as the vector-space cosine match, that retrieves documents of different lengths with roughly equal chances, will not optimally retrieve useful documents from such a collection. We present a modified technique—pivoted cosine normalization—that attempts to match the likelihood of retrieving documents of all lengths to the likelihood of their relevance, and show that this technique yields significant improvements in retrieval effectiveness.
Keywords
Computer Science
Information Processing & ManagementTerm-weighting approaches in automatic text retrieval
9,532 Citations1988Gerard Salton, Chris Buckley
This paper summarizes the insights gained in automatic term weighting, and provides baseline single term indexing models with which other more elaborate content analysis procedures can be compared.
Text REtrieval ConferenceOkapi at TREC
2,229 Citations1994Stephen Robertson, Steve Walker +3 more
Much of the work involved investigating plausible methods of applying Okapi-style weighting to phrases, and expansion using terms from the top documents retrieved by a pilot search on topic terms was used.
Journal of the American Society for Information ScienceRelevance weighting of search terms
2,068 Citations1976Stephen Robertson, Karen Spärck Jones
This paper examines statistical techniques for exploiting relevance information to weight search terms using information about the distribution of index terms in documents in general and shows that specific weighted search methods are implied by a general probabilistic theory of retrieval.
Automatic Query expansion using SMART : TREC 3
467 Citations1994Chris Buckley, Gerard Salton +2 more
This work continues the work in TREC 3, performing runs in the routing, ad-hoc, and foreign language environments, with a major focus on massive query expansion, adding from 300 to 530 terms to each query.
ACM SIGIR ForumInference Networks for Document Retrieval
425 Citations2017Howard R. Turtle, W. Bruce Croft
The use of inference networks to support document retrieval and a network-basead retrieval model is described and compared to conventional probabilistic and Boolean models.
Journal of the American Society for Information ScienceA theory of term importance in automatic text analysis
388 Citations1975Gerard Salton, Chul‐Su Yang +1 more
Most existing automatic content analysis and indexing techniques are based on word frequency characteristics applied largely in an ad hoc manner, but terms exhibiting high occurence frequencies in individual documents are often useful for high recall performance, whereas terms with low frequency in the whole collection are useful forhigh precision.
Overview of the third text Retrieval conference (TREC-3)
379 Citations1995Donna Harman
Evaluates new technologies in text retrieval with 34 papers: indexing structures, fragmentation schemes, probabilistic retrieval, latent semantic indexing, interactive document retrieval, & much more.
Subtopic structuring for full-length document access
330 Citations1993Marti A. Hearst, Christian Plaunt
It is argued that the advent of large volumes of full-length text, as opposed to short texts like abstracts and newswire, should be accompanied by corresponding new approaches to information access and a partition of the text into coherent multi-paragraph units that represent the pattern of subtopics that comprise the text.
Inference networks for document retrieval
212 Citations1989Howard R. Turtle, W. Bruce Croft
The use of inference networks to support document retrieval and a network-based retrieval model is described and compared to conventional probabilistic and Boolean models.
eCommons (Cornell University)Length Normalization in Degraded Text Collections
66 Citations1995Amit Singhal, Gerard Salton +1 more
This study examines the effects of the well known cosinenormalization method in the presence of OCR errors and proposes a new, more robust, normalization method that yields significant improvements in retrieval effectiveness over cosine normalization.
Document Retrieval and Routing Using the INQUERY System.
66 Citations1994John Broglio, James P. Callan +2 more
In the TREC experiments this year, a number of new techniques were introduced for both the ad-hoc retrieval and routing runs, and experiments with Spanish retrieval were carried out.
The importance of proper weighting methods
55 Citations1993Chris Buckley
Evidence is presented that good weighting methods are more important than the feature selection process and it is suggested that the two need to go hand-in-hand in order to be effective.
TREC-3 Ad-Hoc, Routing Retrieval and Thresholding Experiments using PIRCS.
39 Citations1994K. L. Kwok, Laszlo Grunfeld +1 more
The PIRCS retrieval system has been upgraded in TREC-3 to handle the full English collections of 2 GB in an efficient manner and recurrent spreading of activation in the network is used to implement query learning and expansion based on the best-ranked subdocuments of an initial retrieval.
eCommons (Cornell University)Automatic Text Theme Generation and the Analysis of Text Structure
20 Citations1994Gerard Salton, Amit Singhal
In this study, text theme generation and text summarization are related to text struture and it is shown that useful text derivatives are obtainable for texts with diverse structural characteristics.
Automatic text decomposition and structuring
16 Citations1994Gerard Salton, James Allan
Sophisticated text similarity measurements are used to determine relationships between natural-language texts and text segments, leading to improved text access and utilization.
Elsevier eBooksHuman Language Technology
8 Citations2006Graeme Hirst
Human language technology is the name given collectively to the various areas of technology that involve applications to tasks in which language is central, and three subfields are recognized: speech technology, information or document retrieval, and natural language processing.