What is the Role of NLP in Text Retrieval?
Text, speech and language technologyPublished 1 January 1999
Karen Spärck Jones
Citations91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is concluded that LMI is not needed for effective retrieval, but has other important roles within information-selection systems.
Abstract
This paper addresses the value of linguistically-motivated indexing (LMI) for document and text retrieval. After reviewing the basic concepts involved and the assumptions on which LMI is based, namely that complex index descriptions and terms are necessary, I consider past and recent research on LMI, and specifically on automated LMI via NLP. Experiments in the first phase of research, to the late eighties, did not demonstrate value in LMI, but were very limited; but the much larger tests of the Nineties, with full text, have not done so either. My conclusion is that LMI is not needed for effective retrieval, but has other important roles within information-selection systems.
Keywords
Computer Science
Program electronic library and information systemsAn algorithm for suffix stripping
8,136 Citations1980Martin Porter
An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL, and performs slightly better than a much more elaborate system with which it has been compared.
Aslib ProceedingsThe CRANFIELD TESTS ON INDEX LANGUAGE DEVICES
470 Citations1967Cyril W. Cleverdon
It appeared that the most important consideration was the specificity of the index terms; within the context of the conditions existing in this test, single‐word terms were more effective than concept terms or a controlled vocabulary.
ACM Transactions on Information SystemsLexical ambiguity and information retrieval
407 Citations1992Robert Krovetz, W. Bruce Croft
An analysis of lexical ambiguity in information retrieval test collections and experiments to determine the utility of word meanings for separating relevant from nonrelevant documents show that there is considerable ambiguity even in a specialized database.
Journal of the American Society for Information ScienceHow effective is suffixing?
388 Citations1991Donna Harman
Three measures were selected which evaluate performance at given rank cutoff points, such as those cor- responding to a screenful of document titles,such as those responding to the lists of the top ranked documents.
Journal of the American Society for Information ScienceStemming algorithms: A case study for detailed evaluation
371 Citations1996David A. Hull
A case study of stemming algorithms is described which describes a number of novel approaches to evaluation and demonstrates their value.
Scholarworks (University of Massachusetts Amherst)Representation and Learning in Information Retrieval
361 Citations1991David Lewis
A new theoretical model for text classification systems, including systems for document retrieval, automated indexing, electronic mail filtering, and similar tasks, is introduced, suggesting that the poor statistical characteristics of a syntactic indexing phrase representation negate its desirable semantic characteristics.
Text, speech and language technologyNatural Language Information Retrieval
324 Citations1999Tomek Strzalkowski
The use of phrases and structured queries in information retrieval
265 Citations1991W. Bruce Croft, Howard R. Turtle +1 more
The results show that using phrases in this way can improve performance, and that phrases that are automatically extracted from a natural language query perform nearly as well as manually selected phrases.
Cornell University eBooksExperiments in Automatic Phrase Indexing For Document Retrieval: A Comparison of Syntactic and Non-Syntactic Methods
185 Citations1987Joel L. Fagan
A 78-year-old man with palpitations, dizziness, and presyncope was referred to the authors' institution for evaluation and a diagnosis of right ventricular dysplasia was made, and the patient underwent an electrophysiological study to evaluate hisPalpitations and Presyncope.
Information Processing & ManagementTREC and TIPSTER experiments with inquery
159 Citations1995James P. Callan, W. Bruce Croft +1 more
Improvements to the probabilistic information retrieval system based upon a Bayesian inference network model are described, including transforming forms-based specifications of information needs into complex structured queries, automatic query expansion, automatic recognition of features in documents, relevance feedback, and simulated document routing.
Information Processing & ManagementGenerating and evaluating domain-oriented multi-word terms from texts
136 Citations1993Fred J. Damerau
This paper uses a surrogate for experimentation by relying on pre-existing dictionaries as indicators of domain relevance and an attempt is made to provide a more objective evaluation of the selection procedures used.
Journal of the American Society for Information ScienceThe effectiveness of a nonsyntactic approach to automatic phrase indexing for document retrieval
132 Citations1989Joel L. Fagan
It is not likely that phrase indexing of this kind will prove to be an important method of enhancing the performance of automatic document indexing and retrieval systems in operational environments, and a general syntactic analysis facility may be required.
Journal of DocumentationAUTOMATIC SEARCH TERM VARIANT GENERATION
127 Citations1984Karen Spärck Jones, John Tait
The paper describes research designed to improve automatic pre‐coordinate term indexing by applying powerful general‐purpose language analysis techniques to identify term sources in requests, and to generate variant expressions of the concepts involved for document text searching.
Journal of the American Society for Information ScienceFASIT: A fully automatic syntactically based indexing system
99 Citations1983Martín Dillon, Ann Gray
FASIT (Fully Automatic Syntactically based Indexing of Text) identifies content bearing textual units without a full parse, and, without using semantic criteria, groups these units Into quaslsynonymous sets.
Journal of the American Society for Information ScienceA new comparison between conventional indexing (MEDLARS) and automatic text processing (SMART)
90 Citations1972Gerard Salton
It is concluded that no technical justification exists for maintaining controlled, manual indexing in operational retrieval environments through conventional retrieval and automatic text analysis methods (SMART).
Psychology Press eBooksText-based intelligent Systems
87 Citations2014Paul S. Jacobs
This chapter discusses Text Representation for Intelligent Text Retrieval: A Classification-Oriented View, and Intelligent High-Volume Text Processing Using Shallow, Domain-Specific Techniques.
Information Processing & ManagementNatural language information retrieval
77 Citations1995Tomek Strzalkowski
Information Processing & ManagementCLARIT-TREC experiments
64 Citations1995David A. Evans, Robert G. Lefferts
Cet article decrit la preparation and the resultats des experimentations mises en place sur le systeme de recherche CLARIT en ce qui concerne la recherchy ad hoc manuelle.
The automatic indexing system AIR/PHYS - from research to applications
60 Citations1988P. Biebricher, Norbert Fuhr +3 more
An appropriate indexing approach and the corresponding structure of the AIR/PHYS system are described, and the conditions of the application as well as problems of further development are discussed.
Information Processing & ManagementTopic parsing: Accounting for text macro structures in full-text analysis
57 Citations1990Udo Hahn
This paper introduces a parser which is based on the conceptual knowledge of its domain and is organized as a collection of distributed lexicalized grammar modules (word experts) which communicate through message-passing.
Experiments on incorporating syntactic processing of user queries into a document retrieval strategy
56 Citations1988Alan F. Smeaton, C. J. van Rijsbergen
Investigating whether linguistic processes can be used as part of a document retrieval strategy by predefining a level of syntactic analysis of user queries only, suggests that the approach of using linguistic processing in retrieval, is valid.
Information Processing & ManagementAutomatic routing and retrieval using smart: TREC-2
51 Citations1995Chris Buckley, James Allan +1 more
This work continues the work in the TREC 2 environment, performing both routing and ad-hoc experiments on combining global similarities, giving an overall indication of how a document matches a query, with local similarities identifying a smaller part of the document that matches the query.
Robust text processing in automated information retrieval
32 Citations1994Tomek Strzalkowski
It is demonstrated that the use of syntactic compounds in the representation of database documents as well as in the user queries, coupled with an appropriate term weighting strategy, can considerably improve the effectiveness of retrospective search.
Information Storage and RetrievalA technique for machine-aided indexing
31 Citations1973Paul H. Klingbiel
The technique for machine-aided indexing (MAI) developed at the Defense Documentation Center (DDC) is illustrated on a randomly chosen abstract so that the reader can more fully explore this technique and form his own opinion of the applicability and versatility of this particular procedure.
Natural language techniques for intelligent information retrieval
28 Citations1988P. S. Jacob, Lisa F. Rau
This work provides an analysis of areas in which natural language and information retrieval come together, and describes a system that joins the two fields by combining technology, choice of application area, and knowledge acquisition techniques.
Information Processing & ManagementMachine-aided indexing at NASA
17 Citations1994June P. Silvester, Michael T. Genuardi +1 more
The NASA Lexical Dictionary (NLD) is a machine-aided indexing system used online at the National Aeronautics and Space Administration's Center for AeroSpace Information (CASI) and automatically suggests a set of candidate terms from NASA's controlled vocabulary for any designated natural language text input.
Information Storage and RetrievalNegotiation of inquiries in an on-line retrieval system
14 Citations1968Donald J. Hillman
A prototype retrieval system with three major components for textprocessing, connectivity and decision operations, based on a distinguishable subtheory, is discussed, with a major goal to develop procedures whereby research workers can conduct an on-line dialogue via terminals with a body of scientific information.
Information Processing & ManagementEvaluation of machine-aided indexing†
6 Citations1976Paul H. Klingbiel, Catherine C. Rinker
The following is a report of several comparisons designed to measure the retrieval effectiveness of MAI and manual indexing procedures under normal operational conditions.