Word pairs in language modeling for information retrieval
Published 26 April 2004
Carmen Alvarez, Philippe Langlais, Jian‐Yun Nie
Citations17
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work proposes a new language modeling approach to information retrieval that incorporates lexical affinities, or pairs of words that occur near each other, without a constraint on word order.
Abstract
Previous language modeling approaches to information retrieval have focused primarily on single terms. The use of bigram models has been studied, but the restriction on word order and adjacency may not be justified for information retrieval. We propose a new language modeling approach to information retrieval that incorporates lexical a#nities, or pairs of words that occur near each other, without a constraint on word order. The use of compound terms in the vector space model has been shown to outperform the vector model with only single terms (Nie & Dufort, 2002). We explore the use of compound terms in a language modeling approach, and compare our results with the vector space model, and unigram and bigram language model approaches.
Keywords
Computer Science
ACM SIGIR ForumA Language Modeling Approach to Information Retrieval
2,532 Citations2017Jay Ponte, W. Bruce Croft
It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection and provide further proof of concept for the use of language models for retrieval tasks.
ACM SIGIR ForumRelevance-Based Language Models
1,449 Citations2017Victor Lavrenko, W. Bruce Croft
This work proposes a novel technique for estimating a relevance model with no training data and demonstrates that it can produce highly accurate relevance models, addressing important notions of synonymy and polysemy.
ACM SIGIR ForumDocument Language Models, Query Models, and Risk Minimization for Information Retrieval
774 Citations2017John Lafferty, ChengXiang Zhai
A framework for information retrieval that combines document models and query models using a probabilistic ranking function based on Bayesian decision theory is presented and an operational retrieval model that extends recent developments in the language modeling approach to information retrieval is suggested.
ACM SIGIR ForumInformation Retrieval as Statistical Translation
631 Citations2017Adam Berger, John Lafferty
A simple, well motivated model of the document-to-query translation process is proposed, and an algorithm for learning the parameters of this model in an unsupervised manner from a collection of documents is described.
A general language model for information retrieval
545 Citations1999Fei Song, W. Bruce Croft
A new language model for information retrieval is presented, which is based on a range of data smoothing techniques, including the Good-Turning estimate, curve-fitting functions, and model combinations, and can be easily extended to incorporate probabilities of phrases such as word pairs and word triples.
IEEE Transactions on Software EngineeringAn information retrieval approach for automatically constructing software libraries
445 Citations1991Yoelle Maarek, Daniel M. Berry +1 more
A technology for automatically assembling large software libraries which promote software reuse by helping the user locate the components closest to her/his needs is described.
The Computer JournalProbabilistic Models in Information Retrieval
308 Citations1992Norbert Fuhr
An introduction and survey over probabilistic information retrieval (IR) is given: the probability-ranking principle shows that optimum retrieval quality can be achieved under certain assumptions; a conceptual model for IR along with the corresponding event space clarify the interpretation of the Probabilistic parameters involved.
A general language model for information retrieval (poster abstract)
143 Citations1999Fei Song, W. Bruce Croft
According to this new paradigm, each document is viewed as a language sample, and a query as a generation process, retrieved documents are ranked based on the probabilities of producing a query from the corresponding language models of these documents.
Biterm language models for document retrieval
101 Citations2002Munirathnam Srikanth, Rohini K. Srihari
A general language model was proposed that combined bigram language models with Good-Turing estimate and corpus-based smoothing of unigram probabilities to demonstrate better performance of language models against vector space or probabilistic retrieval models for document retrieval.
Title language model for information retrieval
94 Citations2002Rong Jin, Alex Hauptmann +1 more
In the experiments with four different TREC document collections, the title language model for information retrieval with the new smoothing method outperforms both the traditional language model and the vector space model for IR significantly.
Term-specific smoothing for the language modeling approach to information retrieval
90 Citations2002Djoerd Hiemstra
The new language modeling approach is shown to explain a number of practical facts of today's information retrieval systems that are not very well explained by the current state of information retrieval theory, including stop words, mandatory terms, coordination level ranking and retrieval using phrases.
CWI's Institutional Repository (Centrum Wiskunde & Informatica)Relating the new language models of information retrieval to the traditional retrieval models
58 Citations2000Djoerd Hiemstra, Arjen P. de Vries
A Comparison Of Language Modeling And Probabilistic Text Information Retrieval Approaches To Monophonic Music Retrieval.
51 Citations2000Jeremy Pickens
The goal of this paper is to explore some of the capabilities and limitations of current text information retrieval systems as applied to the task of music retrieval.
The Weaver System for Document Retrieval.
18 Citations1999Adam Berger, John Lafferty
The architecture and philosophy of the Weaver system, which represents a departure from traditional approaches to retrieval, are described, and its performance in the TREC-8 ad hoc evaluation is discussed.