Novelty and diversity in information retrieval evaluation
Published 20 July 2008
Charles L. A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher
Citations951
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper develops a framework for evaluation that systematically rewards novelty and diversity into a specific evaluation measure, based on cumulative gain, and demonstrates the feasibility of this approach using a test collection based on the TREC question answering track.
Abstract
Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track.
Keywords
Computer Science
ACM Transactions on Information SystemsCumulated gain-based evaluation of IR techniques
4,634 Citations2002Kalervo Järvelin, Jaana Kekäläinen
This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.
Learning to rank using gradient descent
2,759 Citations2005Chris Burges, Tal Shaked +5 more
RankNet is introduced, an implementation of these ideas using a neural network to model the underlying ranking function, and test results on toy data and on data from a commercial internet search engine are presented.
ACM SIGIR ForumThe Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
2,210 Citations2017Jaime Carbonell, Jade Goldstein
A method for combining query-relevance with information-novelty in the context of text retrieval and summarization and preliminary results indicate some benefits for MMR diversity ranking in document retrieval and in single document summarization.
The use of MMR, diversity-based reranking for reordering documents and producing summaries
2,144 Citations1998Jaime Carbonell, Jade Goldstein
This paper presents a method for combining query-relevance with information-novelty in the context of text retrieval and summarization, and preliminary results indicate some benefits for MMR diversity ranking in document retrieval and in single document summarization.
ACM SIGIR ForumA taxonomy of web search
1,923 Citations2002Andrei Broder
This taxonomy of web searches is explored and how global search engines evolved to deal with web-specific needs is discussed.
Accurately interpreting clickthrough data as implicit feedback
1,383 Citations2005Thorsten Joachims, Laura Granka +3 more
It is concluded that clicks are informative but biased, and while this makes the interpretation of clicks as absolute relevance judgments difficult, it is shown that relative preferences derived from clicks are reasonably accurate on average.
Journal of DocumentationTHE PROBABILITY RANKING PRINCIPLE IN IR
1,071 Citations1977Stephen Robertson
It is shown that the principle that documents should be ranked in order of the probability of relevance or usefulness can be justified under certain assumptions, but that in cases where these assumptions do not hold, the principle is not valid.
Improving web search ranking by incorporating user behavior information
1,041 Citations2006Eugene Agichtein, Eric Brill +1 more
It is shown that incorporating user behavior data can significantly improve ordering of top results in real web search setting, improving the accuracy of a competitive web search ranking algorithms by as much as 31% relative to the original performance.
Information Processing & ManagementA probabilistic model of information retrieval: development and comparative experiments
947 Citations2000Karen Spärck Jones, Steve Walker +1 more
Retrieval evaluation with incomplete information
733 Citations2004Chris Buckley, Ellen M. Voorhees
It is shown that current evaluation measures are not robust to substantially incomplete relevance judgments, and a new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets.
A support vector method for optimizing average precision
720 Citations2007Yisong Yue, Thomas Finley +2 more
This work presents a general SVM learning algorithm that efficiently finds a globally optimal solution to a straightforward relaxation of MAP, and shows its method to produce statistically significant improvements in MAP scores.
Predicting query performance
669 Citations2002Steve Cronen-Townsend, Yun Zhou +1 more
It is suggested that clarity scores measure the ambiguity of a query with respect to a collection of documents and show that they correlate positively with average precision in a variety of TREC test sets.
ACM SIGIR ForumBeyond Independent Relevance
567 Citations2015ChengXiang Zhai, William W. Cohen +1 more
A framework for evaluating subtopic retrieval is proposed which generalizes the traditional precision and recall metrics by accounting for intrinsic topic difficulty as well as redundancy in documents and a maximal marginal relevance (MMR) ranking strategy is proposed.
Learning user interaction models for predicting web search result preferences
467 Citations2006Eugene Agichtein, Eric Brill +2 more
This work presents a real-world study of modeling the behavior of web search users to predict web search result preferences and generalizes the approach to model user behavior beyond clickthrough, which results in higher preference prediction accuracy than models based on clickthrough information alone.
ACM Transactions on Information SystemsAn information-theoretic approach to automatic query expansion
366 Citations2001Claudio Carpineto, Renato De Mori +2 more
This work presents a computationally simple and theoretically justified method for assigning scores to candidate expansion terms within Rocchio's framework for query reweigthing, and discusses the effect on retrieval effectiveness of the main parameters involved in automatic query expansion.
Less is more
319 Citations2006Harr Chen, David R. Karger
Variations in relevance judgments and the measurement of retrieval effectiveness
293 Citations1998Ellen M. Voorhees
Very high correlations were found among the rankings of systems produced using different relevance judgment sets, indicating that the comparative evaluation of retrieval performance is stable despite substantial differences in relevance judgments, and thus reaffirm the use of the TREC collections as laboratory tools.
Probabilistic Relevance Models Based on Document and Query Generation
209 Citations2003John Lafferty, ChengXiang Zhai
A unified account of the probabilistic semantics underlying the language modeling approach and the traditional Probabilistic model for information retrieval is given, showing that the two approaches can be viewed as being equivalent probabilistically.
Lecture notes in computer scienceQuery Difficulty, Robustness, and Selective Application of Query Expansion
199 Citations2004Giambattista Amati, Claudio Carpineto +1 more
There is increasing interest in improving the robustness of IR systems, i.e. their effectiveness on difficult queries, and a selective application of QE would be a natural answer to obtain a more robust retrieval system.
The relationship between IR effectiveness measures and user satisfaction
158 Citations2007Azzah Al‐Maskari, Mark Sanderson +1 more
How users' satisfaction correlates with the effectiveness of Google as quantified by IR measures such as precision and the suite of Cumulative Gain measures (CG, DCG, NDCG) is looked at.
Information Processing & ManagementA risk minimization framework for information retrieval
149 Citations2005ChengXiang Zhai, John Lafferty
A probabilistic information retrieval framework in which the retrieval problem is formally treated as a statistical decision problem, and how this framework can unify existing retrieval models and accommodate systematic development of new retrieval models is discussed.
Information SystemsQuery performance prediction
147 Citations2005Ben He, Iadh Ounis
Six predictors of query performance are studied, which can be generated prior to the retrieval process without the use of relevance scores, showing that these predictors can be useful to infer query performance in practical applications.
Reliable information retrieval evaluation with incomplete and biased judgements
118 Citations2007Stefan Büttcher, Charles L. A. Clarke +2 more
This work compares the performance of this method with other approaches to the problem of incomplete judgements, such as bpref, and shows that the proposed method leads to higher evaluation accuracy, especially if the set of manual judgements is rich in documents, but highly biased against some systems.
Information Processing & ManagementBeyond topicality
84 Citations1982Bert R. Boyce
Topicality is an operationally necessary but insufficient condition for requestor judged relevance and a two stage system which generates a high recall set and discards imprecise documents by measuring their distance from a seed document can be iterated to provide topicality feedback without user input.
Redundant documents and search effectiveness
63 Citations2005Yaniv Bernstein, Justin Zobel
It is shown that content-equivalent documents have a significant effect on the search experience: it was found that 16.6% of all relevant documents in runs submitted to the TREC 2004 terabyte track were redundant.
Information Storage and RetrievalA searching procedure for information retrieval
58 Citations1964William Goffman
A search procedure for an information retrieval system is developed whereby the answer to a question is obtained by maximizing an evaluation function of the system's output in terms of the probability of relevance.
ACM SIGIR ForumAmbiguous requests
47 Citations2007Karen Spärck-Jones, Stephen Robertson +1 more
Retrieval system experimentation has assumed that user requests represent a single information need, but search engine experience demonstrates that this assumption is far from holding in the real world.
Web Search Engine Evaluation Using Clickthrough Data and a User Model
45 Citations2007Georges Dupret, Vanessa Murdock +1 more
A toy model is illustrated with a toy model that once the user behavior is agreed upon, the human assessment can be eliminated and the engine performance can be evaluated based on the clickthrough data of past users.
White Rose Research Online (University of Leeds, The University of Sheffield, University of York)Ambiguous requests: implications for retrieval tests
5 Citations2007Karen Spärck-Jones, Steve Robertson +1 more
Retrieval system experimentation has assumed that user requests represent a single information need, but search engine experience demonstrates that this assumption is far from holding in the real world.