Web Search Engine Evaluation Using Clickthrough Data and a User Model

Published 1 January 2007

Georges Dupret, Vanessa Murdock, Benjamin Piwowarski

Citations45

TL;DR

A toy model is illustrated with a toy model that once the user behavior is agreed upon, the human assessment can be eliminated and the engine performance can be evaluated based on the clickthrough data of past users.

Abstract

Traditional search engine evaluation relies on a list of query document pairs along with a score reflecting the document relevance to the query. The score is generally a human assessment, but nothing is said explicitly about the actual user behavior. In this paper we illustrate with a toy model that once the user behavior is agreed upon, the human assessment can be eliminated and the engine performance can be evaluated based on the clickthrough data of past users. 1.

Keywords

Computer Science

ACM Transactions on Information SystemsCumulated gain-based evaluation of IR techniques

4,634 Citations2002Kalervo Järvelin, Jaana Kekäläinen

This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.

Optimizing search engines using clickthrough data

3,898 Citations2002Thorsten Joachims

The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking.

Accurately interpreting clickthrough data as implicit feedback

1,383 Citations2005Thorsten Joachims, Laura Granka +3 more

It is concluded that clicks are informative but biased, and while this makes the interpretation of clicks as absolute relevance judgments difficult, it is shown that relative preferences derived from clicks are reasonably accurate on average.

Eye-tracking analysis of user behavior in WWW search

700 Citations2004Laura Granka, Thorsten Joachims +1 more

Web Watcher: A Tour Guide for the World Wide Web.

681 Citations1997Thorsten Joachims, Dayne Freitag +1 more

The learning algorithms used by WebWatcher, experimental results showing their e ectiveness, and lessons learned from this case study in Web tour guide agents are described.

Query chains

478 Citations2005Filip Radlinski, Thorsten Joachims

A novel approach for using clickthrough data to learn ranked retrieval functions for web search results by using query chains to generate new types of preference judgments from search engine logs, thus taking advantage of user intelligence in reformulating queries.

Learning user interaction models for predicting web search result preferences

467 Citations2006Eugene Agichtein, Eric Brill +2 more

This work presents a real-world study of modeling the behavior of web search users to predict web search result preferences and generalizes the approach to model user behavior beyond clickthrough, which results in higher preference prediction accuracy than models based on clickthrough information alone.

Query word deletion prediction

81 Citations2003Rosie Jones, Daniel C. Fain

This work provides algorithms which perform substantially better than the baseline in predicting which word should be deleted from a reformulated query, for increasing query coverage in the context of web search on small high-quality collections.

The information retrieval seriesImplicit Feedback: Using Behavior to Infer Relevance

69 Citations2006Diane Kelly

This chapter presented a framework for understanding and studying how behavior can be used as implicit relevance feedback, and a discussion of key issues and problems associated with implicit feedback research and identified challenges for future research.

ArXiv.orgMinimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs

53 Citations2006Filip Radlinski, Thorsten Joachims

A simple method is introduced to modify the presentation of search results that provably gives relevance judgments that are unaffected by presentation bias under reasonable assumptions and can be guaranteed to converge to an ideal ranking given sufficient data.

View PDF