INEX 2006 Evaluation Measures

Lecture notes in computer sciencePublished 18 August 2007

Mounia Lalmas, Gabriella Kazai, Jaap Kamps, Jovan Pehcevski, Benjamin Piwowarski, Stephen Robertson

DOI10.1007/978-3-540-73888-6_3

Source URL

Citations44

SJR quartileQ2

SJR score0.35

SNIP0.55

TL;DR

This paper describes the official measures of retrieval effectiveness employed at the ad hoc track of INEX 2006 and their implications for retrieval effectiveness in the future.

Abstract

This paper describes the official measures of retrieval effectiveness employed at the ad hoc track of INEX 2006.

Keywords

Computer Science

ACM Transactions on Information SystemsCumulated gain-based evaluation of IR techniques

4,634 Citations2002Kalervo Järvelin, Jaana Kekäläinen

This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.

Journal of the American Society for Information Science and TechnologyUsing graded relevance assessments in IR evaluation

244 Citations2002Jaana Kekäläinen, Kalervo Järvelin

It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents, and a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance is proposed.

View PDF

Evaluation by highly relevant documents

228 Citations2001Ellen M. Voorhees

To explore the role highly relevant documents play in retrieval system evaluation, assessors for the TREC-9 web track used a three-point relevance scale and also selected best pages for each topic, confirming the hypothesis that different retrieval techniques work better for retrievinghighly relevant documents.

Lecture notes in computer scienceAdvances in XML Information Retrieval and Evaluation

126 Citations2006Norbert Fuhr, Saadia Malik +1 more

The INEX Environment is used as a Test Bed for Various User Models for XML Retrieval and the Interpretation of CAS is studied.

Comparative Evaluation of XML Information Retrieval Systems: 5th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2006 Dagstuhl Castle, Germany, December 17-20, 2006 Revised and Selected Papers

86 Citations2005Norbert Fuhr, Mounia Lalmas +1 more

ACM Transactions on Information SystemseXtended cumulated gain measures for the evaluation of content-oriented XML retrieval

47 Citations2006Gabriella Kazai, Mounia Lalmas

The evaluation demonstrates that the XCG measures are stable and reliable, and in particular, that the novel measures of effort-precision and gain-recall (ep/gr) show comparable behavior to established IR measures like precision and recall.

Evaluation in (XML) information retrieval

37 Citations2006Benjamin Piwowarski, Georges Dupret

Lecture notes in computer scienceHiXEval: Highlighting XML Retrieval Evaluation

28 Citations2006Jovan Pehcevski, James A. Thom

A correlation analysis of the rank orderings obtained by the new metric and two XCG metrics shows that the orderings are strongly correlated, which demonstrates the usefulness of the proposed metric for evaluation of XML retrieval performance.

Lecture notes in computer scienceOverview of INEX 2006

26 Citations2007Saadia Malik, Andrew Trotman +2 more

An overview of the work carried out as part of INEX 2006 is provided, in the form of a large XML test collection and appropriate scoring methods, for the evaluation of content-oriented XML retrieval systems.

Lecture notes in computer scienceINEX 2002 - 2006: Understanding XML Retrieval Evaluation

23 Citations2007Mounia Lalmas, Anastasios Tombros

This paper describes the evaluation methodology developed in INEX, with particular focus on how evaluation metrics and the notion of relevance are treated.