An Overview of Evaluation Campaigns in Multimedia Retrieval
The Kluwer international series on information retrievalPublished 1 January 2010
Suzanne Little, Ainhoa Llorente, Stefan Rüger
Citations8
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The value of ImageCLEF, and related evaluation conferences, for the multimedia IR researcher is examined as providing not only a forum for assessing and comparing outcomes but also serving to promote research aims, provide practical guidance, and inspire research directions.
Abstract
This chapter presents an academic and research perspective on the impact and importance of ImageCLEF and similar evaluation workshops in multimedia information retrieval (MIR). Three main themes are examined: the position of ImageCLEF compared with other evaluation conferences; general views on the usefulness of evaluation conferences and possible alternatives, and the impact and real–world meaning of evaluation metrics used within ImageCLEF. We examine the value of ImageCLEF, and related evaluation conferences, for the multimedia IR researcher as providing not only a forum for assessing and comparing outcomes but also serving to promote research aims, provide practical guidance (e.g. standard data sets) and inspire research directions.
Keywords
Computer Science
IEEE Transactions on Pattern Analysis and Machine IntelligenceContent-based image retrieval at the end of the early years
6,023 Citations2000A.W.M. Smeulders, Marcel Worring +3 more
The working conditions of content-based retrieval: patterns of use, types of pictures, the role of semantics, and the sensory gap are discussed, as well as aspects of system engineering: databases, system architecture, and evaluation.
ACM Transactions on Information SystemsCumulated gain-based evaluation of IR techniques
4,634 Citations2002Kalervo Järvelin, Jaana Kekäläinen
This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.
Evaluation campaigns and TRECVid
1,224 Citations2006Alan F. Smeaton, Paul Over +1 more
An introduction to information retrieval (IR) evaluation from both a user and a system perspective is given, high-lighting that system evaluation is by far the most prevalent type of evaluation carried out.
Novelty and diversity in information retrieval evaluation
951 Citations2008Charles L. A. Clarke, Maheedhar Kolla +5 more
This paper develops a framework for evaluation that systematically rewards novelty and diversity into a specific evaluation measure, based on cumulative gain, and demonstrates the feasibility of this approach using a test collection based on the TREC question answering track.
ACM Transactions on Information SystemsRank-biased precision for measurement of retrieval effectiveness
628 Citations2008Alistair Moffat, Justin Zobel
A new effectiveness metric, rank-biased precision, is introduced that is derived from a simple model of user behavior, is robust if answer rankings are extended to greater depths, and allows accurate quantification of experimental uncertainty, even when only partial relevance judgments are available.
How reliable are the results of large-scale information retrieval experiments?
577 Citations1998Justin Zobel
A detailed empirical investigation of the TREC results shows that the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found.
ACM SIGIR ForumEvaluating Evaluation Measure Stability
567 Citations2017Chris Buckley, Ellen M. Voorhees
A novel way of examining the accuracy of the evaluation measures commonly used in information retrieval experiments is presented, which validates several of the rules-of-thumb experimenters use and challenges other beliefs, such as the common evaluation measures are equally reliable.
Information Processing & ManagementVariations in relevance judgments and the measurement of retrieval effectiveness
505 Citations2000Ellen M. Voorhees
Journal of the American Society for Information Science and TechnologyThe concept of relevance in IR
473 Citations2003Pia Borlund
The concept of relevance as viewed and applied in the context of IR evaluation is introduced, by presenting an overview of the multidimensional and dynamic nature of the concept.
Lecture notes in computer scienceThe Philosophy of Information Retrieval Evaluation
385 Citations2002Ellen M. Voorhees
The fundamental assumptions and appropriate uses of the Cranfield paradigm, especially as they apply in the context of the evaluation conferences, are reviewed.
Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo GakkaishiThe music information retrieval evaluation exchange (2005-2007): A window into music information retrieval research
296 Citations2008J. Stephen Downie
The background, structure, challenges, and contributions of MIREX are looked at and it is indicated that there are groups of systems that perform equally well within various MIR tasks.
Information Processing & ManagementDimensions of relevance
242 Citations2000Erica Cosijn, Peter Ingwersen
It is shown that motivational relevance, as a manifestation of relevance, should not be viewed as part of a linear, objective–subjective scale of relevances, but rather as an attribute of relevance.
ACM Transactions on Information SystemsSound and complete relevance assessment for XML retrieval
241 Citations2008Benjamin Piwowarski, Andrew Trotman +1 more
Investigations into the creation of sound and complete relevance assessments for the evaluation of content-oriented XML retrieval as carried out at INEX, the evaluation campaign for XML retrieval are described.
Ranking retrieval systems without relevance judgments
236 Citations2001Ian Soboroff, Charles Nicholas +1 more
The initial results of a new evaluation methodology which replaces human relevance judgments with a randomly selected mapping of documents to topics are proposed, which are referred to aspseudo-relevance judgments.
Evaluation of evaluation in information retrieval
227 Citations1995Tefko Saračević
A critical and historical analysis of evaluations of IR systems and processes, and issues related to systems under evaluation, and evaluation criteria, measures, measuring instruments, and methodologies are examined.
The Journal of Experimental MedicineInformation Retrieval Research
196 Citations1982Robert N Oddy
The electrocardiographic records taken during vagus stimulation by pressure from children suffering with chronic heart disease have shown that the stimulation of the vagi in these cases is strikingly effectual, leading to the belief that an important feature in the pathological physiology ofThese cases is hypertonus of the cardiac accelerator nerves.
Signals and communication technologyHigh-Level Feature Detection from Video in TRECVid: A 5-Year Retrospective of Achievements
174 Citations2008Alan F. Smeaton, Paul Over +1 more
This chapter summarizes the work done on the TRECVid high-level feature task, showing the progress made year-on-year, and provides a fairly comprehensive statement on where the state-of-the-art is regarding this important task.
Why batch and user evaluations do not give the same results
156 Citations2001Andrew Turpin, William Hersh
Assessment of the TREC Interactive Track showed that while the queries entered by real users into systems yielding better results in batch studies gave comparable gains in ranking of relevant documents for those users, they did not translate into better performance on specific tasks.
CERES (Cranfield University)Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices
153 Citations1966Cyril W. Cleverdon, J. Mills +1 more
An essential requirement of the project involved cooperation of a large number of research scientists, and the response to the request was most satisfactory, and I acknowledge with thanks the generous assistance of some two hundred scientists.
On relevance weights with little relevance information
144 Citations1997Stephen Robertson, Steve Walker
This research highlights the need to understand more fully the role of emotion in the development of Syetema, as well as the role that language and social media have in this process.
Information Processing & ManagementThe state of retrieval system evaluation
133 Citations1992Gerard Salton
An attempt is made to review the state of retrieval evaluation and to separate certain misgivings about the design of retrieval tests from conclusions that can legitimately be drawn from the evaluation results.
Do batch and user evaluations give the same results?
132 Citations2000William Hersh, Andrew Turpin +5 more
The results showed the weighting scheme giving beneficial results in batch studies did not do so with real users, and other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of papers seen by the users were identified.
Towards an information logic
127 Citations1989C. J. van Rijsbergen
This paper argues that much of the research work in the past has been steps in the direction of a logic for IR, and presents a picture of IR, describing bits of a puzzle which may fit together to point to a new framework within which a computational model or logic could be described.
Information Processing & ManagementOn the reliability of information retrieval metrics based on graded relevance
114 Citations2006Tetsuya Sakai
These experiments show that (Average) Normalised Discounted Cumulative Gain at document cut-off l are the best among the rank-based graded-relevance metrics, provided that l is large, and that Q-measure is the best choice if one requires a recall-based grading metric that is highly correlated with Average Precision.
Lecture notes in computer scienceGeoCLEF 2007: The CLEF 2007 Cross-Language Geographic Information Retrieval Track Overview
114 Citations2008Thomas Mandl, Fredric C. Gey +7 more
GeoCLEF ran as a regular track for the second time within the Cross Language Evaluation Forum (CLEF) 2007 to test and evaluate cross-language geographic information retrieval (GIR): retrieval for topics with a geographic specification.
Journal of the American Society for Information ScienceThe dilemma of measurement in information retrieval research
84 Citations1996David Ellis
It is argued that the dilemma of measurement has remained intractable even given the different assumptions of the different approaches for three connected reasons—the nature of the subject matter of the field; the nature of relevance judgment; and thenature of cognition and knowledge.
IEEE MultimediaVideOlympics: Real-Time Evaluation of Multimedia Retrieval Systems
47 Citations2008Cees G. M. Snoek, Marcel Worring +4 more
Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft)Performance measures for multilabel evaluation: a case study in the area of image classification
40 Citations2010Stefanie Nowak, Hanna Lukashevich +2 more
Users lost (summary)
34 Citations1997Tafko Saracevic
The approach of dealing with the broader context of information science when considering information retrieval (IR) is justified and personal reflections of the author related to the major events and issues that formed his professional life and research agenda are included.
Video Retrieval Using Search and Browsing.
34 Citations2004Daniel Heesch, Peter Howarth +5 more
Overall search performance falls behind that of last year, a result which the authors would like to attribute to the stronger emphasis on motion in this year’s search task.
ACM SIGIR ForumUsers lost
29 Citations1997Tefko Saračević
The approach of dealing with the broader context of information science when considering information retrieval (IR) is justified and personal reflections of the author related to the major events and issues that formed his professional life and research agenda are included.
Lecture notes in computer scienceThe INEX 2006 Multimedia Track
27 Citations2007Thijs Westerveld, Roelof van Zwol
A brief overview of the multimedia track, it’s collection tasks and goals, and the results and the approaches of the participating groups are presented.
Lecture notes in computer scienceExplaining User Performance in Information Retrieval: Challenges to IR Evaluation
24 Citations2009Kalervo Järvelin
The paper makes three points of significance for IR research: The Cranfield paradigm of IR evaluation seems to lose power when one looks at human instead of system performance, and searcher strategies have not been sufficiently described and cannot therefore be properly understood, supported nor evaluated.
Place disambiguation with co-occurrence models.
22 Citations2006Simon Overell
The methods for generating and applying co-occurrence models for the purpose of place name disambiguation, the use of named entity recognition tools and text indexing applications and ways in which the system could be optimised are described.
Lecture notes in computer scienceMedical Image Retrieval Using Texture, Locality and Colour
17 Citations2005Peter Howarth, Alexei Yavlinsky +2 more
A content-based approach was followed, and Tiling a Gabor texture feature to add localisation information proved to be particularly effective, with the retrieval system coming third in the automatic retrieval task.
Synthesis lectures on information concepts, retrieval, and servicesMultimedia Information Retrieval
16 Citations2010Stefan Rüger
The tutorial emphasises state-of-the-art research in the area of multimedia information retrieval, which gives an indication of the research and development trends and, thereby, a glimpse of the future world.
Lecture notes in computer scienceINEX 2005 Multimedia Track
14 Citations2006Roelof van Zwol, Gabriella Kazai +1 more
The track was successful in realizing its objective to provide a pilot evaluation platform for the evaluation of retrieval strategies for XML-based multimedia documents and test approaches for the retrieval of XML fragments using a combination of content-based text and image retrieval techniques.
Multi-timescale Video Shot-Change Detection.
8 Citations2001Marcus Jerome Pickering, Stefan Rüger
Semantic Video Annotation using Background Knowledge and Similarity-based Video Retrieval.
8 Citations2008Ainhoa Llorente, Srđan Zagorac +6 more
The experiments for the High-level Feature Extraction (FE) and Search (SE) tasks and participation in the TRECVID evaluation were a valuable learning experience for the group.
MMIS at ImageCLEF 2008: Experiments combining Different Evidence Sources.
6 Citations2008Simon Overell, Ainhoa Llorente +6 more
It is concluded that text is the dominant feature in this collection of images because of the relative performances of dierence evidences in ImageCLEFwiki and the failure to improve over a baseline.
CLEF (Working Notes)MMIS at ImageCLEF 2009: Non-parametric Density Estimation Algorithms
4 Citations2009Ainhoa Llorente, Suzanne Little +1 more
The MMIS group submitted runs to the Photo Annotation task based on two non-parametric density estimation models, one based on ROC curves and the other in a hierarchical measure proposed by the organisers.
Automated Content Based Video Retrieval.
2 Citations2009Srđan Zagorac, Ainhoa Llorente +3 more