Performance evaluation in content-based image retrieval: overview and proposals
Pattern Recognition LettersPublished 1 April 2001
Henning Müller, Wolfgang Müller, David Squire, Stéphane Marchand‐Maillet, Thierry Pun
Citations551
SJR quartileQ1
SJR score1.00
SNIP1.43
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The advantages and shortcomings of the performance measures currently used in CBIR are discussed and proposals for a standard test suite similar to that used in IR at the annual Text REtrieval Conference (TREC), are presented.
Abstract
Evaluation of retrieval performance is a crucial problem in content-based image retrieval (CBIR). Many different methods for measuring the performance of a system have been created and used by researchers. This article discusses the advantages and shortcomings of the performance measures currently used. Problems such as defining a common image database for performance comparisons and a means of getting relevance judgments (or ground truth) for queries are explained. The relationship between CBIR and information retrieval (IR) is made clear, since IR researchers have decades of experience with the evaluation problem. Many of their solutions can be used for CBIR, despite the differences between the fields. Several methods used in text retrieval are explained. Proposals for performance measures and means of developing a standard test suite for CBIR, similar to that used in IR at the annual Text REtrieval Conference (TREC), are presented.
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Psychological ReviewFeatures of similarity.
7,287 Citations1977Amos Tversky
The metric and dimensional assumptions that underlie the geometric representation of similarity are questioned on both theoretical and empirical grounds and a set of qualitative assumptions are shown to imply the contrast model, which expresses the similarity between objects as a linear combination of the measures of their common and distinctive features.
ComputerQuery by image and video content: the QBIC system
3,058 Citations1995Myron Flickner, Harpreet Sawhney +10 more
The QBIC system is described and its query capabilities are demonstrated, which allows queries on large image and video databases based on example images, user-constructed sketches and drawings, selected color and texture patterns, camera and object motion, and other graphical information.
Color- and texture-based image segmentation using EM and its application to content-based image retrieval
480 Citations2002Serge Belongie, Chad Carson +2 more
A new image representation is presented which provides a transformation from the raw pixel data to a small set of image regions which are coherent in color and texture space based on segmentation using the expectation-maximization algorithm on combined color andtexture features.
Information Processing & ManagementAll users of information retrieval systems are not created equal: An exploration into individual differences
193 Citations1989Christine L. Borgman
A specific study that sought to identify both technical aptitudes and personality characteristics that were related to academic orientation variables previously found to predict IR performance found that academic orientation was related to technical aptitude as measured by standardized achievement tests and coursework.
CERES (Cranfield University)Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices
153 Citations1966Cyril W. Cleverdon, J. Mills +1 more
An essential requirement of the project involved cooperation of a large number of research scientists, and the response to the request was most satisfactory, and I acknowledge with thanks the generous assistance of some two hundred scientists.
Information Processing & ManagementThe state of retrieval system evaluation
133 Citations1992Gerard Salton
An attempt is made to review the state of retrieval evaluation and to separate certain misgivings about the design of retrieval tests from conclusions that can legitimately be drawn from the evaluation results.
Archive ouverte UNIGE (University of Geneva)Content-based query of image databases, inspirations from text retrieval: inverted files, frequency-based weights and relevance feedback
131 Citations1998David Squire, Wolfgang Müller +2 more
Strategies for positive and negative relevance feedback in image retrieval
119 Citations2002Henning Müller, Wolfgang Müller +3 more
This study highlights the utility of negative feedback, especially over several feedback steps, and compares a variety of strategies for positive and negative feedback.
Lecture notes in computer scienceEvaluation in Information Retrieval
83 Citations2000Stephen Robertson
The components of a traditional laboratory-style evaluation experiment in information retrieval (as exemplified by TREC) are summarized, and some of the issues around this form of experiment are discussed.
Applying perceptual grouping to content-based image retrieval: building images
80 Citations2003Iqbal Qasim, J.K. Aggarwall
This paper presents an application of perceptual grouping rules for content-based image retrieval in a Bayesian framework for the retrieval of building images, and the results obtained are presented.
Target testing and the PicHunter Bayesian multimedia retrieval system
73 Citations2002Ingemar J. Cox, Matthew L. Miller +2 more
A simple, quantifiable measure of an image retrieval system's effectiveness, "target testing", is proposed, in which effectiveness is measured as the average number of images that a user must examine in searching for a given random target.
A framework for learning query concepts in image classification
69 Citations2003Aparna Lakshmi Ratan, Oded Maron +2 more
The bag generator is extended to generate more complex instances using multiple cues on segmented high resolution images and it is shown that this method can be used to learn certain object class concepts in addition, to natural scenes.
Electronic workshops in computingSearching for Photos - Journalists’ Practices in PictorialIR
59 Citations1998Marjo Markkula, Eero Sormunen
The study addresses different types of search topics common in journalistic illustration tasks, journalists’ searching behaviour and the criteria they apply in selecting photos, and suggests that browsing is an essential strategy in accessing digital photo archives.
Multimedia Tools and ApplicationsBenchmarking Multimedia Databases
53 Citations1997Arcot Desai Narasimhalu, Mohan Kankanhalli +1 more
The problem of benchmarking multimedia databases is analyzed in this paper and a methodology to suggest a methodology is suggested.
Graph-theoretic clustering for image grouping and retrieval
38 Citations2003Selim Aksoy, R.M. Haralick
A graph-theoretic approach for image retrieval is introduced by formulating the database search as a graph clustering problem by using a constraint that retrieved images should be consistent with each other (close in the feature space) as well as being individually similar to the query image.
Face image retrieval using HMMs
38 Citations2003Aleix M. Martı́nez
A new face recognition system that can be used to index (and thus retrieve) images and videos of a database of faces, based on identifying frontal faces, that allows reasonable variability in facial expressions, illumination conditions, and occlusions caused by eye-wear or items of clothing such as scarves.
Archive ouverte UNIGE (University of Geneva)A Comparison of Human and Machine Assessments of Image Similarity for the Organization of Image Databases
36 Citations1997David Squire, Thierry Pun
Development of measures of the agreement between two partitionings of an image set are developed and it is shown that it is vital to take chance agreements into account.
Testing for human perceptual categories in a physician-in-the-loop CBIR system for medical imagery
29 Citations2003Chi‐Ren Shyu, A.C. Kak +2 more
The article represents an alternative to the scattershot approach to initial feature extraction by describing the perceptual categories that the physicians claim to use for classifying images as belonging to different diseases and describing the specific low-level features that need to be extracted to determine the presence or the absence of the various perceptual categories.
The customized-queries approach to CBIR using EM
26 Citations2003Jennifer Dy, Carla E. Brodley +3 more
An approach called the "customized-queries" approach (CQA) to content-based image retrieval and an algorithm called FSSEM that performs feature selection and clustering simultaneously that radically improves the retrieval precision over the traditional approach that performs retrieval using a single feature vector.
Lecture notes in computer scienceFilter Image Browsing
26 Citations1999J. Vendrig, Marcel Worring +1 more
An extensive evaluation of the methods presented using a user simulation shows that Filter Image Browsing requires less effort from the user.
Retrieval performance improvement through low rank corrections
19 Citations2003Dorin Comaniciu, Peter Meer +2 more
To reduce the amount of computations and the size of logical database entry, the Bhattacharyya distance is approximate, taking into account that most of the energy in the feature space is often restricted to a low dimensional subspace.
Efficient content-based retrieval: experimental results
16 Citations2003Andrew P. Berman, Linda G. Shapiro
A method for using the triangle inequality algorithm for combinations of distance measures, thus allowing for database systems which combine flexibility and speed.
A graph based object description for information retrieval in digital image and video libraries
16 Citations2003Burak Özer, Marilyn Wolf +1 more
The goal of this study is to investigate the shape matching method based on a relational graph of objects, with respect to its accuracy, efficiency and scalability, and to investigate a new description for cars, using relational graphs.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Hunting moving targets: extension to Bayesian methods in multimedia databases</title>
15 Citations1999Wolfgang Mueller, David Squire +2 more
Cox et al. have proposed a Bayesian method to learn the user's performances during each query, which aims to bridge the gap between the level of abstraction of the formulation of a query and that of the desired result.
Image database querying using a multi-scale localized color representation
15 Citations2003Ullas Gargi, Rangachar Kasturi
A computationally efficient method to encode the spatial concentration of colors in images based on local histogram intersection is introduced, which allows image matching at multiple resolutions and partial matching of image regions without requiring image segmentation or cluster approximations.
A fast image retrieval algorithm with automatically extracted discriminant features
15 Citations2003Wey-Shiuan Hwang, Juyang Weng +2 more
This work recast the problem of classification as a regression problem so that the classification and regression (numerical values as output) are unified and the HDR tree automatically forms clusters in the input space guided by the desired output, which produces discriminant spaces.
Improved stochastic modeling of shapes for content-based image retrieval
11 Citations2003Stefan Müller, Gerhard Rigoll
An integrated approach to shape and color based retrieval, where the cues: color and shape, are both utilized in a local rather than a global way, as well as a novel deformation-tolerant method based on (pseudo-) two dimensional stochastic models.
Probabilistic retrieval: new insights and experimental results
11 Citations2003Nuno Vasconcelos, Andrew Lippman
New experimental results are presented, providing evidence that probabilistic retrieval has superior performance and a unified representation for texture and color is introduced.
Inexact graph retrieval
9 Citations2003Benoît Huet, Edwin R. Hancock
A Bayesian matching algorithm that draws on edge consistency and node attribute similarity is used to determine the a posteriori probability of a query graph for each of the candidate matches in the database.