A survey of methods for image annotation
Journal of Visual Languages & ComputingPublished 31 January 2008
Allan Hanbury
Citations160
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Three image annotation approaches are reviewed: free text annotation, keyword annotation and annotation based on ontologies, which discusses the creation of keyword vocabularies for use in automated image annotation evaluation.
Abstract
In order to evaluate automated image annotation and object recognition algorithms, ground truth in the form of a set of images correctly annotated with text describing each image is required. In this paper, three image annotation approaches are reviewed: free text annotation, keyword annotation and annotation based on ontologies. The practical aspects of image annotation are then considered. We discuss the creation of keyword vocabularies for use in automated image annotation evaluation. As direct manual annotation of images requires much time and effort, we also review various methods to make the creation of ground truth more efficient. An overview of annotated image datasets for computer vision research is provided.
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Knowledge AcquisitionA translation approach to portable ontology specifications
12,428 Citations1993Thomas Gruber
This paper describes a mechanism for defining ontologies that are portable over representation systems, basing Ontolingua itself on an ontology of domain-independent, representational idioms.
The Journal of Higher EducationWomen, Fire, and Dangerous Things: What Categories Reveal about the Mind
7,437 Citations1988Terence Odlin, George Lakoff
IEEE Transactions on Pattern Analysis and Machine IntelligenceContent-based image retrieval at the end of the early years
6,023 Citations2000A.W.M. Smeulders, Marcel Worring +3 more
The working conditions of content-based retrieval: patterns of use, types of pictures, the role of semantics, and the sensory gap are discussed, as well as aspects of system engineering: databases, system architecture, and evaluation.
International Journal of LexicographyIntroduction to WordNet: An On-line Lexical Database<sup>*</sup>
4,207 Citations1990George A. Miller, Richard Beckwith +3 more
Standard alphabetical procedures for organizing lexical information put together words that are spelled alike and scatter words with similar or related meanings haphazardly through the list.
International Journal of Computer VisionLabelMe: A Database and Web-Based Tool for Image Annotation
4,205 Citations2007Bryan Russell, Antonio Torralba +2 more
A web-based tool that allows easy image annotation and instant sharing of such annotations is developed and a large dataset that spans many object categories, often containing multiple instances over a wide variety of images is collected.
Visual categorization with bags of keypoints
4,147 Citations2004G. Csurka
Labeling images with a computer game
2,222 Citations2004Luis von Ahn, Laura Dabbish
A new interactive system: a game that is fun and can be used to create valuable output that addresses the image-labeling problem and encourages people to do the work by taking advantage of their desire to be entertained.
IEEE Transactions on Pattern Analysis and Machine IntelligenceSIMPLIcity: semantics-sensitive integrated matching for picture libraries
2,181 Citations2001James Z. Wang, Jia Li +1 more
SIMPLIcity (semantics-sensitive integrated matching for picture libraries), an image retrieval system, which uses semantics classification methods, a wavelet-based approach for feature extraction, and integrated region matching based upon image segmentation to improve retrieval.
The PASCAL visual object classes challenge 2006 (VOC2006) results
2,082 Citations2006Mark Everingham, Andrew Zisserman +2 more
Computer Vision and Image UnderstandingLearning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories
2,052 Citations2007Li Fei-Fei, Rob Fergus +1 more
The incremental algorithm is compared experimentally to an earlier batch Bayesian algorithm, as well as to one based on maximum-likelihood, which have comparable classification performance on small training sets, but incremental learning is significantly faster, making real-time learning feasible.
John Wiley & Sons, Inc. eBooksIntroduction to MPEG-7: Multimedia Content Description Interface
1,172 Citations2002Phillipe Salembier, Thomas Sikora +1 more
This book has been designed as a unique tutorial in the new MPEG 7 standard covering content creation, content distribution and content consumption, and presents a comprehensive overview of the principles and concepts involved in the complete range of Audio Visual material indexing, metadata description, information retrieval and browsing.
Lecture notes in computer scienceTextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation
1,153 Citations2006Jamie Shotton, John Winn +2 more
A new approach to learning a discriminative model of object classes, incorporating appearance, shape and context information efficiently, is proposed, which is used for automatic visual recognition and semantic segmentation of photographs.
IEEE Transactions on Pattern Analysis and Machine IntelligenceAutomatic linguistic indexing of pictures by a statistical modeling approach
1,087 Citations2003Jia Li, James Z. Wang
This paper implemented and tested the ALIP (Automatic Linguistic Indexing of Pictures) system on a photographic image database of 600 different concepts, each with about 40 training images and demonstrated the good accuracy of the system and its high potential in linguistic indexing of photographic images.
Object categorization by learned universal visual dictionary
840 Citations2005John Winn, Antonio Criminisi +1 more
An optimally compact visual dictionary is learned by pair-wise merging of visual words from an initially large dictionary, and a novel statistical measure of discrimination is proposed which is optimized by each merge operation.
Lecture notes in computer scienceThe 2005 PASCAL Visual Object Classes Challenge
767 Citations2006Mark Everingham, Andrew Zisserman +32 more
This chapter provides details of the datasets, algorithms used by the teams, evaluation criteria, and results achieved in the PASCAL Visual Object Classes Challenge.
IEEE Transactions on Image ProcessingImage classification for content-based indexing
758 Citations2001Aditya Vailaya, Mário A. T. Figueiredo +2 more
The goal is to combine multiple two-class classifiers into a single hierarchical classifier, and it is demonstrated that a small vector quantizer can be used to model the class-conditional densities of the features, required by the Bayesian methodology.
SpringerReferenceVisual Information Retrieval
740 Citations2012Alberto Del Bimbo, Marko Grobelnik +2 more
This work applied the Kernel Canonical Correlation Analysis to the Japanese-English cross-language information retrieval and classification and the results were encouraging.
Indoor-outdoor image classification
639 Citations2002Martin Szummer, Rosalind W. Picard
This work systematically studied the features of: histograms in the Ohta color space; multiresolution, simultaneous autoregressive model parameters; and coefficients of a shift-invariant DCT to show how high-level scene properties can be inferred from classification of low-level image features.
Journal of Machine Learning ResearchImage Categorization by Learning and Reasoning with Regions
633 Citations2004Yixin Chen, James Z. Wang
This paper presents a new learning technique, which extends Multiple-Instance Learning (MIL), and its application to the problem of region-based image categorization, and provides experimental results on an image categorizing problem and a drug activity prediction problem.
Nucleic Acids ResearchGoPubMed: exploring PubMed with the Gene Ontology
557 Citations2005Andreas Doms, Michael Schroeder
GoPubMed, a web server which allows users to explore PubMed search results with the Gene Ontology (GO), a hierarchically structured vocabulary for molecular biology, gives an overview of the literature abstracts by categorizing abstracts according to the GO and thus allowing users to quickly navigate through the Abstracts by category.
Peekaboom
515 Citations2006Luis von Ahn, Ruoran Liu +1 more
Peekaboom is an entertaining web-based game that can help computers locate objects in images and is an example of a new, emerging class of games, which not only bring people together for leisure purposes, but also exist to improve artificial intelligence.
IEEE Transactions on Pattern Analysis and Machine IntelligenceReal-Time Computerized Annotation of Pictures
511 Citations2008Jia Li, James Z. Wang
Matching Words and Pictures
459 Citations2003Kobus Barnard, Pinar Duygulu +8 more
Comparative study of textural analysis techniques to characterise tissue from intravascular ultrasound
403 Citations2002William H. Nailon, Stephen McLaughlin +2 more
The results show the ability of the texture analysis techniques used to discriminate clot lesions, and highlights the advantage of using the raw data over the scan-converted data in assessing thrombus composition in vitro.
Gazou Denshi Gakkaishi/Gazō Denshi Gakkaishi/Gazo Denshi GakkaishiVisual Information Retrieval
397 Citations1998Chua Tat-Seng, Wai-Chee Low +1 more
The development of a multi-layered framework for visual information retrieval that integrates current works on content based information retrieval at various level of abstractions – as pixels, as features, as objects, or as abstract concepts is discussed.
Lecture notes in computer scienceToward Category-Level Object Recognition
294 Citations2006Jean Ponce, Martial Hebert +2 more
This book is the outcome of two workshops that brought together about 40 prominent vision and machine learning researchers interested in the fundamental and applicative aspects of object recognition, as well as representatives of industry to promote the creation of an international object recognition community.
IEEE Intelligent SystemsOntology-based photo annotation
283 Citations2001A.T. Schreiber, Barbara Dubbeldam +2 more
The authors describe their approach and the tool they developed to make annotating photos and searching for specific images more intelligent.
Lecture notes in computer scienceA Statistical Model for General Contextual Object Recognition
266 Citations2004Peter Carbonetto, Nando de Freitas +1 more
Experiments show that spatial context considerably improves the accuracy of object recognition, and an approximate EM algorithm that uses loopy belief propagation in the inference step and iterative scaling on the pseudo-likelihood approximation in the parameter update step converges to good local solutions.
Feature generation for text categorization using world knowledge
232 Citations2005Evgeniy Gabrilovich, Shaul Markovitch
Improved machine learning algorithms for text categorization with generated features based on domain-specific and common-sense knowledge are enhanced, addressing the two main problems of natural language processing--synonymy and polysemy.
Lecture notes in computer scienceAdapted Vocabularies for Generic Visual Categorization
211 Citations2006Florent Perronnin, Christopher R. Dance +2 more
A novel and practical approach to GVC is proposed, which describes the content of all the considered classes of images, and class vocabularies obtained through the adaptation of the universal vocabulary using class-specific data.
Lecture notes in computer scienceDataset Issues in Object Recognition
207 Citations2006Jean Ponce, Torres Berg +11 more
Current datasets are lacking in several respects, and this paper discusses some of the lessons learned from existing efforts, as well as innovative ways to obtain very large and diverse annotated datasets.
Real-time computerized annotation of pictures
196 Citations2006Jia Li, James Z. Wang
Animals on the Web
181 Citations2006Tamara L. Berg, David Forsyth
This work demonstrates a method for identifying images containing categories of animals using a clustering method applied to text on web pages and shows unequivocal evidence that visual information improves performance for this task.
Clustering art
160 Citations2005Kobus Barnard, Pınar Duygulu +1 more
This work extends a recently developed method for learning the semantics of image databases using text and pictures and uses WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.
Lecture notes in computer scienceThe Truth about Corel - Evaluation in Image Retrieval
160 Citations2002Henning Müller, Stéphane Marchand‐Maillet +1 more
This article compares different ways of evaluating the performance of content-based image retrieval systems using a subset of the Corel images with the same CBIRSan d the same set of evaluation measures to show how easy it is to get differing results, even when using the same image collection, thesame CBIRS and the same performance measures.
Ontology-based image retrieval
142 Citations2003Eero Hyvönen, Avril Styrman +1 more
This paper considers the situation when a user is faced with an image repository whose content is complicated and semantically unknown to some extent and shows how ontologies can then be of help to the user in formulating the information need, the query, and the answers.
The effects of segmentation and feature choice in a translation model of object recognition
62 Citations2003Kobus Barnard, Pınar Duygulu +3 more
It is shown that word prediction can be markedly improved by providing supervised information for a relatively small number of regions together with large quantities of unsupervised information, which allows a better and more discriminative choice of features and breaks possible symmetries.
Toward bridging the annotation-retrieval gap in image search by a generative modeling approach
57 Citations2006Ritendra Datta, Weina Ge +2 more
An annotation-driven image retrieval approach that is demonstrated to effectively handle cases of partially tagged and completely untagged image databases, multiple keyword queries, and example based queries with or without tags, all in near-realtime.
Art Libraries JournalIconclass: an iconographic classification system
54 Citations1983Leendert D. Couprie
In this paper the classification as such is briefly explained, mainly with the help of two examples: scenes from the stories of St. Antony Abbot and Hercules and the alphabetical index to the classification is given.
FigshareLSCOM Lexicon Definitions and Annotations (Version 1.0)
54 Citations2018Lyndon Kennedy, Alexander G. Hauptmann
The DTO sponsored LSCOM workshop is developing an expanded multimedia concept lexicon on the order of 1000, with 834 initial selected concepts, which consists of 449 unique concepts over the entire TRECVID 2005 development set.
Lecture notes in computer scienceA Correlation Approach for Automatic Image Annotation
50 Citations2006David R. Hardoon, Craig Saunders +2 more
This work proposes an image annotation method that learns a direct mapping from image descriptors to keywords and compares the semantic based methods of Latent Semantic Indexing and Kernel Canonical Correlation Analysis, as well as using a recently proposed vector label based learning method known as Maximum Margin Robot.
Lecture notes in computer scienceBenchmarking for Content-Based Visual Information Search
43 Citations2000Clement H. C. Leung, Horace H. S. Ip
This paper presents concrete proposals concerning the development of a standardised benchmark suite, and by adopting an extensible framework, it is able to cater for a wide variety of applications paradigms and to lend itself to incremental refinement.
International Journal of Computer VisionEvaluation of Localized Semantics: Data, Methodology, and Experiments
37 Citations2007Kobus Barnard, Quanfu Fan +5 more
A new data set of 1014 images with manual segmentations and semantic labels for each segment is presented, together with a methodology for using this kind of data for recognition evaluation, and four algorithms which learn to label image regions from weakly labeled data are evaluated.
Lecture notes in computer scienceTowards a Large Scale Concept Ontology for Broadcast Video
34 Citations2004Alexander G. Hauptmann
This work states that if a rich enough set of intermediate textual descriptors in the form of large lexicons and taxonomic classification schemes are defined, then robust and general-purpose semantic content annotation and retrieval will be enabled through these descriptors.
Usage-oriented multimedia information retrieval technological evaluation
14 Citations2006Christian Fluhr, Pierre-Alain Moëllic +1 more
The purpose behind these campaigns is to minimize the gap between technology evaluation and user-oriented evaluation in the field of information retrieval.
TU/e Research PortalExtracting an ontology of portrayable objects from Wordnet
13 Citations2005Svitlana Zinger, Chr Millet +4 more
A large-scale image ontology is described using the WordNet lexical resources based on English words identifying portrayable objects, which will be a useful tool for solving the object recognition task.
Zenodo (CERN European Organization for Nuclear Research)Evaluating Content Based Image Retrieval Techniques With The One Million Images Clic Test Bed
10 Citations2007Pierre-Alain Moëllic, Patrick Hède +2 more
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Testing a vocabulary for image indexing and ground truthing</title>
9 Citations2001Corinne Joergensen, P. S. Joergensen
The development of an image indexing vocabulary is described, and a conceptual structure which relates the features of the vocabulary to the requirements of image annotation is presented.
Building a Multi-Modal Thesaurus from Annotated Images
4 Citations2006Hichem Frigui, Joshua Caudill
An unsupervised approach to learn associations between low-level visual features and keywords is proposed to build a multi-modal thesaurus that could serve as a foundation for hybrid navigation and search algorithms.
Relating words and image segments on multiple layers for effective browsing and retrieval
2 Citations2005Andrea Kutics, Akihiko Nakagawa +3 more
Results of experiments suggests that step by step semantic inference on consecutive layers of image-word association helps to improve accuracy of retrieval and browsing.