BabyTalk: Understanding and Generating Simple Image Descriptions
IEEE Transactions on Pattern Analysis and Machine IntelligencePublished 31 May 2013
Girish Kulkarni, Visruth Premraj, Vicente Ordóñez, Sagnik Dhar, Siming Li, Yejin Choi
Citations874
SJR quartileQ1
SJR score3.91
SNIP5.99
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The proposed system to automatically generate natural language descriptions from images is very effective at producing relevant sentences for images and generates descriptions that are notably more true to the specific image content than previous work.
Abstract
We present a system to automatically generate natural language descriptions from images. This system consists of two parts. The first part, content planning, smooths the output of computer vision-based detection and recognition algorithms with statistics mined from large pools of visually descriptive text to determine the best content words to use to describe an image. The second step, surface realization, chooses words to construct natural language sentences based on the predicted content and general statistics from natural language. We present multiple approaches for the surface realization step and evaluate each using automatic measures of similarity to human generated reference descriptions. We also collect forced choice human evaluations between descriptions from the proposed generation system and descriptions from competing approaches. The proposed system is very effective at producing relevant sentences for images. It also generates descriptions that are notably more true to the specific image content than previous work.
Keywords
Computer Science
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
61,841 Citations2009Jia Deng, Wei Dong +4 more
A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
2,945 Citations2009Jia Deng, Wei Dong +4 more
IEEE Transactions on Pattern Analysis and Machine Intelligence80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
1,976 Citations2008Antonio Torralba, Rob Fergus +1 more
For certain classes that are particularly prevalent in the dataset, such as people, this work is able to demonstrate a recognition performance comparable to class-specific Viola-Jones style detectors.
2009 IEEE Conference on Computer Vision and Pattern RecognitionDescribing objects by their attributes
1,928 Citations2009Ali Farhadi, Ian Endres +2 more
This paper proposes to shift the goal of recognition from naming to describing, and introduces a novel feature selection method for learning attributes that generalize well across categories.
2009 IEEE Conference on Computer Vision and Pattern RecognitionLearning to detect unseen object classes by between-class attribute transfer
1,632 Citations2009Christoph H. Lampert, Hannes Nickisch +1 more
The experiments show that by using an attribute layer it is indeed possible to build a learning object detection system that does not require any training images of the target classes, and assembled a new large-scale dataset, “Animals with Attributes”, of over 30,000 animal images that match the 50 classes in Osherson's classic table of how strongly humans associate 85 semantic attributes with animal classes.
Lecture notes in computer scienceObject Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
1,608 Citations2002Pınar Duygulu, Kobus Barnard +2 more
This work shows how to cluster words that individually are difficult to predict into clusters that can be predicted well, and cannot predict the distinction between train and locomotive using the current set of features, but can predict the underlying concept.
Automatic evaluation of summaries using N-gram co-occurrence statistics
1,588 Citations2003Chin-Yew Lin, Eduard Hovy
The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
Attribute and simile classifiers for face verification
1,382 Citations2009Neeraj Kumar, Alexander C. Berg +2 more
Two novel methods for face verification using binary classifiers trained to recognize the presence or absence of describable aspects of visual appearance and a new data set of real-world images of public figures acquired from the internet.
Lecture notes in computer scienceEvery Picture Tells a Story: Generating Sentences from Images
1,249 Citations2010Ali Farhadi, Mohsen Hejrati +5 more
A system that can compute a score linking an image to a sentence, which can be used to attach a descriptive sentence to a given image, or to obtain images that illustrate a given sentence.
IEEE Transactions on Pattern Analysis and Machine IntelligenceConvergent Tree-Reweighted Message Passing for Energy Minimization
1,119 Citations2006Vladimir Kolmogorov
This paper develops a modification of the recent technique proposed by Wainwright et al. (Nov. 2005), called sequential tree-reweighted message passing, which outperforms both the ordinary belief propagation and tree- reweighted algorithm in both synthetic and real problems.
International Journal of Computer VisionTextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context
1,052 Citations2007Jamie Shotton, John Winn +2 more
A new approach for learning a discriminative model of object classes, incorporating texture, layout, and context information efficiently, which gives competitive and visually pleasing results for objects that are highly textured, highly structured, and even articulated.
Im2Text: Describing Images Using 1 Million Captioned Photographs
944 Citations2011Vicente Ordóñez, Girish Kulkarni +1 more
A new objective performance measure for image captioning is introduced and methods incorporating many state of the art, but fairly noisy, estimates of image content are developed to produce even more pleasing results.
IEEE Transactions on Pattern Analysis and Machine IntelligenceBabyTalk: Understanding and Generating Simple Image Descriptions
874 Citations2013Girish Kulkarni, Visruth Premraj +6 more
The proposed system to automatically generate natural language descriptions from images is very effective at producing relevant sentences for images and generates descriptions that are notably more true to the specific image content than previous work.
IEEE Transactions on Information TheoryMAP Estimation Via Agreement on Trees: Message-Passing and Linear Programming
670 Citations2005Martin J. Wainwright, Tommi Jaakkola +1 more
This work develops and analyze methods for computing provably optimal maximum a posteriori probability (MAP) configurations for a subclass of Markov random fields defined on graphs with cycles and establishes a connection between a certain LP relaxation of the mode-finding problem and a reweighted form of the max-product (min-sum) message-passing algorithm.
Journal of Experimental Psychology GeneralPerceiving, remembering, and communicating structure in events.
626 Citations2001Jeffrey M. Zacks, Barbara Tversky +1 more
Both segmentation and descriptions support the hierarchical bias hypothesis in event perception: Observers spontaneously encoded the events in terms of partonomic hierarchies and indicate that these knowledge structures may be organized around object/action units.
Hello! My name is... Buffy'' -- Automatic Naming of Characters in TV Video
567 Citations2006Mark Everingham, Josef Šivic +1 more
It is demonstrated that high precision can be achieved by combining multiple sources of information, both visual and textual, by automatic generation of time stamped character annotation by aligning subtitles and transcripts.
Baby talk: Understanding and generating simple image descriptions
536 Citations2011Girish Kulkarni, Visruth Premraj +5 more
A system to automatically generate natural language descriptions from images that exploits both statistics gleaned from parsing large quantities of text data and recognition algorithms from computer vision that is very effective at producing relevant sentences for images.
Collecting Image Annotations Using Amazon's Mechanical Turk
524 Citations2010Cyrus Rashtchian, Peter Young +2 more
It is found that the use of a qualification test provides the highest improvement of quality, whereas refining the annotations through follow-up tasks works rather poorly.
Matching Words and Pictures
459 Citations2003Kobus Barnard, Pinar Duygulu +8 more
Lecture notes in computer scienceAutomatic Attribute Discovery and Characterization from Noisy Web Data
430 Citations2010Tamara L. Berg, Alexander C. Berg +1 more
This work focuses on discovering attributes and their visual appearance, and is as agnostic as possible about the textual description, and characterizes attributes according to their visual representation: global or local, and type: color, texture, or shape.
Learning Visual Attributes
397 Citations2007Vittorio Ferrari, Andrew Zisserman
It is shown that attributes can be learnt starting from a text query to Google image search, and can then be used to recognize the attribute and determine its spatial extent in novel real-world images.
Composing Simple Image Descriptions using Web-scale N-grams
360 Citations2011Siming Li, Girish Kulkarni +3 more
A simple yet effective approach to automatically compose image descriptions given computer vision based inputs and using web-scale n-grams, which indicates that it is viable to generate simple textual descriptions that are pertinent to the specific content of an image, while permitting creativity in the description -- making for more human-like annotations than previous approaches.
International Journal of Computer VisionNatural Language Description of Human Activities from Video Images Based on Concept Hierarchy of Actions
356 Citations2002Atsuhiro Kojima, Takeshi Tamura +1 more
A method for describing human activities from video images based on concept hierarchies of actions based on semantic primitives, which demonstrates the performance of the proposed method by several experiments.
Names and faces in the news
346 Citations2004Tamara L. Berg, Alexander C. Berg +6 more
It is shown quite good face clustering is possible for a dataset of inaccurately and ambiguously labelled face images, obtained by applying a face finder to approximately half a million captioned news images.
Corpus-Guided Sentence Generation of Natural Images
343 Citations2011Yezhou Yang, Ching L. Teo +2 more
Experimental results show that the strategy of combining vision and language produces readable and descriptive sentences compared to naive strategies that use vision alone.
Collective Generation of Natural Image Descriptions
332 Citations2012Полина Кузнецова, Vicente Ordóñez +3 more
A holistic data-driven approach to image description generation, exploiting the vast amount of (noisy) parallel image data and associated natural language descriptions available on the web to generate novel descriptions for query images.
Proceedings of the IEEEI2T: Image Parsing to Text Description
318 Citations2010Benjamin Yao, Xiong Yang +3 more
An image parsing to text description (I2T) framework that generates text descriptions of image and video content based on image understanding and uses automatic methods to parse image/video in specific domains and generate text reports that are useful for real-world applications.
Discriminative models for multi-class object layout
282 Citations2009Chaitanya Desai, Deva Ramanan +1 more
2009 IEEE Conference on Computer Vision and Pattern RecognitionUnderstanding videos, constructing plots learning a visually grounded storyline model from annotated videos
241 Citations2009Abhinav Gupta, Praveen Srinivasan +2 more
An Integer Programming framework for action recognition and storyline extraction using the storyline model and visual groundings learned from training data is formulated.
2009 IEEE Conference on Computer Vision and Pattern Recognition“Who are you?” - Learning person specific classifiers from video
225 Citations2009Josef Šivic, Mark Everingham +1 more
A character specific multiple kernel classifier which is able to learn the features best able to discriminate between the characters is reported, demonstrating significantly increased coverage and performance with respect to previous methods on this material.
Learning Models for Object Recognition from Natural Language Descriptions
210 Citations2009Josiah Wang, Katja Markert +1 more
This work proposes natural language processing methods for extracting salient visual attributes from natural language descriptions to use as ‘templates’ for the object categories, and applies vision methods to extract corresponding attributes from test images.
International Journal of Computer VisionOPTIMOL: Automatic Online Picture Collection via Incremental Model Learning
195 Citations2009Li-Jia Li, Li Fei-Fei
This paper presents a novel object recognition algorithm that performs automatic dataset collecting and incremental model learning simultaneously, and adapts a non-parametric latent topic model and proposes an incremental learning framework.
Harvesting Image Databases from the Web
194 Citations2007Florian Schroff, Antonio Criminisi +1 more
A multi-modal approach employing both text, meta data and visual features is used to gather many, high-quality images from the Web to automatically generate a large number of images for a specified object class.
Animals on the Web
181 Citations2006Tamara L. Berg, David Forsyth
This work demonstrates a method for identifying images containing categories of animals using a clustering method applied to text on web pages and shows unequivocal evidence that visual information improves performance for this task.
Lecture notes in computer scienceBeyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers
180 Citations2008Abhinav Gupta, Larry S. Davis
This work simultaneously learns the visual features defining "nouns" and the differentialVisual features defining such "binary-relationships" using an EM-based approach and constrain the correspondence problem by exploiting additional language constructs to improve the learning process from weakly labeled data.
Clustering art
160 Citations2005Kobus Barnard, Pınar Duygulu +1 more
This work extends a recently developed method for learning the semantics of image databases using text and pictures and uses WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.
Communications of the ACMUsing the forest to see the trees
129 Citations2010Antonio Torralba, Kevin Murphy +1 more
A probabilistic framework for encoding the relationships between context and object properties is used and it is shown how an integrated system provides improved performance, viewed as a significant step toward general purpose machine vision systems.
Generating Image Descriptions Using Dependency Relational Patterns
85 Citations2010Ahmet Aker, Robert Gaizauskas
The results show that summaries biased by dependency pattern models lead to significantly higher ROUGE scores than both n-gram language models reported in previous work and also Wikipedia baseline summaries.
Edinburgh Research Explorer (University of Edinburgh)How Many Words Is a Picture Worth? Automatic Caption Generation for News Images
75 Citations2010Yansong Feng, Mirella Lapata
Experimental results show that an abstractive model defined over phrases is superior to extractive methods.
Image region entropy
74 Citations2005Keiji Yanai, Kobus Barnard
A new method to measure "visualness" of concepts, that is, what extent concepts have visual characteristics, which performs probabilistic region selection for images which are labeled as concept "X" or "non-X", and computes an entropy measure which represents "visuals" of concept.
Unsupervised Learning of Visual Sense Models for Polysemous Words
51 Citations2008Kate Saenko, Trevor Darrell
A new method that uses a dictionary to learn models of visual word sense from a large collection of unlabeled web data and the use of LDA to discover a latent sense space makes the model robust despite the very limited nature of dictionary definitions.
Template-Filtered Headline Summarization
25 Citations2004Liang Zhou, Eduard Hovy
This paper describes the first attempt toward solving the problem of headline summarization with a system that generates key headline clusters and fine-tunes them using templates.
Proceedings of the AAAI Conference on Artificial IntelligenceUsing Closed Captions as Supervision for Video Activity Recognition
25 Citations2010Sonal Gupta, Raymond J. Mooney
It is demonstrated that combining linguistic analysis and automatically trained activity recognizers can significantly improve the precision of video retrieval.
Journal of VisionWhat do we see when we glance at a scene?
19 Citations2004Li Fei-Fei, Christof Koch +2 more
2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition WorkshopsUsing closed captions to train activity recognizers that improve video retrieval
15 Citations2009Sonal Gupta, Raymond J. Mooney
It is demonstrated that combining linguistic analysis and automatically trained activity recognizers can significantly improve the precision of video retrieval.
Finding visual concepts by web image mining
13 Citations2006Keiji Yanai, Kobus Barnard
The method performs probabilistic region selection for images and computes an entropy measure which represents "visualness" of concepts which enables mining image data on the Web with this new application of "Web image mining".