Every Picture Tells a Story: Generating Sentences from Images
Lecture notes in computer sciencePublished 1 January 2010
Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier
Citations1,249
SJR quartileQ2
SJR score0.35
SNIP0.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A system that can compute a score linking an image to a sentence, which can be used to attach a descriptive sentence to a given image, or to obtain images that illustrate a given sentence.
Abstract
Humans can prepare concise descriptions of pictures, focusing on what they find important. We demonstrate that automatic methods can do so too. We describe a system that can compute a score linking an image to a sentence. This score can be used to attach a descriptive sentence to a given image, or to obtain images that illustrate a given sentence. The score is obtained by comparing an estimate of meaning obtained from the image to one obtained from the sentence. Each estimate of meaning comes from a discriminative procedure that is learned using data. We evaluate on a novel dataset consisting of human-annotated images. While our underlying estimate of meaning is impoverished, it is sufficient to produce very good quantitative results, evaluated with a novel score that can account for synecdoche.
Keywords
Computer Science
An Information-Theoretic Definition of Similarity
3,662 Citations1998Dekang Lin
This work presents an informationtheoretic definition of similarity that is applicable as long as there is a probabilistic model and demonstrates how this definition can be used to measure the similarity in a number of different domains.
Learning realistic human actions from movies
3,493 Citations2008Ivan Laptev, Marcin Marszałek +2 more
A new method for video classification that builds upon and extends several recent ideas including local space-time features,space-time pyramids and multi-channel non-linear SVMs is presented and shown to improve state-of-the-art results on the standard KTH action dataset.
A discriminatively trained, multiscale, deformable part model
2,882 Citations2008Pedro F. Felzenszwalb, David McAllester +1 more
A discriminatively trained, multiscale, deformable part model for object detection, which achieves a two-fold improvement in average precision over the best performance in the 2006 PASCAL person detection challenge and outperforms the best results in the 2007 challenge in ten out of twenty categories.
Lecture notes in computer scienceObject Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
1,608 Citations2002Pınar Duygulu, Kobus Barnard +2 more
This work shows how to cluster words that individually are difficult to predict into clusters that can be predicted well, and cannot predict the distinction between train and locomotive using the current set of features, but can predict the underlying concept.
Progress in brain researchChapter 2 Building the gist of a scene: the role of global image features in recognition
1,396 Citations2006Aude Oliva, Antonio Torralba
It is shown that the structure of a scene image can be estimated by the mean of global image features, providing a statistical summary of the spatial layout properties (Spatial Envelope representation) of the scene.
What, where and who? Classifying events by scene and object recognition
794 Citations2007Li-Jia Li, Li Fei-Fei
This paper uses a number of sport games such as snow boarding, rock climbing or badminton to demonstrate event classification and proposes a first attempt to classify events in static images by integrating scene and object categorizations.
Modeling mutual context of object and human pose in human-object interaction activities
612 Citations2010Bangpeng Yao, Li Fei-Fei
A new random field model is proposed to encode the mutual context of objects and human poses in human-object interaction activities and it is shown that this mutual context model significantly outperforms state-of-the-art in detecting very difficult objects andhuman poses.
IEEE Transactions on Pattern Analysis and Machine IntelligenceObserving Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition
528 Citations2009Abhinav Gupta, Aniruddha Kembhavi +1 more
This work presents a Bayesian approach which applies spatial and functional constraints on each of the perceptual elements for coherent semantic interpretation and demonstrates the use of such constraints in recognition of actions from static images without using any motion information.
Collecting Image Annotations Using Amazon's Mechanical Turk
524 Citations2010Cyrus Rashtchian, Peter Young +2 more
It is found that the use of a qualification test provides the highest improvement of quality, whereas refining the annotations through follow-up tasks works rather poorly.
2009 IEEE Conference on Computer Vision and Pattern RecognitionTowards total scene understanding: Classification, annotation and segmentation in an automatic framework
496 Citations2009Li-Jia Li, Richard Socher +1 more
A fully automatic learning framework that is able to learn robust scene models from noisy Web data such as images and user tags from Flickr.com that significantly outperforms state-of-the-art algorithms.
Learning structured prediction models
462 Citations2005Ben Taskar, Vassil Chatalbashev +2 more
This work considers large margin estimation in a broad range of prediction models where inference involves solving combinatorial optimization problems, for example, weighted graph-cuts or matchings, and relies on the expressive power of convex optimization problems to compactly capture inference or solution optimality in structured prediction models.
Content-based image retrieval
432 Citations2005Ritendra Datta, Jia Li +1 more
Some of the key contributions in the current decade related to image retrieval and automated image annotation are discussed, spanning 120 references, and a study on the trends in volume and impact of publications in the field with respect to venues/journals and sub-topics is concluded.
WordsEye
381 Citations2001Bob Coyne, Richard Sproat
The linguistic analysis and depiction techniques used by WordsEye are described along with some general strategies by which more abstract concepts are made depictable.
Proceedings of the IEEEI2T: Image Parsing to Text Description
318 Citations2010Benjamin Yao, Xiong Yang +3 more
An image parsing to text description (I2T) framework that generates text descriptions of image and video content based on image understanding and uses automatic methods to parse image/video in specific domains and generate text reports that are useful for real-world applications.
Linguistically motivated large-scale NLP with C&C and boxer
254 Citations2007James Curran, Stephen Clark +1 more
An NLP system which is based on syntactic and semantic formalisms from theoretical linguistics, and which is used to analyse the entire Gigaword corpus in less than 5 days using only 18 processors, represents a break-through in NLP technology.
Objects in Action: An Approach for Combining Action Understanding and Object Perception
251 Citations2007Abhinav Gupta, Larry S. Davis
A Bayesian approach is presented which unifies the inference processes involved in object classification and localization, action understanding and perception of object reaction, and which can segment and recognize actions which are either too subtle to perceive or too hard to recognize using motion features alone.
2009 IEEE Conference on Computer Vision and Pattern RecognitionUnderstanding videos, constructing plots learning a visually grounded storyline model from annotated videos
241 Citations2009Abhinav Gupta, Praveen Srinivasan +2 more
An Integer Programming framework for action recognition and storyline extraction using the storyline model and visual groundings learned from training data is formulated.
Lecture notes in computer scienceBeyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers
180 Citations2008Abhinav Gupta, Larry S. Davis
This work simultaneously learns the visual features defining "nouns" and the differentialVisual features defining such "binary-relationships" using an EM-based approach and constrain the correspondence problem by exploiting additional language constructs to improve the learning process from weakly labeled data.
Clustering art
160 Citations2005Kobus Barnard, Pınar Duygulu +1 more
This work extends a recently developed method for learning the semantics of image databases using text and pictures and uses WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.
Meta-analysis of face recognition algorithms
52 Citations2003P. Jonathon Phillips, Elaine M. Newton
A meta-analysis of performance results of face recognition algorithms in the literature shows that control scores are predictive of performance of novel algorithms at statistically significant levels.
Who’s Doing What: Joint Modeling of Names and Verbs for Simultaneous Face and Pose Annotation
36 Citations2009Jie Luo, Barbara Caputo +1 more
A joint model for simultaneously solving the image-caption correspondences and learning visual appearance models for the face and pose classes occurring in the corpus can be used to recognize people and actions in novel images without captions.
Lecture notes in computer scienceImproving People Search Using Query Expansions
25 Citations2008Thomas Mensink, Jakob Verbeek
To improve search results by exploiting faces of other people that co-occur frequently with the queried person, query expansion is used to provide a query-specific relevant set of `negative' examples which should be separated from the potentially positive examples in the text-based result set.
Cambridge University Press eBooksWords and Pictures: Categories, Modifiers, Depiction, and Iconography
12 Citations2009David Forsyth, Tamara L. Berg +5 more
Meta-analysis of face recognition algorithms
5 Citations2001P. Jonathon Phillips, Elaine M. Newton