Image retrieval using scene graphs
Published 1 June 2015
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein
Citations1,097
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A conditional random field model that reasons about possible groundings of scene graphs to test images and shows that the full model can be used to improve object localization compared to baseline methods and outperforms retrieval methods that use only objects or low-level image features.
Abstract
This paper develops a novel framework for semantic image retrieval based on the notion of a scene graph. Our scene graphs represent objects (“man”, “boat”), attributes of objects (“boat is white”) and relationships between objects (“man standing on boat”). We use these scene graphs as queries to retrieve semantically related images. To this end, we design a conditional random field model that reasons about possible groundings of scene graphs to test images. The likelihoods of these groundings are used as ranking scores for retrieval. We introduce a novel dataset of 5,000 human-generated scene graphs grounded to images and use this dataset to evaluate our method for image retrieval. In particular, we evaluate retrieval using full scene graphs and small scene subgraphs, and show that our method outperforms retrieval methods that use only objects or low-level image features. In addition, we show that our full model can be used to improve object localization compared to baseline methods.
Keywords
Computer Science
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
61,841 Citations2009Jia Deng, Wei Dong +4 more
A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
International Journal of Computer VisionDistinctive Image Features from Scale-Invariant Keypoints
55,266 Citations2004David Lowe
This paper presents a method for extracting distinctive invariant features from images that can be used to perform reliable matching between different views of an object or scene and can robustly identify objects among clutter and occlusion while achieving near real-time performance.
Lecture notes in computer scienceMicrosoft COCO: Common Objects in Context
42,219 Citations2014Tsung-Yi Lin, Michael Maire +6 more
A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.
International Journal of Computer VisionImageNet Large Scale Visual Recognition Challenge
40,207 Citations2015Olga Russakovsky, Jia Deng +10 more
The creation of this benchmark dataset and the advances in object recognition that have been possible as a result are described, and the state-of-the-art computer vision accuracy with human accuracy is compared.
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
31,752 Citations2014Ross Girshick, Jeff Donahue +2 more
This paper proposes a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 -- achieving a mAP of 53.3%.
International Journal of Computer VisionThe Pascal Visual Object Classes (VOC) Challenge
19,539 Citations2009Mark Everingham, Luc Van Gool +3 more
Are we ready for autonomous driving? The KITTI vision benchmark suite
14,407 Citations2012Andreas Geiger, P Lenz +1 more
The autonomous driving platform is used to develop novel challenging benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM and 3D object detection, revealing that methods ranking high on established datasets such as Middlebury perform below average when being moved outside the laboratory to the real world.
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Caffe
11,194 Citations2014Yangqing Jia, Evan Shelhamer +6 more
Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures.
International Journal of Computer VisionModeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
6,410 Citations2001Aude Oliva, Antonio Torralba
The performance of the spatial envelope model shows that specific information about object shape or identity is not a requirement for scene categorization and that modeling a holistic representation of the scene informs about its probable semantic category.
International Journal of Computer VisionSelective Search for Object Recognition
6,118 Citations2013Jasper Uijlings, Koen E. A. van de Sande +2 more
This paper introduces selective search which combines the strength of both an exhaustive search and segmentation, and shows that its selective search enables the use of the powerful Bag-of-Words model for recognition.
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
4,863 Citations1999John Platt
The output of a lassi(cid:12)er should be a alibrated posterior probability to enable post-pro essing and a method to train a kernel lassi with a logit link and a regularized maximum likelihood is proposed.
International Journal of Computer VisionLabelMe: A Database and Web-Based Tool for Image Annotation
4,205 Citations2007Bryan Russell, Antonio Torralba +2 more
A web-based tool that allows easy image annotation and instant sharing of such annotations is developed and a large dataset that spans many object categories, often containing multiple instances over a wide variety of images is collected.
SUN database: Large-scale scene recognition from abbey to zoo
3,202 Citations2010Jianxiong Xiao, James Hays +3 more
This paper proposes the extensive Scene UNderstanding (SUN) database that contains 899 categories and 130,519 images and uses 397 well-sampled categories to evaluate numerous state-of-the-art algorithms for scene recognition and establish new bounds of performance.
IEEE Transactions on Pattern Analysis and Machine IntelligenceProduct Quantization for Nearest Neighbor Search
2,885 Citations2010H. Jegou, Matthijs Douze +1 more
This paper introduces a product quantization-based approach for approximate nearest neighbor search to decompose the space into a Cartesian product of low-dimensional subspaces and to quantize each subspace separately.
Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories
2,482 Citations2005Li Fei-Fei, Rob Fergus +1 more
Transactions of the Association for Computational LinguisticsFrom image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
2,461 Citations2014Peter Young, Alice Lai +2 more
This work proposes to use the visual denotations of linguistic expressions to define novel denotational similarity metrics, which are shown to be at least as beneficial as distributional similarities for two tasks that require semantic inference.
2009 IEEE Conference on Computer Vision and Pattern RecognitionDescribing objects by their attributes
1,928 Citations2009Ali Farhadi, Ian Endres +2 more
This paper proposes to shift the goal of recognition from naming to describing, and introduces a novel feature selection method for learning attributes that generalize well across categories.
IEEE Transactions on Pattern Analysis and Machine IntelligenceAttribute-Based Classification for Zero-Shot Visual Object Categorization
1,601 Citations2013Christoph H. Lampert, Hannes Nickisch +1 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceAggregating Local Image Descriptors into Compact Codes
1,535 Citations2011H. Jegou, Florent Perronnin +4 more
This paper first presents and evaluates different ways of aggregating local image descriptors into a vector and shows that the Fisher kernel achieves better performance than the reference bag-of-visual words approach for any given vector dimension.
The Role of Context for Object Detection and Semantic Segmentation in the Wild
1,436 Citations2014Roozbeh Mottaghi, Xianjie Chen +6 more
A novel deformable part-based model is proposed, which exploits both local context around each candidate detection as well as global context at the level of the scene, which significantly helps in detecting objects at all scales.
Journal of Artificial Intelligence ResearchFraming Image Description as a Ranking Task: Data, Models and Evaluation Metrics
1,329 Citations2013Micah Hodosh, Peter Young +1 more
This work proposes to frame sentence-based image annotation as the task of ranking a given pool of captions, and introduces a new benchmark collection, consisting of 8,000 images that are each paired with five different captions which provide clear descriptions of the salient entities and events.
IEEE Transactions on Pattern Analysis and Machine IntelligenceMeasuring the Objectness of Image Windows
1,231 Citations2012Bogdan Alexe, Thomas Deselaers +1 more
A generic objectness measure, quantifying how likely it is for an image window to contain an object of any class, and uses objectness as a complementary score in addition to the class-specific model, which leads to fewer false positives.
Relative attributes
893 Citations2011Devi Parikh, Kristen Grauman
This work proposes a generative model over the joint space of attribute ranking outputs, and proposes a novel form of zero-shot learning in which the supervisor relates the unseen object category to previously seen objects via attributes (for example, ‘bears are furrier than giraffes’).
Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval
861 Citations2007Ondřej Chum, James Philbin +3 more
This paper brings query expansion into the visual domain via two novel contributions: strong spatial constraints between the query image and each result allow us to accurately verify each return, suppressing the false positives which typically ruin text-based query expansion.
Transactions of the Association for Computational LinguisticsGrounded Compositional Semantics for Finding and Describing Images with Sentences
831 Citations2014Richard Socher, Andrej Karpathy +3 more
The DT-RNN model, which uses dependency trees to embed sentences into a vector space in order to retrieve images that are described by those sentences, outperform other recursive and recurrent neural networks, kernelized CCA and a bag-of-words baseline on the tasks of finding an image that fits a sentence description and vice versa.
Large-scale image retrieval with compressed Fisher vectors
761 Citations2010Florent Perronnin, Yan Liu +2 more
This article shows why the Fisher representation is well-suited to the retrieval problem: it describes an image by what makes it different from other images, and why it should be compressed to reduce their memory footprint and speed-up the retrieval.
arXiv (Cornell University)Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
726 Citations2014Andrej Karpathy, Armand Joulin +1 more
Baby talk: Understanding and generating simple image descriptions
536 Citations2011Girish Kulkarni, Visruth Premraj +5 more
A system to automatically generate natural language descriptions from images that exploits both statistics gleaned from parsing large quantities of text data and recognition algorithms from computer vision that is very effective at producing relevant sentences for images.
2009 IEEE Conference on Computer Vision and Pattern RecognitionTowards total scene understanding: Classification, annotation and segmentation in an automatic framework
496 Citations2009Li-Jia Li, Richard Socher +1 more
A fully automatic learning framework that is able to learn robust scene models from noisy Web data such as images and user tags from Flickr.com that significantly outperforms state-of-the-art algorithms.
NEIL: Extracting Visual Knowledge from Web Data
450 Citations2013Xinlei Chen, Abhinav Shrivastava +1 more
NEIL (Never Ending Image Learner), a computer program that runs 24 hours per day and 7 days per week to automatically extract visual knowledge from Internet data, is proposed in an attempt to develop the world's largest visual structured knowledge base with minimum human labeling effort.
Recognition using visual phrases
449 Citations2011Mohammad Amin Sadeghi, Ali Farhadi
It is shown that a visual phrase detector significantly outperforms a baseline which detects component objects and reasons about relations, even though visual phrase training sets tend to be smaller than those for objects.
Lecture notes in computer scienceAutomatic Attribute Discovery and Characterization from Noisy Web Data
430 Citations2010Tamara L. Berg, Alexander C. Berg +1 more
This work focuses on discovering attributes and their visual appearance, and is as agnostic as possible about the textual description, and characterizes attributes according to their visual representation: global or local, and type: color, texture, or shape.
Conference of the European Chapter of the Association for Computational LinguisticsMidge: Generating Image Descriptions From Computer Vision Detections
430 Citations2012Margaret Mitchell, Jesse Dodge +8 more
A novel generation system that composes humanlike descriptions of images from computer vision detections by leveraging syntactically informed word co-occurrence statistics and automatically generating some of the most natural image descriptions to date.
Lecture notes in computer scienceEfficient Object Category Recognition Using Classemes
420 Citations2010Lorenzo Torresani, Martin Szummer +1 more
A new descriptor for images is introduced which allows the construction of efficient and compact classifiers with good accuracy on object category recognition, and allows object-category queries to be made against image databases using efficient classifiers such as linear support vector machines.
Learning Visual Attributes
397 Citations2007Vittorio Ferrari, Andrew Zisserman
It is shown that attributes can be learnt starting from a text query to Google image search, and can then be used to recognize the attribute and determine its spatial extent in novel real-world images.
Lecture notes in computer scienceGeodesic Object Proposals
349 Citations2014Philipp Krähenbühl, Vladlen Koltun
An approach for identifying a set of candidate objects in a given image that can be used for object recognition, segmentation, and other object-based image parsing tasks is presented.
Conference on Computational Natural Language LearningStanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task
347 Citations2011Heeyoung Lee, Yves Peirsman +4 more
A Multi-Pass Sieve for Coreference Resolution
326 Citations2010Karthik Raghunathan, Heeyoung Lee +5 more
This work proposes a simple coreference architecture based on a sieve that applies tiers of deterministic coreference models one at a time from highest to lowest precision, and outperforms many state-of-the-art supervised and unsupervised models on several standard corpora.
Exploiting hierarchical context on a large database of object categories
324 Citations2010Myung Jin Choi, Joseph J. Lim +2 more
This paper introduces a new dataset with images that contain many instances of different object categories and proposes an efficient model that captures the contextual information among more than a hundred ofobject categories and shows that the context model can be applied to scene understanding tasks that local detectors alone cannot solve.
Learning Everything about Anything: Webly-Supervised Visual Concept Learning
307 Citations2014Santosh Divvala, Ali Farhadi +1 more
A fully-automated approach for learning extensive models for a wide range of variations within any concept, which leverages vast resources of online books to discover the vocabulary of variance, and intertwines the data collection and modeling steps to alleviate the need for explicit human supervision in training the models.
arXiv (Cornell University)The New Data and New Challenges in Multimedia Research.
249 Citations2015Bart Thomée, David A. Shamma +6 more
The rationale behind the creation of the YFCC100M, the largest public multimedia collection that has ever been released, is explained, as well as the implications the dataset has for science, research, engineering, and development.
Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora
231 Citations2010Richard Socher, Li Fei-Fei
A semi-supervised model which segments and annotates images using very few labeled images and a large unaligned text corpus to relate image regions to text labels and outperforms the state-of-the-art in annotation.
International Journal of Computer VisionDiscriminative Models for Multi-Class Object Layout
212 Citations2011Chaitanya Desai, Deva Ramanan +1 more
A unified model for multi-class object recognition is introduced that casts the problem as a structured prediction task and how to formulate learning as a convex optimization problem is shown.
Bringing Semantics into Focus Using Visual Abstraction
205 Citations2013C. Lawrence Zitnick, Devi Parikh
This paper creates 1,002 sets of 10 semantically similar abstract scenes with corresponding written descriptions and thoroughly analyzes this dataset to discover semantically important features, the relations of words to visual features and methods for measuring semantic similarity.
Spatial-bag-of-features
202 Citations2010Yang Cao, Changhu Wang +3 more
The proposed retrieval framework works well in image retrieval task owing to the encoding of geometric information of objects for capturing objects' spatial transformation, the supervised feature selection and combination strategy for enhancing the discriminative power, and the representation of bag-of-features for effective image matching and indexing for large scale image retrieval.
Crowdsourcing annotations for visual object detection
201 Citations2012Hao Su, Jia Deng +1 more
This paper studies strategies to crowd-source bounding box annotations and presents a system that consists of three simple sub-tasks — a drawing task, a quality verification task and a coverage veri-cation task that is scalable, accurate, and cost-effective.
Attribute-centric recognition for cross-category generalization
193 Citations2010Ali Farhadi, Ian Endres +1 more
This work introduces a new dataset that provides annotation for sharing models of appearance and correlation across categories and uses it to learn part and category detectors that serve as the visual basis for an integrated model of objects.
COSTA: Co-Occurrence Statistics for Zero-Shot Classification
192 Citations2014Thomas Mensink, Efstratios Gavves +1 more
The main contribution is COSTA, which exploits co-occurrences of visual concepts in images for knowledge transfer, which estimates a classifier for a new label, as a weighted combination of related classes, using the co-Occurrences to define the weight.
Understanding Indoor Scenes Using 3D Geometric Phrases
189 Citations2013Wongun Choi, Yu-Wei Chao +2 more
A hierarchical scene model for learning and reasoning about complex indoor scenes which is computationally tractable, can be learned from a reasonable amount of training data, and avoids oversimplification is presented.
Lecture notes in computer scienceIntroduction to a Large-Scale General Purpose Ground Truth Database: Methodology, Annotation Tool and Benchmarks
188 Citations2007Benjamin Yao, Xiong Yang +1 more
A large scale general purpose image database with human annotated ground truth consisting of more than 636,748 annotated images and video frames is presented.
arXiv (Cornell University)Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
181 Citations2014Andrej Karpathy, Armand Joulin +1 more
This work introduces a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data and introduces a structured max-margin objective that allows this model to explicitly associate fragments across modalities.
Lecture notes in computer scienceBeyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers
180 Citations2008Abhinav Gupta, Larry S. Davis
This work simultaneously learns the visual features defining "nouns" and the differentialVisual features defining such "binary-relationships" using an EM-based approach and constrain the correspondence problem by exploiting additional language constructs to improve the learning process from weakly labeled data.
Context Driven Scene Parsing with Attention to Rare Classes
170 Citations2014Shuicheng Yan, Brian Price +2 more
This paper focuses on rare object classes, which play an important role in achieving richer semantic understanding of visual scenes, compared to common background classes, and makes two novel contributions: rare class expansion and semantic context description.
Learning the Visual Interpretation of Sentences
168 Citations2013C. Lawrence Zitnick, Devi Parikh +1 more
This paper extracts predicate tuples that contain two nouns and a relation from sentences to generate novel scenes depicting the sentences' visual meaning by sampling from the Conditional Random Field (CRF).
DeepDive: Web-scale Knowledge-base Construction using Statistical Learning and Inference
167 Citations2012Feng Niu, Ce Zhang +2 more
An end-to-end (live) demonstration system called DeepDive is presented that performs knowledge-base construction (KBC) from hundreds of millions of web pages and addresses the scalability challenges to achieve web-scale KBC.
Transactions of the Association for Computational LinguisticsJointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World
164 Citations2013Jayant Krishnamurthy, Thomas Kollar
This paper introduces Logical Semantics with Perception (LSP), a model for grounded language acquisition that learns to map natural language statements to their referents in a physical environment and finds that LSP outperforms existing, less expressive models that cannot represent relational language.
Learning Spatial Knowledge for Text to 3D Scene Generation
158 Citations2014Anne Lynn S. Chang, Manolis Savva +1 more
The main innovation of this work is to show how to augment explicit constraints with learned spatial knowledge to infer missing objects and likely layouts for the objects in the scene.
Characterizing structural relationships in scenes using graph kernels
156 Citations2011Matthew Fisher, Manolis Savva +1 more
This paper shows how to represent scenes as graphs that encode models and their semantic relationships, and shows that incorporating structural relationships allows the method to provide a more relevant set of results when compared against previous approaches to model context search.
Scene Parsing by Integrating Function, Geometry and Appearance Models
152 Citations2013Yibiao Zhao, Song‐Chun Zhu
The proposed approach not only significantly widens the scope of indoor scene parsing algorithm from the segmentation and the 3D recovery to the functional object recognition, but also yields improved overall performance.
Visual Semantic Search: Retrieving Videos via Complex Textual Queries
149 Citations2014Dahua Lin, Sanja Fidler +2 more
This paper first parse the sentential descriptions into a semantic graph, which is then matched to visual concepts using a generalized bipartite matching algorithm, and learns the importance of each term using structure prediction.
PiCoDes: Learning a Compact Code for Novel-Category Recognition
97 Citations2011Alessandro Bergamo, Lorenzo Torresani +1 more
PICODES: a very compact image descriptor which nevertheless allows high performance on object category recognition and an alternation scheme and convex upper bound which demonstrate excellent performance in practice are presented.
Query Adaptive Similarity for Large Scale Object Retrieval
85 Citations2013Danfeng Qin, Christian Wengert +1 more
This paper presents a probabilistic framework for modeling the feature to feature similarity measure, and proposes a function to score the individual contributions into an image to image similarity within the probabilism framework.
Predicting Object Dynamics in Scenes
83 Citations2014David F. Fouhey, C. Lawrence Zitnick
This paper learns from sequences of abstract images gathered using crowd-sourcing to overcome a lack of densely annotated spatiotemporal data, and demonstrates qualitatively and quantitatively that their models produce plausible scene predictions on both the abstract images, as well as natural images taken from the Internet.
arXiv (Cornell University)OpenGM: A C++ Library for Discrete Graphical Models
79 Citations2012Bjoern Andres, Thorsten Beier +1 more
OpenGM is a C++ template library for defining discrete graphical models and performing inference on these models, using a wide range of state-of-the-art algorithms, and its algorithms, HDF5 file format and command line tools are modular and extendible.
Lirias (KU Leuven)Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task
75 Citations2012Heeyoung Lee, Yves Peirsman +4 more
Detecting objects in large image collections and videos by efficient subimage retrieval
64 Citations2009Christoph H. Lampert
An extensive evaluation on several datasets shows that ESR is not only very fast, but it also achieves excellent detection accuracies thereby improving over previous systems for object-based image retrieval.
Efficient indexing for large scale visual search
39 Citations2009Xiao Zhang, Zhiwei Li +3 more