Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
arXiv (Cornell University)Published 10 February 2015Open access
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Salakhutdinov, Ruslan
Citations7,525
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An attention based model that automatically learns to describe the content of images is introduced that can be trained in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound.
Abstract
Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
arXiv (Cornell University)Very Deep Convolutional Networks for Large-Scale Image Recognition
75,540 Citations2014Karen Simonyan, Andrew Zisserman
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
Going deeper with convolutions
46,825 Citations2015Christian Szegedy, Wei Liu +7 more
A deep convolutional neural network architecture codenamed Inception is proposed that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).
Lecture notes in computer scienceMicrosoft COCO: Common Objects in Context
42,219 Citations2014Tsung-Yi Lin, Michael Maire +6 more
A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
UvA-DARE (University of Amsterdam)Auto-Encoding Variational Bayes
15,628 Citations2013Diederik P. Kingma, Max Welling
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
Nature reviews. NeuroscienceControl of goal-directed and stimulus-driven attention in the brain
12,975 Citations2002Maurizio Corbetta, Gordon L. Shulman
Evidence for partially segregated networks of brain areas that carry out different attentional functions is reviewed, finding that one system is involved in preparing and applying goal-directed selection for stimuli and responses, and the other is specialized for the detection of behaviourally relevant stimuli.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
Show and tell: A neural image caption generator
6,363 Citations2015Oriol Vinyals, Alexander Toshev +2 more
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image.
arXiv (Cornell University)Practical Bayesian Optimization of Machine Learning Algorithms
5,612 Citations2012Jasper Snoek, Hugo Larochelle +1 more
This work describes new algorithms that take into account the variable cost of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation and shows that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms.
Long-term recurrent convolutional networks for visual recognition and description
5,282 Citations2015Jeff Donahue, Lisa Anne Hendricks +5 more
A novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and shows such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.
Deep visual-semantic alignments for generating image descriptions
4,978 Citations2015Andrej Karpathy, Li Fei-Fei
A model that generates natural language descriptions of images and their regions using a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding is presented.
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)LOL: An Investigation into Cybernetic Humor, or: Can Machines Laugh?
3,084 Citations2016Alex Graves, Gervasi, Vincenzo +1 more
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time.
arXiv (Cornell University)Stochastic Backpropagation and Approximate Inference in Deep Generative Models
2,644 Citations2014Danilo Jimenez Rezende, Shakir Mohamed +1 more
Transactions of the Association for Computational LinguisticsFrom image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
2,461 Citations2014Peter Young, Alice Lai +2 more
This work proposes to use the visual denotations of linguistic expressions to define novel denotational similarity metrics, which are shown to be at least as beneficial as distributional similarities for two tasks that require semantic inference.
arXiv (Cornell University)Recurrent Neural Network Regularization
2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more
This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
2,080 Citations2014Michael Denkowski, Alon Lavie
Meteor Universal brings language specific evaluation to previously unsupported target languages by automatically extracting linguistic resources from the bitext used to train MT systems and using a universal parameter set learned from pooling human judgments of translation quality from several language directions.
Recurrent Continuous Translation Models
1,345 Citations2013Nal Kalchbrenner, Phil Blunsom
A class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units are introduced.
Journal of Artificial Intelligence ResearchFraming Image Description as a Ranking Task: Data, Models and Evaluation Metrics
1,329 Citations2013Micah Hodosh, Peter Young +1 more
This work proposes to frame sentence-based image annotation as the task of ranking a given pool of captions, and introduces a new benchmark collection, consisting of 8,000 images that are each paired with five different captions which provide clear descriptions of the salient entities and events.
From captions to visual concepts and back
1,322 Citations2015Hao Fang, Saurabh Gupta +10 more
This paper uses multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives, and develops a maximum-entropy language model.
arXiv (Cornell University)Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
1,316 Citations2014Ryan Kiros, Ruslan Salakhutdinov +1 more
This work introduces the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder, and shows that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic.
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
1,068 Citations2014Jeff Donahue, Lisa Anne Hendricks +5 more
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
arXiv (Cornell University)Recurrent Models of Visual Attention
999 Citations2014Volodymyr Mnih, Nicolas Heess +2 more
arXiv (Cornell University)DRAW: A Recurrent Neural Network For Image Generation
963 Citations2015Karol Gregor, Ivo Danihelka +3 more
Describing Videos by Exploiting Temporal Structure
943 Citations2015Li Yao, Atousa Torabi +5 more
This work proposes an approach that successfully takes into account both the local and global temporal structure of videos to produce descriptions and proposes a temporal attention mechanism that allows to go beyond local temporal modeling and learns to automatically select the most relevant temporal segments given the text-generating RNN.
IEEE Transactions on Pattern Analysis and Machine IntelligenceBabyTalk: Understanding and Generating Simple Image Descriptions
874 Citations2013Girish Kulkarni, Visruth Premraj +6 more
The proposed system to automatically generate natural language descriptions from images is very effective at producing relevant sentences for images and generates descriptions that are notably more true to the specific image content than previous work.
International Conference on Learning RepresentationsHow to Construct Deep Recurrent Neural Networks
577 Citations2014Razvan Pascanu, Çaǧlar Gülçehre +2 more
Multimodal Neural Language Models
566 Citations2014Ryan Kiros, Ruslan Salakhutdinov +1 more
This work introduces two multimodal neural language models: models of natural language that can be conditioned on other modalities and imagetext modelling, which can generate sentence descriptions for images without the use of templates, structured prediction, and/or syntactic trees.
Conference of the European Chapter of the Association for Computational LinguisticsMidge: Generating Image Descriptions From Computer Vision Detections
430 Citations2012Margaret Mitchell, Jesse Dodge +8 more
A novel generation system that composes humanlike descriptions of images from computer vision detections by leveraging syntactically informed word co-occurrence statistics and automatically generating some of the most natural image descriptions to date.
Learning to combine foveal glimpses with a third-order Boltzmann machine
427 Citations2010Hugo Larochelle, Geoffrey E. Hinton
A model based on a Boltzmann machine with third-order connections that can learn how to accumulate information about a shape over several fixations is described, showing that it can perform at least as well as a model trained on whole images.
Composing Simple Image Descriptions using Web-scale N-grams
360 Citations2011Siming Li, Girish Kulkarni +3 more
A simple yet effective approach to automatically compose image descriptions given computer vision based inputs and using web-scale n-grams, which indicates that it is viable to generate simple textual descriptions that are pertinent to the specific content of an image, while permitting creativity in the description -- making for more human-like annotations than previous approaches.
Corpus-Guided Sentence Generation of Natural Images
343 Citations2011Yezhou Yang, Ching L. Teo +2 more
Experimental results show that the strategy of combining vision and language produces readable and descriptive sentences compared to naive strategies that use vision alone.
Artificial IntelligenceThe dropout learning algorithm
341 Citations2014Pierre Baldi, Peter Sadowski
A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates, which allows a complete analysis of the ensemble averaging properties ofdropout in linear networks.
Collective Generation of Natural Image Descriptions
332 Citations2012Полина Кузнецова, Vicente Ordóñez +3 more
A holistic data-driven approach to image description generation, exploiting the vast amount of (noisy) parallel image data and associated natural language descriptions available on the web to generate novel descriptions for query images.
Image Description using Visual Dependency Representations
294 Citations2013Desmond Elliott, Frank Keller
In an image description task, two template-based description generation models that operate over visual dependency representations outperform approaches that rely on object proximity or corpus information to generate descriptions on both automatic measures and on human judgements.
Transactions of the Association for Computational LinguisticsT<scp>ree</scp>T<scp>alk</scp>: Composition and Compression of Trees for Image Descriptions
280 Citations2014Полина Кузнецова, Vicente Ordóñez +2 more
This work is the first to show the empirical benefit of automatically generalized captions for composing natural image descriptions, and significantly better performance than previous approaches for both image caption generalization and generation.
Neural ComputationLearning Where to Attend with Deep Architectures for Image Tracking
200 Citations2012Misha Denil, Loris Bazzani +2 more
An attentional model for simultaneous object tracking and recognition that is driven by gaze data is discussed, and a straightforward extension of the existing approach to the partial information setting results in poor performance, and an alternative method based on modeling the reward surface as a gaussian process is proposed.
Bern Open Repository and Information System (University of Bern)IGS Technical Report 2012
186 Citations2013Rolf Dach, Yoomin Jean
arXiv (Cornell University)Learning a Recurrent Visual Representation for Image Caption Generation
179 Citations2014Xinlei Chen, C. Lawrence Zitnick
This paper uses a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction and evaluates the approach on several tasks.
arXiv (Cornell University)The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
164 Citations2013Lex Weaver, Nigel Tao
arXiv (Cornell University)Learning Generative Models with Visual Attention
86 Citations2013Yichuan Tang, Nitish Srivastava +1 more
A deep-learning based generative framework using attention that can robustly attend to the face region of novel test subjects and can learn generative models of new faces from a novel dataset of large images where the face locations are not known.