Show, Attend and Tell: Neural Image Caption Generation with Visual\n Attention
arXiv (Cornell University)Published 10 February 2015Open access
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov
Citations1,754
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Inspired by recent work in machine translation and object detection, we\nintroduce an attention based model that automatically learns to describe the\ncontent of images. We describe how we can train this model in a deterministic\nmanner using standard backpropagation techniques and stochastically by\nmaximizing a variational lower bound. We also show through visualization how\nthe model is able to automatically learn to fix its gaze on salient objects\nwhile generating the corresponding words in the output sequence. We validate\nthe use of attention with state-of-the-art performance on three benchmark\ndatasets: Flickr8k, Flickr30k and MS COCO.\n
Keywords
Computer Science
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
arXiv (Cornell University)Very Deep Convolutional Networks for Large-Scale Image Recognition
75,540 Citations2014Karen Simonyan, Andrew Zisserman
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Lecture notes in computer scienceMicrosoft COCO: Common Objects in Context
42,219 Citations2014Tsung-Yi Lin, Michael Maire +6 more
A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
Nature reviews. NeuroscienceControl of goal-directed and stimulus-driven attention in the brain
12,975 Citations2002Maurizio Corbetta, Gordon L. Shulman
Evidence for partially segregated networks of brain areas that carry out different attentional functions is reviewed, finding that one system is involved in preparing and applying goal-directed selection for stimuli and responses, and the other is specialized for the detection of behaviourally relevant stimuli.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
Deep visual-semantic alignments for generating image descriptions
4,978 Citations2015Andrej Karpathy, Li Fei-Fei
A model that generates natural language descriptions of images and their regions using a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding is presented.
arXiv (Cornell University)Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
3,282 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Transactions of the Association for Computational LinguisticsFrom image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
2,461 Citations2014Peter Young, Alice Lai +2 more
This work proposes to use the visual denotations of linguistic expressions to define novel denotational similarity metrics, which are shown to be at least as beneficial as distributional similarities for two tasks that require semantic inference.
arXiv (Cornell University)Recurrent Neural Network Regularization
2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more
This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
arXiv (Cornell University)Recurrent Models of Visual Attention
2,272 Citations2014Volodymyr Mnih, Nicolas Heess +2 more
A novel recurrent neural network model that is capable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution is presented.
arXiv (Cornell University)Going Deeper with Convolutions
1,381 Citations2014Christian Szegedy, Wei Liu +7 more
Journal of Artificial Intelligence ResearchFraming Image Description as a Ranking Task: Data, Models and Evaluation Metrics
1,329 Citations2013Micah Hodosh, Peter Young +1 more
This work proposes to frame sentence-based image annotation as the task of ranking a given pool of captions, and introduces a new benchmark collection, consisting of 8,000 images that are each paired with five different captions which provide clear descriptions of the salient entities and events.
arXiv (Cornell University)Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
1,316 Citations2014Ryan Kiros, Ruslan Salakhutdinov +1 more
This work introduces the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder, and shows that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic.
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
1,068 Citations2014Jeff Donahue, Lisa Anne Hendricks +5 more
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
Digital Access to Scholarship at Harvard (DASH) (Harvard University)Practical Bayesian Optimization of Machine Learning Algorithms
989 Citations2012Jasper Snoek, Hugo Larochelle +1 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceBabyTalk: Understanding and Generating Simple Image Descriptions
874 Citations2013Girish Kulkarni, Visruth Premraj +6 more
The proposed system to automatically generate natural language descriptions from images is very effective at producing relevant sentences for images and generates descriptions that are notably more true to the specific image content than previous work.
arXiv (Cornell University)Multiple Object Recognition with Visual Attention
701 Citations2014Jimmy Ba, Volodymyr Mnih +1 more
Multimodal Neural Language Models
566 Citations2014Ryan Kiros, Ruslan Salakhutdinov +1 more
This work introduces two multimodal neural language models: models of natural language that can be conditioned on other modalities and imagetext modelling, which can generate sentence descriptions for images without the use of templates, structured prediction, and/or syntactic trees.
Learning to combine foveal glimpses with a third-order Boltzmann machine
427 Citations2010Hugo Larochelle, Geoffrey E. Hinton
A model based on a Boltzmann machine with third-order connections that can learn how to accumulate information about a shape over several fixations is described, showing that it can perform at least as well as a model trained on whole images.
Composing Simple Image Descriptions using Web-scale N-grams
360 Citations2011Siming Li, Girish Kulkarni +3 more
A simple yet effective approach to automatically compose image descriptions given computer vision based inputs and using web-scale n-grams, which indicates that it is viable to generate simple textual descriptions that are pertinent to the specific content of an image, while permitting creativity in the description -- making for more human-like annotations than previous approaches.
Artificial IntelligenceThe dropout learning algorithm
341 Citations2014Pierre Baldi, Peter Sadowski
A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates, which allows a complete analysis of the ensemble averaging properties ofdropout in linear networks.
Collective Generation of Natural Image Descriptions
332 Citations2012Полина Кузнецова, Vicente Ordóñez +3 more
A holistic data-driven approach to image description generation, exploiting the vast amount of (noisy) parallel image data and associated natural language descriptions available on the web to generate novel descriptions for query images.
Image Description using Visual Dependency Representations
294 Citations2013Desmond Elliott, Frank Keller
In an image description task, two template-based description generation models that operate over visual dependency representations outperform approaches that rely on object proximity or corpus information to generate descriptions on both automatic measures and on human judgements.
Transactions of the Association for Computational LinguisticsT<scp>ree</scp>T<scp>alk</scp>: Composition and Compression of Trees for Image Descriptions
280 Citations2014Полина Кузнецова, Vicente Ordóñez +2 more
This work is the first to show the empirical benefit of automatically generalized captions for composing natural image descriptions, and significantly better performance than previous approaches for both image caption generalization and generation.
Neural ComputationLearning Where to Attend with Deep Architectures for Image Tracking
200 Citations2012Misha Denil, Loris Bazzani +2 more
An attentional model for simultaneous object tracking and recognition that is driven by gaze data is discussed, and a straightforward extension of the existing approach to the partial information setting results in poor performance, and an alternative method based on modeling the reward surface as a gaussian process is proposed.
arXiv (Cornell University)Show and Tell: A Neural Image Caption Generator
186 Citations2014Oriol Vinyals, Alexander Toshev +2 more
arXiv (Cornell University)Learning a Recurrent Visual Representation for Image Caption Generation
179 Citations2014Xinlei Chen, C. Lawrence Zitnick
This paper uses a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction and evaluates the approach on several tasks.
arXiv (Cornell University)The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
164 Citations2013Lex Weaver, Nigel Tao
arXiv (Cornell University)From Captions to Visual Concepts and Back
91 Citations2014Hao Fang, Saurabh Gupta +10 more
arXiv (Cornell University)Learning Generative Models with Visual Attention
86 Citations2013Yichuan Tang, Nitish Srivastava +1 more
A deep-learning based generative framework using attention that can robustly attend to the face region of novel test subjects and can learn generative models of new faces from a novel dataset of large images where the face locations are not known.