Action Recognition using Visual Attention
arXiv (Cornell University)Published 12 November 2015Open access
Shikhar Sharma, Ryan Kiros, Ruslan Salakhutdinov
Citations358
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A soft attention based model using multi-layered Recurrent Neural Networks with Long Short-Term Memory units which are deep both spatially and temporally for action recognition in videos.
Abstract
We propose a soft attention based model for the task of action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units which are deep both spatially and temporally. Our model learns to focus selectively on parts of the video frames and classifies videos after taking a few glimpses. The model essentially learns which parts in the frames are relevant for the task at hand and attaches higher importance to them. We evaluate the model on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyze how the model focuses its attention depending on the scene and the action being performed.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
2009 IEEE Conference on Computer Vision and Pattern RecognitionImageNet: A large-scale hierarchical image database
61,841 Citations2009Jia Deng, Wei Dong +4 more
A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
Going deeper with convolutions
46,825 Citations2015Christian Szegedy, Wei Liu +7 more
A deep convolutional neural network architecture codenamed Inception is proposed that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
arXiv (Cornell University)Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
7,525 Citations2015Kelvin Xu, Jimmy Ba +6 more
An attention based model that automatically learns to describe the content of images is introduced that can be trained in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
Show and tell: A neural image caption generator
6,363 Citations2015Oriol Vinyals, Alexander Toshev +2 more
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image.
Long-term recurrent convolutional networks for visual recognition and description
5,282 Citations2015Jeff Donahue, Lisa Anne Hendricks +5 more
A novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and shows such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.
Reading digits in natural images with unsupervised feature learning
4,546 Citations2024Yuval Netzer
A new benchmark dataset for research use is introduced containing over 600,000 labeled digits cropped from Street View images, and variants of two recently proposed unsupervised feature learning methods are employed, finding that they are convincingly superior on benchmarks.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
3,526 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
arXiv (Cornell University)Recurrent Neural Network Regularization
2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more
This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
arXiv (Cornell University)Recurrent Models of Visual Attention
2,272 Citations2014Volodymyr Mnih, Nicolas Heess +2 more
A novel recurrent neural network model that is capable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution is presented.
Hybrid speech recognition with Deep Bidirectional LSTM
1,794 Citations2013Alex Graves, Navdeep Jaitly +1 more
The hybrid approach with DBLSTM appears to be well suited for tasks where acoustic modelling predominates, and the improvement in word error rate over the deep network is modest, despite a great increase in framelevel accuracy.
arXiv (Cornell University)Unsupervised Learning of Video Representations using LSTMs
1,666 Citations2015Nitish Srivastava, Elman Mansimov +1 more
Oxford University Research Archive (ORA) (University of Oxford)Two-Stream Convolutional Networks for Action Recognition in Videos
1,476 Citations2014Karen Simonyan, Andrew Zisserman
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
arXiv (Cornell University)Multiple Object Recognition with Visual Attention
701 Citations2014Jimmy Ba, Volodymyr Mnih +1 more
Modeling video evolution for action recognition
609 Citations2015Basura Fernando, Efstratios Gavves +3 more
The proposed method to capture video-wide temporal information for action recognition postulate that a function capable of ordering the frames of a video temporally captures well the evolution of the appearance within the video.
arXiv (Cornell University)Deep Image: Scaling up Image Recognition
330 Citations2015Wu Ren, Shengen Yan +3 more
A state-of-the-art image recognition system, Deep Image, developed using end-to-end deep learning, which achieves excellent results on multiple challenging computer vision benchmarks.
What do 15,000 object categories tell us about classifying and localizing actions?
206 Citations2015Mihir Jain, Jan van Gemert +1 more
It is shown that objects matter for actions, and are often semantically relevant as well, and the first in-depth study of encoding objects for actions and it is established that actions have object preferences.
arXiv (Cornell University)Describing Videos by Exploiting Temporal Structure
189 Citations2015Li Yao, Atousa Torabi +5 more
Exploiting Image-trained CNN Architectures for Unconstrained Video Classification
176 Citations2015Shengxin Zha, Florian Luisier +3 more
The proposed late fusion of CNN- and motion-based features can further increase the mean average precision (mAP) on MED'14 from 34.95% to 38.74% and achieves the state-of-the-art classification performance on the challenging UCF-101 dataset.
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
141 Citations2015Subhashini Venugopalan, Huijuan Xu +4 more
This paper proposes to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure, to create sentence descriptions of open-domain videos with large vocabularies.
DL-SFA: Deeply-Learned Slow Feature Analysis for Action Recognition
121 Citations2014Lin Sun, Kui Jia +4 more
This paper uses a two-layered SFA learning structure with 3D convolution and max pooling operations to scale up the method to large inputs and capture abstract and structural features from the video.
arXiv (Cornell University)Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
83 Citations2015Serena Yeung, Olga Russakovsky +4 more
arXiv (Cornell University)Object Detection Networks on Convolutional Feature Maps
35 Citations2015Shaoqing Ren, Kaiming He +3 more
arXiv (Cornell University)Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition
21 Citations2014Zhenzhong Lan, Ming Lin +3 more