Convolutional Learning of Spatio-temporal Features
Lecture notes in computer sciencePublished 1 January 2010Open access
Graham W. Taylor, Rob Fergus, Yann LeCun, Christoph Bregler
Citations651
SJR quartileQ2
SJR score0.35
SNIP0.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A model that learns latent representations of image sequences from pairs of successive images is introduced, allowing it to scale to realistic image sizes whilst using a compact parametrization.
Abstract
We address the problem of learning good features for understanding video data. We introduce a model that learns latent representations of image sequences from pairs of successive images. The convolutional architecture of our model allows it to scale to realistic image sizes whilst using a compact parametrization. In experiments on the NORB dataset, we show our model extracts latent "flow fields" which correspond to the transformation between the pair of input frames. We also use our model to extract low-level motion features in a multi-stage architecture for action recognition, demonstrating competitive performance on both the KTH and Hollywood2 datasets.
Keywords
Computer Science
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
Vision ResearchSparse coding with an overcomplete basis set: A strategy employed by V1?
3,720 Citations1997Bruno A. Olshausen, David J. Field
These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli.
Learning realistic human actions from movies
3,493 Citations2008Ivan Laptev, Marcin Marszałek +2 more
A new method for video classification that builds upon and extends several recent ideas including local space-time features,space-time pyramids and multi-channel non-linear SVMs is presented and shown to improve state-of-the-art results on the standard KTH action dataset.
Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004.Recognizing human actions: a local SVM approach
2,939 Citations2004Christian Schüldt, Ivan Laptev +1 more
Behavior Recognition via Sparse Spatio-Temporal Features
2,474 Citations2006Piotr Dollár, Vincent Rabaud +2 more
It is shown that the direct 3D counterparts to commonly used 2D interest point detectors are inadequate, and an alternative is proposed, and a recognition algorithm based on spatio-temporally windowed data is devised.
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
2,446 Citations2009Honglak Lee, Roger Grosse +2 more
The convolutional deep belief network is presented, a hierarchical generative model which scales to realistic image sizes and is translation-invariant and supports efficient bottom-up and top-down probabilistic inference.
The MIT Press eBooksEfficient sparse coding algorithms
2,393 Citations2007Honglak Lee, Alexis Battle +2 more
These algorithms are applied to natural images and it is demonstrated that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons.
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Online dictionary learning for sparse coding
2,111 Citations2009Julien Mairal, Francis Bach +2 more
A new online optimization algorithm for dictionary learning is proposed, based on stochastic approximations, which scales up gracefully to large datasets with millions of training samples, and leads to faster performance and better dictionaries than classical batch algorithms for both small and large datasets.
Learning methods for generic object recognition with invariance to pose and lighting
1,332 Citations2004Yann LeCun, Fu Jie Huang +1 more
A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second and proved impractical, while convolutional nets yielded 16/7% error.
Evaluation of local spatio-temporal features for action recognition
1,313 Citations2009Heng Wang, Muhammad Muneeb Ullah +3 more
It is demonstrated that regular sampling of space-time features consistently outperforms all testedspace-time interest point detectors for human actions in realistic settings and is a consistent ranking for the majority of methods over different datasets.
Space-time interest points
1,301 Citations2003Laptev, Tony Lindeberg
This work builds on the idea of the Harris and Forstner interest point operators and detects local structures in space-time where the image values have significant local variations in both space and time to detect spatio-temporal events.
2009 IEEE Conference on Computer Vision and Pattern RecognitionActions in context
1,188 Citations2009Marcin Marszałek, Ivan Laptev +1 more
This paper automatically discover relevant scene classes and their correlation with human actions, and shows how to learn selected scene classes from video without manual supervision and develops a joint framework for action and scene recognition and demonstrates improved recognition of both in natural video.
The MIT Press eBooksEfficient Learning of Sparse Representations with an Energy-Based Model
1,079 Citations2007Marc’Aurelio Ranzato, Christopher S. Poultney +2 more
A novel unsupervised method for learning sparse, overcomplete features using a linear encoder, and a linear decoder preceded by a sparsifying non-linearity that turns a code vector into a quasi-binary sparse code vector.
An empirical evaluation of deep architectures on problems with many factors of variation
1,030 Citations2007Hugo Larochelle, Dumitru Erhan +3 more
A series of experiments indicate that these models with deep architectures show promise in solving harder learning problems that exhibit many factors of variation.
Sparse deep belief net model for visual area V2
881 Citations2007Honglak Lee, Chaitanya Ekanadham +1 more
An unsupervised learning model is presented that faithfully mimics certain properties of visual area V2 and the encoding of these more complex "corner" features matches well with the results from the Ito & Komatsu's study of biological V2 responses, suggesting that this sparse variant of deep belief networks holds promise for modeling more higher-order features.
Lecture notes in computer scienceAn Efficient Dense and Scale-Invariant Spatio-Temporal Interest Point Detector
875 Citations2008Geert Willems, Tinne Tuytelaars +1 more
This paper presents for the first time spatio-temporal interest points that are at the same time scale-invariant (both spatially and temporally) and densely cover the video content and can be computed efficiently.
Multiscale conditional random fields for image labeling
843 Citations2004Xuming He, Richard S. Zemel +1 more
An approach to include contextual features for labeling images, in which each pixel is assigned to one of a finite set of labels, are incorporated into a probabilistic framework, which combines the outputs of several components.
A Biologically Inspired System for Action Recognition
718 Citations2007Hueihan Jhuang, T. Serre +2 more
The approach builds on recent work on object recognition based on hierarchical feedforward architectures and extends a neurobiological model of motion processing in the visual cortex and finds that sparse features in intermediate stages outperform dense ones and that using a simple feature selection approach leads to an efficient system that performs better with far fewer features.
PLoS Computational BiologyWhy is Real-World Visual Object Recognition Hard?
547 Citations2008Nicolas Pinto, David Cox +1 more
It is shown that a simple V1-like model—a neuroscientist's “null” model—outperforms state-of-the-art object recognition systems (biologically inspired and otherwise) on a standard, ostensibly natural image recognition test.
3D Object Recognition with Deep Belief Nets
304 Citations2009Vinod Nair, Geoffrey E. Hinton
A new type of top-level model for Deep Belief Nets is introduced, a third-order Boltzmann machine, trained using a hybrid algorithm that combines both generative and discriminative gradients that substantially outperforms shallow models such as SVMs.
Neural ComputationLearning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines
231 Citations2010Roland Memisevic, Geoffrey E. Hinton
A low-rank approximation to this interaction tensor that uses a sum of factors, each of which is a three-way outer product, which allows efficient learning of transformations between larger image patches and demonstrates the learning of optimal filter pairs from various synthetic and real image sequences.
Learning Multilevel Distributed Representations for High-Dimensional Sequences
192 Citations2007Ilya Sutskever, Geoffrey E. Hinton
A new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems are described, and their performance is demonstrated using synthetic video sequences of two balls bouncing in a box.
2009 IEEE Conference on Computer Vision and Pattern RecognitionStacks of convolutional Restricted Boltzmann Machines for shift-invariant feature learning
186 Citations2009Mohammad Norouzi, Mani Ranjbar +1 more
The convolutional RBM (C-RBM), a variant of the RBM model in which weights are shared to respect the spatial structure of images, is developed, which learns a set of features that can generate the images of a specific object class.
Unsupervised Learning of Image Transformations
184 Citations2007Roland Memisevic, Geoffrey E. Hinton
A probabilistic model for learning rich, distributed representations of image transformations that develops domain specific motion features, in the form of fields of locally transformed edge filters, and can fantasize new transformations on previously unseen images.
2009 IEEE Conference on Computer Vision and Pattern RecognitionActions in context
110 Citations2009Marcin Marszałek, Ivan Laptev +1 more
Learning Transformational Invariants from Natural Movies
62 Citations2008Charles F. Cadieu, Bruno A. Olshausen
A hierarchical, probabilistic model that learns to extract complex motion from movies of the natural environment that encoded transformational invariants, which are selective for the speed and direction of a moving pattern, but are invariant to its spatial structure.
Recursive Sparse, Spatiotemporal Coding
25 Citations2009Thomas Dean, Rich Washington +1 more
The new approach to learning sparse, spatiotemporal codes in which the number of basis vectors, their orientations, velocities and the size of their receptive fields change over the duration of unsupervised training meets or exceeds the performance of state-of-the-art activity-recognition methods.
The Visual ComputerThree-dimensional oil painting reconstruction with stroke based rendering
16 Citations2007Kyong Joon Lee, Dong Hwan Kim +2 more
An algorithm to generate a three-dimensional (3-D) oil painting using artistic filters and sample 3-Dstructures is presented, which would show a natural illumination change as the position of the light source varies, providing a realistic effect to the image.