Pose-Sensitive Embedding by Nonlinear NCA Regression
Published 6 December 2010
Graham W. Taylor, Rob Fergus, George Williams, Ian Spiro, Christoph Bregler
Citations31
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel method for learning a nonlinear embedding based on several extensions to the Neighborhood Component Analysis (NCA) framework is achieved, enabling it to scale to realistically-sized images and to evaluate quantitatively against other embedding methods.
Abstract
This paper tackles the complex problem of visually matching people in similar pose but with different clothes, background, and other appearance changes. We achieve this with a novel method for learning a nonlinear embedding based on several extensions to the Neighborhood Component Analysis (NCA) framework. Our method is convolutional, enabling it to scale to realistically-sized images. By cheaply labeling the head and hands in large video databases through Amazon Mechanical Turk (a crowd-sourcing service), we can use the task of localizing the head and hands as a proxy for determining body pose. We apply our method to challenging real-world data and show that it can generalize beyond hand localization to infer a more general notion of body pose. We evaluate our method quantitatively against other embedding methods. We also demonstrate that realworld performance can be improved through the use of synthetic data. 1
Keywords
Computer ScienceEngineering
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
International Journal of Computer VisionModeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
6,410 Citations2001Aude Oliva, Antonio Torralba
The performance of the spatial envelope model shows that specific information about object shape or identity is not a requirement for scene categorization and that modeling a holistic representation of the scene informs about its probable semantic category.
Dimensionality Reduction by Learning an Invariant Mapping
5,057 Citations2006Raia Hadsell, Sumit Chopra +1 more
This work presents a method - called Dimensionality Reduction by Learning an Invariant Mapping (DrLIM) - for learning a globally coherent nonlinear function that maps the data evenly to the output manifold.
IEEE Transactions on Pattern Analysis and Machine IntelligencePfinder: real-time tracking of the human body
4,147 Citations1997Christopher R. Wren, A. Azarbayejani +2 more
A discriminatively trained, multiscale, deformable part model
2,882 Citations2008Pedro F. Felzenszwalb, David McAllester +1 more
A discriminatively trained, multiscale, deformable part model for object detection, which achieves a two-fold improvement in average precision over the best performance in the 2006 PASCAL person detection challenge and outperforms the best results in the 2007 challenge in ten out of twenty categories.
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
2,446 Citations2009Honglak Lee, Roger Grosse +2 more
The convolutional deep belief network is presented, a hierarchical generative model which scales to realistic image sizes and is translation-invariant and supports efficient bottom-up and top-down probabilistic inference.
What is the best multi-stage architecture for object recognition?
2,155 Citations2009Kevin Jarrett, Koray Kavukcuoglu +2 more
It is shown that using non-linearities that include rectification and local contrast normalization is the single most important ingredient for good accuracy on object recognition benchmarks and that two stages of feature extraction yield better accuracy than one.
Neighbourhood Components Analysis
1,729 Citations2004Jacob Goldberger, Geoffrey E. Hinton +2 more
A novel method for learning a Mahalanobis distance measure to be used in the KNN classification algorithm that directly maximizes a stochastic variant of the leave-one-out KNN score on the training set.
Lecture notes in computer scienceHuman Detection Using Oriented Histograms of Flow and Appearance
1,653 Citations2006Navneet Dalal, Bill Triggs +1 more
A detector for standing and moving people in videos with possibly moving cameras and backgrounds is developed, testing several different motion coding schemes and showing empirically that orientated histograms of differential optical flow give the best overall performance.
International Journal of Computer VisionHumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion
1,175 Citations2009Leonid Sigal, Alexandru O. Bălan +1 more
A baseline algorithm for 3D articulated tracking that uses a relatively standard Bayesian framework with optimization in the form of Sequential Importance Resampling and Annealed Particle Filtering is described, and a variety of likelihood functions, prior models of human motion and the effects of algorithm parameters are explored.
Near-Optimal Hashing Algorithms for Approximate Nearest Neighbor in High Dimensions
1,099 Citations2006Alexandr Andoni, Piotr Indyk
Poselets: Body part detectors trained using 3D human pose annotations
994 Citations2009Lubomir Bourdev, Jitendra Malik
A new dataset, H3D, is built of annotations of humans in 2D photographs with 3D joint information, inferred using anthropometric constraints, to address the classic problems of detection, segmentation and pose estimation of people in images with a novel definition of a part, a poselet.
Computer Vision and Image UnderstandingVision-based human motion analysis: An overview
830 Citations2007Ronald Poppe
The characteristics of human motion analysis are discussed to highlight trends in the domain and to point out limitations of the current state of the art.
2009 IEEE Conference on Computer Vision and Pattern RecognitionPictorial structures revisited: People detection and articulated pose estimation
817 Citations2009Mykhaylo Andriluka, Stefan Roth +1 more
This paper proposes a generic approach based on the pictorial structures framework, and demonstrates that such a model is equally suitable for both detection and pose estimation tasks, outperforming the state of the art on three recently proposed datasets.
Fast pose estimation with parameter-sensitive hashing
764 Citations2003Shakhnarovich, Viola +1 more
A new algorithm is introduced that learns a set of hashing functions that efficiently index examples relevant to a particular estimation task, and can rapidly and accurately estimate the articulated pose of human figures from a large database of example images.
Small codes and large image databases for recognition
739 Citations2008Antonio Torralba, Rob Fergus +1 more
The goal is to develop efficient image search and scene matching techniques that are not only fast, but also require very little memory, enabling their use on standard hardware or even on handheld devices.
IEEE Transactions on Pattern Analysis and Machine IntelligenceRecovering 3D human pose from monocular images
729 Citations2006Ankur Agarwal, Bill Triggs
A learning-based method for recovering 3D human body pose from single images and monocular image sequences, embedded in a novel regressive tracking framework, using dynamics from the previous state estimate together with a learned regression value to disambiguate the pose.
PLoS Computational BiologyWhy is Real-World Visual Object Recognition Hard?
547 Citations2008Nicolas Pinto, David Cox +1 more
It is shown that a simple V1-like model—a neuroscientist's “null” model—outperforms state-of-the-art object recognition systems (biologically inspired and otherwise) on a standard, ostensibly natural image recognition test.
NatureSelf-organizing neural network that discovers surfaces in random-dot stereograms
508 Citations1992Suzanna Becker, Geoffrey E. Hinton
The authors' simulations show that when the learning procedure is applied to adjacent patches of two-dimensional images, it allows a neural network that has no prior knowledge of the third dimension to discover depth in random dot stereograms of curved surfaces.
Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure
438 Citations2007Ruslan Salakhutdinov, Geoffrey E. Hinton
CogPrints (University of Southampton)Notes on Convolutional Neural Networks
395 Citations2006Jake Bouvrie
The derivation and implementation of convolutional neural networks are discussed, followed by an extension which allows one to learn sparse combinations of feature maps, and small snippets of MATLAB code are given to accompany the equations.
Deep learning from temporal coherence in video
355 Citations2009Hossein Mobahi, Ronan Collobert +1 more
A learning method for deep architectures that takes advantage of sequential data, in particular from the temporal coherence that naturally exists in unlabeled video recordings, and is used to improve the performance on a supervised task of interest.
Lecture notes in computer scienceEstimating Human Body Configurations Using Shape Context Matching
351 Citations2002Greg Mori, Jitendra Malik
The problem is to take a single two-dimensional image containing a human body, locate the joint positions, and use these to estimate the body configuration and pose in three-dimensional space.
Strike a Pose: Tracking People by Finding Stylized Poses
304 Citations2005Deva Ramanan, David Forsyth +1 more
A person detector that quite accurately detects and localizes limbs of people in lateral walking poses is built, and an algorithm for finding and kinematically tracking multiple people in long sequences is developed.
PLoS Computational BiologyA High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation
254 Citations2009Nicolas Pinto, David Doukhan +2 more
Large-scale privacy protection in Google Street View
251 Citations2009Andrea Frome, German Cheung +7 more
This work presents a system that combines a standard sliding-window detector tuned for a high recall, low-precision operating point with a fast post-processing stage that is able to remove additional false positives by incorporating domain-specific information not available to the sliding- window detector.
arXiv (Cornell University)Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
196 Citations2010Koray Kavukcuoglu, Marc’Aurelio Ranzato +1 more
This work proposes a simple and efficient algorithm to learn basis functions, which provides a fast and smooth approximator to the optimal representation, achieving even better accuracy than exact sparse coding algorithms on visual object recognition tasks.
Inferring 3D structure with a statistical image-based shape model
195 Citations2003Grauman, Shakhnarovich +1 more
The shape model enables accurate estimation of structure despite segmentation errors or missing views in the input silhouettes, and it works even with only a single input view, using a training set of thousands of pedestrian images generated from a synthetic model.
2009 IEEE Conference on Computer Vision and Pattern RecognitionStacks of convolutional Restricted Boltzmann Machines for shift-invariant feature learning
186 Citations2009Mohammad Norouzi, Mani Ranjbar +1 more
The convolutional RBM (C-RBM), a variant of the RBM model in which weights are shared to respect the spatial structure of images, is developed, which learns a set of features that can generate the images of a specific object class.
Unsupervised Learning of Image Transformations
184 Citations2007Roland Memisevic, Geoffrey E. Hinton
A probabilistic model for learning rich, distributed representations of image transformations that develops domain specific motion features, in the form of fields of locally transformed edge filters, and can fantasize new transformations on previously unseen images.
2009 IEEE Conference on Computer Vision and Pattern RecognitionLearning sign language by watching TV (using weakly aligned subtitles)
163 Citations2009Patrick Buehler, Andrew Zisserman +1 more
This work proposes a distance function to match signing sequences which includes the trajectory of both hands, the hand shape and orientation, and properly models the case of hands touching and shows that by optimizing a scoring function based on multiple instance learning, it is able to extract the sign of interest from hours of signing footage, despite the very weak and noisy supervision.
Automatic basis function construction for approximate dynamic programming and reinforcement learning
162 Citations2006Philipp W. Keller, Shie Mannor +1 more
This work proposes to use neighborhood component analysis (Goldberger et al., 2005), a dimensionality reduction technique created for supervised learning, in order to map a high-dimensional state space to a low-dimensional space, based on the Bellman error, or on the temporal difference (TD) error.
2009 IEEE Conference on Computer Vision and Pattern RecognitionPose search: Retrieving people using their pose
154 Citations2009Vittorio Ferrari, Manuel J. Marín‐Jiménez +1 more
This work proposes and investigates several pose descriptors which are person, clothing, background and lighting independent, and improves the performance over existing methods for localizing upper body layout on unconstrained video.
Boostmap: a method for efficient approximate similarity rankings
150 Citations2004Vassilis Athitsos, Jonathan Alon +2 more
The experiments indicate that BoostMap compares favorably with existing embedding methods that have been employed in computer vision and database applications, i.e., FastMap and Bourgain embeddings.
Adaptive pose priors for pictorial structures
136 Citations2010Benjamin Sapp, Chris Jordan +1 more
This work develops an effective shape-based kernel for upper-body pose similarity and proposes a leave-one-out loss function for learning a sparse subset of exemplars for kernel regression estimates from a learned sparse set of exemplar.
Transfer Learning in Sign language
124 Citations2007Ali Farhadi, David Forsyth +1 more
This work builds word models for ASL that transfer between different signers and different aspects and demonstrates transfer learning in two scenarios: from avatar to a frontally viewed human signer and from an avatar tohuman signer in a 3/4 view.
A Convolutional Neural Network Hand Tracker
90 Citations1994Steven J. Nowlan, John Platt
A system that can track a hand in a sequence of video frames and recognize hand gestures in a user-independent manner and is designed to operate in real time with existing hardware is described.
Lecture notes in computer sciencePittPatt Face Detection and Tracking for the CLEAR 2007 Evaluation
22 Citations2008M.C. Nechyba, Louis Brandy +1 more
Pittsburgh Pattern Recognition's participation in the face detection and tracking tasks for the CLEAR 2007 evaluation is described, and the three main stages of processing are explained: frame-based face detection; motion-based tracking; and track filtering.