Learning to predict where humans look
Published 1 September 2009Open access
Tilke Judd, Krista A. Ehinger, Frédo Durand, Antonio Torralba
Citations2,042
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper collects eye tracking data of 15 viewers on 1003 images and uses this database as training and testing examples to learn a model of saliency based on low, middle and high-level image features.
Abstract
For many applications in graphics, design, and human computer interaction, it is essential to understand where humans look in a scene. Where eye tracking devices are not a viable option, models of saliency can be used to predict fixation locations. Most saliency approaches are based on bottom-up computation that does not consider top-down image semantics and often does not match actual eye movements. To address this problem, we collected eye tracking data of 15 viewers on 1003 images and use this database as training and testing examples to learn a model of saliency based on low, middle and high-level image features. This large database of eye tracking data is publicly available with this paper.
Keywords
Computer Science
International Journal of Computer VisionModeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
6,410 Citations2001Aude Oliva, Antonio Torralba
The performance of the spatial envelope model shows that specific information about object shape or identity is not a requirement for scene categorization and that modeling a holistic representation of the scene informs about its probable semantic category.
International Journal of Computer VisionLabelMe: A Database and Web-Based Tool for Image Annotation
4,205 Citations2007Bryan Russell, Antonio Torralba +2 more
A web-based tool that allows easy image annotation and instant sharing of such annotations is developed and a large dataset that spans many object categories, often containing multiple instances over a wide variety of images is collected.
Saliency Detection: A Spectral Residual Approach
3,427 Citations2007Xiaodi Hou, Liqing Zhang
A simple method for the visual saliency detection is presented, independent of features, categories, or other forms of prior knowledge of the objects, and a fast method to construct the corresponding saliency map in spatial domain is proposed.
Vision ResearchA saliency-based search mechanism for overt and covert shifts of visual attention
3,193 Citations2000L. Itti, Christof Koch
A detailed computer implementation of a saliency map scheme is described, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner, which is applied to common psychophysical stimuli as well as to a very demanding visual search task.
A discriminatively trained, multiscale, deformable part model
2,882 Citations2008Pedro F. Felzenszwalb, David McAllester +1 more
A discriminatively trained, multiscale, deformable part model for object detection, which achieves a two-fold improvement in average precision over the best performance in the 2006 PASCAL person detection challenge and outperforms the best results in the 2007 challenge in ten out of twenty categories.
Robust real-time object detection.
2,256 Citations2001Paul Viola
Journal of VisionSUN: A Bayesian framework for saliency using natural statistics
1,234 Citations2008Lingyun Zhang, Matthew H. Tong +3 more
In the model, saliency is computed locally, which is consistent with the neuroanatomy of the early visual system and results in an efficient algorithm with few free parameters, which provides a straightforward explanation for many search asymmetries observed in humans.
MPG.PuRe (Max Planck Society)Large Scale Multiple Kernel Learning
1,212 Citations2006Sören Sonnenburg, Gunnar Rätsch +2 more
It is shown that the proposed multiple kernel learning algorithm can be rewritten as a semi-infinite linear program that can be efficiently solved by recycling the standard SVM implementations, and generalize the formulation and the method to a larger class of problems, including regression and one-class classification.
Proceedings - International Conference on Image ProcessingThe steerable pyramid: a flexible architecture for multi-scale derivative computation
1,062 Citations2002Eero P. Simoncelli, William T. Freeman
An architecture for efficient and accurate linear decomposition of an image into scale and orientation subbands and the construction and implementation of the transform is described.
Journal of VisionThe central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions
1,011 Citations2007Benjamin W. Tatler
The endurance of the central fixation bias irrespective of the distribution of image features, or the observer's task, implies one of three possible explanations: first, the center of the screen may be an optimal location for early information processing of the scene, or second, the central bias reflects a tendency to re-center the eye in its orbit.
Robust real-time face detection
895 Citations2005Paul Viola, Michael Jones
Journal of VisionSaliency, attention, and visual search: An information theoretic approach
855 Citations2009Neil D. B. Bruce, John K. Tsotsos
It is demonstrated that a variety of visual search behaviors appear as emergent properties of the model and therefore basic principles of coding and information transmission are demonstrated.
ACM Transactions on GraphicsImproved seam carving for video retargeting
745 Citations2008Michael Rubinstein, Ariel Shamir +1 more
The Caltech Institute Archives (California Institute of Technology)Predicting human gaze using low-level saliency combined with face detection
462 Citations2007Moran Cerf, Jonathan Harel +2 more
It is demonstrated that a combined model of face detection and low-level saliency significantly outperforms a low- level model in predicting locations humans fixate on, based on eye-movement recordings of humans observing photographs of natural scenes, most of which contained at least one person.
Stylization and abstraction of photographs
434 Citations2002Doug DeCarlo, Anthony Santella
This work describes a computational approach to stylizing and abstracting photographs that explicitly responds to the design goal of good information design and represents a new alternative for non-photorealistic rendering both in its visual style, in its approach to visual form, and in its techniques for interaction.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Real-time foveated multiresolution system for low-bandwidth video communication</title>
370 Citations1998Wilson S. Geisler, Jeffrey S. Perry
This work has developed a foveated multiresolution pyramid video coder/decoder which runs in real-time on a general purpose computer and includes zero-tree coding.
Gaze-based interaction for semi-automatic photo cropping
352 Citations2006Anthony Santella, Maneesh Agrawala +3 more
An interactive method for cropping photographs given minimal information about important content location, provided by eye tracking is presented, enabling applications such as automatic snapshot recomposition, adaptive documents, and thumbnailing.
Vision ResearchPredicting visual fixations on video based on low-level visual features
297 Citations2007Olivier Le Meur, Patrick Le Callet +1 more
The proposed model shows significant improvement over the selected benchmarking models (except the centered model), and conclusions are drawn regarding both the influence of low-level visual features over time and the central bias in an eye tracking experiment.
PubMedModeling Search for People in 900 Scenes: A combined source model of eye guidance.
290 Citations2009Krista A. Ehinger, B. Hidalgo-Sotelo +2 more
Vision ResearchA simple saliency model predicts a number of motion popout phenomena
242 Citations1999Ruth Rosenholtz
A simple model for predicting ease of search for a unique motion, based upon a quantitative measure of target saliency, predicts a number of well-known motion search phenomena and suggests that one control for target Saliency as well as target discriminability when drawing conclusions about visual system mechanisms from search experiments.
ACM Transactions on GraphicsStylization and abstraction of photographs
227 Citations2002Doug DeCarlo, Anthony Santella
IEEE Transactions on Image ProcessingFoveation scalable video coding with automatic fixation selection
191 Citations2003Zhou Wang, Ligang Lu +1 more
A foveation scalable video coding (FSVC) algorithm which supplies good quality-compression performance as well as effective rate scalability, and is adaptable to different applications, such as knowledge-based video coding and video communications over time-varying, multiuser and interactive networks.
The MIT Press eBooksA Nonparametric Approach to Bottom-Up Visual Saliency
187 Citations2007Wolf Kienzle, Felix A. Wichmann +2 more
The model is rather simplistic and essentially parameter-free, and contrasts recent developments in the field that usually aim at higher prediction rates at the cost of additional parameters and increasing model complexity, and in fact learns image features that resemble findings from several previous studies.
IEEE Transactions on Pattern Analysis and Machine IntelligenceEsaliency (Extended Saliency): Meaningful Attention Using Stochastic Image Modeling
140 Citations2009Tamar Avraham, Michael Lindenbaum
A validated stochastic model is proposed to estimate the probability that an image part is of interest and is referred to as saliency and thus specify saliency in a mathematically well-defined sense.
Advances in psychologyAttention and communication: Eye-movement-based research paradigms
51 Citations1996Boris M. Velichkovsky, Marc Pomplun +1 more