A Bayesian View on Multimodal Cue Integration
Published 4 December 2005Open access
Marc O. Ernst
Citations186
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This chapter reviews a model that in the statistical sense describes an optimal integration mechanism for integrating sensory information and points out how this integration scheme can be incorporated in a larger framework using Bayesian decision theory (BDT).
Abstract
Abstract We perceive our own body and the world surrounding us via multiple sources of sensory information derived from several modalities, including vision, touch, and audition. To enable interactions with the environment, this information has to converge into a coherent and unambiguous multimodal percept of the body and the world. But how does the brain come up with such a unique percept? In this chapter, I review a model that in the statistical sense describes an optimal integration mechanism. The benefit of integrating sensory information comes from a reduction in variance of the final perceptual estimate. Furthermore, I point out how this integration scheme can be incorporated into a larger framework using Bayesian decision theory (BDT).
Keywords
PsychologyNeuroscience
Signal detection theory and psychophysics
9,998 Citations1966David Marvin Green, John A. Swets
This book discusses statistical decision theory and sensory processes in signal detection theory and psychophysics and describes how these processes affect decision-making.
NatureHumans integrate visual and haptic information in a statistically optimal fashion
4,935 Citations2002Marc O. Ernst, Martin S. Banks
The nervous system seems to combine visual and haptic information in a fashion that is similar to a maximum-likelihood integrator, and this model behaved very similarly to humans in a visual–haptic task.
NatureBayesian integration in sensorimotor learning
2,086 Citations2004Konrad P. Körding, Daniel M. Wolpert
This work shows that subjects internally represent both the statistical distribution of the task and their sensory uncertainty, combining them in a manner consistent with a performance-optimizing bayesian process.
Current BiologyThe Ventriloquist Effect Results from Near-Optimal Bimodal Integration
1,854 Citations2004David Alais, David C. Burr
This study investigates spatial localization of audio-visual stimuli and finds that for severely blurred visual stimuli, the reverse holds: sound captures vision while for less blurred stimuli, neither sense dominates and perception follows the mean position.
Trends in Cognitive SciencesMerging the senses into a robust percept
1,847 Citations2004Marc O. Ernst, HH Bülthoff
It is shown that, depending on the type of information, different combination and integration strategies are used and that prior knowledge is often required for interpreting the sensory signals.
Cambridge University Press eBooksPerception as Bayesian Inference
1,468 Citations1996David C. Knill, David C. Knill +14 more
Nature NeuroscienceMotion illusions as optimal percepts
1,010 Citations2002Yair Weiss, Eero P. Simoncelli +1 more
A model of visual motion perception using standard estimation theory, under the assumptions that there is noise in the initial measurements and slower motions are more likely to occur than faster ones, is formulated and found that specific instantiation of such a velocity estimator can account for a wide variety of psychophysical phenomena.
Vision ResearchMeasurement and modeling of depth cue combination: in defense of weak fusion
996 Citations1995Michael S. Landy, Laurence T. Maloney +2 more
This paper argues that the MWF model is consistent with previous experimental results and is a parsimonious summary of these results, and describes experimental methods, analogous to perturbation analysis, that permit us to analyze depth cue combination in novel ways.
Journal of NeurophysiologyIntegration of Proprioceptive and Visual Position-Information: An Experimentally Supported Model
749 Citations1999Robert J. van Beers, Anne C. Sittig +1 more
The proposed model can explain the unexpectedly small sizes of the variable errors in the localization of a seen hand that were reported earlier and implies that the CNS has knowledge about the direction-dependent precision of the proprioceptive and visual information.
Data Fusion for Sensory Information Processing Systems
527 Citations1990James J. Clark, Alan Yuille
The role of data Fusion in Sensory Systems, Bayesian Sensory Information Processing, and Temporal Aspects of Data Fusion are introduced.
ScienceCombining Sensory Information: Mandatory Fusion Within, but Not Between, Senses
499 Citations2002James Hillis, Marc O. Ernst +2 more
It is reported that single-cue information is indeed lost when cues from within the same sensory modality (disparity and texture gradients in vision) are combined, but not when different modalities (vision and haptics) are Combined.
Journal of VisionSlant from texture and disparity cues: Optimal cue combination
433 Citations2004James Hillis, Simon J. Watt +2 more
There is a growing empirical consensus that MLE provides a good quantitative account of cue combination and that sensory information is used in a manner that maximizes the precision of perceptual estimates.
Vision ResearchDo humans optimally integrate stereo and texture information for judgments of surface slant?
423 Citations2003David C. Knill, Jeffrey A. Saunders
The results are consistent with the hypothesis that humans optimally combine the two cues to surface slant, with cue weights proportional to the subjective reliability of the cues.
Experimental Brain ResearchThe precision of proprioceptive position sense
398 Citations1998Robert J. van Beers, Anne C. Sittig +1 more
The derived spatial probability distributions which describe the precision of localization on the basis of three different sources of information suggest that hand positions closer to the shoulder are localized more precisely than positions further away and that joint angles are represented in the central nervous system more precise than they are consciously perceived.
Nature NeuroscienceExperience can change the 'light-from-above' prior
391 Citations2004Wendy J. Adams, Erich W. Graf +1 more
It is shown that the 'light-from-above' prior, used to extract information about shape from shading is modified in response to active experience with the scene, demonstrating that priors are constantly adapted by interactiveExperience with the environment.
Current Opinion in NeurobiologyBayesian models of object perception
310 Citations2003Daniel Kersten, Alan Yuille
Advances in Bayesian models of computer vision and in the measurement and modeling of natural image statistics are providing the tools to test and constrain theories of human object perception, which are having an impact on the interpretation of cortical function.
Perception & PsychophysicsThe ventriloquist effect does not depend on the direction of deliberate visual attention
255 Citations2000Paul Bertelson, Jean Vroomen +2 more
It is concluded that ventriloquism largely reflects automatic sensory interactions, with little or no role for deliberate spatial attention.
Vision ResearchWeighted linear cue combination with possibly correlated error
241 Citations2003İpek Oruç, Laurence T. Maloney +1 more
Six out of eight observers did better with combined cues than with either cue alone, and performance was consistent with optimal combination of uncorrelated cues.
Perception & PsychophysicsThe ventriloquist effect does not depend on the direction of automatic visual attention
196 Citations2001Jean Vroomen, Paul Bertelson +1 more
Ventriloquism can be dissociated from exogenous visual attention and appears to reflect sensory interactions with little role for the direction of visual spatial attention.
Current BiologyViewing Geometry Determines How Vision and Haptics Combine in Size Perception
173 Citations2003Sergei Gepshtein, Martin S. Banks
Combined size estimates are finer than is possible with either vision or haptics alone; indeed, they approach statistical optimality.
Experimental Brain ResearchFeeling what you hear: auditory signals can modulate tactile tap perception
165 Citations2004Jean‐Pierre Bresciani, Marc O. Ernst +4 more
Results show that when provided with auditory and tactile sensory signals that are likely to be generated by the same stimulus, the central nervous system (CNS) tends to automatically integrate these signals.
Vision ResearchDiscrimination of planar surface slant from texture: human and ideal observers compared
148 Citations1998David C. Knill
Subjects' discrimination performance improves dramatically with increasing surface slant, tracking the performance of the ideal observers; subjects can integrate texture information over a large range of visual angles; and comparisons between human subjects and ideal observers show that the human observers rely to some degree on foreshortening information.
Journal of the Optical Society of America AIdeal cue combination for localizing texture-defined edges
139 Citations2001Michael S. Landy, Haruyuki Kojima
An ideal-observer model, previously used in studies of depth perception and color constancy, was fitted to the data and this model does account for most trends in the data.
Vision ResearchPerception of 3D surface orientation from skew symmetry
86 Citations2001Jeffrey A. Saunders, David C. Knill
An explanation of the results involving the optimal integration of stereo and skew symmetry is proposed, and a Bayesian model is presented that can account for the pattern of biases.
Bulletin of Environmental Contamination and ToxicologyHigh-level Vision as Statistical Inference
80 Citations1999Daniel Kersten
International Journal of Computer VisionHow Optimal Depth Cue Integration Depends on the Task
47 Citations2000Paul Schrater, Daniel Kersten
Journal of VisionBayesian combination of ambiguous shape cues
38 Citations2004Wendy J. Adams, Pascal Mamassian
The data confirm that different cues can disambiguate each other and are well modeled by a Bayesian approach incorporating a prior for convexity.
Journal of VisionWhat is an inter-sensory object? Optimal combination of vision and touch depends on their spatial coincidence
1 Citations2004Sergei Gepshtein, Johannes Burge +2 more