Dopamine Neurons Can Represent Context-Dependent Prediction Error
NeuronPublished 1 January 2004Open access
Hiroyuki Nakahara, H. Itoh, Reiko Kawagoe, Yoriko Takikawa, Okihide Hikosaka
Citations291
SJR quartileQ1
SJR score6.75
SNIP2.95
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is found that monkey DA neurons can encode a context-dependent prediction error, which can be simulated only by implementing the relevant context into the TD model.
Abstract
Midbrain dopamine (DA) neurons are thought to encode reward prediction error. Reward prediction can be improved if any relevant context is taken into account. We found that monkey DA neurons can encode a context-dependent prediction error. In the first noncontextual task, a light stimulus was randomly followed by reward, with a fixed equal probability. The response of DA neurons was positively correlated with the number of preceding unrewarded trials and could be simulated by a conventional temporal difference (TD) model. In the second contextual task, a reward-indicating light stimulus was presented with the probability that, while fixed overall, was incremented as a function of the number of preceding unrewarded trials. The DA neuronal response then was negatively correlated with this number. This history effect corresponded to the prediction error based on the conditional probability of reward and could be simulated only by implementing the relevant context into the TD model.
Keywords
NeuroscienceBiochemistry, Genetics and Molecular Biology
ScienceA Neural Substrate of Prediction and Reward
9,610 Citations1997Wolfram Schultz, Peter Dayan +1 more
Findings in this work indicate that dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events can be understood through quantitative theories of adaptive optimizing control.
Journal of NeurophysiologyPredictive Reward Signal of Dopamine Neurons
4,582 Citations1998Wolfram Schultz
Dopamine systems may have two functions, the phasic transmission of reward information and the tonic enabling of postsynaptic neurons.
Journal of NeuroscienceA framework for mesencephalic dopamine systems based on predictive Hebbian learning
2,114 Citations1996P. Read Montague, Peter Dayan +1 more
A theoretical framework is developed that shows how mesencephalic dopamine systems could distribute to their targets a signal that represents information about future expectations and shows that, through a simple influence on synaptic plasticity, fluctuations in dopamine release can act to change the predictions in an appropriate manner.
ScienceDiscrete Coding of Reward Probability and Uncertainty by Dopamine Neurons
2,100 Citations2003Christopher D. Fiorillo, Philippe N. Tobler +1 more
Using distinct stimuli to indicate the probability of reward, it was found that the phasic activation of dopamine neurons varied monotonically across the full range of probabilities, supporting past claims that this response codes the discrepancy between predicted and actual reward.
IEEE Transactions on Bio-medical ElectronicsA Method of Measuring Eye Movemnent Using a Scieral Search Coil in a Magnetic Field
1,835 Citations1963D. A. Robinson
With the subject exposed to an alternating magnetic field, eye position may be accurately recorded from the voltage generated in a coil of wire embedded in a scleral contact lens worn by the subject.
NatureNeural correlates of decision variables in parietal cortex
1,757 Citations1999Michael L. Platt, Paul W. Glimcher
The data indicate that a decision-theoretic model may provide a powerful new framework for studying the neural processes that intervene between sensation and action.
Psychological ReviewToward a modern theory of adaptive networks: Expectation and prediction.
1,457 Citations1981Richard S. Sutton, Andrew G. Barto
The adaptive element presented learns to increase its response rate in anticipation of increased stimulation, producing a conditioned response before the occurrence of the unconditioned stimulus, and is in strong agreement with the behavioral data regarding the effects of stimulus context.
ScienceThe Basal Ganglia and Adaptive Motor Control
1,325 Citations1994Ann M. Graybiel, Toshihiko Aosaki +2 more
The basal ganglia are neural structures within the motor and cognitive control circuits in the mammalian forebrain and are interconnected with the neocortex by multiple loops that have a distributed modular architecture resembling local expert architectures of computational learning models during sensorimotor learning.
The MIT Press eBooksModels of Information Processing in the Basal Ganglia
1,187 Citations1994
Part 1 Fundamentals: information processing in modular circuits linking basal ganglia and cerebral cortex, James C. Houk context-dependent activity in primate striatum reflecting past and future behavioural events and a model of how the basal ganglia generate and use neural signals that predict reinforcement are presented.
Nature NeuroscienceDopamine neurons report an error in the temporal prediction of reward during learning
1,130 Citations1998Jeffrey R. Hollerman, Wolfram Schultz
Dopamine neuron responses reflected the changes in reward prediction during individual learning episodes; dopamine neurons were activated by rewards during early trials, but activation was progressively reduced as performance was consolidated and rewards became more predictable.
NatureDopamine responses comply with basic assumptions of formal learning theory
1,042 Citations2001Waelti Pascale, Anthony Dickinson +1 more
The data indicate that the use of analytical tests derived from formal behavioural learning theory provides a powerful approach for studying the role of single neurons in learning.
Journal of NeurophysiologyResponses of monkey dopamine neurons during learning of behavioral reactions
993 Citations1992T. Ljungberg, Paul Apicella +1 more
D dopamine neurons responded simultaneously to the sight of primary food reward and to the conditioned stimulus associated with reward, thus demonstrating specificity for the behavioral significance of stimuli.
Cognitive Brain ResearchNeuropharmacology of timing and time perception
941 Citations1996Warren H. Meck
A psychological model of duration discrimination that differentiates the speed of an internal clock used for the registration of current sensory input from the speedOf the memory-storage process used forThe representation of the durations of prior stimulus events has proven useful in integrating these findings.
NatureA cellular mechanism of reward-related learning
899 Citations2001John N. J. Reynolds, Brian I. Hyland +1 more
It is proposed that stimulation of the substantia nigra when the lever is pressed induces a similar potentiation of cortical inputs to the striatum, positively reinforcing the learning of the behaviour by the rats.
Neural ComputationTD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
800 Citations1994Gerald Tesauro
The latest version of TD-Gammon is now estimated to play at a strong master level that is extremely close to the world's best human players.
ACM SIGART BulletinDyna, an integrated architecture for learning, planning, and reacting
767 Citations1991Richard S. Sutton
Dyna is an AI architecture that integrates learning, planning, and reactive execution that relies on machine learning methods for learning from examples, yet is not tied to any particular method.
Neural NetworksMetalearning and neuromodulation
687 Citations2002Kenji Doya
A computational theory on the roles of the ascending neuromodulatory systems from the viewpoint that they mediate the global signals that regulate the distributed learning mechanisms in the brain is presented.
NatureReward expectancy in primate prefrental neurons
685 Citations1996Masataka Watanabe
It is reported that the delay activity in prefrontal neurons is dependent also on the particular reward received for the behavioural response, and to the way the reward is given.
ScienceRole for Cingulate Motor Area Cells in Voluntary Movement Selection Based on Reward
658 Citations1998Keisetsu Shima, Jun Tanji
It is suggested that cells in the rostral cingulate motor area, one of the higher order motor areas in the cortex, play a part in processing the reward information for motor selection.
Nature NeuroscienceExpectation of reward modulates cognitive signals in the basal ganglia
642 Citations1998Reiko Kawagoe, Yoriko Takikawa +1 more
The results indicate that the caudate contributes to the determination of oculomotor outputs by connecting motivational values (for example, expectation of reward) to visual information.
The MIT Press eBooksA Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement
550 Citations1994James C. Houk, James L. Adams +1 more
This chapter contains sections titled: Introduction, Dopamine Neurons, Organization of Strtosomal Modules, Mechanism of Responsiveness to Predictors of Reinforcement, Correspondence with the Theory of Adaptive Critics, and Relation to the Actor-Critic Architecture.
Journal of NeurophysiologyImportance of unpredictability for reward responses in primate dopamine neurons
543 Citations1994Jacques Mirenowicz, Wolfram Schultz
D dopamine neurons responded to liquid reward during learning but not when task performance was established, at which time the neuronal response occurred to the conditioned, reward-predicting, movement-triggering stimulus.
NeuronCoding of Cognitive Magnitude
533 Citations2003Andreas Nieder, Earl K. Miller
Certain cognitive and perceptual/sensory representations share the same fundamental mechanisms and neural coding schemes, and nonlinear compression was observed during the acquisition phase of the task and maintained through the memory phase with no further compression.
ScienceAnterior Cingulate: Single Neuronal Signals Related to Degree of Reward Expectancy
514 Citations2002Munetaka Shidara, Barry J. Richmond
As monkeys perform schedules containing several trials with a visual cue indicating reward proximity, their error rates decrease as the number of remaining trials decreases, suggesting that their motivation and/or reward expectancy increases as the reward approaches.
NeuronEffect of Expected Reward Magnitude on the Response of Neurons in the Dorsolateral Prefrontal Cortex of the Macaque
450 Citations1999Matthew I. Leon, Michael N. Shadlen
Reward-related enhancement was evident throughout the memory period and was most pronounced when the remembered target appeared in the neuron's response field, and the mixture of neural signals representing spatial working memory and reward expectation appears to be a distinct feature of area 46.
The MIT Press eBooksAdaptive Critics and the Basal Ganglia
450 Citations1994Andrew G. Barto
This chapter contains sections titled Reinforcement Learning, The Actor-Critic Architecture, Imminence Weighting, An Input's Value, Learning to Predict, The Adaptive Critic Learning Rule, Effective Reinforcement, and The Actor Learning Rule.
NatureA neural correlate of response bias in monkey caudate nucleus
436 Citations2002Johan Lauwereyns, Katsumi Watanabe +2 more
This work identifies neurons in the monkey caudate nucleus that create a spatially selective response bias depending on the expected gain, and concludes that this neuronal mechanism creates an advance bias that favours a spatial response when it is associated with a high reward value.
Annual Review of NeuroscienceSequential Organization of Multiple Movements: Involvement of Cortical Motor Areas
433 Citations2001Jun Tanji
The issue of motor selection to arrange multiple movements in an appropriate temporal order is dealt with, rather than the issue of constructing spatio-temporal structures in a single action.
NaturePerformance monitoring by the supplementary eye field
402 Citations2000Veit Stuphorn, Tracy Taylor +1 more
ScienceNeuronal Correlates of Goal-Based Motor Selection in the Prefrontal Cortex
391 Citations2003Kenji Matsumoto, Wataru Suzuki +1 more
A significant number of neurons in the medial prefrontal cortex were activated, after cue presentation and before motor execution, only by particular action-reward combinations, likely to underlie goal-based action selection.
Journal of NeurophysiologyDopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements
351 Citations1990Ranulfo Romo, W Schultz
Response to door opening in this task was similar to the touch response during self-initiated movements in the same neurons in terms of latency, duration, and magnitude, and responses appeared to be related to the appetitive properties of the object being touched rather than the object itself.
NatureAntisaccade performance predicted by neuronal activity in the supplementary eye field
337 Citations1997Madeleine Schlag-Rey, Nelly Amador +2 more
It is shown, using recordings from the supplementary eye field (a frontal cortex oculomotor centre) in monkeys, that visual and movement neurons retain the same spatial selectivity across randomly mixed pro- and antisaccade trials, suggesting a mechanism through which voluntary antisaccades commands can override reflexive glances.
NatureBee foraging in uncertain environments using predictive hebbian learning
318 Citations1995P. Read Montague, Peter Dayan +2 more
A model of bee foraging in uncertain environments based on a neuron with widespread projections to odour processing regions of the honeybee brain and a predictive form of hebbian synaptic plasticity is constructed, showing how neuromodulatory influences can be used to bias actions and control synaptic Plasticity in a way that goes beyond standard correlational mechanisms.
Journal of NeuroscienceCorrelated Coding of Motivation and Outcome of Decision by Dopamine Neurons
305 Citations2003Takemasa Satoh, Sadamu Nakai +2 more
The dual correlated coding of motivation and REEs suggested the involvement of the dopamine system, both in reinforcement in more elaborate ways than currently proposed and in motivational function in reward-based decision-making and learning.
Experimental Brain ResearchModulation of saccadic eye movements by predicted reward outcome
296 Citations2002Yoriko Takikawa, Reiko Kawagoe +3 more
A memory-guided saccade task in which only one fixed direction out of four was rewarded, called the one-direction-rewarded task (1DR), found that the parameters of saccades changed depending on whether or not the saccADE was followed by reward.
Nature NeuroscienceAnterior cingulate activity during routine and non-routine sequential behaviors in macaques
295 Citations2000Emmanuel Procyk, Yuji Tanaka +1 more
Neurons activated with sequence components (68%) differed in activity between search and repetition, and task-related activity occurred in a regime of memory-based motor performance in which attention to action is less necessary.
NeuronMonitoring and Control of Action by the Frontal Lobes
286 Citations2002Jeffrey D. Schall, Veit Stuphorn +1 more
These processes are investigated by recording neural activity in the frontal lobe of macaque monkeys performing a countermanding task and appear not to control directly movement initiation but instead signal the production of errors, the anticipation and delivery of reinforcement, and the presence of processing conflict.
Journal of NeurophysiologyNeuronal Activity in Medial Frontal Cortex During Learning of Sequential Procedures
266 Citations1998Kae Nakamura, Katsuyuki Sakai +1 more
The results suggest that the pre-SMA, rather than SMA, is more involved in the acquisition of new sequential procedures.
Proceedings of the National Academy of SciencesRole for cells in the presupplementary motor area in updating motor plans.
241 Citations1996Kohei Shima, Hajime Mushiake +2 more
The data suggest a role of pre-SMA cells in updating motor plans for subsequent temporally ordered movements, not common in the SMA and not found in the primary motor cortex.
Journal of Cognitive NeuroscienceA Computational Model of How the Basal Ganglia Produce Sequences
230 Citations1998Gregory S. Berns, Terrence J. Sejnowski
The behavior of the model was found to be sensitive to the ratio of the striatal-nigral learning rate to the STN-GP learning rate, and the degree of striatal inhibition of the globus pallidus had a significant influence on both learning and the ability to select an action.
Journal of NeuroscienceVisual and Anticipatory Bias in Three Cortical Eye Fields of the Monkey during an Adaptive Decision-Making Task
215 Citations2002Brian C. Coe, Kazuya Tomihara +2 more
The results suggest that although all three cortical eye fields reflect attentional and intentional aspects of sensorimotor processing, SEF plays an earlier and perhaps more cognitive role in internally guided decision-making processes for saccades.
Journal of NeurophysiologyInfluence of Reward Expectation on Visuospatial Processing in Macaque Lateral Prefrontal Cortex
204 Citations2002Shunsuke Kobayashi, Johan Lauwereyns +3 more
Previous findings suggesting that the LPFC exerts dual influences based on predicted reward outcome: improvement of memory-guided saccades and suppression of inappropriate behavior (when reward is not expected) are extended.
Experimental Brain ResearchLearning of sequential movements by neural network model with dopamine-like reinforcement signal
201 Citations1998Roland E. Suri, Wolfram Schultz
This study explored a neural network model that used a reward-prediction error signal strongly resembling dopamine responses for learning movement sequences and efficiently allowed the model to learn long sequences.
Journal of Cognitive NeuroscienceParallel Cortico-Basal Ganglia Mechanisms for Acquisition and Execution of Visuomotor Sequences—A Computational Approach
196 Citations2001Hiroyuki Nakahara, Kenji Doya +1 more
A computational model based on the hypothesis that the visual and motor loops achieve both the quick acquisition of novel sequences and the robust execution of well-learned sequences is examined and found that the dual mechanism with the coordinator was superior to the single (visual or motor) mechanism.
Journal of NeurophysiologyResponses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey
185 Citations1987Wolfram Schultz, Ranulfo Romo
The results show that midbrain DA neurons projecting to the striatum respond to noxious somatosensory input in the anesthetized monkey, providing evidence for an involvement of DAergic neurotransmission in the representation of exteroceptiveinput in the brain.
Neural ComputationTemporal Difference Model Reproduces Anticipatory Neural Activity
169 Citations2001Roland E. Suri, Wolfram Schultz
This study demonstrates that the prediction signal of the TD model reproduces characteristics of cortical and striatal anticipatory neural activity, and suggests that tonic anticipatory activities may reflect prediction signals that are involved in the processing of dopamine neuron activity.
Journal of NeurophysiologyReward-Predicting Activity of Dopamine and Caudate Neurons—A Possible Mechanism of Motivational Control of Saccadic Eye Movement
165 Citations2004Reiko Kawagoe, Yoriko Takikawa +1 more
Activity of midbrain dopamine neurons and caudate projection neurons is examined while monkeys were performing a one-direction-rewarded version of memory-guided saccade task to suggest that DA neurons, with their connection to CD neurons, modulate the spatially selective signals in CD neurons in the reward-predicting manner andCD neurons in turn modulate saccades parameters with their polysynaptic connections to the oculomotor brain stem.
Current Opinion in NeurobiologyNeural correlates of decisions
154 Citations2002Michael L. Platt
These observations demonstrate that a single sensory cue, such as the distress call of a juvenile monkey, can generate multiple different behavioral responses, depending on context, learned associations, and memory for the outcome of prior actions.
Journal of NeurophysiologyReward-Dependent Spatial Selectivity of Anticipatory Activity in Monkey Caudate Neurons
129 Citations2002Yoriko Takikawa, Reiko Kawagoe +1 more
Using a memory-guided saccade task, a group of neurons in the monkey caudate nucleus showed activity before presentation of an instruction cue stimulus, suggesting that each CD neuron indicates a particular position-reward association prospectively, usually with contralateral preference.
NeuronFeature-Based Anticipation of Cues that Predict Reward in Monkey Caudate Nucleus
126 Citations2002Johan Lauwereyns, Yoriko Takikawa +6 more
NeuronA Neural Correlate of Oculomotor Sequences in Supplementary Eye Field
99 Citations2002Xiaofeng Lu, Masako Matsuzawa +1 more
An oculomotor sequence task in which the monkey had to choose the target solely by the sequential context, not by the current stimulus combination, suggests that the SEF contributes to the generation of saccades in many learned sequences.
The Journal of Comparative NeurologyMultiple striatal representation in primate substantia nigra
70 Citations1994André Parent, Lili‐Naz Hazrati
The pattern of arborization of the striatonigral fibers in the squirrel monkey was studied with Phaseolus vulgaris‐leucoagglutinin (PHA‐L) and biocytin as anterograde tracers and indicated that striatal information is conveyed to the substantia nigra in a highly ordered fashion through multiple segregated channels.
The MIT Press eBooksMotivated Reinforcement Learning
50 Citations2002Peter Dayan
The standard reinforcement learning view of the involvement of neuromodulatory systems in instrumental conditioning includes a rather straightforward conception of motivation as prediction of sum future reward.
Experimental Brain ResearchAnticipatory responses of dopamine neurons and cortical neurons reproduced by internal model
33 Citations2001Roland E. Suri
Comparison of the model architecture with biological neural networks suggests that chains of neurons with tonic anticipatory activity may underlie the formation of novel associative chains.
Maker Innovations SeriesReinforcement Learning
19 Citations2024G. R. Kanagachidambaresan, N. Bharathi
The MIT Press eBooksModeling the Roles of Basal Ganglia in Timing and Sequencing Saccadic Eye Movements
14 Citations1994Michael A. Arbib, Peter Ford Dominey
This chapter contains sections titled: Basal Ganglia and the Control of Working Memory, Corticostriatal Plasticity, Notes, References.