Solving Deep Memory POMDPs with Recurrent Policy Gradients
Lecture notes in computer sciencePublished 1 January 2007
Daan Wierstra, A. Foerster, Jan Peters, Jürgen Schmidhuber
Citations146
SJR quartileQ2
SJR score0.35
SNIP0.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Recurrent Policy Gradients, a modelfree reinforcement learning (RL) method creating limited-memory stochastic policies for partially observable Markov decision problems (POMDPs) that require long-term memories of past observations is presented.
Abstract
This paper presents Recurrent Policy Gradients, a model-free reinforcement learning (RL) method creating limited-memory sto-chastic policies for partially observable Markov decision problems (POMDPs) that require long-term memories of past observations. The approach involves approximating a policy gradient for a Recurrent Neural Network (RNN) by backpropagating return-weighted characteristic eligibilities through time. Using a "Long Short-Term Memory" architecture, we are able to outperform other RL methods on two important benchmark tasks. Furthermore, we show promising results on a complex car driving simulation task.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
Policy Gradient Methods for Reinforcement Learning with Function Approximation
4,982 Citations1999Richard S. Sutton, David McAllester +2 more
This paper proves for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.
Proceedings of the IEEEBackpropagation through time: what it does and how to do it
4,904 Citations1990Paul J. Werbos
This paper first reviews basic backpropagation, a simple method which is now being widely used in areas like pattern recognition and fault diagnosis, and describes further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations or true recurrent networks, and other practical issues which arise with this method.
Proceedings of the International Joint Conference on Neural Networks, 2003.
1,673 Citations2003
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
1,261 Citations2009
This chapter contains sections titled: Introduction Exponential Error Decay Dilemma: Avoiding Aradient Decay Prevents Long-Term Latching Remedies and Conclusion.
Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.A combined SVM and LDA approach for classification
987 Citations2006Tao Xiong, Vladimir Cherkassky
It is shown that existing SVM software can be used to solve the SVM/LDA formulation and empirical comparisons of the proposed algorithm with SVM and LDA using both synthetic and real world benchmark data are presented.
IEEE Transactions on Neural NetworksLearning to trade via direct reinforcement
512 Citations2001John Moody, Matthew Saffell
It is demonstrated how direct reinforcement can be used to optimize risk-adjusted investment returns (including the differential Sharpe ratio), while accounting for the effects of transaction costs.
Policy Gradient Methods for Robotics
511 Citations2006Jan Peters, Stefan Schaal
An overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field is given and how the most recently developed methods can significantly improve learning performance is shown.
Elsevier eBooksLearning Without State-Estimation in Partially Observable Markovian Decision Processes
328 Citations1994Satinder Singh, Tommi Jaakkola +1 more
A new framework for learning without state-estimation in POMDPs is developed by including stochastic policies in the search space, and by defining the value or utility of a distribution over states.
Lecture notes in computer scienceNatural Actor-Critic
310 Citations2005Jan Peters, Sethu Vijayakumar +1 more
Neural NetworksA stochastic reinforcement learning algorithm for learning real-valued functions
294 Citations1990Vijaykumar Gullapalli
A stochastic reinforcement learning algorithm for learning functions with continuous outputs using a connectionist network that learns to perform an underconstrained positioning task using a simulated 3 degree-of-freedom robot arm.
Robotics and Autonomous SystemsBiped dynamic walking using reinforcement learning
196 Citations1997Hamid Benbrahim, Judy A. Franklin
This paper presents some results from a study of biped dynamic walking using reinforcement learning, where a hardware biped robot was built, a new reinforcement learning algorithm as well as a new learning architecture were developed.
Reinforcement Learning with Long Short-Term Memory
193 Citations2001Bram Bakker
Model-free RL-LSTM using Advantage (λ) learning and directed exploration can solve non-Markovian tasks with long-term dependencies between relevant events.
arXiv (Cornell University)Learning Finite-State Controllers for Partially Observable Environments
192 Citations2013Nicolas Meuleau, Leonid Peshkin +2 more
Evolving neural network controllers for unstable systems
187 Citations2002Alexis P. Wieland
The author describes how genetic algorithms were used to create recurrent neural networks to control a series of unstable systems, including network controllers with two, one, and zero inputs, and variations of the pole balancing problem.
Journal of Artificial Intelligence ResearchExperiments with Infinite-Horizon, Policy-Gradient Estimation
139 Citations2001J. Baxter, Peter L. Bartlett +1 more
Algorithms that perform gradient ascent of the average reward in a partially observable Markov decision process (POMDP) based on GPOMDP, an algorithm introduced in a companion paper (Baxter & Bartlett, 2001), which computes biased estimates of the performance gradient in POMDPs.
Direct gradient-based reinforcement learning
96 Citations2002J. Baxter, Peter L. Bartlett
An algorithm for computing approximations to the gradient of the average reward from a single sample path of a controlled partially observable Markov decision process is presented and it is proved that the algorithm converges with probability 1.
Reinforcement learning and its application to control
78 Citations1992Vijaykumar Gullapalli
It is argued that for certain types of problems the latter approach, of which reinforcement learning is an example, can yield faster, more reliable learning, while the former approach is relatively inefficient.
ANU Open Research (Australian National University)Policy-Gradient Algorithms for Partially Observable Markov Decision Processes
75 Citations2003Douglas Aberdeen
This thesis develops several improved algorithms for learning policies with memory in an infinite-horizon setting including an application written for the Bunyip cluster that won the international Gordon-Bell prize for price/performance in 2001.
Neural Information Processing SystemsFast Online Policy Gradient Learning with SMD Gain Vector Adaptation
29 Citations2005Jin Yu, Douglas Aberdeen +1 more
This work improves its robustness and speed of convergence with stochastic meta-descent, a gain vector adaptation method that employs fast Hessian-vector products that outperform previously employed online stochastics, offline conjugate, and natural policy gradient methods.
Toward effective combination of off-line and on-line training in ADP framework
14 Citations2007Danil Prokhorov
This work introduces the approach of combining proven off-line methods of training for robustness with a group of on-line method, and illustrates importance of using recurrent neural networks for both controller/actor and critic.