A Generalized Natural Actor-Critic Algorithm
Neural Information Processing SystemsPublished 7 December 2009
Tetsuro Morimura, Eiji Uchibe, Junichiro Yoshimoto, Kenji Doya
Citations13
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A generalized Natural Gradient that linearly interpolates the two FIMs is described and an efficient implementation for the gNG learning based on a theory of the estimating function, the generalized Natural Actor-Critic (gNAC) algorithm is proposed.
Abstract
Policy gradient Reinforcement Learning (RL) algorithms have received substantial attention, seeking stochastic policies that maximize the average (or discounted cumulative) reward. In addition, extensions based on the concept of the Natural Gradient (NG) show promising learning efficiency because these regard metrics for the task. Though there are two candidate metrics, Kakade's Fisher Information Matrix (FIM) for the policy (action) distribution and Morimura's FIM for the state-action joint distribution, but all RL algorithms with NG have followed Kakade's approach. In this paper, we describe a generalized Natural Gradient (gNG) that linearly interpolates the two FIMs and propose an efficient implementation for the gNG learning based on a theory of the estimating function, the generalized Natural Actor-Critic (gNAC) algorithm. The gNAC algorithm involves a near optimal auxiliary function to reduce the variance of the gNG estimates. Interestingly, the gNAC can be regarded as a natural extension of the current state-of-the-art NAC algorithm [1], as long as the interpolating parameter is appropriately selected. Numerical experiments showed that the proposed gNAC algorithm can estimate gNG efficiently and outperformed the NAC algorithm.
Keywords
Computer Science
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
MIT Press eBooksIntroduction to Reinforcement Learning
6,868 Citations1998Richard S. Sutton, Andrew G. Barto
In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning.
Neural ComputationNatural Gradient Works Efficiently in Learning
2,700 Citations1998Шун-ичи Амари
The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters.
ScholarlyCommons (University of Pennsylvania)A Natural Policy Gradient
725 Citations2001Sham M. Kakade
This work provides a natural gradient method that represents the steepest descent direction based on the underlying structure of the parameter space and shows drastic performance improvements in simple MDPs and in the more challenging MDP of Tetris.
SIAM Journal on Control and OptimizationOnActor-Critic Algorithms
694 Citations2003Vijay R. Konda, John N. Tsitsiklis
This article proposes and analyzes a class of actor-critic algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic.
Policy Gradient Methods for Robotics
511 Citations2006Jan Peters, Stefan Schaal
An overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field is given and how the most recently developed methods can significantly improve learning performance is shown.
Applied Physics Letters10.1162/1532443041827907
446 Citations2000
The new algorithm, least-squares policy iteration (LSPI), learns the state-action value function which allows for action selection without a model and for incremental policy improvement within a policy-iteration framework.
Lecture notes in computer scienceNatural Actor-Critic
310 Citations2005Jan Peters, Sethu Vijayakumar +1 more
Machine LearningTechnical Update: Least-Squares Temporal Difference Learning
297 Citations2002Justin A. Boyan
This paper updates Bradtke and Barto's work in three significant ways: first, it presents a simpler derivation of the LSTD algorithm; second, it generalizes from λ = 0 to arbitrary values of λ; at the extreme of κ, the resulting new algorithm is shown to be a practical, incremental formulation of supervised linear regression.
Neural NetworksA stochastic reinforcement learning algorithm for learning real-valued functions
294 Citations1990Vijaykumar Gullapalli
A stochastic reinforcement learning algorithm for learning functions with continuous outputs using a connectionist network that learns to perform an underconstrained positioning task using a simulated 3 degree-of-freedom robot arm.
Stochastic policy gradient reinforcement learning on a simple 3D biped
270 Citations2005Russ Tedrake, Tianbao Zhang +1 more
This work presents a learning system which is able to quickly and reliably acquire a robust feedback control policy for 3D dynamic walking from a blank-slate using only trials implemented on the physical robot.
FigshareCovariant Policy Search
159 Citations2018J. Andrew Bagnell, Jeff Schneider
This work proposes a natural metric on controller parameterization that results from considering the manifold of probability distributions over paths induced by a stochastic controller that leads to a covariant gradient ascent rule.
Incremental Natural Actor-Critic Algorithms
154 Citations2007Shalabh Bhatnagar, Mohammad Ghavamzadeh +2 more
The results extend prior two-timescale convergence results for actor-critic methods by using temporal difference learning in the actor and by incorporating natural gradients, and they extend prior empirical studies of natural actor- Criterion methods by providing the first convergence proofs and the first fully incremental algorithms.
The MIT Press eBooksNatural Actor-Critic for Road Traffic Optimisation
92 Citations2007Silvia Richter, Douglas Aberdeen +1 more
A policy-gradient reinforcement learning approach is used to directly optimise the traffic signals, mapping currently deployed sensor observations to control signals and extending natural-actor critic approaches to work for distributed and online infinite-horizon problems.
BernoulliInformation Geometry of Estimating Functions in Semi-Parametric Statistical Models
76 Citations1997Шун-ичи Амари, Motoaki Kawanabe
Journal of Statistical Planning and InferenceUnbiased statistical estimation functions for parameters in presence of nuisance parameters
36 Citations1984B. Chandrasekar, B. K. Kale
Neural ComputationDerivatives of Logarithmic Stationary Distributions for Policy Gradient Reinforcement Learning
19 Citations2009Tetsuro Morimura, Eiji Uchibe +3 more
A method for estimating the log stationary state distribution derivative (LSD) as a useful form of the derivative of the stationary state distributions through backward Markov chain formulation and a temporal difference learning framework is proposed.
Lecture notes in computer scienceA New Natural Policy Gradient by Stationary Distribution Metric
13 Citations2008Tetsuro Morimura, Eiji Uchibe +2 more
A Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, is proposed and derived, and a new NPG named "Natural State-action Gradient"(NSG) is derived).
A semiparametric statistical approach to model-free policy evaluation
9 Citations2008Tsuyoshi Ueno, Motoaki Kawanabe +3 more
LSTD-based policy evaluation from the new view-point of semiparametric statistical inference is discussed, showing that the estimator can be obtained from a particular estimating function which guarantees its convergence to the true value asymptotically, without specifying a model of the environment.
Lecture notes-monograph seriesOptimal Instrumental Variable Estimation for Linear Models With Stochastic Regressors Using Estimating Functions
5 Citations1997A. C. Singh, R. Prabhakar Rao