A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients
IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews)Published 1 November 2012Open access
I. Grondman, Lucian Buşoniu, Gabriel A. D. Lopes, Robert Babuška
Citations1,005
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The workings of the natural gradient is described, which has made its way into many actor-critic algorithms over the past few years, and a review of several standard and natural actor-critic algorithms is given.
Abstract
International audience
Keywords
Computer ScienceEngineering
Machine LearningQ-learning
8,970 Citations1992Christopher J. Watkins, Peter Dayan
This paper presents and proves in detail a convergence theorem forQ-learning based on that outlined in Watkins (1989), showing that Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action- values are represented discretely.
Journal of Artificial Intelligence ResearchReinforcement Learning: A Survey
8,831 Citations1996Leslie Pack Kaelbling, Michael L. Littman +1 more
Central issues of reinforcement learning are discussed, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
IEEE Transactions on Systems Man and CyberneticsFuzzy logic in control systems: fuzzy logic controller. I
5,581 Citations1990C.C. Lee
This paper presents a survey of the FLC; describes a general methodology for constructing an FLC and assessing its per formance; and points to problems which need further research.
Policy Gradient Methods for Reinforcement Learning with Function Approximation
4,982 Citations1999Richard S. Sutton, David McAllester +2 more
This paper proves for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.
IBM Journal of Research and DevelopmentSome Studies in Machine Learning Using the Game of Checkers
4,355 Citations1959Arthur L. Samuel
A new signature-table technique is described together with an improved book-learning procedure which is thought to be much superior to the linear polynomial method and to permit the program to look ahead to a much greater depth than it otherwise could do.
Machine LearningLearning to Predict by the Methods of Temporal Differences
3,894 Citations1988Richard S. Sutton
This article introduces a class of incremental learning procedures specialized for prediction-that is, for using past experience with an incompletely known system to predict its future behavior, and proves their convergence and optimality for special cases and relate them to supervised-learning methods.
Machine LearningLearning to predict by the methods of temporal differences
2,758 Citations1988Richard S. Sutton
Neural ComputationNatural Gradient Works Efficiently in Learning
2,700 Citations1998Шун-ичи Амари
The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters.
IEEE Transactions on Automatic ControlMultivariate stochastic approximation using a simultaneous perturbation gradient approximation
2,282 Citations1992James C. Spall
The paper presents an SA algorithm that is based on a simultaneous perturbation gradient approximation instead of the standard finite-difference approximation of Keifer-Wolfowitz type procedures that can be significantly more efficient than the standard algorithms in large-dimensional problems.
IEEE Transactions on Systems Man and CyberneticsFuzzy logic in control systems : fuzzy logic controller. Part II
2,227 Citations1990Chuen Chien Lee
The fuzzy logic controller (FLC) based on fuzzy logic provides a means of converting a linguistic control strategy based on expert knowledge into an automatic control strategy.
AutomaticaOnline actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
1,625 Citations2010Kyriakos G. Vamvoudakis, Frank L. Lewis
This paper presents an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks, and calls this ‘synchronous’ policy iteration.
On-line Q-learning Using Connectionist Systems
1,497 Citations1994G. A. Rummery
IEEE Transactions on Automatic ControlAn analysis of temporal-difference learning with function approximation
1,202 Citations1997John N. Tsitsiklis, Benjamin Van Roy
It is proved that divergence may occur when updates are not based on trajectories of the Markov chain, which reconciles positive and negative results that have been discussed in the literature, regarding the soundness of temporal-difference learning.
Scientific Repository (Petra Christian University)PILCO: A Model-Based and Data-Efficient Approach to Policy Search
1,076 Citations2011Marc Peter Deisenroth, Carl Edward Rasmussen
PILCO reduces model bias, one of the key problems of model-based reinforcement learning, in a principled way by learning a probabilistic dynamics model and explicitly incorporating model uncertainty into long-term planning.
Elsevier eBooksResidual Algorithms: Reinforcement Learning with Function Approximation
953 Citations1995Leemon C. Baird
Both direct and residual gradient algorithms are shown to be special cases of residual algorithms, and it is shown that residual algorithms can combine the advantages of each approach.
Reinforcement Learning and Dynamic Programming Using Function Approximators
921 Citations2010Lucian Buşoniu, Robert Babuška +2 more
Reinforcement Learning and Dynamic Programming Using Function Approximators provides a comprehensive and unparalleled exploration of the field of RL and DP, with a focus on continuous-variable problems.
Neural NetworksReinforcement learning of motor skills with policy gradients
849 Citations2008Jan Peters, Stefan Schaal
This paper examines learning of complex motor skills with human-like limbs, and combines the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning with the theory of stochastic policy gradient learning.
Synthesis lectures on artificial intelligence and machine learningAlgorithms for Reinforcement Learning
749 Citations2010Csaba Szepesvári
This book focuses on those algorithms of reinforcement learning that build on the powerful theory of dynamic programming, and gives a fairly comprehensive catalog of learning problems, and describes the core ideas, followed by the discussion of their theoretical properties and limitations.
NeurocomputingNatural Actor-Critic
736 Citations2008Jan Peters, Stefan Schaal
This paper investigates a novel model-free reinforcement learning architecture, the Natural Actor-Critic, where the actor updates are based on stochastic policy gradients employing Amari's natural gradient approach, while the critic obtains both the natural policy gradient and additional parameters of a value function simultaneously by linear regression.
ScholarlyCommons (University of Pennsylvania)A Natural Policy Gradient
725 Citations2001Sham M. Kakade
This work provides a natural gradient method that represents the steepest descent direction based on the underlying structure of the parameter space and shows drastic performance improvements in simple MDPs and in the more challenging MDP of Tetris.
SIAM Journal on Control and OptimizationOnActor-Critic Algorithms
694 Citations2003Vijay R. Konda, John N. Tsitsiklis
This article proposes and analyzes a class of actor-critic algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic.
Machine LearningLinear Least-Squares algorithms for temporal difference learning
639 Citations1996Steven J. Bradtke, Andrew G. Barto
Two new temporal diffence algorithms based on the theory of linear least-squares function approximation, LS TD and RLS TD, are introduced and prove probability-one convergence when it is used with a function approximator linear in the adjustable parameters.
AutomaticaNatural actor–critic algorithms
565 Citations2009Shalabh Bhatnagar, Richard Sutton +2 more
Four new reinforcement learning algorithms based on actor-critic, natural-gradient and function-approximation ideas are presented, and the first convergence proofs and the first fully incremental algorithms are provided.
Proceedings of the AAAI Conference on Artificial IntelligenceRelative Entropy Policy Search
478 Citations2010Jan Peters, Katharina Mülling +1 more
The Relative Entropy Policy Search (REPS) method is suggested and can be shown to work well on typical reinforcement learning benchmark problems and yields an exact update step.
Elsevier eBooksStable Function Approximation in Dynamic Programming
455 Citations1995Geoffrey J. Gordon
A proof of convergence is provided for a wide class of temporal difference methods involving function approximators such as k-nearest-neighbor, and it is shown experimentally that these methods can be useful.
Machine LearningFeature-based methods for large scale dynamic programming
452 Citations1996John N. Tsitsiklis, Benjamin Van Roy
A methodological framework is developed and algorithms that employ two types of feature-based compact representations; that is, representations that involve feature extraction and a relatively simple approximation architecture are developed.
Machine LearningPolicy search for motor primitives in robotics
438 Citations2010Jens Kober, Jan Peters
Adaptive linear quadratic control using policy iteration
409 Citations2005Steven J. Bradtke, B. Erik Ydstie +1 more
The stability and convergence results for dynamic programming-based reinforcement learning applied to linear quadratic regulation (LQR) are presented and the specific algorithm is based on Q-learning and it is proven to converge to an optimal controller.
Journal of Artificial Intelligence ResearchInfinite-Horizon Policy-Gradient Estimation
333 Citations2001J. Baxter, P. L. Bartlett
GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies, is introduced.
INFORMS journal on computingReinforcement Learning: A Tutorial Survey and Recent Advances
320 Citations2008Abhijit Gosavi
This overview of reinforcement learning is aimed at uncovering the mathematical roots of this science so that readers gain a clear understanding of the core concepts and are able to use them in their own research.
Machine LearningTechnical Update: Least-Squares Temporal Difference Learning
297 Citations2002Justin A. Boyan
This paper updates Bradtke and Barto's work in three significant ways: first, it presents a simpler derivation of the LSTD algorithm; second, it generalizes from λ = 0 to arbitrary values of λ; at the extreme of κ, the resulting new algorithm is shown to be a practical, incremental formulation of supervised linear regression.
Neural NetworksA stochastic reinforcement learning algorithm for learning real-valued functions
294 Citations1990Vijaykumar Gullapalli
A stochastic reinforcement learning algorithm for learning functions with continuous outputs using a connectionist network that learns to perform an underconstrained positioning task using a simulated 3 degree-of-freedom robot arm.
Reinforcement Learning for Humanoid Robotics
286 Citations2003Jan Peters, Sethu Vijayakumar +1 more
This paper discusses different approaches of reinforcement learning in terms of their applicability in humanoid robotics, and demonstrates that ‘vanilla’ policy gradient methods can be significantly improved using the natural policy gradient instead of the regular policy gradient.
Gradient Descent for General Reinforcement Learning
239 Citations1998Leemon C. Baird, Andrew Moore
A simple learning rule is derived, the VAPS algorithm, which can be instantiated to generate a wide range of new reinforcement-learning algorithms, and allows policy-search and value-based algorithms to be combined, thus unifying two very different approaches to reinforcement learning into a single Value and Policy Search algorithm.
SIAM Journal on Control and OptimizationActor-Critic--Type Learning Algorithms for Markov Decision Processes
237 Citations1999Vijaymohan R. Konda, Vivek S. Borkar
Algorithms for learning the optimal policy of a Markov decision process (MDP) based on simulated transitions are formulated and analyzed, which are variants of the well-known "actor-critic" (or "adaptive critic") algorithm in the artificial intelligence literature.
Likelilood ratio gradient estimation
223 Citations1987Peter W. Glynn
Two applications settings are described, namely Monte Carlo optimization and statistical analysis of complex stochastic systems and how these methods apply to general discrete-event simulations is indicated.
Robotics and Autonomous SystemsBiped dynamic walking using reinforcement learning
196 Citations1997Hamid Benbrahim, Judy A. Franklin
This paper presents some results from a study of biped dynamic walking using reinforcement learning, where a hardware biped robot was built, a new reinforcement learning algorithm as well as a new learning architecture were developed.
An analysis of reinforcement learning with function approximation
193 Citations2008Francisco S. Melo, Sean Meyn +1 more
The convergence properties of several variations of Q-learning when combined with function approximation are analyzed, extending the analysis of TD-learning in (Tsitsiklis & Van Roy, 1996a) to stochastic control settings.
Information and ControlAn adaptive optimal controller for discrete-time Markov environments
182 Citations1977Ian H. Witten
It is shown that, under certain conditions, the adaptive controller's actions eventually become optimal for the particular control task with which it is faced, in the sense that they maximize the expected reward obtained in the future.
FigshareCovariant Policy Search
159 Citations2018J. Andrew Bagnell, Jeff Schneider
This work proposes a natural metric on controller parameterization that results from considering the manifold of probability distributions over paths induced by a stochastic controller that leads to a covariant gradient ascent rule.
Why natural gradient?
156 Citations2002S. Amari, S.C. Douglas
This paper outlines an alternative technique, termed natural gradient adaptation, that overcomes the poor convergence properties of gradient adaptation in many cases and is asymptotically Fisher-efficient for maximum likelihood estimation tasks.
Incremental Natural Actor-Critic Algorithms
154 Citations2007Shalabh Bhatnagar, Mohammad Ghavamzadeh +2 more
The results extend prior two-timescale convergence results for actor-critic methods by using temporal difference learning in the actor and by incorporating natural gradients, and they extend prior empirical studies of natural actor- Criterion methods by providing the first convergence proofs and the first fully incremental algorithms.
AutomaticaAverage cost temporal-difference learning
142 Citations1999John N. Tsitsiklis, Benjamin Van Roy
A variant of temporal-di ! erence learning that approximates average and di ! erential costs of an irreducible aperiodic Markov chain and provides a bound on the resulting approximation error that exhibits an interesting dependence on the ` mixing time a of the Markov chain.
Gradient Optimization and Nonlinear Control
138 Citations1976L. Hasdorff
The book represents an introduction to computation in control by an iterative, gradient, numerical method, where linearity is not assumed, and conjugate gradient descent is used.
IEEE Transactions on Neural NetworksContinuous-Time Adaptive Critics
124 Citations2007T. Hanselmann, Lyle Noakes +1 more
A continuous-time formulation of an adaptive critic design (ACD) is investigated, where backpropagation through time (BPTT) and real-time recurrent learning (RTRL) are prevalent and second-order actor adaptation using Newton's method is established for fast actor convergence for a general plant and critic.
Neural NetworksReinforcement learning for a biped robot based on a CPG-actor-critic method
117 Citations2007Yutaka Nakamura, Takeshi Mori +2 more
Computer simulations show that training of the CPG can be successfully performed by the proposed CPG-actor-critic method, thus allowing the biped robot to not only walk stably but also adapt to environmental changes.
IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)Impedance Learning for Robotic Contact Tasks Using Natural Actor-Critic Algorithm
103 Citations2009Byungchan Kim, Jooyoung Park +2 more
This paper considers a learning strategy of motor skill for robotic contact tasks based on a human motor control theory and machine learning schemes that optimizes the performance of the contact tasks in uncertain conditions of the environment.
Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228)Reinforcement learning of walking behavior for a four-legged robot
93 Citations2003Hajime Kimura, T. Yamashita +1 more
An action selection scheme for actor-critic algorithms, in which the actor selects a continuous action from its bounded action space by using the normal distribution, shows the robot successfully learns to walk in practical learning steps.
The MIT Press eBooksNatural Actor-Critic for Road Traffic Optimisation
92 Citations2007Silvia Richter, Douglas Aberdeen +1 more
A policy-gradient reinforcement learning approach is used to directly optimise the traffic signals, mapping currently deployed sensor observations to control signals and extending natural-actor critic approaches to work for distributed and online infinite-horizon problems.
Evaluation of Policy Gradient Methods and Variants on the Cart-Pole Benchmark
76 Citations2007Martin Riedmiller, Jan Peters +1 more
This paper evaluates different versions from the three main kinds of model-free policy gradient methods, i.e., finite difference gradients, 'vanilla' policy gradients and natural policy gradient methods, using the cart pole regulator benchmark.
Autonomous RobotsLearning model-free robot control by a Monte Carlo EM algorithm
72 Citations2009Nikos Vlassis, Marc Toussaint +2 more
A Monte Carlo EM algorithm (MCEM) for control learning that searches directly in the space of controller parameters using information obtained from randomly generated robot trajectories, related to, and generalizes, the PoWER algorithm of Kober and Peters.
Information SciencesA fuzzy Actor–Critic reinforcement learning network
67 Citations2007Xuesong Wang, Yuhu Cheng +1 more
A new fuzzy Actor-Critic reinforcement learning network (FACRLN) based on a fuzzy radial basis function (FRBF) neural network is proposed, which is able to adjust its structure and parameters in an adaptive way with a novel self-organizing approach according to the complexity of the task and the progress in learning.
Systems & Control LettersAn actor–critic algorithm with function approximation for discounted cost constrained Markov decision processes
67 Citations2010Shalabh Bhatnagar
This article develops the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints and proves the asymptotic almost sure convergence of the algorithm to a locally optimal policy.
IEEE Transactions on Fuzzy SystemsA convergent actor-critic-based frl algorithm with application to power management of wireless transmitters
64 Citations2003H.R. Berenji, David Vengerov
This work proves that a fuzzy rulebase actor satisfies the necessary conditions that guarantee the convergence of its parameters to a local optimum, and provides the first convergence proof for fuzzy reinforcement learning (FRL).
Non-parametric policy gradients
60 Citations2008Kristian Kersting, Kurt Driessens
A non-parametric policy gradient approach - called NPPG - that can deal with propositional, continuous, and relational domains in a unified way, and experimental results show that it can even improve on established results.
Finite-Sample Analysis of LSTD
49 Citations2010Alessandro Lazaric, Mohammad Ghavamzadeh +1 more
The problem of policy evaluation in reinforcement learning, i.e., learning the value function of a fixed policy, using the least-squares temporal-difference (LSTD) learning algorithm, is considered and a finite-sample analysis of LSTD is reported.
IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)A fuzzy reinforcement learning approach to power control in wireless transmitters
48 Citations2005David Vengerov, Nicholas Bambos +1 more
This work presents a new distributed fuzzy reinforcement learning algorithm (ACFRL-2) capable of adequately solving a class of problems to which the power control problem belongs, and shows that the algorithm converges almost deterministically to a neighborhood of optimal parameter values, as opposed to a very noisy stochastic convergence of earlier algorithms.
Systems & Control LettersA sensitivity formula for risk-sensitive cost and the actor–critic algorithm
46 Citations2001Vivek S. Borkar
The algorithm is based on a ‘sensitivity formula’ for the risk sensitive cost and is shown to converge with probability one to the desired solution of the finite Markov chains.
IEEE Transactions on Automatic ControlA Distributed Actor-Critic Algorithm and Applications to Mobile Sensor Network Coordination Problems
43 Citations2010Paris Pennesi, Ioannis Ch. Paschalidis
A distributed actor-critic method that orchestrates the coordination of multiple agents solving a general class of a Markov decision problem and uses a consensus-like algorithm for updating agents' policy parameters is introduced.
Optimality of Reinforcement Learning Algorithms with Linear Function Approximation
42 Citations2002Ralf Schoknecht
This paper shows that each of the solutions is optimal with respect to a specific objective function and characterises the different solutions as images of the optimal exact value function under different projection operations.
IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)Reinforcement Learning for Resource Allocation in LEO Satellite Networks
40 Citations2007Wipawee Usaha, Javier Barria
Two solution methods based on reinforcement learning (RL) are proposed in order to circumvent the computational burden of DP and can achieve up to 56% higher average revenue over existing routing methods used in LEO satellite networks with reasonable storage and computational requirements.
ePrints@IISc (Indian Institute of Science)Natural Actor - Critic Algorithms
32 Citations2009Shalabh Bhatnagar, Richard S. Sutton +2 more
Reinforcement learning applications in dynamic pricing of retail markets
29 Citations2003C.V.L. Raju, Y. Narahari +1 more
This paper considers a single seller market and a two seller market, and formulates the dynamic pricing problem in the RL framework in a setting that easily generalizes to markets with more than two sellers.
Real-time learning: a ball on a beam
28 Citations2003H. Benbrahim, J. Doleac +2 more
In the Real-Time Learning Laboratory at GTE Laboratories, machine learning algorithms are being implemented on hardware testbeds with modified connectionist actor-critic system applied to a ball balancing task.
Learning Control Under Extreme Uncertainty
25 Citations1992Vijaykumar Gullapalli
The results indicate that direct reinforcement learning can be used to learn a robust reactive control strategy that results in skillful peg-in-hole insertions.
Lecture notes in computer scienceAn RLS-Based Natural Actor-Critic Algorithm for Locomotion of a Two-Linked Robot Arm
21 Citations2005Jooyoung Park, Jong-Ho Kim +1 more
An actor-critic type algorithm utilizing the RLS(recursive least-squares) method, which is one of the most efficient techniques for adaptive signal processing, together with natural policy gradient, showed better performance than the conventional stochastic gradient ascent algorithm.
Neural ComputationDerivatives of Logarithmic Stationary Distributions for Policy Gradient Reinforcement Learning
19 Citations2009Tetsuro Morimura, Eiji Uchibe +3 more
A method for estimating the log stationary state distribution derivative (LSD) as a useful form of the derivative of the stationary state distributions through backward Markov chain formulation and a temporal difference learning framework is proposed.
Application of actor-critic learning to adaptive state space construction
17 Citations2005Yuhu Cheng, Jianqiang Yi +1 more
An adaptive control scheme based on normalized radial basis function under the structure of actor-critic and input layer and hidden layer of NRBF network are shared by the actor and critic to adopt reinforcement learning for complicated and continuous systems.
A Multi-agent Reinforcement Learning using Actor-Critic methods
14 Citations2008Chungui Li, Meng Wang +1 more
The value of agentpsilas temporal best-response strategy instead of the value of an equilibria is introduced, and the algorithm uses the linear programming to compute Q values when there are multi Nash equilibrium in the games.
Lecture notes in computer scienceFitted Natural Actor-Critic: A New Algorithm for Continuous State-Action MDPs
14 Citations2008Francisco S. Melo, Manuel Lopes
A new algorithm, fitted natural actor-critic(FNAC), is proposed that extends the work in [1] to allow for general function approximation and data reuse and combines the appealing features of both approaches while overcoming their main weaknesses.
Urban Traffic Signal Learning Control Using Fuzzy Actor-Critic Methods
13 Citations2009Chungui Li, Meng Wang +3 more
Lecture notes in computer scienceA New Natural Policy Gradient by Stationary Distribution Metric
13 Citations2008Tetsuro Morimura, Eiji Uchibe +2 more
A Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, is proposed and derived, and a new NPG named "Natural State-action Gradient"(NSG) is derived).
Neural Information Processing SystemsA Generalized Natural Actor-Critic Algorithm
13 Citations2009Tetsuro Morimura, Eiji Uchibe +2 more
A generalized Natural Gradient that linearly interpolates the two FIMs is described and an efficient implementation for the gNG learning based on a theory of the estimating function, the generalized Natural Actor-Critic (gNAC) algorithm is proposed.
An actor-critic method using Least Squares Temporal Difference learning
12 Citations2009Ioannis Ch. Paschalidis, Keyong Li +1 more
This paper uses a Least Squares Temporal Difference algorithm in an actor-critic framework where the actor and the critic operate concurrently, and proves the convergence of the process.
Importance sampling actor-critic algorithms
8 Citations2006Jason Williams, John W. Fisher +1 more
IS can be used with temporal difference methods to estimate a cost function parameter for one policy using the entire history of system interactions incorporating many different policies, and the resulting algorithm is applied to improving gradient estimates in a policy gradient optimization.
Lecture notes in computer scienceBasis Expansion in Natural Actor Critic Methods
7 Citations2008Sertan Girgin, Pierre‐Marie Preux
This paper proposes a new approach which uses cascade-correlation learning architecture for automatically constructing a set of basis functions within the context of Natural Actor-Critic algorithms, which allow more complex policies to be represented, and consequently improve the performance of the resulting policies.
Two steps natural actor critic learning for underwater cable tracking
7 Citations2010Andrés El-Fakdi, Marc Carreras +1 more
The natural actor-critic (NAC) algorithm has been selected to solve the action selection problem of an autonomous robot in a cable tracking task and aims to take advantage of policy gradient and value function techniques for fast convergence.
FigsharePolicy Search in Kernel Hilbert Space
7 Citations2018J. Andrew Bagnell, Jeff Schneider
This work shows how policy search (with or without the additional guidance of value-functions) in a Reproducing Kernel Hilbert Space gives a simple and rigorous extension of the technique to non-parametric settings and investigates a new class of algorithms which generalize REINFORCE-style likelihood ratio methods to yield both online and batch techniques that perform gradient search in a function space of policies.
Transactions of the Institute of Systems Control and Information EngineersStochastic Optimization
6 Citations2023Bruno Hideki Fukushima-Kimura, Yoshinori Kamijima +2 more
It is observed that the SCA performed better than the Glauber dynamics in some special cases, while the $\varepsilon$-SCA showed the highest performance in all scenarios.
Natural gradient actor-critic algorithms using random rectangular coarse coding
4 Citations2008Hajime Kimura
A quantitative analysis of the random coarse coding comparing with regular-grid approaches is shown, and a new approach is presented that combines the natural gradient actor-critic with the random rectangular coarse coding.
…