PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Scientific Repository (Petra Christian University)Published 28 June 2011Open access
Marc Peter Deisenroth, Carl Edward Rasmussen
Citations1,076
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
PILCO reduces model bias, one of the key problems of model-based reinforcement learning, in a principled way by learning a probabilistic dynamics model and explicitly incorporating model uncertainty into long-term planning.
Abstract
In this paper, we introduce pilco, a practical, data-efficient model-based policy search method. Pilco reduces model bias, one of the key problems of model-based reinforcement learning, in a principled way. By learning a probabilistic dynamics model and explicitly incorporating model uncertainty into long-term planning, pilco can cope with very little data and facilitates learning from scratch in only a few trials. Policy evaluation is performed in closed form using state-ofthe-art approximate inference. Furthermore, policy gradients are computed analytically for policy improvement. We report unprecedented learning efficiency on challenging and high-dimensional control tasks. 1. Introduction and Related
Keywords
Computer SciencePhysics and Astronomy
Journal of Electronic ImagingPattern Recognition and Machine Learning
21,976 Citations2007Christopher Bishop
Probability Distributions, linear models for Regression, Linear Models for Classification, Neural Networks, Graphical Models, Mixture Models and EM, Sampling Methods, Continuous Latent Variables, Sequential Data are studied.
The MIT Press eBooksGaussian Processes for Machine Learning
10,409 Citations2005Carl Edward Rasmussen, Christopher K. I. Williams
The book provides a long-needed, systematic and unified treatment of theoretical and practical aspects of GPs in machine learning, targeted at researchers and students in machine learning and applied statistics.
Neural ComputationReinforcement Learning in Continuous Time and Space
989 Citations2000Kenji Doya
A reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action based on the Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems is presented and algorithms for estimating value functions and improving policies with the use of function approximators are derived.
Lecture notes in computer scienceNeural Fitted Q Iteration – First Experiences with a Data Efficient Neural Reinforcement Learning Method
758 Citations2005Martin Riedmiller
NFQ, an algorithm for efficient and effective training of a Q-value function represented by a multi-layer perceptron, is introduced and it is shown empirically, that reasonably few interactions with the plant are needed to generate control policies of high quality.
Policy Gradient Methods for Robotics
511 Citations2006Jan Peters, Stefan Schaal
An overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field is given and how the most recently developed methods can significantly improve learning performance is shown.
Autonomous helicopter control using reinforcement learning policy search methods
277 Citations2002J. Andrew Bagnell, Jeff Schneider
This work considers algorithms that evaluate and synthesize controllers under distributions of Markovian models and demonstrates the presented learning control algorithm by flying an autonomous helicopter and shows that the controller learned is robust and delivers good performance in this real-world domain.
A comparison of direct and model-based reinforcement learning
242 Citations2002Christopher G. Atkeson, J.C. Santamaría
This paper compares direct reinforcement learning (no explicit model) and model-based reinforcement learning on a simple task: pendulum swing up and finds that in this task model- based approaches support reinforcement learning from smaller amounts of training data and efficient handling of changing goals.
Using inaccurate models in reinforcement learning
216 Citations2006Pieter Abbeel, Morgan Quigley +1 more
This paper presents a hybrid algorithm that requires only an approximate model, and only a small number of real-life trials, and achieves near-optimal performance in the real system, even when the model is only approximate.
Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
216 Citations2011Marc Peter Deisenroth, Carl Edward Rasmussen +1 more
It is demonstrated how a low-cost off-the-shelf robotic system can learn closed-loop policies for a stacking task in only a handful of trials-from scratch.
Gaussian Processes in Reinforcement Learning
215 Citations2003Malte Kuß, Carl Edward Rasmussen
It is speculated that the intrinsic ability of GP models to characterise distributions of functions would allow the method to capture entire distributions over future values instead of merely their expectation, which has traditionally been the focus of much of reinforcement learning.
NeurocomputingGaussian process dynamic programming
202 Citations2009Marc Peter Deisenroth, Carl Edward Rasmussen +1 more
This article introduces Gaussian process dynamic programming (GPDP), an approximate value function-based RL algorithm, and proposes to learn probabilistic models of the a priori unknown transition dynamics and the value functions on the fly.
Proceedings - IEEE International Conference on Robotics and Automation/ProceedingsGaussian Processes and Reinforcement Learning for Identification and Control of an Autonomous Blimp
198 Citations2007Jonathan Ko, Daniel J. Klein +2 more
This paper shows how the GP-enhanced model can be used in conjunction with reinforcement learning to generate a blimp controller that is superior to those learned with ODE or GP models alone.
Bayes meets bellman: the Gaussian process approach to temporal difference learning
171 Citations2003Yaakov Engel, Shie Mannor +1 more
A novel Bayesian approach to the problem of value function estimation in continuous state spaces by imposing a Gaussian prior over value functions and assuming aGaussian noise model is presented.
Repository KITopen (Karlsruhe Institute of Technology)Efficient Reinforcement Learning using Gaussian Processes
170 Citations2010Marc Peter Deisenroth
First, PILCO, a fully Bayesian approach for efficient RL in continuous-valued state and action spaces when no expert knowledge is available is introduced, and principled algorithms for robust filtering and smoothing in GP dynamic systems are proposed.
Energy and passivity based control of the double inverted pendulum on a cart
151 Citations2002Wei Zhong, Helmut Röck
2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03).Propagation of uncertainty in Bayesian kernel models - application to multiple-step ahead forecasting
133 Citations2003Joaquin Quiñonero Candela, Agathe Girard +2 more
This work derives novel analytic expressions for the predictive mean and variance for Gaussian kernel shapes under the assumption of a Gaussian input distribution in the static case, and of a recursive Gaussian predictive density in iterative forecasting.
HAL (Le Centre pour la Communication Scientifique Directe)Reinforcement Learning Using Neural Networks, with Applications to Motor Control
84 Citations2002Rémi Coulom
The continuous TD(lambda) algorithm is refined to handle situations with discontinuous states and controls, and the vario-eta algorithm is proposed as a simple but efficient method to perform gradient descent.
Exploiting Model Uncertainty Estimates for Safe Dynamic Control Learning
78 Citations1996Jeff Schneider
This paper addresses the case where the system must be prevented from having catastrophic failures during learning, and proposes a new algorithm adapted from the dual control literature and use Bayesian locally weighted regression models with dynamic programming.
AutomaticaDual adaptive control of nonlinear stochastic systems using neural networks
44 Citations1998Simon G. Fabri, Visakan Kadirkamanathan
A suboptimal dual adaptive system is developed for control of stochastic, nonlinear, discrete time plants that are affine in the control input and a control law is developed that takes into consideration the uncertainty of the parameter e stimates, thereby eliminating the need to perform prior open-loop plant identification.
Dynamics and ControlNonlinear Modeling and Control of a Unicycle
44 Citations1999Yoav Naveh, P. Bar‐Yoseph +1 more
NeurocomputingVariational Bayesian learning of nonlinear hidden state-space models for model predictive control
34 Citations2009Tapani Raiko, Matti Tornio
The simulations indicate that the proposed stochastic approach to identification and model predictive control in nonlinear hidden state-space models is able to find a representation of the system state that makes control easier especially under high noise.
Model-free off-policy reinforcement learning in continuous environment
13 Citations2005Paweł Wawrzyński, Andrzej Pacut
An algorithm of reinforcement learning in continuous state and action spaces that utilizes the entire history of agent-environment interaction to construct a control policy that is several times shorter than the one required by other algorithms.
Lecture notes in computer scienceIncorporating Domain Models into Bayesian Optimization for RL
11 Citations2010Aaron Wilson, Alan Fern +1 more
This paper uses the observed data to learn approximate transitions models that allow for Monte-Carlo predictions of policy returns that are incorporated into the Bayesian Optimization framework as a type of prior on policy returns, which can better inform the BO process.