A comparison of direct and model-based reinforcement learning
Published 22 November 2002
Christopher G. Atkeson, J.C. Santamaría
Citations242
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper compares direct reinforcement learning (no explicit model) and model-based reinforcement learning on a simple task: pendulum swing up and finds that in this task model- based approaches support reinforcement learning from smaller amounts of training data and efficient handling of changing goals.
Abstract
This paper compares direct reinforcement learning (no explicit model) and model-based reinforcement learning on a simple task: pendulum swing up. We find that in this task model-based approaches support reinforcement learning from smaller amounts of training data and efficient handling of changing goals.
Keywords
Computer Science
Journal of Artificial Intelligence ResearchReinforcement Learning: A Survey
8,831 Citations1996Leslie Pack Kaelbling, Michael L. Littman +1 more
Central issues of reinforcement learning are discussed, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state.
Machine LearningSimple statistical gradient-following algorithms for connectionist reinforcement learning
7,386 Citations1992Ronald J. Williams
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units that are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reInforcement tasks, and they do this without explicitly computing gradient estimates.
Artificial Intelligence ReviewLocally Weighted Learning
1,684 Citations1997Christopher G. Atkeson, Andrew Moore +1 more
The survey discusses distance functions, smoothing parameters, weighting functions, local model structures, regularization of the estimates and bias, assessing predictions, handling noisy data and outliers, improving the quality of predictions by tuning fit parameters, and applications of locally weighted learning.
Communications of the ACMTemporal difference learning and TD-Gammon
1,485 Citations1995Gerald Tesauro
The domain of complex board games such as Go, chess, checkers, Othello, and backgammon has been widely regarded as an ideal testing ground for exploring a variety of concepts and approaches in artificial intelligence and machine learning.
Elsevier eBooksIntegrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming
1,351 Citations1990Richard S. Sutton
Results are shown for a simple Dyna-PI system that simultaneously learns by trial and error, learns a world model, and plans optimal routes using the evolving world model and it is shown that Dyna-Q architectures are easy to adapt for use in changing environments.
Artificial IntelligenceLearning to act using real-time dynamic programming
1,109 Citations1995Andrew G. Barto, Steven J. Bradtke +1 more
An algorithm based on dynamic programming, which is called Real-Time DP, is introduced, by which an embedded system can improve its performance with experience and illuminate aspects of other DP-based reinforcement learning methods such as Watkins'' Q-Learning algorithm.
Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
1,065 Citations1995Richard S. Sutton
It is concluded that reinforcement learning can work robustly in conjunction with function approximators and that there is little justi cation at present for avoiding the case of general Reinforcement Learning and Function Approximation.
ACM SIGART BulletinDyna, an integrated architecture for learning, planning, and reacting
767 Citations1991Richard S. Sutton
Dyna is an AI architecture that integrates learning, planning, and reactive execution that relies on machine learning methods for learning from examples, yet is not tied to any particular method.
Machine LearningPrioritized Sweeping: Reinforcement Learning with Less Data and Less Time
710 Citations1993Andrew Moore, Christopher G. Atkeson
This work presents a new algorithm, prioritized sweeping, for efficient prediction and control of stochastic Markov systems, which successfully solves large state-space real-time problems with which other methods have difficulty.
Artificial Intelligence ReviewLocally Weighted Learning for Control
538 Citations1997Christopher G. Atkeson, Andrew Moore +1 more
There are ways in which locally weighted learning, a type of lazy learning, has been applied by us to control tasks, and various forms that control tasks can take, are explained.
IEEE Control SystemsReinforcement learning is direct adaptive optimal control
513 Citations1992Richard S. Sutton, Andrew G. Barto +1 more
Reinforcement learning methods are presented as a computationally simple, direct approach to the adaptive optimal control of nonlinear systems.
Adaptive linear quadratic control using policy iteration
409 Citations2005Steven J. Bradtke, B. Erik Ydstie +1 more
The stability and convergence results for dynamic programming-based reinforcement learning applied to linear quadratic regulation (LQR) are presented and the specific algorithm is based on Q-learning and it is proven to converge to an optimal controller.
Machine LearningPrioritized sweeping: Reinforcement learning with less data and less time
387 Citations1993Andrew Moore, Christopher G. Atkeson
This work presents a new algorithm, prioritized sweeping, for efficient prediction and control of stochastic Markov systems, which successfully solves large state-space real-time problems with which other methods have difficulty.
A reinforcement learning approach to job-shop scheduling
378 Citations1995Wei Zhang, Thomas G. Dietterich
Reinforcement learning methods are applied to learn domain-specific heuristics for job shop scheduling to suggest that reinforcement learning can provide a new method for constructing high-performance scheduling systems.
Neural NetworksA stochastic reinforcement learning algorithm for learning real-valued functions
294 Citations1990Vijaykumar Gullapalli
A stochastic reinforcement learning algorithm for learning functions with continuous outputs using a connectionist network that learns to perform an underconstrained positioning task using a simulated 3 degree-of-freedom robot arm.
Machine LearningThe Parti-game Algorithm for Variable Resolution Reinforcement Learning in Multidimensional State-spaces
262 Citations1995Andrew Moore, Christopher G. Atkeson
Parti-game is a new algorithm for learning feasible trajectories to goal regions in high dimensional continuous state-spaces and applies techniques from game-theory and computational geometry to efficiently and adaptively concentrate high resolution only on critical areas.
RePEc: Research Papers in EconomicsHandbook of Computational Economics
253 Citations1996Hans M. Amman, John Rust +1 more
This paper presents a meta-modelling framework for general equilibrium modelling for policy analysis and forecasting of dynamic linear economies and some of the methods used in this framework came from previous work on dynamic dynamic programming in economics.
Elsevier eBooksPlanning by Incremental Dynamic Programming
145 Citations1991Richard S. Sutton
The basic results and ideas of dynamic programming as they relate most directly to the concerns of planning in AI are presented, which form the theoretical basis for the incremental planning methods used in the integrated architecture Dyna.
Machine LearningThe parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces
138 Citations1995Andrew Moore, Christopher G. Atkeson
Parti-game is a new algorithm for learning feasible trajectories to goal regions in high dimensional continuous state-spaces and applies techniques from game-theory and computational geometry to efficiently and adaptively concentrate high resolution only on critical areas.
Learning tasks from a single demonstration
129 Citations2002Christopher G. Atkeson, Stefan Schaal
This paper explores an approach to learning from demonstration based on learning an optimization criterion from the demonstration and a task model from repeated attempts to perform the task, and using the learned criterion and model to compute an appropriate robot movement.
Reinforcement Learning Applied to Linear Quadratic Regulation
117 Citations1992Steven J. Bradtke
An algorithm based on Q-learning is described that is proven to converge to the optimal controller for a large class of LQR problems, an important class of control problems involving continuous state and action spaces and requiring a simple type of non-linear function approximator.
Reinforcement Learning With High-Dimensional, Continuous Actions
89 Citations1993III. Baird, Klopf Leemon C. +1 more
neural information processing systemsLearning to Control an Unstable System with Forward Modeling
89 Citations1989Michael I. Jordan, Robert A. Jacobs
This work extends previous work by considering how this methodology can be applied to the optimization of quantities that are distal not only in space but also in time.
Minimum-time control of the Acrobot
83 Citations2002Gary Boone
A direct search algorithm for finding swingup trajectories for the Acrobot is described, which uses a lookahead search that maximizes theAcrobot's total energy in an N-step window.
Using Local Trajectory Optimizers to Speed Up Global Optimization in Dynamic Programming
81 Citations1993Christopher G. Atkeson
This work uses second order local trajectory optimization to generate locally optimal plans and local models of the value function and its derivatives, and maintains global consistency of the local Models of thevalue function, guaranteeing that the locally optimal Plans are actually globally optimal.
Efficient reinforcement learning
74 Citations1994Claude-Nicolas Fiechter
A new formal model for studying reinforcement learning, based on Valiant's PAC framework, that requires the learner to produce a policy whose expected value from the initial state is ε-close to that of the optimal policy, with probability no less than 1−δ.
Efficient reinforcement learning: model-based Acrobot control
51 Citations2002Gary Boone
This paper explores the planning speed and data efficiency of explicitly learning models, as well as using heuristic knowledge to aid the search for solutions and reduce the amount of data required from the real robot.
DSpace@MIT (Massachusetts Institute of Technology)Associative reinforcement learning for optimal control
25 Citations1991Peter Millington
Three connectionist implementations of dynamic programming for optimal control: a preliminary comparative analysis
5 Citations2002Hugues Bersini, Vittorio Gorrini
Three optimal control methodologies all relying on neural network for their universal approximation capabilities and on dynamic programming for substituting the time-integral optimization by a succession of time-local optimizations are presented in this paper and applied on the same elementary rendezvous problem.
To swing up an inverted Pendulum using stochastic real-valued Reinforcement Learning
2 Citations1994Anette Standfuss, R. Eckmiller
This paper deals with the problem of learning to swing up an inverted pendulum, which belongs to the class of highly nonlinear, non-minimum phase control problems without a general control methodology.