Model-Free Imitation Learning with Policy Optimization
arXiv (Cornell University)Published 26 May 2016Open access
Jonathan Ho, Jayesh K. Gupta, Stefano Ermon
Citations41
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Under the apprenticeship learning formalism, this work develops alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert.
Abstract
In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. Such algorithms are therefore not directly applicable to large, high-dimensional environments, and their performance can significantly degrade if the planning problems are not solved to optimality. Under the apprenticeship learning formalism, we develop alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert. Our approach, based on policy gradients, scales to large continuous environments with guaranteed convergence to local minima.
Keywords
Computer ScienceDecision Sciences
Policy Gradient Methods for Reinforcement Learning with Function Approximation
4,982 Citations1999Richard S. Sutton, David McAllester +2 more
This paper proves for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.
Apprenticeship learning via inverse reinforcement learning
2,875 Citations2004Pieter Abbeel, Andrew Y. Ng
This work thinks of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and gives an algorithm for learning the task demonstrated by the expert, based on using "inverse reinforcement learning" to try to recover the unknown reward function.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)Maximum Entropy Inverse Reinforcement Learning
2,061 Citations2018Brian D. Ziebart, Andrew L. Maas +2 more
A probabilistic approach based on the principle of maximum entropy that provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods is developed.
American Journal of Veterinary ResearchPharmacokinetics of a novel formulation of ivermectin after administration to goats
1,782 Citations2006Aranzazu González, Ana M. Sahagún +4 more
The commercial formulation used in this study is a good option to consider when administering ivermectin to goats because of the high absorption, which is characterized by high values of F.
arXiv (Cornell University)A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
1,313 Citations2010Stéphane Ross, Geoffrey J. Gordon +1 more
Neural NetworksReinforcement learning of motor skills with policy gradients
849 Citations2008Jan Peters, Stefan Schaal
This paper examines learning of complex motor skills with human-like limbs, and combines the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning with the theory of stochastic policy gradient learning.
Maximum margin planning
630 Citations2006Nathan Ratliff, J. Andrew Bagnell +1 more
This work learns mappings from features to cost so an optimal policy in an MDP with these cost mimics the expert's behavior, and demonstrates a simple, provably efficient approach to structured maximum margin learning, based on the subgradient method, that leverages existing fast algorithms for inference.
Approximately Optimal Approximate Reinforcement Learning
590 Citations2002Sham M. Kakade, John Langford
Machine LearningSearch-based structured prediction
436 Citations2009Hal Daumé, John Langford +1 more
Searn is an algorithm for integrating search and learning to solve complex structured prediction problems such as those that occur in natural language, speech, computational biology, and vision and comes with a strong, natural theoretical guarantee: good performance on the derived classification problems implies goodperformance on the structured prediction problem.
Nonlinear Inverse Reinforcement Learning with Gaussian Processes
262 Citations2011Sergey Levine, Zoran Popović +1 more
A probabilistic algorithm that allows complex behaviors to be captured from suboptimal stochastic demonstrations, while automatically balancing the simplicity of the learned reward structure against its consistency with the observed actions.
A Game-Theoretic Approach to Apprenticeship Learning
226 Citations2007Syed Umar, Robert E. Schapire
A new algorithm is given that is computationally faster, is easier to implement, and can be applied even in the absence of an expert, and it is shown that this algorithm may produce a policy that is substantially better than the expert's.
Autonomous RobotsLearning to search: Functional gradient techniques for imitation learning
206 Citations2009Nathan Ratliff, David Silver +1 more
The work presented extends the Maximum Margin Planning (MMP) framework to admit learning of more powerful, non-linear cost functions, and demonstrates practical real-world performance with three applied case-studies including legged locomotion, grasp planning, and autonomous outdoor unstructured navigation.
Apprenticeship learning using linear programming
191 Citations2008Syed Umar, Michael Bowling +1 more
This work shows how to frame apprenticeship learning as a linear programming problem, and shows that using an off-the-shelf LP solver to solve this problem results in a substantial improvement in running time over existing methods---up to two orders of magnitude faster in the authors' experiments.
Neural Information Processing SystemsLearning from Limited Demonstrations
72 Citations2013Beomjoon Kim, Amir massoud Farahmand +2 more
This work proves an upper bound on the Bellman error of the estimate computed by APID at each iteration, and shows empirically that APID outperforms pure Approximate Policy Iteration, a state-of-the-art LfD algorithm, and supervised learning in a variety of scenarios, including when very few and/or suboptimal demonstrations are available.
Machine LearningTraining parsers by inverse reinforcement learning
69 Citations2009Gergely Neu, Csaba Szepesvári
This paper presents a number of recent incremental IRL algorithms in a unified framework and map them to parser training algorithms, allowing them to recover some existing parsing algorithms, as well as to obtain a new one.
FigshareAn Invitation to Imitation
39 Citations2018J. Andrew Bagnell
This work represents a summary from a very personal perspective of research on computationally effective methods for learning to imitate behavior to serve two audiences: to engage machine learning experts in the challenges of imitation learning and the interesting theoretical and practical distinctions with more familiar frameworks like statistical supervised learning theory.
Proceedings of the AAAI Conference on Artificial IntelligenceLearning Large-Scale Dynamic Discrete Choice Models of Spatio-Temporal Preferences with Application to Migratory Pastoralism in East Africa
29 Citations2015Stefano Ermon, Yexiang Xue +9 more
The problem of inferring agents' preferences from observed movement trajectories is considered as an Inverse Reinforcement Learning (IRL) problem and a probabilistic approach is taken and generative models that can be used to simulate behavior under new circumstances such as changes in resource availability, access policies, or climate are considered.