Learning CPG sensory feedback with policy gradient for biped locomotion for a full-body humanoid
National Conference on Artificial IntelligencePublished 9 July 2005
Gen Endo, Jun Morimoto, Takamitsu Matsubara, Jun Nakanishi, Gordon Cheng
Citations34
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is demonstrated that an appropriate feed-back controller can be acquired within a thousand trials by numerical simulations and the obtained controller in numerical simulation achieves stable walking with a physical robot in the real world.
Abstract
This paper describes a learning framework for a central pattern generator based biped locomotion controller using a policy gradient method. Our goals in this study are to achieve biped walking with a 3D hardware humanoid, and to develop an efficient learning algorithm with CPG by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feed-back controller can be acquired within a thousand trials by numerical simulations and the obtained controller in numerical simulation achieves stable walking with a physical robot in the real world. Numerical simulations and hardware experiments evaluated walking velocity and stability. Furthermore, we present the possibility of an additional online learning using a hardware robot to improve the controller within 200 iterations.
Keywords
Biochemistry, Genetics and Molecular BiologyEngineering
Policy Gradient Methods for Reinforcement Learning with Function Approximation
4,982 Citations1999Richard S. Sutton, David McAllester +2 more
This paper proves for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.
Artificial IntelligencePlanning and acting in partially observable stochastic domains
4,129 Citations1998Leslie Pack Kaelbling, Michael L. Littman +1 more
A novel algorithm for solving pomdps off line and how, in some cases, a finite-memory controller can be extracted from the solution to a POMDP is outlined.
Neural ComputationReinforcement Learning in Continuous Time and Space
989 Citations2000Kenji Doya
A reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action based on the Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems is presented and algorithms for estimating value functions and improving policies with the use of function approximators are derived.
Biological CyberneticsSustained oscillations generated by mutually inhibiting neurons with adaptation
747 Citations1985Kiyotoshi Matsuoka
This paper mathematically discusses sustained oscillations generated by mutual inhibition of the neurons which are represented by a continuous-variable model with a kind of fatigue or adaptation effect and suggests that the adaptation of the neuron plays a very important role for the appearance of the oscillations.
SIAM Journal on Control and OptimizationOnActor-Critic Algorithms
694 Citations2003Vijay R. Konda, John N. Tsitsiklis
This article proposes and analyzes a class of actor-critic algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic.
Biological CyberneticsA model of the neuro-musculo-skeletal system for human locomotion
560 Citations1995Gentaro Taga
Using a computer simulation, it is found that locomotion emerged as a stable limit cycle that was generated by the global entrainment between the musculo-skeletal system, the neural system, and the environment.
Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems
353 Citations1994Tommi Jaakkola, Satinder Singh +1 more
This work proposes and analyze a new learning algorithm to solve a certain class of non-Markov decision problems and operates in the space of stochastic policies, a space which can yield a policy that performs considerably better than any deterministic policy.
Reinforcement Learning for Humanoid Robotics
286 Citations2003Jan Peters, Sethu Vijayakumar +1 more
This paper discusses different approaches of reinforcement learning in terms of their applicability in humanoid robotics, and demonstrates that ‘vanilla’ policy gradient methods can be significantly improved using the natural policy gradient instead of the regular policy gradient.
Stochastic policy gradient reinforcement learning on a simple 3D biped
270 Citations2005Russ Tedrake, Tianbao Zhang +1 more
This work presents a learning system which is able to quickly and reliably acquire a robust feedback control policy for 3D dynamic walking from a blank-slate using only trials implemented on the physical robot.
Robotics and Autonomous SystemsBiped dynamic walking using reinforcement learning
196 Citations1997Hamid Benbrahim, Judy A. Franklin
This paper presents some results from a study of biped dynamic walking using reinforcement learning, where a hardware biped robot was built, a new reinforcement learning algorithm as well as a new learning architecture were developed.
Experimental Studies of a Neural Oscillator for Biped Locomotion with QRIO
124 Citations2006Gen Endo, Jun Nakanishi +2 more
This paper presents the achievement of 3D biped locomotion using a neural oscillator applied to a humanoid robot, QRIO, employing reduced number of neural oscillators as the CPG model, along with a task space Cartesian coordinate system and utilizing entrainment property to establish stable walking gait.
Mechanical system of a small biped entertainment robot
81 Citations2004Toru Ishida, Y. Kuroki +1 more
This paper discusses more about mechanical system, which is important and original for a small biped entertainment robot, which will be used, in home environment, and proposes the ideas against falling-over which makes the robot as safe as possible.
National Conference on Artificial IntelligenceReinforcement learning for a CPG-driven biped robot
66 Citations2004Takeshi Mori, Yutaka Nakamura +2 more
This study proposes a learning scheme for a CPG controller called a C PG-actor-critic model, whose learning algorithm is based on a policy gradient method, and applies this method to autonomous acquisition of biped locomotion by a biped robot simulator.
Learning Sensory Feedback to CPG with Policy Gradient for Biped Locomotion
30 Citations2006Takamitsu Matsubara, Jun Morimoto +3 more
The goal in this study is to develop an efficient learning algorithm by reducing the dimensionality of the state space used for learning by using a policy gradient method for a CPG-based biped locomotion controller.
The MIT Press eBooksBiologically Inspired Adaptive Dynamic Walking of a Quadruped Robot
25 Citations2004Hiroshi Kimura, Yasuhiro Fukuoka +1 more
Development of sensor system of a small biped entertainment robot
21 Citations2004Toru Ishida, Y. Kuroki
The sensor system of this robot is reported which is important and essential for a small biped entertainment robot which can be used in the home environment and the safety for humans is considered.