New inference strategies for solving Markov Decision Processes using reversible jump MCMC
arXiv (Cornell University)Published 9 May 2012Open access
Matthias Hoffman, Hendrik Kueck, Nando de Freitas, Arnaud Doucet
Citations8
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper builds on previous work which uses inferences techniques, in particular Markov Chain Monte Carlo methods, to solve parameterized control problems and introduces a new target distribution which is able to incorporate more reward information from sampled trajectories.
Abstract
In this paper we build on previous work which uses inferences techniques, in particular Markov Chain Monte Carlo (MCMC) methods, to solve parameterized control problems. We propose a number of modifications in order to make this approach more practical in general, higher-dimensional spaces. We first introduce a new target distribution which is able to incorporate more reward information from sampled trajectories. We also show how to break strong correlations between the policy parameters and sampled trajectories in order to sample more freely. Finally, we show how to incorporate these techniques in a principled manner to obtain estimates of the optimal policy.
Keywords
Computer ScienceMathematics
BiometrikaReversible jump Markov chain Monte Carlo computation and Bayesian model determination
5,929 Citations1995Peter J. Green
A new framework for the construction of reversible Markov chain samplers that jump between parameter subspaces of differing dimensionality is proposed, which is flexible and entirely constructive, and should have wide applicability in model determination problems.
IEEE Transactions on Pattern Analysis and Machine IntelligenceMean shift, mode seeking, and clustering
3,823 Citations1995Yizong Cheng
Mean shift, a simple interactive procedure that shifts each data point to the average of data points in its neighborhood is generalized and analyzed and makes some k-means like clustering algorithms its special cases.
Probabilistic inference for solving discrete and continuous state Markov Decision Processes
502 Citations2006Marc Toussaint, Amos Storkey
An Expectation Maximization algorithm for computing optimal policies that actually optimizes the discounted expected future return for arbitrary reward functions and without assuming an ad hoc finite total time is presented.
arXiv (Cornell University)PEGASUS: A Policy Search Method for Large MDPs and POMDPs
369 Citations2013Andrew Y. Ng, Michael I. Jordan
This work proposes a new approach to the problem of searching a space of policies for a Markov decision process (MDP) or a partially observable Markov decisions process (POMDP), given a model, based on the following observation: Any (PO)MDP can be transformed into an "equivalent" POMDP in which all state transitions are deterministic.
Springer tracts in advanced roboticsPolicy Search for Motor Primitives in Robotics
231 Citations2013Jens Kober, Jan Peters
Handbook of statisticsSimulation Based Optimal Design
189 Citations2005Peter Müller
Simulation based approaches to optimal design is reviewed, with an emphasis on problems that cast optimal design as formal decision problems, and some examples of important alternative, nondecision theoretic approaches are shown.
Planning by Probabilistic Inference.
184 Citations2003Hagai Attias
A new approach is presented to the problem of planning under uncertainty in a probabilistic generative model involving actions and states, and the toolbox of inference techniques are brought to bear on the planning problem.
Statistics and ComputingMarginal maximum a posteriori estimation using Markov chain Monte Carlo
105 Citations2002Randal Douc, Simon Godsill +1 more
A simple and novel MCMC strategy, called State-Augmentation for Marginal Estimation (SAME), which leads to MMAP estimates for Bayesian models and illustrates the simplicity and utility of the approach for missing data interpolation in autoregressive time series and blind deconvolution of impulsive processes.
Lancaster EPrints (Lancaster University)Non-centred parameterisations for hierarchical models and data augmentation.
88 Citations2003Gareth O. Roberts, Omiros Papaspiliopoulos +1 more
This paper will investigate the construction of non-centered methods by the use of state space expansion techniques, and will introduce methods for devising partially non- centered parameterisations, many of which are data-dependent.
MPG.PuRe (Max Planck Society)Probabilistic inference for solving (PO) MDPs
83 Citations2006Marc Toussaint, Stefan Harmeling +1 more
The approach is based on an equivalence between maximization of the expected future return in the time-unlimited MDP and likelihood maximization in a related mixture of finite-time MDPs, which allows to use expectation maximization (EM) for computing optimal policies, using arbitrary inference techniques in the E-step.
arXiv (Cornell University)Hierarchical POMDP Controller Optimization by Likelihood Maximization
75 Citations2012Marc Toussaint, Laurent Charlin +1 more
Applied Mathematics and ComputationOn solving integral equations using Markov chain Monte Carlo methods
46 Citations2010Randal Douc, Adam M. Johansen +1 more
An original approach to the solution of Fredholm equations of the second kind is proposed, which interprets the standard Von Neumann expansion of the solution as an expectation with respect to a probability distribution defined on a union of subspaces of variable dimension.
TUbilio (Technical University of Darmstadt)An Expectation Maximization Algorithm for Continuous Markov Decision Processes with Arbitrary Reward
30 Citations2009Matthew D. Hoffman, Nando de Freitas +2 more
A new expectation maximization algorithm is derived for policy optimization in linear Gaussian Markov decision processes, where the reward function is parameterized in terms of a flexible mixture of Gaussians, which is more flexible and general than closed-form solutions.
Planning and Acting in Uncertain Environments using Probabilistic Inference
26 Citations2006Deepak Kumar Verma, Rajesh P. N. Rao
The proposed framework for planning and action selection based on probabilistic inference in graphical models utilizes the most probable explanation (MPE) of variables in a graphical model, allowing tractable and efficient inference of actions.
Proceedings - IEEE International Conference on Robotics and Automation/ProceedingsReinforcement Learning for Operational Space Control
21 Citations2007Jan Peters, Stefan Schaal
A generalization of the EM-based reinforcement learning framework suggested by Dayan and Hinton is used, which reduces the problem of learning with immediate rewards to a reward-weighted regression problem with an adaptive, integrated reward transformation for faster convergence.
Using EM for Reinforcement Learning
20 Citations2000Peter Dayan, Geoffrey E. Hinton
This work discsus Hinton’s (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent, and shows circumstances under which applying the RPP is guaranteed to increase the mean return.