A Distributed Actor-Critic Algorithm and Applications to Mobile Sensor Network Coordination Problems
IEEE Transactions on Automatic ControlPublished 20 January 2010
Paris Pennesi, Ioannis Ch. Paschalidis
Citations43
SJR quartileQ1
SJR score3.80
SNIP2.59
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A distributed actor-critic method that orchestrates the coordination of multiple agents solving a general class of a Markov decision problem and uses a consensus-like algorithm for updating agents' policy parameters is introduced.
Abstract
We introduce and establish the convergence of a distributed actor-critic method that orchestrates the coordination of multiple agents solving a general class of a Markov decision problem. The method leverages the centralized single-agent actor-critic algorithm of and uses a consensus-like algorithm for updating agents' policy parameters. As an application and to validate our approach we consider a reward collection problem as an instance of a multi-agent coordination problem in a partially known environment and subject to dynamical changes and communication constraints.
Keywords
Computer Science
IEEE Transactions on Automatic ControlCoordination of groups of mobile autonomous agents using nearest neighbor rules
8,330 Citations2003Ali Jadbabaie, Jie Lin +1 more
IEEE Transactions on Robotics and AutomationCoverage Control for Mobile Sensing Networks
2,661 Citations2004Jorge Cortés, Sonia Martı́nez +2 more
This paper proposes gradient descent algorithms for a class of utility functions which encode optimal coverage and sensing policies which are adaptive, distributed, asynchronous, and verifiably correct.
IEEE Transactions on Automatic ControlDistributed asynchronous deterministic and stochastic gradient optimization algorithms
2,021 Citations1986John N. Tsitsiklis, Dimitri P. Bertsekas +1 more
DSpace@MIT (Massachusetts Institute of Technology)Actor-critic algorithms
1,811 Citations2002Vijay R. Konda, John N. Tsitsiklis
This thesis proposes and studies actor-critic algorithms which combine the above two approaches with simulation to find the best policy among a parameterized class of policies, and proves convergence of the algorithms for problems with general state and decision spaces.
PROBLEMS IN DECENTRALIZED DECISION MAKING AND COMPUTATION
1,360 Citations1984Ιωάννης Τσιτσικλής
A scheme whereby a set of decision makers (processors) exchange and update tentative decisions which minimize a common cost function, given information they possess is considered; it is shown that they are guaranteed to converge to consensus.
IEEE Transactions on Automatic ControlAn analysis of temporal-difference learning with function approximation
1,202 Citations1997John N. Tsitsiklis, Benjamin Van Roy
It is proved that divergence may occur when updates are not based on trajectories of the Markov chain, which reconciles positive and negative results that have been discussed in the literature, regarding the soundness of temporal-difference learning.
Convergence in Multiagent Coordination, Consensus, and Flocking
867 Citations2006Vincent D. Blondel, Julien M. Hendrickx +2 more
An old distributed algorithm for reaching consensus that has received a fair amount of recent attention is discussed, in which a number of agents exchange their values asynchronously and form weighted averages with (possibly outdated) values possessed by their neighbors.
SIAM Journal on Control and OptimizationOnActor-Critic Algorithms
694 Citations2003Vijay R. Konda, John N. Tsitsiklis
This article proposes and analyzes a class of actor-critic algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic.
IEEE Transactions on Automatic ControlSimulation-based optimization of Markov reward processes
315 Citations2001Peter Marbach, John N. Tsitsiklis
Distributed Cooperative coverage Control of Sensor Networks
172 Citations2006Wei Li, Christos G. Cassandras
A distributed coverage control scheme for cooperating mobile sensor networks that incorporates communication costs into the coverage control problem, viewing the sensor network as a multi-source, single-basestation data collection network.
Solving sensor network coverage problems by distributed asynchronous actor-critic methods
3 Citations2007Paris Pennesi, Ioannis Ch. Paschalidis
A distributed actor-critic method is proposed to solve the reward collection problem, where both the positions and the values of the rewards change with time, and establish its convergence and adaptation capabilities.