Planning for Decentralized Control of Multiple Robots Under UncertaintyChristopher Amato1, George D. Konidaris1, Gabriel Cruz1

Christopher A. Maynor2, Jonathan P. How2 and Leslie P. Kaelbling1

1CSAIL, 2LIDSMIT

Cambridge, MA 02139

Abstract

We describe a probabilistic framework for synthesizing con-trol policies for general multi-robot systems, given environ-ment and sensor models and a cost function. Decentral-ized, partially observable Markov decision processes (Dec-POMDPs) are a general model of decision processes wherea team of agents must cooperate to optimize some objective(specified by a shared reward or cost function) in the presenceof uncertainty, but where communication limitations meanthat the agents cannot share their state, so execution mustproceed in a decentralized fashion. While Dec-POMDPs aretypically intractable to solve for real-world problems, recentresearch on the use of macro-actions in Dec-POMDPs hassignificantly increased the size of problem that can be prac-tically solved as a Dec-POMDP. We describe this generalmodel, and show how, in contrast to most existing methodsthat are specialized to a particular problem class, it can syn-thesize control policies that use whatever opportunities forcoordination are present in the problem, while balancing offuncertainty in outcomes, sensor information, and informationabout other agents. We use three variations on a warehousetask to show that a single planner of this type can generatecooperative behavior using task allocation, direct communi-cation, and signaling, as appropriate.

IntroductionThe decreasing cost and increasing sophistication of recentlyavailable robot hardware has the potential to create manynew opportunities for applications where teams of relativelycheap robots can be deployed to solve real-world problems.Practical methods for coordinating such multi-robot teamsare therefore becoming critical. A wide range of approacheshave been developed for solving specific classes of multi-robot problems, such as task allocation [15], navigation in aformation [5], cooperative transport of an object [20], coor-dination with signaling [6] or communication under variouslimitations [33]. Broadly speaking, the current state of theart in multi-robot research is to hand-design special-purposecontrollers that are explicitly designed to exploit some prop-erty of the environment or produce a specific desirable be-havior. Just as in the single-robot case, it would be muchmore desirable to instead specify a world model and a costmetric, and then have a general-purpose planner automati-cally derive a controller that minimizes cost, while remain-ing robust to the uncertainty that is fundamental to real robot

systems [37].The decentralized partially observable Markov decision

process (Dec-POMDP) is a general framework for repre-senting multiagent coordination problems. Dec-POMDPshave been studied in fields such as control [1, 23], opera-tions research [8] and artificial intelligence [29]. Like theMDP [31] and POMDP [17] models that it extends, the Dec-POMDP model is very general, considering uncertainty inoutcomes, sensors and information about the other agents,and aims to optimize policies against a a general cost func-tion. Dec-POMDP problems are often characterized by in-complete or partial information about the environment andthe state of other agents due to limited, costly or unavailablecommunication. Any problem where multiple agents sharea single overall reward or cost function can be formalized asa Dec-POMDP, which means a good Dec-POMDP solverwould allow us to automatically generate control policies(including policies over when and what to communicate) forvery rich decentralized control problems, in the presence ofuncertainty. Unfortunately, this generality comes at a cost:Dec-POMDPs are typically infeasible to solve except forvery small problems [3].

One reason for the intractability of solving large Dec-POMDPs is that current approaches model problems at a lowlevel of granularity, where each agent’s actions are primitiveoperations lasting exactly one time step. Recent research hasaddressed the more realistic MacDec-POMDP case whereeach agent has macro-actions: temporally extended actionswhich may require different amounts of time to execute [3].MacDec-POMDPs cannot be reduced to Dec-POMDPs dueto the asynchronous nature of decision-making in this con-text — some agents may be choosing new macro-actionswhile others are still executing theirs. This enables sys-tems to be modeled so that coordination decisions only oc-cur at the level of deciding which macro-actions to execute.MacDec-POMDPs retain the ability to coordinate agentswhile allowing near-optimal solutions to be generated forsignificantly larger problems than would be possible usingother Dec-POMDP-based methods.

Macro-actions are a natural model for the modular con-trollers often sequenced to obtain robot behavior. Themacro-action approach leverages expert-designed or learnedcontrollers for solving subproblems (e.g., navigating to awaypoint or grasping an object), bridging the gap between

traditional robotics research and work on Dec-POMDPs.This approach has the potential to produce high-quality gen-eral solutions for real-world heterogeneous multi-robot co-ordination problems by automatically generating control andcommunication policies, given a model.

The goal of this paper is to present this general frame-work for solving decentralized cooperative partially observ-able robotics problems and provide the first demonstrationof such a method running on real robots. We begin byformally describing the Dec-POMDP model, its solutionand relevant properties, and describe MacDec-POMDPs anda memory-bounded algorithm for solving them. We thendescribe a process for converting a robot domain into aMacDec-POMDP model, solving it, and using the solu-tion to produce a SMACH [9] finite-state machine task con-troller. Finally, we use three variants of a warehouse task toshow that a MacDec-POMDP planner allows coordinationbehaviors to emerge automatically by optimizing the avail-able macro-actions (allocating tasks, using direct commu-nication, and employing signaling, as appropriate). We be-lieve the MacDec-POMDP represents a foundational algo-rithmic framework for generating solutions for a wide rangeof multi-robot systems.

Decentralized, Partially Observable MarkovDecision Processes

Dec-POMDPs [8] generalize partially observable Markovdecision processes to the multiagent, decentralized setting.Multiple agents operate under uncertainty based on (possi-bly different) partial views of the world, with execution un-folding over a bounded or unbounded sequence of steps. Ateach step, every agent chooses an action (in parallel) basedpurely on locally observable information, resulting in an im-mediate reward and an observation being obtained by eachindividual agent. The agents share a single reward or costfunction, so they should cooperate to solve the task, but theirlocal views mean that operation is decentralized during exe-cution.

As depicted in Fig. 1, a Dec-POMDP [8] involves mul-tiple agents that operate under uncertainty based on differ-ent streams of observations. We focus on solving sequen-tial decision-making problems with discrete time steps andstochastic models with finite states, actions, and observa-tions, though the model can be extended to continuous prob-lems. A key assumption is that state transitions are Marko-vian, meaning that the state at time t depends only on thestate and events at time t − 1. The reward is typically onlyused as a way to specify the objective of the problem and isnot observed during execution.

More formally, a Dec-POMDP is described by a tuple〈I, S, Ai, T,R, Ωi, O, h〉, where

• I is a finite set of agents.

• S is a finite set of states with designated initial state dis-tribution b0.

• Ai is a finite set of actions for each agent iwithA = ×iAithe set of joint actions, where × is the Cartesian productoperator.

Environment

a1

o1 an

on

r

Figure 1: Representation of n agents in a Dec-POMDP set-ting with actions ai and observations oi for each agent ialong with a single reward r.

• T is a state transition probability function, T : S × A ×S → [0, 1], that specifies the probability of transitioningfrom state s ∈ S to s′ ∈ S when the actions ~a ∈ A aretaken by the agents. Hence, T (s,~a, s′) = Pr(s′|~a, s).

• R is a reward function: R : S × A → R, the immediatereward for being in state s ∈ S and taking the actions~a ∈ A.

• Ωi is a finite set of observations for each agent, i, withΩ = ×iΩi the set of joint observations.

• O is an observation probability function: O : Ω×A×S →[0, 1], the probability of seeing observations ~o ∈ Ω givenactions ~a ∈ A were taken which results in state s′ ∈ S.Hence O(~o,~a, s′) = Pr(~o|~a, s′).

• h is the number of steps until the problem terminates,called the horizon.

Note that while the actions and observation are factored,the state need not be. This flat state representation allowsmore general state spaces with arbitrary state informationoutside of an agent (such as target information or environ-mental conditions). Because the full state is not directly ob-served, it may be beneficial for each agent to remember ahistory of its observations. Specifically, we can consider anaction-observation history for agent i as

HAi = (s0i , a

1i , . . . , s

l−1i , ali).

Unlike in POMDPs, it is not typically possible to calculate acentralized estimate of the system state from the observationhistory of a single agent, because the system state dependson the behavior of all of the agents.

SolutionsA solution to a Dec-POMDP is a joint policy—a set of poli-cies, one for each agent in the problem. Since each policy isa function of history, rather than of a directly observed state,it is typically represented as either a policy tree, where thevertices indicate actions to execute and the edges indicatetransitions conditioned on an observation, or as a finite statecontroller which executes in a similar manner. An exampleof each is given in Figure 2.

As in the POMDP case, the goal is to maximize the to-tal cumulative reward, beginning at some initial distribution

a1

a2

a2 a1 a1

o1 o2

o1 o1 o2o2

a3

a3

(a)o1

o1 o2

o2

a2a1

(b)

Figure 2: A single agent’s policy represented as (a) a policytree and (b) a finite-state controller with initial state shownwith a double circle.

over states b0. In general, the agents do not observe the ac-tions or observations of the other agents, but the rewards,transitions, and observations depend on the decisions of allagents. The work discussed in this paper (and the vast ma-jority of work in the Dec-POMDP community) considers thecase where the model is assumed to be known to all agents.

The value of a joint policy, π, from state s is

V π(s) = E

[h−1∑t=0

γtR(~at, st)|s, π

],

which represents the expected value of the immediate rewardfor the set of agents summed for each step of the problemgiven the action prescribed by the policy until the horizon isreached. In the finite-horizon case, the discount factor, γ, istypically set to 1. In the infinite-horizon case, as the numberof steps is infinite, the discount factor γ ∈ [0, 1) is includedto maintain a finite sum and h = ∞. An optimal policybeginning at state s is π∗(s) = arg maxπ V

π(s).Unfortunately, large problem instances remain in-

tractable: some advances have been made in optimal algo-rithms [1, 2, 4, 10, 12, 27], but optimally solving a Dec-POMDP is NEXP-complete, so most approaches that scalewell make very strong assumptions about the domain (e.g.,assuming a large amount of independence between agents)[13, 24, 26] and/or have no guarantees about solution quality[28, 34, 38].

Macro-Actions for Dec-POMDPsDec-POMDPs typically require synchronous decision-making: every agent repeatedly determines which action toexecute, and then executes it within a single time step. Thisrestriction is problematic for robot domains for two reasons.First, robot systems are typically endowed with a set of con-trollers, and planning consists of sequencing the executionof those controllers. However, due to both environmentaland controller complexity, the controllers will almost alwaysexecute for an extended period, and take differing amountsof time to run. Synchronous decision-making would thusrequire us to wait until all robots have completed their con-troller execution before we perform the next action selec-tion, which is suboptimal and may not even always be pos-sible (since the robots do not know the system state and stay-ing in place may be difficult in some domains). Second, the

planning complexity of a Dec-POMDP is doubly exponen-tial in the horizon. A planner that must try to reason about allof the robots’ possible policies at every time step will onlyever be able to make very short plans.

Recent research has extended the Dec-POMDP model toplan using options, or temporally extended actions [3]. ThisMacDec-POMDP formulation models a group of robots thatmust plan by sequencing an existing set of controllers, en-abling planning at the appropriate level to compute near-optimal solutions for problems with significantly longerhorizons and larger state-spaces.

We can gain additional benefits by exploiting knownstructure in the multi-robot problem. For instance, most con-trollers only depend on locally observable information anddo not require coordination. For example, consider a con-troller that navigates a robot to a waypoint. Only local in-formation is required for navigation—the robot may detectother robots but their presence does not change its objective,and it simply moves around them—but choosing the targetwaypoint likely requires the planner to consider the loca-tions and actions of all robots. Macro-actions with indepen-dent execution allow coordination decisions to be made onlywhen necessary (i.e., when choosing macro-actions) ratherthan at every time step. Because we build on top of Dec-POMDPs, macro-action choice may depend on history, butduring execution macro-actions may depend only on a sin-gle observation, depend on any number of steps of history, oreven represent the actions of a set of robots. That is, macro-actions are very general and can be defined in such a wayto take advantage of the knowledge available to the robotsduring execution.

ModelWe first consider macro-actions that only depend on a sin-gle robot’s information. This is an extension the optionsframework [36] to multi-agent domains while dealing withthe lack of synchronization between agents. The optionsframework is a formal model of a macro-actions [36] thathas been very successful in aiding representation and solu-tions in single robot domains [19]. A MacDec-POMDP withlocal options is defined as a Dec-POMDP where we also as-sume Mi represents a finite set of options for each agent, i,with M = ×iMi the set of joint options [3]. A local optionis defined by the tuple:

Mi = (βmi, Imi

, πmi),

consisting of stochastic termination condition βmi: HA

i →[0, 1], initiation set Imi

⊂ HAi and option policy πmi

:HAi ×Ai → [0, 1]. Note that this representation uses action-

observation histories of an agent in the terminal and initia-tion conditions as well as the option policy. Simpler casescan consider reactive policies that map single observationsto actions as well as termination and initiation sets that de-pend only on single observations. This is especially appro-priate when the agent has knowledge about aspects of thestate necessary for option execution (e.g., its own locationwhen navigating to a waypoint causing observations to belocation estimates). As we later discuss, initiation and termi-

nal conditions can depend on global states (e.g., also endingexecution based on unobserved events).

Because it may be beneficial for agents to remember theirhistories when choosing which option to execute, we con-sider policies that remember option histories (as opposed toaction-observation histories). We define an option history as

HMi = (h0i ,m

1i , . . . , h

l−1i ,ml

i),

which includes both the action-observation histories wherean option was chosen and the selected options themselves.The option history also provides an intuitive representationfor using histories within options. It is more natural for op-tion policies and termination conditions to depend on his-tories that begin when the option is first executed (action-observation histories) while the initiation conditions woulddepend on the histories of options already taken and theirresults (option histories). While a history over primitive ac-tions also provides the number of steps that have been ex-ecuted in the problem (because it includes actions and ob-servations at each step), an option history may require manymore steps to execute than the number of options listed. Wecan also define a (stochastic) local policy, µi : HM

i ×Mi →[0, 1] that depends on option histories. We then define a jointpolicy for all agents as µ.

Because option policies are built out of primitive actions,we can evaluate policies in a similar way to other Dec-POMDP-based approaches. Given a joint policy, the primi-tive action at each step is determined by the high level pol-icy which chooses the option and the option policy whichchooses the action. The joint policy and option policies canthen be evaluated as:

V µ(s) = E

[h−1∑t=0

γtR(~at, st)|s, π, µ

].

For evaluation in the case where we define a set of optionswhich use observations (rather than histories) for initiation,termination and option policies (while still using option his-tories to choose options) see Amato, Konidaris and Kael-bling [3].

AlgorithmsBecause Dec-POMDP algorithms produce policies mappingagent histories to actions, they can be extended to consideroptions instead of primitive actions. Two such algorithmshave been extended [3], but other extensions are possible.

In these approaches, deterministic polices are generatedwhich are represented as policy trees (as shown in Figure2). A policy tree for each agent defines a policy that canbe executed based on local information. The root node de-fines the option to choose in the known initial state, and an-other option is assigned to each of the legal terminal statesof that option; this continues for the depth of the tree. Such atree can be evaluated up to a desired (low-level) horizon us-ing the policy evaluation given above, which may not reachsome nodes of the tree due to the differing execution timesof some options.

A simple exhaustive search method can be used to gen-erate hierarchically optimal deterministic policies. This al-gorithm is similar in concept to the dynamic programming

m1 m2

(a) Step 1

m1

m1 m1

βs1 βs2

m1

m1 m2

βs1 βs2

m2

m1 m1

βs1

βs2

m1

βs3

m2

m1 m2

βs1

βs2

m1

βs3

(b) Step 2 of DP

Figure 3: Policies for a single agent after (a) one step and(b) two steps of dynamic programming using options m1

and m2 and (deterministic) terminal states as βs.

algorithm used in Dec-POMDPs [16], but full evaluation andpruning (removing dominated policies) are not used. Insteadthe structure of options is exploited to reduce the space ofpolicies considered. That is, to generate deterministic poli-cies, trees are built up as in Figure 3. Trees of increasingdepth are constructed until all of the policies are guaran-teed to terminate before the desired horizon. When all poli-cies are sufficiently long, all combinations of these policiescan be evaluated as above (by flattening out the polices intoprimitive action Dec-POMDP policies, starting from someinitial state and proceeding until h). The combination withthe highest value at the initial belief state, b0, is a hierarchi-cally optimal policy. Note that the benefit of this approachis that only legal policies are generated using the initiationand terminal conditions for options.

Memory-bounded dynamic programming (MBDP) [34]has also been extended to use options as shown in Algo-rithm 1. This approach bounds the number of trees that aregenerated by the method above as only a finite number ofpolicy trees are retained (given by parameter MaxTrees) ateach tree depth. To increase the tree depth to t + 1, all pos-sible trees are considered that choose some option and thenhave the trees retained from depth t as children. Trees arechosen by evaluating them at states that are reachable usinga heuristic policy that is executed for the first h− t− 1 stepsof the problem. A set of MaxTrees states is generated andthe highest-valued trees for each state are kept. This processcontinues, using shorter heuristic policies until all combina-tions of the retained trees reach horizon h. Again, the setof trees with the highest value at the initial belief state isreturned.

The MBDP-based approach is potentially suboptimal be-cause a fixed number of trees are retained, and trees op-timized at the states provided by the heuristic policy maybe suboptimal (because the heuristic policy may be subop-timal and the algorithm assumes the states generated by theheuristic policy are known initial states for the remainingpolicy tree). Nevertheless, since the number of policies ateach step is bounded by MaxTrees, MBDP has time andspace complexity linear in the horizon. As a result, this ap-proach has been shown to work well in many large MacDec-POMDPs [3].

Solving Multi-Robot Problems withMacDec-POMDPs

The MacDec-POMDPs framework is a natural way to rep-resent and generate behavior for general multi-robot sys-tems. A high-level description of this process is given in

Algorithm 1 Option-based memory bounded dynamic pro-gramming

1: function OPTIONMBDP(MaxTrees,h,Hpol)2: t← 03: someTooShort← true4: µt ← ∅5: repeat6: µt+1 ←GeneateNextStepTrees(µt)7: Compute V µt+1

8: µt+1 ← ∅9: for all k ∈MaxTrees do

10: sk ← GenerateState(Hpol,h− t− 1)11: µt+1 ← µt+1 ∪ arg maxµt+1 V

µt+1(sk)12: end for13: t← t+ 114: µt ← µt+1

15: someTooShort←testLength(µt)16: until someTooShort = false17: return µt18: end function

Figure 4. We assume an abstract model of the system isgiven in the form of macro-action representations, which in-clude the associated policies as well as initiation and termi-nal conditions. These macro-actions are controllers oper-ating in (possibly) continuous time with continuous actionsand feedback, but their operation is discretized for use withthe planner. This discretization represents an underlying dis-crete Dec-POMDP which consists of the primitive actions,states of the system and the associated rewards. The Dec-POMDP methods discussed above typically assume a fullmodel is given, but in this work, we make the more realis-tic assumption that we can simulate the macro-actions in anenvironment that is similar to the real-world domain. As aresult, we do not need a full representation of the underly-ing Dec-POMDP and use the simulator to test macro-actioncompletion and evaluate policies. In the future, we planto remove this underlying Dec-POMDP modeling and in-stead represent the macro-action initiation, termination andpolicies using features directly in the continuous robot state-space. In practice, models of each macro-action’s behaviorcan be generated by executing the corresponding controllerfrom a variety of initial conditions (which is how our modeland simulator was constructed in the experiment section).Given the macro-actions and simulator, the planner then au-tomatically generates a solution which optimizes the valuefunction with respect to the uncertainty over outcomes, sen-sor information and other agents. This solution comes inthe form of SMACH controllers [9] which are hierarchicalstate machines for use in a ROS [32] environment. Eachnode in the SMACH controller represents a macro-actionwhich is executed on the robot and each edge correspondsto a terminal condition. In this way, the trees in Figure 3can be directly translated into SMACH controllers, one foreach robot. Our system is thus able to automatically gen-erate SMACH controllers, which are typically designed byhand, for complex, general multi-robot systems.

Op#mized controllers for each robot (in SMACH format)

System descrip#on (macro-­‐ac#ons, dynamics, sensor uncertainty, rewards/costs)

Planner (solving the MacDec-­‐POMDP)

Figure 4: A high level system diagram.

It is also worth noting that our approach can incorporateexisting solutions for more restricted scenarios as macro-actions. For example, our approach can build on the largeamount of research in single and multi-robot systems thathas gone into solving difficult problems such as navigationin a formation [5] or cooperative transport of an object [20].The solutions to these problems could be represented asmacro-actions in our framework, building on existing re-search to solve even more complex multi-robot problems.

Planning using MacDec-POMDPs in theWarehouse Domain

We test our methods in a warehousing scenario using a setof iRobot Creates (Figure 5), and demonstrate how the samegeneral model and solution methods can be applied in ver-sions of this domain with different communication capabil-ities. This is the first time that Dec-POMDP-based meth-ods have been used to solve large multi-robot domains. Wedo not compare with other methods because other Dec-POMDP cannot solve problems of this size and currentmulti-robot methods cannot automatically derive solutionsfor these multifaceted problems. The results demonstratethat our methods can automatically generate the appropri-ate motion and communication behavior while consideringuncertainty over outcomes, sensor information and otherrobots.

The Warehouse DomainWe consider three robots in a warehouse that are tasked withfinding and retrieving boxes of two different sizes: largeand small. Robots can navigate to known depot locations(rooms) to retrieve boxes and bring them back to a desig-nated drop-off area. The larger boxes can only be movedeffectively by two robots (if a robot tries to pick up the largebox by itself, it will move to the box, but fail to pick it up).While the locations of the depots are known, the contents(the number and type of boxes) are unknown. Our plannergenerates a SMACH controller for each of the robots offlinewhich are then executed online in a decentralized manner.

In each scenario, we assumed that each robot could ob-serve its own location, see other robots if they were within(approximately) one meter, observe the nearest box when

Figure 5: The warehouse domain with three robots.

in a depot and observe the size of the box if it is holdingone. These observations were implemented within a Viconsetup to allow for system flexibility, but the solutions wouldwork in any setting in which these observations are gener-ated. In the simulator that is used by the planner to generateand evaluate solutions, the resulting state space of the prob-lem includes the location of each robot (discretized into ninepossible locations) and the location of each of the boxes (ina particular depot, with a particular robot or at the goal). Theprimitive actions are to move in four different directions aswell as pickup, drop and communication actions. Note thatthis primitive state and action representation is used for eval-uation purposes and not actually implemented on the robots(which just utilize the SMACH controllers). Higher fidelitysimulators could also be used. The three-robot version ofthis scenario has 1,259,712,000 states, which is several or-ders of magnitude larger than problems typically solvable byDec-POMDP solvers. These problems are solved using theoption-based MBDP algorithm initialized with a hand codedheuristic policy. Experiments were run on a single core ofa 2.5 GHz machine with 8GB of memory. Exact solutiontimes were not calculated, but average solution times for thepolicies presented below were approximately one hour.

In our Dec-POMDP model, navigation has a smallamount of noise in the amount of time required to move tolocations (reflecting the real-world dynamics): this noise in-creases when the robots are pushing the large box (reflectingthe need for slower movements and turns in this case). Wedefined macro-actions that depend only on the observationsabove, but option selection depends on the history of op-tions executed and observations seen as a result (the optionhistory).

Scenario 1: No CommunicationIn the first scenario, we consider the case where robots couldnot communicate with each other. Therefore, all cooperationis based on the controllers that are generated by the planner(which knows the controllers generated for all robots whenplanning offline) and observations of the other robots (whenexecuting online). The macro-actions were as follows:• Go to depot 1.• Go to depot 2.

d2

m ps

d1

d1 m

d1 m

d1 m

goal

g m

, D2

m dr

m pl

goal

g m

, D1

m dr

, D2

, goal

, , D1

, goal

m pl

goal

g m

, D1

m dr

, , D1

, goal

, D1

, D1

d1

[repeat for 6 more steps ]

Macro-actions d1=depot 1 d2=depot 2

g=goal (drop-off area) ps=pick up small box pl=pick up large box

dr=drop box

Figure 7: Path executed in policy trees. Only macro-actionsexecuted (nodes) and observations seen (edges, with boxesand robots given pictorially) are shown.

• Go to the drop-off area.

• Pick up the small box.

• Pick up the large box.

• Drop off a box.

The depot macro-actions are applicable anywhere and ter-minate when the robot is within the walls of the appropriatedepot. The drop-off and drop macro-actions are only appli-cable if the robot is holding a box, and the pickup macro-actions are only applicable when the robot observes a box ofthe particular type. Navigation is stochastic in the amount oftime that will be required to succeed (as mentioned above).Picking up the small box was assumed to succeed determin-istically, but this easily be changed if the pickup mechanismis less robust. These macro-actions correspond to naturalchoices for robot controllers.

This case1 (seen in Figure 6 along with a depiction of theexecuted policy in Figure 7) uses only two robots to moreclearly show the optimized behavior in the absence of com-munication. The policy generated by the planner assigns onerobot to go to each of the depots (Figure 6(a)). The robotsthen observe the contents of the depots they are in (Figure6(b)). If there are two robots in the same room as a largebox, they will push it back to the goal. If there is only onerobot in a depot and there is a small box to push, the robot

1Videos for all scenarios can be seen athttp://youtu.be/istb8TIp_jw

(a) Two robots set out for differ-ent depots.

(b) The robots observe the boxesin their depots (large on left,small on right).

(c) White robot moves to thelarge box and green robot movesto the small one.

(d) White robot waits whilegreen robot pushes the small box.

(e) Green robot drops the box offat the goal.

(f) Green robot goes to the de-pot 1 and sees the other robot andlarge box.

(g) Green robot moves to helpthe white robot.

(h) The two robots push the largebox back to the goal.

Figure 6: Video captures from the no communication version of the warehouse problem.

will push the small box (Figure 6(c)). If the robot is in adepot with a large box and no other robots, it will stay in thedepot, waiting for another robot to come and help push thebox (Figure 6(d)). In this case, once the the other robot isfinished pushing the small box (Figure 6(e)), it goes back tothe depots to check for other boxes or robots that need help(Figure 6(f)). When it sees another robot and the large boxin the depot on the left (depot 1), it attempts to help pushthe large box (Figure 6(g)) and the two robots are successfulpushing the large box to the goal (Figure 6(h)). In this case,the planner has generated a policy in a similar fashion to taskallocation—two robots go to each room, and then search forhelp needed after pushing any available boxes. However,in our case this behavior was generated by an optimizationprocess that considered the different costs of actions, the un-certainty involved and the results of those actions into thefuture.

Scenario 2: Local CommunicationIn scenario 2, robots can communicate when they are withinone meter of each other. The macro-actions are the sameas above, but we added ones to communicate and wait forcommunication. The resulting macro-action set is:

• Go to depot 1.

• Go to depot 2.

• Go to the drop-off area.

• Pick up the small box.

• Pick up the large box.

• Drop off a box.

• Go to an area between the depots (the “waiting room”).

• Wait in the waiting room for another robot.

• Send signal #1.

• Send signal #2.

Here, we allow the robots to choose to go to a “waitingroom” which is between the two depots. This permits therobots to possibly communicate or receive communicationsbefore committing to one of the depots. The waiting-roommacro-action is applicable in any situation and terminateswhen the robot is between the waiting room walls. The de-pot macro-actions are now only applicable in the waitingroom, while the drop-off, pick up and drop macro-actionsremain the same. The wait macro-action is applicable inthe waiting room and terminates when the robot observesanother robot in the waiting room. The signaling macro-actions are applicable in the waiting room and are observ-able by other robots that are within approximately a meterof the signaling robot. Note that we do not specify whatsending each communication signal means.

The results for this domain are shown in Figure 8. Wesee that the robots go to the waiting room (Figure 8(a)) (be-cause we required the robots to be in the waiting room beforechoosing to move to a depot) and then two of the robots goto depot 2 (the one on the right) and one robot goes to de-pot 1 (the one on the left) (Figure 8(b)). Note that becausethere are three robots, the choice for the third robot is ran-dom while one robot will always be assigned to each of thedepots. Because there is only a large box to push in depot 1,the robot in this depot goes back to the waiting room to tryto find another robot to help it push the box (Figure 8(c)).The robots in depot 2 see two small boxes and they chooseto push these back to the goal (Figure 8(d)). Once the smallboxes are dropped off (Figure 8(e)), one of the robots returnsto the waiting room (Figure 8(f)) and then is recruited by theother robot to push the large box back to the goal (Figure8(g)). The robots then successfully push the large box backto the goal (Figure 8(h)). Note that in this case the plan-ning process determines how the signals should be used to

(a) The three robots begin mov-ing to the waiting room.

(b) One robot goes to depot 1 andtwo robots go to depot 2. The de-pot 1 robot sees a large box.

(c) The depot 1 robot saw a largebox, so it moved to the wait-ing room while the other robotspushed the small boxes.

(d) The depot 1 robot waits withthe other robots push the smallboxes.

(e) The two robots drop off thesmall boxes at the goal while theother robot waits.

(f) The green robot goes to thewaiting room to try to receive anysignals.

(g) The white robot sent signal#1 when it saw the green robotand this signal is interpreted as aneed for help in depot 1.

(h) The two robots in depot 1push the large box back to thegoal.

Figure 8: Video captures from the limited communication version of the warehouse problem.

perform communication.

Scenario 3: Global CommunicationIn the last scenario, the robots can use signaling (rather thandirect communication). In this case, there is a switch in eachof the depots that can turn on a blue or red light. This lightcan be seen in the waiting room and there is another lightswitch in the waiting room that can turn off the light. (Thelight and switch were simulated in software and not incorpo-rated in the physical domain.) As a result, the macro-actionsin this scenario were as follows:

• Go to depot 1.

• Go to depot 2.

• Go to the drop-off area.

• Pick up the small box.

• Pick up the large box.

• Drop off a box.

• Go to an area between the depots (the “waiting room”).

• Turn on a blue light.

• Turn on a red light.

• Turn off the light.

The first seven macro-actions are the same as for the com-munication case except we relaxed the assumption that therobots had to go to the waiting room before going to thedepots (making both the depot and waiting room macro-actions applicable anywhere). The macro-actions for turn-ing the lights on are applicable in the depots and the macro-actions for turning the lights off are applicable in the waitingroom. While the lights were intended to signal requests forhelp in each of the depots, we did not assign a particular

color to a particular depot. In fact, we did not assign themany specific meaning, allowing the planner to set them inany way that improves performance.

The results are shown in Figure 9. Because one robotstarted ahead of the others, it was able to go to depot 1 tosense the size of the boxes while the other robots go to thewaiting room (Figure 9(a)). The robot in depot 1 turned onthe light (red in this case, but not shown in the images) tosignify that there is a large box and assistance is needed(Figure 9(b)). The green robot (the first other robot to thewaiting room) sees this light, interprets it as a need for helpin depot 1, and turns off the light (Figure 9(c)). The otherrobot arrives in the waiting room, does not observe a lighton and moves to depot 2 (also Figure 9(c)). The robot indepot 2 chooses to push a small box back to the goal and thegreen robot moves to depot 1 to help the other robot (Fig-ure 9(d)). One robot then pushes the small box back to thegoal while the two robots in depot 1 begin pushing the largebox (Figure 9(e)). Finally, the two robots in depot 1 pushthe large box back to the goal (Figure 9(f)). This behavioris optimized based on the information given to the planner.The semantics of all these signals as well as the movementand signaling decisions were decided on by the planning al-gorithm to maximize value.

Related WorkThere are several frameworks that have been developed formulti-robot decision making in complex domains. For in-stance, behavioral methods have been studied for perform-ing task allocation over time in loosely-coupled [30] ortightly-coupled [35] tasks. These are heuristic in nature andmake strong assumptions about the type of tasks that will becompleted.

One important related class of methods is based on using

(a) One robot starts first andgoes to depot 1 while the otherrobots go to the waiting room.

(b) The robot in depot 1 sees alarge box, so it turns on the redlight (the light is not shown).

(c) The green robot sees the lightfirst, so it turns it off and goesto depot 1 while the white robotgoes to depot 2.

(d) The robots in depot 1 moveto the large box, while the robotin depot 2 begins pushing thesmall box.

(e) The robots in depot 1 beginpushing the large box and therobot in depot 2 pushes a smallbox to the goal.

(f) The robots from depot 1 suc-cessfully push the large box tothe goal.

Figure 9: Video captures from the signaling version of thewarehouse problem.

linear temporal logic (LTL) [7, 21] to specify behavior fora robot; from this specification, reactive controllers that areguaranteed to satisfy the specification can be derived. Thesemethods are appropriate when the world dynamics can beeffectively described non-probabilistically and when thereis a useful discrete characterization of the robot’s desiredbehavior in terms of a set of discrete constraints. When ap-plied to multiple robots, it is necessary to give each robotits own behavior specification. Other logic-based represen-tations for multi-robot systems have similar drawbacks andtypically assume centralized planning and control [22].

Market-based approaches use traded value to establish anoptimization framework for task allocation [11, 15]. Theseapproaches have been used to solve real multi-robot prob-lems [18], but are largely aimed to tightly-coupled tasks,where the robots can communicate through a bidding mech-anism.

Emery-Montemerlo et al. [14] introduced a (coopera-tive) game-theoretic formalization of multi-robot systemswhich resulted in solving a Dec-POMDP. An approximateforward search algorithm was used to generate solutions,but scalability was limited because a (relatively) low-levelDec-POMDP was used. Also, Messias et al. [25] introduce

an MDP-based model where a set of robots with controllersthat can execute for varying amount of time must cooper-ate to solve a problem. However, decision-making in theirsystem is centralized.

ConclusionWe have demonstrated—for the first time—that complexmulti-robot domains can be solved with Dec-POMDP-basedmethods. The MacDec-POMDP model is expressive enoughto capture multi-robot systems of interest, but also simpleenough to be feasible to solve in practice. Our results showthat a general purpose MacDec-POMDP planner can gener-ate cooperative behavior for complex multi-robot domainswith task allocation, direct communication, and signalingbehavior emerging automatically as properties of the solu-tion for the given problem model. Because all coopera-tive multi-robot problems can be modeled as Dec-POMDPs,MacDec-POMDPs represent a powerful tool for automati-cally trading-off various costs, such as time, resource us-age and communication while considering uncertainty in thedynamics, sensors and other robot information. These ap-proaches have great potential to lead to automated solutionmethods for general multi-robot coordination problems withlarge numbers of heterogeneous robots in complex, uncer-tain domains.

In the future, we plan to explore incorporating these state-of-the-art macro-actions into our MacDec-POMDP frame-work as well as examine other types of structure that can beexploited. Other topics we plan to explore include increas-ing scalability by making solution complexity depend on thenumber of agent interactions rather than the domain size,and having robots learn models of their sensors, dynamicsand other robots. These approaches have great potential tolead to automated solution methods for general multi-robotcoordination problems with large numbers of heterogeneousrobots in complex, uncertain domains.

References[1] Amato, C.; Chowdhary, G.; Geramifard, A.; Ure, N. K.;

and Kochenderfer, M. J. 2013. Decentralized control ofpartially observable Markov decision processes. In Pro-ceedings of the Fifty-Second IEEE Conference on Deci-sion and Control, 2398–2405.

[2] Amato, C.; Dibangoye, J. S.; and Zilberstein, S. 2009.Incremental policy generation for finite-horizon DEC-POMDPs. In Proceedings of the Nineteenth InternationalConference on Automated Planning and Scheduling, 2–9.

[3] Amato, C.; Konidaris, G.; and Kaelbling, L. P. 2014.Planning with macro-actions in decentralized POMDPs.In Proceedings of the Thirteenth International Confer-ence on Autonomous Agents and Multiagent Systems.

[4] Aras, R.; Dutech, A.; and Charpillet, F. 2007. Mixedinteger linear programming for exact finite-horizon plan-ning in decentralized POMDPs. In Proceedings ofthe Seventeenth International Conference on AutomatedPlanning and Scheduling, 18–25.

[5] Balch, T., and Arkin, R. C. 1998. Behavior-based for-mation control for multi-robot teams. IEEE Transactionson Robotics and Automation 14(6):926–939.

[6] Beckers, R.; Holland, O.; and Deneubourg, J.-L. 1994.From local actions to global tasks: Stigmergy and collec-tive robotics. In Artificial life IV, volume 181, 189.

[7] Belta, C.; Bicchi, A.; Egerstedt, M.; Frazzoli, E.;Klavins, E.; and Pappas, G. J. 2007. Symbolic plan-ning and control of robot motion [grand challenges ofrobotics]. Robotics & Automation Magazine, IEEE14(1):61–70.

[8] Bernstein, D. S.; Givan, R.; Immerman, N.; and Zilber-stein, S. 2002. The complexity of decentralized control ofMarkov decision processes. Mathematics of OperationsResearch 27(4):819–840.

[9] Bohren, J. 2010. SMACH. http://wiki.ros.org/smach/.

[10] Boularias, A., and Chaib-draa, B. 2008. Exactdynamic programming for decentralized POMDPs withlossless policy compression. In Proceedings of the Eigh-teenth International Conference on Automated Planningand Scheduling.

[11] Dias, M. B., and Stentz, A. T. 2003. A comparativestudy between centralized, market-based, and behavioralmultirobot coordination approaches. In Proceedings ofthe 2003 IEEE/RSJ International Conference on Intelli-gent Robots and Systems (IROS ’03), volume 3, 2279 –2284.

[12] Dibangoye, J. S.; Amato, C.; Buffet, O.; and Charpil-let, F. 2013a. Optimally solving Dec-POMDPs ascontinuous-state MDPs. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intel-ligence.

[13] Dibangoye, J. S.; Amato, C.; Doniec, A.; and Charpil-let, F. 2013b. Producing efficient error-bounded solu-tions for transition independent decentralized MDPs. InProceedings of the Twelfth International Conference onAutonomous Agents and Multiagent Systems.

[14] Emery-Montemerlo, R.; Gordon, G.; Schneider, J.; andThrun, S. 2005. Game theoretic control for robot teams.In Proceedings of the 2005 IEEE International Confer-ence on Robotics and Automation, 1163–1169.

[15] Gerkey, B., and Mataric, M. 2004. A formal analysisand taxonomy of task allocation in multi-robot systems.International Journal of Robotics Research 23(9):939–954.

[16] Hansen, E. A.; Bernstein, D. S.; and Zilberstein, S.2004. Dynamic programming for partially observablestochastic games. In Proceedings of the Nineteenth Na-tional Conference on Artificial Intelligence, 709–715.

[17] Kaelbling, L. P.; Littman, M. L.; and Cassandra, A. R.1998. Planning and acting in partially observable stochas-tic domains. Artificial Intelligence 101:1–45.

[18] Kalra, N.; Ferguson, D.; and Stentz, A. T. 2005. Ho-plites: A market-based framework for planned tight coor-dination in multirobot teams. In Proceedings of the Inter-national Conference on Robotics and Automation, 1170– 1177.

[19] Kober, J.; Bagnell, J. A.; and Peters, J. 2013. Rein-forcement learning in robotics: A survey. The Interna-tional Journal of Robotics Research 32(11):1238 – 1274.

[20] Kube, C., and Bonabeau, E. 2000. Cooperative trans-port by ants and robots. Robotics and Autonomous Sys-tems 30(1-2):85–101.

[21] Loizou, S. G., and Kyriakopoulos, K. J. 2004. Auto-matic synthesis of multi-agent motion tasks based on ltlspecifications. In Decision and Control, 2004. CDC. 43rdIEEE Conference on, volume 1, 153–158. IEEE.

[22] Lundh, R.; Karlsson, L.; and Saffiotti, A. 2008. Au-tonomous functional configuration of a network robotsystem. Robotics and Autonomous Systems 56(10):819–830.

[23] Mahajan, A. 2013. Optimal decentralized control ofcoupled subsystems with control sharing. IEEE Transac-tions on Automatic Control 58:2377–2382.

[24] Melo, F. S., and Veloso, M. 2011. Decentralized MDPswith sparse interactions. Artificial Intelligence.

[25] Messias, J. V.; Spaan, M. T.; and Lima, P. U. 2013.GSMDPs for multi-robot sequential decision-making. InProceedings of the Twenty-Seventh AAAI Conference onArtificial Intelligence.

[26] Nair, R.; Varakantham, P.; Tambe, M.; and Yokoo, M.2005. Networked distributed POMDPs: a synthesis ofdistributed constraint optimization and POMDPs. In Pro-ceedings of the Twentieth National Conference on Artifi-cial Intelligence.

[27] Oliehoek, F. A.; Spaan, M. T. J.; Amato, C.; andWhiteson, S. 2013. Incremental clustering and expansionfor faster optimal planning in Dec-POMDPs. Journal ofArtificial Intelligence Research 46:449–509.

[28] Oliehoek, F. A.; Whiteson, S.; and Spaan, M. T. J.2013. Approximate solutions for factored Dec-POMDPswith many agents. In Proceedings of the Twelfth Interna-tional Conference on Autonomous Agents and MultiagentSystems.

[29] Oliehoek, F. A. 2012. Decentralized POMDPs. InWiering, M., and van Otterlo, M., eds., ReinforcementLearning: State of the Art, volume 12 of Adaptation,Learning, and Optimization. Springer. 471–503.

[30] Parker, L. E. 1998. ALLIANCE: An architecture forfault tolerant multirobot cooperation. IEEE Transactionson Robotics and Automation 14(2):220–240.

[31] Puterman, M. L. 1994. Markov Decision Processes:Discrete Stochastic Dynamic Programming. Wiley-Interscience.

[32] Quigley, M.; Conley, K.; Gerkey, B.; Faust, J.; Foote,T.; Leibs, J.; Wheeler, R.; and Ng, A. Y. 2009. ROS: an

open-source robot operating system. In ICRA workshopon open source software, volume 3.

[33] Rekleitis, I.; Lee-Shue, V.; New, A. P.; and Choset, H.2004. Limited communication, multi-robot team basedcoverage. In Robotics and Automation, 2004. Proceed-ings. ICRA’04. 2004 IEEE International Conference on,volume 4, 3462–3468. IEEE.

[34] Seuken, S., and Zilberstein, S. 2007. Memory-bounded dynamic programming for DEC-POMDPs. InProceedings of the Twentieth International Joint Confer-ence on Artificial Intelligence, 2009–2015.

[35] Stroupe, A. W.; Ravichandran, R.; and Balch, T. 2004.Value-based action selection for exploration and dynamictarget observation with robot teams. In Proceedings ofthe International Conference on Robotics and Automa-tion, volume 4, 4190–4197. IEEE.

[36] Sutton, R. S.; Precup, D.; and Singh, S. 1999. BetweenMDPs and semi-MDPs: A framework for temporal ab-straction in reinforcement learning. Artificial Intelligence112(1):181–211.

[37] Thrun, S.; Burgard, W.; and Fox, D. 2005. Prob-abilistic Robotics (Intelligent Robotics and AutonomousAgents). The MIT Press.

[38] Velagapudi, P.; Varakantham, P. R.; Sycara, K.; andScerri, P. 2011. Distributed model shaping for scalingto decentralized POMDPs with hundreds of agents. InProceedings of the Tenth International Conference on Au-tonomous Agents and Multiagent Systems.