Quick Polytope Approximation of All Correlated Equilibria in Stochastic Games

Liam MacDermed, Karthik S. Narayan, Charles L. Isbell, Lora WeissGeorgia Institute of Technology

Robotics and Intelligent Machines LaboratoryAtlanta, Georgia 30332

liam@cc.gatech.edu, karthik.narayan@gatech.edu, isbell@cc.gatech.edu, lora.weiss@gtri.gatech.edu

Abstract

Stochastic or Markov games serve as reasonable modelsfor a variety of domains from biology to computer se-curity, and are appealing due to their versatility. In thispaper we address the problem of finding the completeset of correlated equilibria for general-sum stochasticgames with perfect information. We present QPACE –an algorithm orders of magnitude more efficient thanprevious approaches while maintaining a guarantee ofconvergence and bounded error. Finally, we validate ourclaims and demonstrate the limits of our algorithm withextensive empirical tests.

1 Introduction and Related WorkStochastic games naturally extend Markov decision pro-cesses (MDPs) in multi-agent reinforcement learning prob-lems. They can model a broad range of interesting problemsincluding oligopoly and monetary policy (Amir 2001), net-work security and utilization (Nguyen, Alpcan, and Basar2010), and phenotype-expression patterns in evolving mi-crobes (Wolf and Arkin 2003). Unfortunately, typical ap-plications of stochastic games are currently limited to verysmall and simple games. Our primary objective in this workis to make it feasible to solve larger and more complexstochastic games.

There have been many approaches to solving stochasticgames in game theory and operations research, mostly fo-cused on two-player, zero sum games or repeated games.One line of work uses a recursive “self-generating set” ap-proach that has a very similar flavor to the work we presenthere (Abreu 1988; Cronshaw 1997; Judd, Yeltekin, and Con-klin 2003; Horner et al. 2010). A recent example of this workis that of Burkov and Chaib-draa (2010) where the set ofsub-game perfect Nash equilibria is found via a repeatedlyfiner grain tiling of the set. Unfortunately, none of these ap-proaches has appealing error or computational guaranteesable to generalize to stochastic games.

In artificial intelligence, the field of multi-agent learninghas produced a particularly promising line of research em-ploying a modified version of Bellman’s dynamic program-ming approach to compute the value function (Greenwald

Copyright c© 2011, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

and Hall 2003; Littman 2001; Hu and Wellman 2003). Thisapproach has lead to a few successes, particularly in thezero-sum case; however, value-function based approacheshave been proven to be insufficient in the general case(Zinkevich, Greenwald, and Littman 2005). In the wake ofthis negative result, a new line of research has emerged thatreplaces the value-function with a multi-dimensional equiv-alent referred to as the achievable set.

Murray and Gordon (2007) derive an intractable algo-rithm for calculating the exact form of the achievable setbased Bellman equation and proved correctness and conver-gence; however, their approximation algorithm is not gener-ally guaranteed to converge with bounded error, even withinfinite time. MacDermed and Isbell (2009) solved theseproblems by targeting ε-equilibria instead of exact equilibriaand by approximating the closed convex hull with a boundednumber of vertices. Their algorithm bounded the final errorintroduced by these approximations. While MacDermed andIsbell’s algorithm scales linearly with the number of states,it becomes infeasible when there are more than two playerswith three actions each.

In this paper we adopt a new representation of achiev-able sets and present a new algorithm, QPACE, that reducesMacDermed and Isbell’s approach to a series of efficientlinear programs while maintaining their theoretical guaran-tees. We prove correctness and conduct extensive empiricaltests demonstrating QPACE to be several orders of magni-tude faster while providing identical results.

2 BackgroundStochastic games extend MDPs to multiple agents by usingvectors (one element per player) to represent actions and re-wards rather than the scalars used in MDPs. In our work, weassume agents are rational and attempt to maximize theirlong term expected utility with discount factor γ. We as-sume agents can communicate freely with each other (cheaptalk). We also assume that agents observe and rememberpast joint-actions, allowing agents to change behavior andthreaten or punish based on each other’s actions (e.g. ”if youdo X, I’ll do Y”).

We formally represent a stochastic game by the tuple〈I, S, s0, A, P,R〉. The components in the tuple include thefinite set of players I , where we let n = |I|; the statespace S, where each s ∈ S corresponds to a normal form

game; a set of actions A; a transition probability functionP : S × An × S → R, describing the probability oftransitioning from one state to another given a joint-action~a = 〈a1, · · · , an〉; and a reward function R : S×An → Rndescribing the reward a state provides to each player after ajoint-action. Agents start in initial state s0. They then repeat-edly choose joint-actions, receive rewards, and transition tonew states.

When solving stochastic games, we target correlatedequilibria (CE), which generalize Nash equilibria. A CEtakes the form of a probability distribution across joint-actions. Agents observe a shared random variable corre-sponding to joint-actions drawn from the CE’s distribution.This informs agents of their prescribed action without di-rectly revealing other agents’ actions; however, informationabout other agents’ actions is revealed in the form of a pos-terior probability. The distribution is a CE when no agenthas an incentive to deviate. CEs only require a shared ran-dom variable, but equilibrium selection without communica-tion is an open problem and beyond the scope of this paper;hence our assumption of cheap talk.

2.1 Achievable SetsThe core idea of our algorithm involves an achievable setfunction V : S → {Rn}, the multi-agent analogy to thesingle agent value-function. As a group of n agents followa joint-policy, each player i ∈ I receives rewards. The dis-counted sum of these rewards is that player’s utility, ui. Thevector ~u ∈ Rn containing these utilities is known as thejoint-utility, or value-vector. Thus, a joint-policy yields ajoint-utility in Rn. If we examine all mixed joint-policiesstarting from state s, discard those not in equilibrium, andcompute all the joint-utilities of the remaining policies wewill have a set of points in Rn: the achievable set.

An achievable set contains all possible joint-utilities thatplayers can receive using policies in equilibrium. Each di-mension represents the utility for a player (Figure 1). Thisdefinition is valid for any equilibrium solution concept, butQPACE can only compute achievable sets representable asconvex polytopes, such as those from CE. Since this setcontains all attainable joint-utilities, it will contain the op-timal joint-utility for any definition of “optimal.” From thisachievable set, an optimal joint-utility can be chosen usinga bargaining solution (Myerson 1991). Once the agents inthe game agree on a joint-utility, a policy can be constructedin a greedy manner for each player that achieves the targetedutilities (Murray and Gordon 2007), similar to the procedurein which a value function can be used to construct a pol-icy in single-agent reinforcement learning. As agents careonly about the utility they achieve and not about the specificpolicy they use, computing the achievable set for each statesolves the stochastic game.

2.2 Augmented Stage Games and ThreatsWhen choosing which action to execute, agents consider notonly their immediate reward but also the long term utility ofthe next state in which they may arrive. The expected joint-utility for each joint-action is known as the continuation util-ity, −→cu~a. Because the agents consider both their immediate

Player 1 Utility

Play

er 2

Uti

lity

(2, 2) (7, 2)7, 2

(2, 7)

2, 7

2, 2

0, 0

R

D

L

U

A) B)

Player 1 Utility

Play

er 2

Uti

lity

(5.25, 5.25)

(7, 2)7, 2

(2, 7)

2, 7

6, 6

0, 0(3.6, 3.6)

C

C

D

D

A) B)

Player 1 Utility

Play

er 2

Uti

lity

(5.25, 5.25)

(7, 2)

(2, 7)

(3.6, 3.6)

A) B)

Player 1 Utility

Play

er 2

Uti

lity

Player 1 Utility

Play

er 2

Uti

lity

7, 2

2, 7

6, 6

0, 0

Chicken

Chicken

Dare

Dare

Player 1 Utility

Play

er 2

Uti

lity (5.25, 5.25)

(7, 2)

(2, 7)

(3.6, 3.6)

A) B)

7, 2

2, 7

6, 6

0, 0

Chicken

Chicken

Dare

Dare

Figure 1: A) The game of chicken. B) The achievable set of corre-lated equilibria for chicken. For each joint-utility within the regionthere exists a correlated equilibrium yielding the given rewards foreach player. For example the point (5.25, 5.25) is achieved by theagents playing (chicken, chicken) with probability 1

2while playing

(dare, chicken), and (chicken, dare) each with probability 14

.

reward and their continuation utility they are in essence play-ing an augmented stage game with payoffs R(s,~a) +−→cu~a.

Assuming that the players cooperate with each other, theychoose to jointly play any equilibrium policy in the succes-sor state s′. Thus, the continuation utility −→cu~as′ of each suc-cessor state may be any point in the achievable set V (s′)of that state. We call the set of possible continuation utili-ties, Q(s,~a), the action achievable set function. When thereis only one successor, i.e. P (s′|s,~a) = 1, then Q(s,~a) =V (s′). When there is more than one successor state, the con-tinuation utility is the expectation over these utilities, i.e.

Q∗(s,~a) ={∑

s′P (s′|s,~a)−→cu~as′ | −→cu~as′ ∈ V ∗(s′)

}(1)

It is advantageous for players to punish other players whodeviate from the agreed upon joint-policy. Players accom-plish this by changing their continuation utility in the eventof defection. The harshest threat possible (which maximizesthe achievable set) is a global grim trigger strategy, where allother players cooperate in choosing policies minimizing thedefecting player’s rewards for the rest of the game. The grimtrigger threat-point (

−→gt~ai) is a continuation utility that play-

ers choose for each joint-action ~a in the event that player idefects. This utility can be calculated as a zero-sum gameusing Nash-Q learning (Hu and Wellman 2003) indepen-dently at initialization. While grim trigger threats can leadto subgame imperfect equilibria, extending our algorithm tocompute subgame perfect equilibria is easy (see Section 5).

3 The QPACE AlgorithmAchievable set based approaches replace the value-function,V (s), in Bellman’s dynamic program with an achievable setfunction – a mapping from state to achievable set. We up-date the achievable set function over a series of iterations,akin to value iteration. Each iteration produces an improvedestimate of the achievable set function using the previous

Figure 2: An example achievable set contraction

iteration’s estimate. The achievable set function is initial-ized to some large over-estimate and the algorithm shrinksthe sets during each iteration until insufficient progress ismade, at which point the current estimate of the achievableset function is returned (Figure 2).

The general outline of QPACE per iteration is:

1. Calculate the action achievable sets Q(s,~a), giving us theset of possible continuation utilities.

2. Construct a set of inequalities that defines the set of cor-related equilibria.

3. Approximately project this feasible set into value-vectorspace by solving a linear program for each hyperplane ofour achievable set polytope V (s).

3.1 Representing Achievable Sets

QPACE represents each achievable set as a set of m linearinequalities with fixed coefficients. Recall that the achiev-able set is closed and convex, and can be thought of as ann-dimensional polytope (one dimension for each player’sutility). Polytopes can be defined as an intersection of half-spaces. Each of the m half-spaces j ∈ H consists of a nor-mal Hj and an offset bj such that the achievable joint-utilityx is restricted by the linear inequality Hjx ≤ bj . The half-space normals form a matrix H = [H1, . . . ,Hm], and theoffsets a vector b = 〈b1, . . . , bm〉. For example, the poly-tope depicted in Figure 3c can be represented by the equa-tion Hx ≤ b where H and b are as shown.

While any achievable set may be represented by an inter-section of half-spaces, it may require an unbounded number.Such a growth in the complexity of achievable sets does in-deed occur in practice; therefore, we compute an approxima-tion using half-spaces sharing the same fixed set of normals.Each of our polytopes differ only in their offsets, so QPACEmust only store the b vector for each V (s) and Q(s,~a). Thenormals H are chosen at initialization to enable a regularapproximation of a Euclidean ball (Figure 3a).

To approximate an achievable set, we find the minimumoffsets such that our approximation completely contains theoriginal set; that is, we retain each hyperplane’s normal butcontract each offset until it barely touches the given poly-tope. This process is shown for the game of chicken in Fig-ure 3. We call this the regular polytope approximation (RPA)of polytope P using normals H where RPA(H,P )j =maxx∈P [Hj · x] for each offset j. We say that H permitserror ε if ‖RPA(H,P )− P‖ ≤ ε. RPA was first presentedin MacDermed and Isbell (2009).

Previous achievable set based methods have representedtheir achievable sets in a number of different ways: con-vex combinations of vertices (Mac Dermed and Isbell 2009;Gordon 2007), polytopes of arbitrary complexity (Judd, Yel-tekin, and Conklin 2003), and as unions of hypercubes(Burkov and Chaib-draa 2010). QPACE’s representation al-lows for quicker computation of each key step in Mac-Dermed and Isbell’s (2009) approach leading to an algo-rithm several orders of magnitude faster than previous al-gorithms.

Player 1 Utility

Play

er 2

Uti

lity

A) B)

C) D)

½

2

2

½-1

1

0

0

0

0

-1

-1

½

½ -

-½-

½-

H =

1

1

0

0

0

0

-1

-1

2

2

2

2

½

½

½

½

½-

½-½-

½-

b =2

3.62

-2.5-7

-5.25-7

-2.5

Player 1 Utility

Play

er 2

Uti

lity

A) B)

C) D)

H =

110

-1-1-101

01110

-1-1-1

b =

23.6

2-2.5

-7-5.25

-7-2.5

H =

1½0

-½-1-½

0½

0½1½0

-½-1-½

b =

75.25

72.5-2

-3.6-2

2.5

3.5

Hx ≤ b

0, 1

0, -1

½, ½

-½, -½

-½, ½

½, -½

1, 0-1, 0

A) B)

C)

H =

1½0

-½-1-½

0½

0½1½0

-½-1-½

b =

75.25

72.5-2

-3.6-2

2.5 Player 1 Utility

Play

er 2

Uti

lity

Hx ≤ b

0, 1

0, -1

½, ½

-½, -½

-½, ½

½, -½

1, 0-1, 0

Figure 3: The QPACE representation. A) A Euclidean ball sur-rounded by half-spaces suitable for a regular polytope approxima-tion (RPA). The normals of each half-space are illustrated. Notethat the half-spaces approximate the inscribed dotted circle. B) Theachievable set from Figure 1b being approximated using the RPAfrom (A). Each half-space of the Euclidean ball is moved such thatit touches the achievable set at one point. C) The resulting polytopeapproximation whose normals and offsets are specified inH and b.

+

C) C)

x2 ≤ 9

x2 ≤ 7

x2 ≤ 2

= Minkowski sum of:

=

2

2

b1 + b2 =1

3.11

-3-9

-5.25-7

-2.5

2

2

b2 =-2

-1.5-1

-0.5-2

-0.5-1

-0.5

75.25

72.5-2

-3.6-2

2.5

21.5

20.5

10.5

10.5

96.75

93

-1-3.1

-13

Figure 4: An example approximate Minkowski sum of RPA poly-topes. Our method is exact in two dimensions and permits only εerror in higher dimensions.

3.2 Computing Action Achievable SetsAt the beginning of each iteration, QPACE calculates theaction-achievable set Q(s,~a) for each state s and each jointaction ~a in the stochastic game. We do so by rewriting Equa-tion 1 using Minkowski addition. Given two sets A andB, their Minkowski sum A + B is defined as A + B ={a+b | a ∈ A, b ∈ B}. Efficient computation of Minkowskiaddition of two convex polytopes of arbitrary dimension d isan open and active problem, with complexity O(|A|d) where|A| is the number of half-spaces; however we can approxi-mate a Minkowski sum efficiently using our representationas the sum of the offsets (Figure 4). We prove this result:

Lemma 3.0.1. Suppose that A and B represent polytopes(Hx ≤ bA) and (Hx ≤ bB) and H permits only ε erroras discussed in Section 3.1. Then the weighted Minkowskisum αA + βB for constants α, β ∈ R+ and α + β = 1 is

approximated by the polytope Hx ≤ α · bA + β · bB with atmost ε relative error.

Proof. Noting RPA(H,A) ≤ ε and RPA(H,B) ≤ ε,

RPA(H,αA+ βB) = maxx∈A

maxy∈B

[α(Hi · x) + β(Hj · y)]

= α ·maxx∈A

[Hj · x] + β ·maxy∈B

[Hj · y]

= α ·RPA(H,A) + β ·RPA(H,B)

≤ ε · (α+ β) ≤ ε,

Note that multiple additions do not increase relative er-ror. Equation 1 rewriten as a weighted Minkowski sum be-comes: Q(s,~a) =

∑s′ P (s

′|s,~a)V (s). To improve the per-formance of subsequent steps we scale the set of continua-tion utilities by the discount factor γ and add the immediatereward (translating the polytope). The final offsets for theaction-achievable sets may be computed for each j ∈ H as:

Q(s,~a)j = R(s,~a) ·Hj + γ∑

s′P (s′|s,~a)V (s′)j (2)

3.3 Defining the Set of Correlated EquilibriaCalculating the set of CE in a normal form stage game (e.g.Figure 1b) is straightforward. A probability distribution Xover joint-actions (with x~a being the probability of choosingjoint-action ~a) is in equilibrium if and only if the reward offollowing the prescribed action is no worse than taking anyother action. More formally, X is a CE if and only if in states, for each player i, for distinct actions α, β ∈ Ai, where~a(α) means joint-action ~a with player i taking action α:∑

~ax~a(α)R(s,~a(α))i ≥

∑~ax~a(α)R(s,~a(β))i (3)

These rationality constraints are linear inequalities andtogether with the probability constraints

∑x~a = 1 and

x~a ≥ 0 define a polytope in Rn. Any point in this polytoperepresents a CE which yields a value-vector

∑~a x~aR(s,~a).

The union of all such value-vectors (the polytope projectedinto value-vector space) is the achievable set of state s whenagents do not consider utilities gained from future states.

Agents do not play a single normal form game. Insteadthey play the augmented game where in order for an agent tomake a decision in the current state, they must know whichutility among the many possible in Q(s,~a) they will receivein each successor state. Therefore, a CE of the augmentedstage game consists of both a probability x~a for each joint-action and an expected continuation utility −→cu~ai for eachplayer and joint-action, resulting in (n + 1)|A|n variables.Given a state s, the set of possible CEs over the variables x~aand −→cu~ai of the augmented game becomes:

For each player i, distinct actions α, β ∈ Ai,∑~a∈An

x~a(α)−−−→cu~a(α) i ≥

∑~a∈An

x~a(α) [−−−→gt~a(β) i +R(s,~a(β))i] (4)

For each joint-action ~a ∈ An, and j ∈ H:−→cu~a ∈ Q(s,~a) (i.e. Hj

−→cu~a ≤ Q(s,~a)j ) (5)

The rationality constraints (4) are quadratic because wehave a variable for the continuation utility (−→cu~ai) which mustbe scaled by our variable for the probability of that joint-action occurring (x~a). We can eliminate this multiplicationby scaling the action-achievable set Q(s,~a) in inequality 5by x~a, making the value of −→cu~ai already include the mul-tiplication by x~a. This gives us our polytope in R(n+1)|A|n

over variables −→cu~ai and x~a of feasible correlated equilibria:

For each player i, distinct actions α, β ∈ Ai,∑~a∈An

−−−→cu~a(α) i ≥∑~a∈An

x~a(α) [−−−→gt~a(β) i +R(s,~a(β))i]∑

~a∈Anx~a = 1 and ∀~a ∈ An, x~a ≥ 0

For each joint-action ~a ∈ An,−→cu~a ∈ x~a Q(s,~a) (i.e. Hj

−→cu~a ≤ x~a Q(s,~a)j )

(6)

3.4 Computing the Achievable Set FunctionThe polytope defined above in (6) is the set of CE injoint-action-probability (x~a) and continuation-utility (−→cu~ai)space; however, we do not ultimately want the set of CEsonly the set of resulting value-vectors V (s). The value thata particular correlated equilibrium presents to a player i is∑~a−→cu~ai. Recall that we are approximating our achievable

sets with a fixed set of half-spaces, where each half-spacetouches a vertex of the exact polytope (Figure 3b). This ver-tex will correspond to the CE with value-vector that maxi-mizes a weighted average of the agents’ utilities, weightedby the half-space’s normal. We can find this CE using a lin-ear program where the feasible set is as defined in (6) and theobjective function is a dot product of the value to each playerand the half-space’s normal. The optimal weighted averagefound will equal the offset for that half-space in the approx-imating polytope. In every state s, we compute one linearprogram for each approximating half-space and update itsoffset to the optimal value found. For each half-space j withnormal Hj we compute offset V (s)j as the maximum ob-jective function value to the following LP:

V (s)j =

{max

∑~a

∑i∈I

Hj,i · −→cu~aisubject to inequalities in (6)

(7)

3.5 Linear Program Solution CachingEvery iteration, QPACE solves one LP (equation 7) for eachhalf-space in every state. Depending on the value of ε, thenumber of half-spaces and hence the number of LPs canbecome very large. Solving these LPs is the main bottle-neck in QPACE. Fortunately, we can dramatically improvethe performance of these LPs by taking advantage of thefact that the solutions of many of the LPs do not tend to“change” significantly from iteration to iteration. More pre-cisely, the maximum offset difference for a given achievableset in consecutive iterations is small. For a fixed state, wecan therefore expect the solution corresponding to the LPof one iteration to be quite close to that of the previous it-eration. LPs typically spend the bulk of their running time

Algorithm 1 The QPACE Algorithm.Inputs: stochastic game G

half-spaces H permitting ε errorOutput: achievable sets {V }

1: for all s ∈ S, j ∈ H do2: V (s)j ⇐ maxs′,~a(R(s

′,~a) ·Hj)/γ3: repeat4: for all s,~a, j ∈ H do5: Q(s,~a)j = equation (2)6: for all s, j ∈ H do7: V (s)j ⇐ LP (7) starting at BFS-cache(s, j)8: BFS-cache(s, j)⇐ optimal BFS of the LP9: until ∀s : V (s)j changed less than ε/2

10: return {V }

searching over a space of basic feasible solutions (BFS), sochoosing an initial BFS close to the optimal one dramaticallyimproves performance. QPACE caches the optimal BFS ofeach state/half-space calculation and uses the solution as thestarting BFS in the next iteration. Empirical tests confirmthat after the first few iterations of QPACE the LPs will findan optimal BFS very near the cached BFS (see Figure 5f).In fact, a majority of cached BFSs are already optimal forthe new LP. Note that even if the optimal BFS from one it-eration is exactly the optimal BFS of the next, the optimalvalues may be different as the offsets change.

This optimization only applies if a BFS from one itera-tion, x∗, is a valid BFS in the next. Every LP has identicalvariables, objectives, and constraints, with subsequent itera-tions differing only in the offsets b; therefore, x∗ is a basicsolution for the new LP. While x∗ is basic, it may not be fea-sible for the new LP. However, LP sensitivity analysis guar-antees that an optimal BFS from one iteration is a BFS ofthe dual problem in the next iteration when only b changes.Therefore, we alternate each iteration between solving theprimal and dual problems.

3.6 CorrectnessIn the QPACE algorithm (Algorithm 1) each step except forline 5 performs an equivalent operation to MacDermed andIsbell’s (2009), albeit using a different representation. TheMinkowski sums in line 5 can now introduce ε error into theaction achievable sets, doubling the potential error inducedat each iteration. To mitigate this factor we run QPACE withtwice the precision when there are three or more players.Convergence and error bounds are therefore the same as inMac Dermed and Isbell (2009). Interestingly even with threeor more players our empirical tests show that LP (7) withexact Minkowski sums and LP (7) with our approximateMinkowski sums give identical results.

4 Empirical ResultsWe ran tests to determine the scalability of QPACE acrossa number of different dimensions: precision (Figure 5a),number of states (Figure 5b), number of joint-actions (Fig-ure 5c), and number of players (Figure 5d). We ran the algo-

rithms over generated games, random in the state transitionand reward functions. Unless otherwise indicated, the gameswere run with 10 states, 2 players, 2 actions each, and withε = 0.05 and γ = 0.9. We used the GLPK library as ourlinear programming solver, and used the FFQ-Learning al-gorithm (Littman 2001) to compute grim trigger threats.

As users specify more precision, the number of hyper-planes in our approximate Euclidean ball increases expo-nentially. So, wall clock time increases exponentially as εdecreases as Figures 5a confirms. The QPACE algorithm be-gins to do worse than MacDermed and Isbell (MI09) for verylow values of ε, especially in QPACE without caching (Fig-ure 5a). To see why, note that if an achievable set can berepresented by only a few verticies or half-spaces (e.g. a hy-percube) than MI09 will only use those verticies necessarywhile QPACE will always use all half-spaces in H .

Both MI09 and QPACE scale linearly with the number ofstates independent of other parameters (Figure 5b), howeverQPACE is about 240 times faster than MI09 with the de-fault parameters. The number of variables used in each LPis linear with respect to the number of joint-actions. LP’s inturn are polynomial with respect to the number of variables.Therefore as expected, both algorithms are polynomial withrespect to joint-actions (Figure 5c). However, QPACE usessignificantly fewer variables per joint-action and thereforehas a lower rate of growth.

QPACE scales much better than MI09 with respect to thenumber of players. The number of joint-actions grows ex-ponentially with the number of players, so both algorithmshave exponential growth in the number of players; however,the number of vertices needed to represent a polytope alsogrows exponentially while the number of half-spaces onlygrows polynomially. Thus, QPACE is able to handle manymore players than MI09.

A natural question to ask is how much each of QPACE’snew aspects contribute to the overall efficiency. The changeof representation permits three key improvements: fewervariables in the linear programs, an elimination of the needto calculate vertices of the polytopes, fast Minkowski addi-tion, and LP solution caching. By comparing QPACE with-out caching to MI09 in Figures 5a-c we observe the im-pact of the first two improvements. We now examine Fig-ures 5e&f to determine the contribution of the last two im-provements.

Both QPACE and MI09 employ Minkowski sums whencomputing the expected continuation utility over futurestates (when transitions are deterministic, neither algorithmcomputes Minkowski sums). As the number of possiblesuccessor states increases, both algorithms perform moreMinkowski additions. Figure 5e graphs the effect of addi-tional successor states for both MI09 and QPACE and showsthat while MI09 suffers from a 10% increases in time whenthe number of successors is large, QPACE is unaffected.

From Figure 5f, we observe that as the number of iter-ations progresses, the average number of LP iterations in-volved decreases quickly for QPACE with caching. The al-gorithm initially takes around 100 LP iterations, and dropsto less than 3 LP iterations by the sixth iteration. On theother hand, QPACE without caching starts at around 20 LP

0.1

1

10

100

1000

0 100 200 300 400

0.1

1

10

100

1000

10000

2 4 6 8 10

0

100

200

300

400

500

600

0 10 20 30

0

20

40

60

80

100

0 20 40 60

Joint-Actions

Wal

l Tim

e (s

)W

all T

ime

(s)

Wal

l Tim

e (s

)

Wal

l Tim

e (s

)W

all T

ime

(s)

Precision (ε) States

Players

IterationSuccessor States

Avg.

LP

Itera

tions

A) B)

D)C)

E) F)

QPACE QPACE - without caching MI09

0

10

20

30

40

0.00020.0020.020.2

0

200

400

600

800

1000

0 5000 10000

Figure 5: Performance evaluation. A-E) An illustration of howour algorithm scales along a number of variables. Note that someaxis are logarithmic. F) Average number of iterations each LP runsfor with and without caching over iterations of QPACE.

iterations and plateaus to a consistent 70 LP iterations. Thegraphs demonstrate that LP caching does in fact make a sig-nificant difference in running time. After the tenth iteration,caching consistently reduces the time per iteration from 1.5seconds down to 0.2 seconds. In the long run, LP cachingcontributes an order of magnitude speed boost.

5 Conclusion and ExtensionsQPACE finds all correlated equilibria of a stochastic gameorders of magnitude more efficiently than previous ap-proaches, while maintaining a guarantee of convergence.Further, no returned equilibria provides more than ε incen-tive to deviate. QPACE’s regular half-space representationand LP caching scheme allows QPACE to solve larger andmore complex games.

QPACE can easily be modified to produce sub-game per-fect equilibria by making the threat points

−→gt~ai variables in

(6) and constraining them in a similar way to the −→cu~ai; wethen employ the same trick described in section 3.3 to re-move the resulting quadratic constraints. QPACE can also be

extended to work in games of imperfect monitoring and im-perfect recall. In these games player’s don’t observe actionsso they can’t use threats and can’t choose different continu-ation points for each joint-action. Thus

−→gt~ai =

−→cu~ai and thevalues ofQ(s,~a)j are interdependent across actions, forcingequations (1) and (6) to become one large set of inequalities.

The empirical results suggest that a good estimate re-quires many fewer half-spaces than a full Euclidean ball. Astraight forward extension would be to start with only a fewhalf-spaces and dynamically add half-spaces as needed.

ReferencesAbreu, D. 1988. On the theory of infinitely repeated gameswith discounting. Econometrica 56(2):383–96.Amir, R. 2001. Stochastic games in economics and relatedfields: an overview. CORE Discussion Papers 2001060.Burkov, A., and Chaib-draa, B. 2010. An approximatesubgame-perfect equilibrium computation technique for re-peated games. CoRR abs/1002.1718.Cronshaw, M. B. 1997. Algorithms for finding repeatedgame equilibria. Computational Economics 10(2):139–68.Gordon, G. J. 2007. Agendas for multi-agent learning. Ar-tificial Intelligence 171(7):392 – 401.Greenwald, A., and Hall, K. 2003. Correlated-q learning. InProc. 20th International Conference on Machine Learning(ICML), 242–249.Horner, J.; Sugaya, T.; Takahashi, S.; and Vieille, N. 2010.Recursivemethods in discounted stochastic games: an algo-rithm for δ → 1 and a folk theorem. Econometrica.Hu, J., and Wellman, M. 2003. Nash q-learning for general-sum stochastic games. In Journal of Machine Learning Re-search 4:1039-1069.Judd, K. L.; Yeltekin, S.; and Conklin, J. 2003. Computingsupergame equilibria. Econometrica 71(4):1239–1254.Littman, M. L. 2001. Friend-or-foe Q-learning in general-sum games. In Proc. 18th International Conf. on MachineLearning (ICML), 322–328.Mac Dermed, L., and Isbell, C. 2009. Solving stochasticgames. In Advances in Neural Information Processing Sys-tems 22. 1186–1194.Murray, C., and Gordon, G. J. 2007. Multi-robot negotia-tion: Approximating the set of subgame perfect equilibria ingeneral-sum stochastic games. In Advances in Neural Infor-mation Processing Systems 19. 1001–1008.Myerson, R. B. 1991. Game Theory: Analysis of Conflict.Harvard University Press, Cambridge.Nguyen, K. C.; Alpcan, T.; and Basar, T. 2010. Stochasticgames for security in networks with interdependent nodes.CoRR abs/1003.2440.Wolf, D. M., and Arkin, A. P. 2003. Motifs, modulesand games in bacteria. Current Opinion in Microbiology6:125134.Zinkevich, M.; Greenwald, A.; and Littman, M. L. 2005.Cyclic equilibria in markov games. In Proceedings of Neu-ral Information Processing Systems.