Post on 08-Aug-2015
Sparse Binary Zero-Sum Games[ACML 2014]
David Auger1 Jialin Liu2 Sylvie Ruette3 David L. St-Pierre4
Olivier Teytaud2
1AlCAAP, Laboratoire PRiSM, Universite de Versailles Saint Quentin-en-Yvelines, France
2TAO, INRIA-CNRS-LRI, Universite Paris-Sud, France
3Laboratoire de Mathematiques, CNRS, Universite Paris-Sud, France
4Universite du Quebec a Trois-Rivieres, Canada
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 1 / 26
Thanks to reviewers for very fruitful comments.
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 2 / 26
Introduction
Two-person zero-sum game MK×K
Nash Equilibrium → O(K 2α) with α > 3
If the Nash is sparse → k × k submatrix
→ O(k3kK logK ) with probability 1− δ (provable)
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 3 / 26
Introduction
Two-person zero-sum game MK×K
Nash Equilibrium → O(K 2α) with α > 3
If the Nash is sparse → k × k submatrix
→ O(k3kK logK ) with probability 1− δ (provable)
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 3 / 26
Zero-sum matrix games
Game defined by matrix M
I choose (privately) i
Simultaneously, you choose j
I earn Mi ,j
You earn −Mi ,j
So this is zero-sum.
Or you earn 1−Mi ,j (so this is 1-sum, equivalent).
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 4 / 26
Nash Equilibrium
Nash Equilibrium (NE)
Zero-sum matrix game M
My strategy = probability distrib. on rows = x
Your strategy = probability distrib. on cols = y
Expected reward = xTMy
There exists x∗, y∗ such that ∀x , y ,
xTMy∗ ≤ x∗TMy∗ ≤ x∗TMy .
(x∗, y∗) is a Nash Equilibrium (no unicity).
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 6 / 26
Ok, I earn Mi ,j , you earn −Mi ,j
Nash: Ok I play i with probability x∗i
How to compute x*?
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 7 / 26
Ok, I earn Mi ,j , you earn −Mi ,j
Nash: Ok I play i with probability x∗i
How to compute x*?
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 7 / 26
Solving Nash
Solution 1: Linear Programming (LP)
1 M ← M + C so that it is positive (without loss of generality)
2 LP: find 0 ≤ u minimizing∑iui such that (MT ) · u ≥ 1
3 x∗ = u/∑iui
=⇒ classical, provably exact, polynomial time
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 8 / 26
Solving Nash
Solution 2: Approximate Nash Equilibrium
Approximate ε-NE
(x∗, y∗) such that
xTMy∗ − ε ≤ x∗TMy∗ ≤ x∗TMy + ε.
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 9 / 26
Solution 1: LP (comp. expensive)
Solution 2: Approximate Nash Equilibrium
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 10 / 26
Solution 1: LP (comp. expensive)
Solution 2: Approximate Nash Equilibrium
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 10 / 26
Computing approximate Nash Equilibrium
Assuming the matrix is of size K × K ...
LP (see reduction from Nash to linear programming in[Von Stengel (2002)]): O(K 2α) with 3 < α ≤ 4
[Grigoriadis and Khachiyan(1995)]:
ε-Nash with expected time O(K log(K)ε2 ), i.e. less than the size of the
matrix!Parallel : O( log2(K)
ε2 ) if using Klog(K) processors
Other algorithms: similar complexity, approximate solution + fixedtime with probability 1− δ
EXP3 ([Auer et al.(1995)])Inf ([Audibert and Bubeck(2009)])
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 11 / 26
Computing approximate Nash Equilibrium
Assuming the matrix is of size K × K ...
LP (see reduction from Nash to linear programming in[Von Stengel (2002)]): O(K 2α) with 3 < α ≤ 4
[Grigoriadis and Khachiyan(1995)]:
ε-Nash with expected time O(K log(K)ε2 ), i.e. less than the size of the
matrix!Parallel : O( log2(K)
ε2 ) if using Klog(K) processors
Other algorithms: similar complexity, approximate solution + fixedtime with probability 1− δ
EXP3 ([Auer et al.(1995)])Inf ([Audibert and Bubeck(2009)])
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 11 / 26
Other tools 1: Hadamard determinant
Hadamard determinant bound([Hadamard(1893)], [Brenner and Cummings(1972)])
Given matrix Mk×k with coefficients in {−1, 0, 1}, then M has
determinant at most kk2 , i.e.
| detM| ≤ kk2 .
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 12 / 26
Other tools 2: Linear programming
Solve
min ax
Mx ≤ c
x ∈ Rd
If there is a finite optimum, then there is a finite optimum x suchthat, for some E with |E | = d ,
∀i ∈ E , Mix = cithe Mi for i in E are linear independent(=⇒ i.e. d lin. indep. constraints are active)
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 13 / 26
Why is this relevant ?
Nash = solution of linear programming problem
x∗: Nash Equilibrium of MK×K
Let us assume that x∗ is unique and has at most k non-zerocomponents (sparsity)
⇒ x∗ = also NE of a k × k submatrix: M ′k×k⇒ x∗ = solution of LP in dimension k⇒ x∗ = solution of k lin. eq. with coefficients in {−1, 0, 1}⇒ x∗ = inv-matrix ∗ vector⇒ x∗ = obtained by “cofactors / det matrix”
⇒ x∗ has denominator at most kk2
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 14 / 26
Why is this relevant ?
Nash = solution of linear programming problem
x∗: Nash Equilibrium of MK×K
Let us assume that x∗ is unique and has at most k non-zerocomponents (sparsity)⇒ x∗ = also NE of a k × k submatrix: M ′k×k⇒ x∗ = solution of LP in dimension k⇒ x∗ = solution of k lin. eq. with coefficients in {−1, 0, 1}⇒ x∗ = inv-matrix ∗ vector⇒ x∗ = obtained by “cofactors / det matrix”
⇒ x∗ has denominator at most kk2
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 14 / 26
How to realise ?
Under assumption that the Nash is sparse
x∗ is rational with “small” denominator
So let us compute an ε-Nash (sublinear time!)
And let us compute its closest approximation with “smalldenominator” (Hadamard)
variants for ε-Nash =⇒ exact Nash
Rounding: switch to closest approximation
Truncation: remove small components and work on the remainingsubmatrix (exact solving)
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 15 / 26
Evil in the details
||y − y∗||∞ ≥ ε does not imply V (y) ≥ V (y∗) + ε;
indeed V (y) ≥ V (y∗) + ||y−y∗||∞k
k2
Results : (if Grigoriadis)
For a K × K matrix with Nash k-sparseExact solution in time O(poly(k) + (K logK )k3k) withtruncation-algorithm
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 16 / 26
Experimental results: two card games
Previous results: ingaming of Urban Rivals
New results: metagaming of Pokemon
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 17 / 26
Ingaming results (Urban Rivals)
Previous work: [Flory and Teytaud(2011)], implementation ofTruncated-EXP3, without proof
Urban Rivals AI= Monte Carlo Tree Search([Coulom (2006)]),using zero-sum matrix gamesas a key component
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 18 / 26
Ingaming results (Urban Rivals)
Previous work: [Flory and Teytaud(2011)], implementation ofTruncated-EXP3, without proof
Results don’t look impressive (∼ 56%), but the game is highlyrandomized =⇒ Reaching 55% is far from being negligible
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 19 / 26
New experiments
Test on Pokemon Deck choice (“metagaming”)
Based on EXP3+truncation
Various versions of EXP3 (6= parameters)
Code available https://www.lri.fr/~teytaud/games.html
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 20 / 26
New experiments
With a poorly tuned EXP3 : truncation brings a huge improvement
100 101 102 103 1040.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
TEXP3 vs EXP3
100 101 102 103 1040.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
TEXP3 vs UniformEXP3 vs Uniform
Figure: Performance in terms of budget T with a poorly tuned EXP3 for thegame of Pokeman using 2 cards.
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 21 / 26
New experiments
With a well-tuned EXP3, truncation brings a significant improvement
100 101 102 103 1040.5
0.51
0.52
0.53
0.54
0.55
0.56
0.57
0.58
TEXP3 vs EXP3
100 101 102 103 1040.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
TEXP3 vs UniformEXP3 vs Uniform
Figure: Performance in terms of budget T with a well-tuned EXP3 for the gameof Pokeman using 2 cards.
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 22 / 26
Conclusions & further work
Proved small improvement, experimentally big improvement.Improving the bound ?
We don’t know k (sparsity level). Adaptive algorithms ?
Proved only with unique Nash (x∗, y∗). Necessary ?
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 23 / 26
Jean-Yeves Audibert and Sebastien Bubeck.
Minimax policies for adversarial and stochastic bandits.In 22th annual conference on learning theory, 2009.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E. Schapire.
Gambling in a rigged casino: the adversarial multi-armed bandit problem.In Proceedings of the 36th Annual Symposium on Foundations of Computer Science. IEEE Computer Society Press, 1995.
Remi Coulom (2006).
Efficient selectivity and backup operators in Monte-Carlo tree search.In Computers and games, 2006.
Joel Brenner and Larry Cummings.
The Hadamard maximum determinant problem.In Amer. Math. Monthly, 1972.
Sebastien Flory and Olivier Teytaud.
Upper confidence trees with short term partial information.In Procedings of EvoGames, 2011.
Michael D. Grigoriadis and Leonid G. Khachiyan.
A sublinear-time randomized approximation algorithm for matrix games.In Operations Research Letters, 1995.
Jacques Hadamard.
Resolution d’une question relative aux determinants.In Bull. Sci. Math., 1893.
Bernhard Von Stengel.
Computing equilibria for two-person games.In Handbook of game theory with economic applications, 2002.
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 24 / 26
Thank you for your attention !
David Auger
David L. St-Pierre
Sylvie Ruette
Olivier Teytaud
Jialin LIU (INRIA-TAO) Sparse Binary Zero-Sum Games 25 / 26