Post on 16-Dec-2015
Krishnendu Chatterjee 1
Graph Games with Reachabillity Objectives: Mixing Chess, Soccer and Poker
Krishnendu Chatterjee
5th Workshop on Reachability Problems,
Genova, Sept 30, 2011
Krishnendu Chatterjee 2
Games on Graphs
Games on graphs.
History Zermelo’s theorem about Chess in 1913 From every configuration
Either player 1 can enforce a win. Or player 2 can enforce a win. Or both players can enforce a draw.
Krishnendu Chatterjee 3
Chess: Games on Graph
Chess is a game on graph. Configuration graph.
Krishnendu Chatterjee 4
Graphs vs. Games
Two interacting players in games: Player 1 (Box) vs Player 2 (Diamond).
Krishnendu Chatterjee 5
Game Graph
Krishnendu Chatterjee 6
Game Graphs
A game graph G= ((S,E), (S1, S2)) Player 1 states (or vertices) S1 and similarly player 2
states S2, and (S1, S2) partitions S.
E is the set of edges. E(s) out-going edges from s, and assume E(s) non-
empty for all s.
Game played by moving tokens: when player 1 state, then player 1 chooses the out-going edge, and if player 2 state, player 2 chooses the outgoing edge.
Krishnendu Chatterjee 7
Game Example
Krishnendu Chatterjee 8
Game Example
Krishnendu Chatterjee 9
Game Example
Krishnendu Chatterjee 10
Strategies
Strategies are recipe how to move tokens or how to extend plays. Formally, given a history of play (or finite sequence of states), it chooses a probability distribution over out-going edges. ¾: S* S1 D(S).
¼: S* S2 ! D(S).
Krishnendu Chatterjee 11
Strategies Strategies are recipe how to move tokens or how to extend plays. Formally, given a
history of play (or finite sequence of states), it chooses a probability distribution over out-going edges.
¾: S* S1 ! D(S).
History dependent and randomized.
History independent: depends only current state (memoryless or positional). ¾: S1 ! D(S)
Deterministic: no randomization (pure strategies). ¾: S* S1 ! S
Deterministic and memoryless: no memory and no randomization (pure and memoryless and is the simplest class).
¾: S1 ! S
Same notations for player 2 strategies ¼.
Krishnendu Chatterjee 12
Objectives
Objectives are subsets of infinite paths, i.e., Ã µ S!.
Reachability: there is a set of good vertices (example check-mate) and goal is to reach them. Formally, for a set T if vertices or states, the objective is the set of paths that visit the target T at least once.
Krishnendu Chatterjee 13
Applications: Verification and Control of Systems
Verification and control of systems
Environment
Controller
M satisfies property (Ã)
E
C
Krishnendu Chatterjee 14
Applications: Verification and Control of Systems
Verification and control of systems
Question: does there exists a controller that against all environment ensures the property.
M satisfies property (Ã)EC || ||
Krishnendu Chatterjee 1616
-synthesis [Church, Ramadge/Wonham, Pnueli/Rosner]
-model checking of open systems
-receptiveness [Dill, Abadi/Lamport]
-semantics of interaction [Abramsky]
-non-emptiness of tree automata [Rabin, Gurevich/ Harrington]
-behavioral type systems and interface automata [deAlfaro/ Henzinger]
-model-based testing [Gurevich/Veanes et al.]
-etc.
Game Models Applications
Krishnendu Chatterjee 17
Reachability Games
Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X.
X
T
Krishnendu Chatterjee 18
Reachability Games
Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X.
X
T
Krishnendu Chatterjee 19
Reachability Games
Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X.
X
T
Krishnendu Chatterjee 20
Reachability Games
Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X.
Fix-point
X
T
Krishnendu Chatterjee 21
Reachability Games
Winning set for a partition: Determinacy Player 1 wins: then no matter what player 2 does,
certainly reach the target. Player 2 wins: then no matter what player 1 does, the
target is never reached.
Memoryless winning strategies.
Can be computed in linear time [Beeri 81, Immerman 81].
Krishnendu Chatterjee 22
Chess Theorem
Zermelo’s Theorem
Win1Win2
Both draw
Krishnendu Chatterjee 23
Game Graphs Till Now
Game graphs we have seen till now
Many rounds (possibly infinite).
Turn-based.
Krishnendu Chatterjee 24
Simultaneous Games
Theory of rational behavior as game theory von Neumann- Morgenstern games Matrix zero-sum games
R P S
R (0,0) (-1,1) (1,-1)
P (1,-1) (0,0) (-1,1)
S (-1,1) (1,-1) (0,0)
Krishnendu Chatterjee 25
Simultaneous Games
Theory of rational behavior as game theory von Neumann- Morgenstern games Matrix zero-sum games
R P S
R (0,0) (-1,1) (1,-1)
P (1,-1) (0,0) (-1,1)
S (-1,1) (1,-1) (0,0)
Krishnendu Chatterjee 26
Simultaneous Games
Example: Prisoners dilemma. Another example.
R L C
R (1,-1) (-1,1) (-1,1)
L (-1,1) (1,-1) (-1,1)
C (-1,1) (-1,1) (1,-1)
Krishnendu Chatterjee 27
Simultaneous Games
Example: Prisoners dilemma. Another example.
R L C
R (1,-1) (-1,1) (-1,1)
L (-1,1) (1,-1) (-1,1)
C (-1,1) (-1,1) (1,-1)
Krishnendu Chatterjee 28
Simultaneous Games
Another example: Penalty shoot-out (Soccer)
R L C
R (1,-1) (-1,1) (-1,1)
L (-1,1) (1,-1) (-1,1)
C (-1,1) (-1,1) (1,-1)
Krishnendu Chatterjee 29
Chess Vs. Soccer (Penalty)
Chess: Turn-based Possibly infinite rounds
Theory of simultaneous games (Soccer) Concurrent One-shot (one-round)
Mix chess and soccer Concurrent games on graphs
Krishnendu Chatterjee 30
Mixing Chess and Soccer: Concurrent Graph Games
Krishnendu Chatterjee 31
Concurrent Game Graphs
A concurrent game graph is a tuple G =(S,M,¡1,¡2,±)
• S is a finite set of states.
• M is a finite set of moves or actions.
• ¡i: S ! 2M n ; is an action assignment function that assigns the non-empty set ¡i(s) of actions to player i at s, where i 2 {1,2}.
• ±: S £ M £ M ! S, is a transition function that given a state and actions of both players gives the next state.
Krishnendu Chatterjee 32
An Example: Snow-ball Game
s Rrun, waithide, throw
hide, wait
run, throw[Everett 57]
Run
Hide
Throw Wait
Krishnendu Chatterjee 33
New Solution Concepts
Sure winning for turn-based.
New solution concepts
Almost-sure winning.
Limit-sure winning.
Krishnendu Chatterjee 34
Almost-sure Winning Example
s Rhead, headtail, tail
head, tailtail, head
Almost-sure winning strategy: say head and tail with probability ½.Randomization is necessary.
Krishnendu Chatterjee 35
Concurrent reachability games: limit-sure
s Rrun, waithide, throw
hide, wait
run, throw[Everett 57]
Move Probabilityrun qhide 1-q (q>0)
Win at s with probability1-q, for all q > 0.
Run
Hide
Throw Wait
Krishnendu Chatterjee 36
Concurrent reachability games: limit-sure
s Rrun, waithide, throw
hide, wait
run, throw
Run
Hide
Throw Wait
[Everett 57]
Move Probabilityrun qhide 1-q (q>0)
Win at s with probability1-q, for all q > 0.
w = 0 1 1
Player 1 cannot achieve w(s) = 1, only w(s) = 1-q for all q > 0.
Krishnendu Chatterjee 39
Results for Concurrent Reachability Games
Sure winning: Deterministic memoryless sufficient. Linear time.
Almost-sure winning: Randomization is necessary. Randomized memoryless is sufficient. Quadratic time algorithm.
Limit-sure winning: Randomization is necessary. Randomized memoryless is sufficient. Quadratic time algorithm.
Results from [dAHK98, CdAH06, CdAH09]
Krishnendu Chatterjee 40
Games Till Now
Turn-based graph games
Concurrent graph games Applications: again verification and synthesis with
synchronous interaction.
Both these games are perfect-information games. Players know the precise state of the game.
The game of Poker: players play but do not know the perfect state of the game.
Krishnendu Chatterjee 41
Summary: Theory of Graph Games
Winning Mode/ Game Graphs
Sure Almost-sure Limit-sure
Turn-based Games (CHESS)
Linear time (PTIME-complete)
Linear-time (PTIME-complete)
Linear-time(PTIME-complete)
Concurrent Games (CHESS+ SOCCER)
Linear time (PTIME-complete)
Quadratic time (PTIME-complete)
Quadratic time(PTIME-complete)
Partial-information Games(CHESS + SOCCER+ POKER)
Krishnendu Chatterjee 42
Mixing Chess and Poker: Partial-information Graph Games
Krishnendu Chatterjee 43
Why Partial-information
Perfect-information: controller knows everything about the system. This is often unrealistic in the design of reactive
systems because • systems have internal state not visible to controller (private
variables)• noisy sensors entail uncertainties on the state of the game
Partial-observationHidden variables = imperfect information.
Sensor uncertainty = imperfect information.
Krishnendu Chatterjee 44
Partial-information Games
A PIG G =(L, A, , O) is as follows L is a finite set of locations (or states). A is a finite set of input letters (or actions). µ L £ A £ L non-deterministic transition relation that
for a state and an action gives the possible next states.
O is the set of observations and is a partition of the state space. The observation represents what is observable.
Perfect-information: O={{l} | l 2 L}.
Krishnendu Chatterjee 45
PIG: Example
a,b
a
b a
b
Krishnendu Chatterjee 46
New Solution Concepts
Sure winning: winning with certainty (in perfect information setting determinacy).
Almost-sure winning: win with probability 1.
Limit-sure winning: win with probability arbitrary close to 1.
We will illustrate the solution concepts with card games.
Krishnendu Chatterjee 47
Card Game 1
Step 1: Player 2 selects a card from the deck of 52 cards and moves it from the deck (player 1 does not know the card).
Step 2: Step 2 a: Player 2 shuffles the deck. Step 2 b: Player 1 selects a card and view it. Step 2 c: Player 1 makes a guess of the secret card or
goes back to Step 2 a.
Player 1 wins if the guess is correct.
Krishnendu Chatterjee 48
Card Game 1
Player 1 can win with probability 1: goes back to Step 2 a until all 51 cards are seen.
Player 1 cannot win with certainty: there are cases (though with probability 0) such that all cards are not seen. Then player 1 either never makes a guess or makes a wrong guess with positive probability.
Krishnendu Chatterjee 49
Card Game 2
Step 1: Player 2 selects a new card from an exactly same deck and puts is in the deck of 52 cards (player 1 does not know the new card). So the deck has 53 cards with one duplicate.
Step 2: Step 2 a: Player 2 shuffles the deck. Step 2 b: Player 1 selects a card and view it. Step 2 c: Player 1 makes a guess of the secret duplicate
card or goes back to Step 2 a.
Player 1 wins if the guess is correct.
Krishnendu Chatterjee 50
Card Game 2
Player 1 can win with probability arbitrary close to 1: goes back to Step 2 a for a long time and then choose the card with highest frequency.
Player 1 cannot win probability 1, there is a tiny chance that not the duplicate card has the highest frequency, but can win with probability arbitrary close to 1, (i.e., for all ² >0, player 1 can win with probability 1- ², in other words the limit is 1).
Krishnendu Chatterjee 51
Sure winning for Reachability
Result from [Reif 79]
Memory is required.
Exponential memory required.
Subset construction: what subsets of states player 1 can be. Reduction to exponential size turn-based games.
EXPTIME-complete.
Krishnendu Chatterjee 53
Almost-sure winning for Reachability
Result from [CDHR 06, CHDR 07]
Standard subset construction fails: as it captures only sure winning, and not same as almost-sure winning.
More involved subset construction is required.
EXPTIME-complete.
Krishnendu Chatterjee 54
Summary: Theory of Graph Games
Winning Mode/ Game Graphs
Sure Almost-sure Limit-sure
Turn-based Games (CHESS)
Linear time (PTIME-complete)
Linear-time (PTIME-complete)
Linear-time(PTIME-complete)
Concurrent Games (CHESS+ SOCCER)
Linear time (PTIME-complete)
Quadratic time (PTIME-complete)
Quadratic time(PTIME-complete)
Partial-information Games(CHESS + SOCCER+ POKER)
EXPTIME-complete EXPTIME-complete
Krishnendu Chatterjee 55
Limit-sure winning for Reachability
Limit-sure winning for reachability is undecidable [GO 10, CH 10].
Reduction from the Post-correspondence problem (PCP).
Krishnendu Chatterjee 56
Mixing Chess, Soccer and Poker
Partial-information concurrent games
Concurrency can be obtained for free (polynomial reduction) for partial-information games.
So all the results for partial-information turn-based games also hold for partial-information concurrent games.
Krishnendu Chatterjee 57
Summary: Theory of Graph Games
Winning Mode/ Game Graphs
Sure Almost-sure Limit-sure
Turn-based Games (CHESS)
Linear time (PTIME-complete)
Linear-time (PTIME-complete)
Linear-time(PTIME-complete)
Concurrent Games (CHESS+ SOCCER)
Linear time (PTIME-complete)
Quadratic time (PTIME-complete)
Quadratic time(PTIME-complete)
Partial-information Games(CHESS + SOCCER+ POKER)
EXPTIME-complete EXPTIME-complete Undecidable.
Krishnendu Chatterjee 58
Strategy Complexity
Krishnendu Chatterjee 59
Classes of strategies
rand. action-invisible
pure
rand. action-visible Classification
according to the power of strategies
Krishnendu Chatterjee 60
Classes of strategies
rand. action-invisible
pure
rand. action-visible Classification
according to the power of strategies
Poly-time reduction from decision problem of rand. act.-vis. to rand. act.-inv.
Krishnendu Chatterjee 61
Known results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (belief)
[CDHR’06(remark), GS’09]
exponential (belief) [GS’09]
pure ? ? ?
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure ? ? ?
Krishnendu Chatterjee 62
Beliefs
• Belief is sufficient.
• Randomized action invisible or visible almost same.
• The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case.
Three prevalent beliefs:
Krishnendu Chatterjee 63
Pure Strategies
• Belief is sufficient.
Proofs• Doubts.
Belief
Krishnendu Chatterjee 64
Pure Strategies
• Belief is sufficient.
Proofs• Doubts
Lesson: Doubt your belief and believe in your doubts!!! See the unexpected.
Belief
Krishnendu Chatterjee 65
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (belief)
[CDHR’06(remark), GS’09]
exponential (belief) [GS’09]
pure ? ? ?
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure ? ? ?
Krishnendu Chatterjee 66
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (more than belief)
exponential (belief) [GS’09]
pure exponential (more than belief) ? ?
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure exponential (more than belief) ? ?
Krishnendu Chatterjee 67
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (more than belief)
exponential (belief) [GS’09]
pure exponential (more than belief) ? ?
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure exponential (more than belief) ? ?
Krishnendu Chatterjee 68
Pure Strategies: Player 1 Perfect, Player 2 Partial (positive)
Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless
Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless
Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless
Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential
Krishnendu Chatterjee 69
Pure Strategies: Player 1 Perfect, Player 2 Partial (positive)
Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless
Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless
Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless
Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential
Pl1 Perfect, Pl 2 Partial: Stochastic, Pure.
Add probabilityRestrict to pure
Pl 2 less informedPl 1 more informed, Pl 2 less informed
Krishnendu Chatterjee 70
Pure Strategies: Player 1 Perfect, Player 2 Partial (positive)
Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless
Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless
Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless
Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential
Pl1 Perfect, Pl 2 Partial: Stochastic, Pure. Non-elementary complete
Add probabilityRestrict to pure
Pl 2 less informedPl 1 more informed, Pl 2 less informed
Krishnendu Chatterjee 71
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (more than belief)
exponential (belief) [GS’09]
pure exponential (more than belief)
non-elementarycomplete
?
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure exponential (more than belief)
non-elementarycomplete
?
Krishnendu Chatterjee 72
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (more than belief)
exponential (belief) [GS’09]
pure exponential (more than belief)
non-elementarycomplete
?
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure exponential (more than belief)
non-elementarycomplete
?
Player 1 wins from more states, but needs more memory !
Krishnendu Chatterjee 73
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (more than belief)
exponential (belief) [GS’09]
pure exponential (more than belief)
non-elementarycomplete
finite (at least non-elementary)
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure exponential (more than belief)
non-elementarycomplete
finite (at least non-elementary)
Krishnendu Chatterjee 74
Player 1 perfect, player 2 partial
• Win from more places.
• Winning strategy is very hard to implement.
Information is useful, but ignorance is bliss !!!
More information:
Krishnendu Chatterjee 75
Reductions for equivalence
Equivalence of the decision problems for almost-sure reachwith pure strategies and rand. act.-inv. strategies• Reduction of rand. act.-inv. to pure choice of a subset of actions (support of prob. dist.)
• Reduction of pure to rand. act.-inv. (holds for almost-sure only)
It follows that the memory requirements for pure hold for rand. act.-inv. as well !
Krishnendu Chatterjee 76
New results
Almost-sureplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis.
exponential (belief) [CDHR’06]
memoryless[BGG’09]
exponential (belief) [BGG’09]
rand. act.-inv.
exponential (more than belief)
finite (at least non-elementary)
pure exponential (more than belief)
non-elementarycomplete
finite (at least non-elementary)
Reachability - Memory requirement (for player 1)
Positiveplayer 1 partialplayer 2 perfect
player 1 perfect
player 2 partial
2-sidedboth partial
rand. act.-vis. memoryless memoryless memoryless
rand. act.-inv. memoryless memoryless memoryless
pure exponential (more than belief)
non-elementarycomplete
finite (at least non-elementary)
Krishnendu Chatterjee 77
Beliefs
• Belief is sufficient.
• Randomized action invisible or visible almost same.
• The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case.
Three prevalent beliefs:
Belief Fails!
[CD11] Chatterjee, Doyen. Partial-Observation Stochastic Games: How to Win when Belief Fails. CoRR abs/1107.2141, July 2011.
Krishnendu Chatterjee 78
The Message
Play Chess; Play Soccer;
But stay away from Poker !!!
Krishnendu Chatterjee 79
Conclusion Theory of graph games
Turn-based, concurrent, and partial-information games. Different solution concepts and different complexity. Several algorithmic questions open.
Partial information games
Problem with clear practical motivation.
Challenging to establish the right frontier of complexity.
Important generalization of perfect-information games.
Unfortunately, undecidable and also high complexity.
Current research: identifying decidable and more efficient sub-classes.
Krishnendu Chatterjee 80
Collaborators
Luca de Alfaro
Laurent Doyen
Thomas A. Henzinger
Jean-Francois Raskin
Krishnendu Chatterjee 81
Thank you !
Questions ?
The end