Generalized and Bounded Policy Iteration for Finitely Nested Interactive POMDPs:

Scaling Up

Ekhlas Sonu, Prashant DoshiDept. of Computer Science

University of Georgia

AAMAS 2012

OverviewWe generalize Bounded Policy Iteration for POMDP to the multiagent decision making framework of Interactive POMDPWe discuss the challenges associated with this generalizationSubstantial scalability achieved using the generalized approach

Introduction: Interactive POMDPInteractive POMDP (Gmytrasiewicz&Doshi,05):

Generalization of POMDP to multiagent settingsApplications

Money Laundering (Ng et al.,10) Lemonade stand game (Wunder et al.,11) Modeling human behavior (Doshi et al.,10), and more…

Differs from Dec-POMDPDec-POMDP: Team of agentsI-POMDP: Individual agent in presence of other agents – cooperative, competitive or neutral settings

Introduction: I-POMDP(Finitely-nested and 2 agents)

I-POMDPi,l =<ISi,l, A, Wi, Ti, Oi, Ri, γ>

Physical States(S)

i

ai/Ti(s, ai, aj, s’)

oi/Oi(s’, ai, aj, oi) , Ri (s, ai, aj)

j

oj/Oj(s’, ai, aj, oj) , Rj (s, ai, aj)

aj/Tj(s, ai, aj, s’)

ISi,l = S X Qj,l-1 S: Set of physical statesQj,l-1 : Set of intentional

models of j at level l-1

A = Ai X Aj

Wi: set of observations of iTi: S X Ai X Aj DSOi: S X Ai X Aj DWi

Ri: S X Ai X Aj R

Interactive state

I-POMDP Belief Update and Value Function

Belief Update:An agent must predict the other agent’s actions by anticipating its updated beliefs over time. Therefore belief update consists of

Updating distribution over physical states: Transition Function, Observation Function of agent i

Updating distribution over dynamic models: Belief update of other agents and its observation function

Value Function:Must incorporate the I-POMDP belief update in computing long term rewards

Solving I-POMDP (Related Work)Previous work: Value iteration algorithms

Interactive particle filtering (I-PF) (Doshi&Gmytrasiewicz,09) nested particle filter: sampled recursive representation of agent’ nested belief

Interactive point-based value iteration (I-PBVI) (Doshi&Perez,08)point based domination check

Iteratively apply Backup Operator:Expensive operatorScale only to toy problems

Over multiple time steps:Curse of historyCurse of dimensionality

Phy. St. (S)b = D(ISi,l)

BackgroundPolicy Iteration

Class of solution algorithms – search policy spaceExponential growth in solution size

Bounded Policy Iteration (Poupart&Boutilier,03)

Fixed solution size (controlled growth)Applied in POMDP & Dec-POMDP

Dec-BPI (Bernstein,Hansen&Zilberstein,05) -- optional correlation device may not be feasible in non-cooperative settings

Contribution: We present the first policy iteration algorithm (approximate) for I-POMDPs : generalization of BPIShow scalability to larger problems

Policy RepresentationPossible representation of policy

Node actionEdge obs

Finite State Controllers(Hansen, 1998)Tree Representation

Node has an infinite horizon policy rooted at itNode has a value vector associated with it which is a linear vector over the entire belief spaceBeliefs are mapped to a node (n) that optimizes the expected reward from that belief:

i.e. argmaxn b ∙ Vn

Finite State ControllerA finite state controller may be defined as:

where:is the set of nodes in the FSC of agent iis the set of edge labels (Wi)

Let:partitions the entire belief space

Policy IterationStarting with an initial controller, iterate over two steps until convergence:

Policy Evaluation:Evaluate Vn for each nodeSolve system of linear equations

Policy Improvement:Construct a better controllerPossibly by adding new nodes

Policy Improvement (Hansen,98)

Apply Backup operator, i.e. construct new nodes with all possible values of action and transition on observation

|A||N||W| new nodesAdd them to the controller

Prune all dominated nodesDrawback: Leads to exponential growth in controller size

V

0 1P(s)Example of policy iteration for a POMDP

Bounded Policy Iteration (BPI) (Poupart&Boutilier,03)

Instead of performing a complete back up, replace a node with a better node

Linear program for partial backupNew node is a convex combination of two backed up nodesChanges in controller: e

:stochastic action policy:stochastic observation policy

Local OptimaThis form of policy improvement is prone to converging to local optimaWhen all nodes are tangents to backed up nodes: e = 0, no improvementEscape technique suggested by Poupart & Boutilier (2003) in BPI

V

0 1P(s)

I-POMDP Generalization: Nested ControllersNested Controllers: Analogous to nested beliefs

Embed recursive reasoning

Starting from level 0 upwards, for each level l, construct a Finite state controller for each frame of each agent ( )

For convenience of representation, let’s assume two agents and each one frame for an agent at each level

Agent i’s level 2 controller:

Agent j’s level 1 controller:

Agent i’s level 0 controller:

Interactive BPI: Policy EvaluationCompute the value vector of each node using the estimate of other agent’s model by solving a system of linear equations:For each ni,l, and interactive state, is=(s, nj,l-1), solve:

I-BPI: Policy Improvement

New vector dominates old vector by e and hence replaces it

e

V

0 1P(s)

Pick a node (ni,l) and perform a partial backup using LP to construct another node (n’i,l) that pointwise dominates ni,l by some e > 0

I-BPI: Policy ImprovementPick a node (ni,l) and perform a partial backup using LP to construct another node that pointwise dominates ni,l by some e > 0

Objective Function: maximize eVariables: Constraints:

Escaping Local Optima

V

0 1

P(s)

bT bR1

bR2

Analogous to escaping for POMDPs

Algorithm: I-BPI1. Starting from Level 0 up to Level l, construct a 1 node controller

for each level with a random action and transition to itself. 2. Reformulate interactive state space and evaluate

L0

L1

Ll

Time

.

.

.

Algorithm: I-BPI3. Starting from Level 0 up to Level l, perform 1 step of back up

operator. Max |Ai(j)| nodes

L0

L1

Ll

Time

.

.

.

Algorithm: I-BPI4. Starting from Level 0 up to Level l, reformulate IS space, perform

policy evaluation followed by policy improvement at each level

L0

L1

Ll

Time

.

.

.

Algorithm: I-BPI5. Repeat step 4 until convergence6. If converged, push nested controller out of local optima by

adding new nodes

L0

L1

Ll

Time

.

.

.

Evaluation

Runtime for algorithm and the average rewards from simulations * Represents expected rewards obtained from vectors

AUAV: 81 states, 5 actions, 4 observationsMoney Laundering: 99 States, 11 actions, 9 ObservationsScales to larger problems...

Evaluation

Simulations results for multiagent tiger problem showing results obtained by simulating performance of agent controllers of various sizes for Levels 1 – 4

DiscussionAdvantages of I-BPI

Is significantly quicker and scales to large problems (100s of states, tens of actions and observations)Mitigates curse of history and curse of dimensionalityImproved solution quality

LimitationsProne to local optima

Escape technique may not work for certain local optima

Not entirely free from curses of history and dimensionality

Future WorkScale to even larger problems and more agentsMealy machine implementation for controllers (Amato et al. 2011)

Thank you…

Poster #731 today at 16:00-17:00 (Panel 98)

Acknowledgement:This research is partially supported by an NSF CAREER grant, #IIS-0845036

Policy ImprovementApply Backup operator, i.e. construct new nodes with all possible values of action and transition to nodes in current controller

|A||N||Z| new nodesAdd them to the controller

|A|

Z1 Z2Z|Z|

|N||N|

|N|

Introduction: POMDPPOMDP: Framework for optimal sequential decision making under uncertainty in single agent settings

<S, A, W, T, O, R, g >

Physical States(S)

a/T(s, a, s’)

z/O(s’, a, z) , R(s, a)

b = D(S)

S: set of statesA: set of actionsZ: set of observations

T: S X A DSO: S X A DZR: S X A R

Objective is to find a policy p that maximizes long term expected rewards:ER = Immediate Reward + discounted future reward

•Agent maintains a belief (b) over physical states•Policy p : b A

g: discount factorh: Horizon

Future WorkExtend approach to problems with even larger dimensions

Extend to problems with more than two agents

Mealy machine implementation of finite state controllers (Amato, et.al; 2011)

I-POMDP Belief Update and Value FunctionBelief Update:

An agent must predict the other agent’s actions by anticipating its updated beliefs over time. Therefore belief update consists of

Updating distribution over physical states: Transition Function, Observation Function of agent i

Updating distribution over dynamic models: Belief update of other agents and its observation function

Value Function:

Solving I-POMDP (Related Work)Previous work: Value iteration algorithms

I-PF (Doshi, Gmytrasiewicz; 2009): particle filter: sampled recursive representation of agent’ nested belief

I-PBVI (Doshi, Perez; 2008):point based domination check

Iteratively apply Backup Operator:Expensive operator

Over multiple time steps:Curse of historyCurse of dimensionality

Phy. St. (S)

s, a/T(s, a, s’)

s’/O(s’, a, z), R(s, a)

b = D(ISi,l)

I-POMDP Generalization: Nested Controllers

Embed recursive reasoningStarting from level 0 upwards, for each level l, construct a Finite state controller for each frame of each agent ( )

For convenience of representation, let’s assume two agents and each one frame for an agent at each level

L 0:

L 1:

.

.

.

.

.

.

.

.

.

.

.

.

L l:

I-POMDP Generalization: Nested Controllers

Embed recursive reasoningStarting from level 0 upwards, for each level l, construct a Finite state controller for each frame of each agent ( )

For convenience of representation, let’s assume two agents and each one frame for an agent at each level