Markov Decision Processes

AIMA: 17.1, 17.2 (excluding 17.2.3), 17.3

Search, planning, MDP

Search

Planning MDP

Factorized staterepresentations Uncertainty &

utility

Actions are unreliableNo hard goalsUtility depends on entire environment history

Planning and MDPs• In addition to actions having costs, we might have goals

with rewards, with the understanding that if you achieve a goal, you get the corresponding reward

• So now, the objective of planning is to find a plan that has the highest net benefit measured as the difference between the cumulative reward for the goals achieved and the cumulative cost of the actions used

• This problem is both easy (since an “empty” plan is a solution, just not a very good one) and hard (since now the “quality of the plan” in terms of its net benefit is more important)

• On top of this, we might also want to say that rewards are not limited to just goals achieved in the final state, but can also be gathered for visiting certain good states on the way

Example MDP

Markov decision process• A sequential decision problem for a fully

observable, stochastic environment with a Markovian transition model and additive rewards is called a MDP– A set of states– A set of actions in each state– A transition model– A reward function

P(j | i, a) = probability of doing action a in i leads to j

What does a solution look like?• The solution should tell the optimal

action to do in each state (called a “Policy”)– Policy is a function from states to

actions– Not a sequence of actions anymore

• because of the non-deterministic actions

– If there are |S| states and |A| actions that we can do at each state, then there are |A||S| policies

Optimal policies depend on rewards

R(s) = -0.04

Horizon & Policy• We said policy is a function from states to

actions, but we sort of lied• Best policy is non-stationary, i.e., depends on

how long the agent has to “live” – which is called “horizon”

• More generally, a policy is a mapping from <state, time-to-death> <action>– So, if we have a horizon of k, then we will have k

policies• If the horizon is infinite, then policies must all be

the same.. (So infinite horizon case is easy!)

Horizon & Policy

We will concentrate on infinite horizon problemsIn which the optimal policy is stationary

Infinite horizon finite horizon, k=3

Stationary preferences• Preferences between state sequences are

stationary:• If two sequences [s0,s1,s2,…] and [s0’,s1’,s2’,…]

begin with the same state (s0=s0’), then the two sequences should be preference-ordered the same way as the sequences [s1,s2,…] and [s1’,s2’,…]

• If you prefer future f1 to f2 starting tomorrow, you should prefer them the same way even if they start today

Utility of state sequence• Define utility of a sequence of states in terms of their

rewards– Assume “stationarity” of preferences– Then, only two reasonable ways to define Utility of a sequence

of states

𝛾≤1

The big picture

Compute the utilities of states

Compute the optimal policy

Utility of a state• The utility of a state is the expected utility of the

state sequences that might follow it• The state sequences depend on the policy that

is executed• If we let st be the state the agent is in after

executing policy π for t steps (note that st is a random variable), then we have

• The true utility of a state is

The expected sum of discounted rewards if the agent executes an optimal policy.

Different with R(s),The short-term reward

Utility of a state

The expected sum of discounted rewards if the agent executes an optimal policy.