1

Chapter 4Discrete time Markov Chain

Learning objectives :• Introduce discrete time Markov Chain• Model manufacturing systems using Markov Chain• Able to evaluate the steady-state performances

Textbook :C. Cassandras and S. Lafortune, Introduction to Discrete

Event Systems, Springer, 2007

2

Plan

• Basic definitions of discrete time Markov Chains • Classification of Discrete Time Markov Chains• Analysis of Discrete Time Markov Chains

3

Basic definitions

of discrete time Markov chains

4

Discrete Time Markov Chain (DTMC)

Definition : a stochastic process with discrete state space and discrete time {Xn, n > 0} is a discrete time Markov Chain (DTMC) iff

P[Xn+1 = j Xn = in, ..., X0 = i0] = P[Xn+1 = j Xn = in] = pij(n)

In a DTMC, the past history impacts on the future evolution of the system via the current state of the system

pij(n) is called transition probability from state i to state j at time n.

5

Discrete Time Markov Chain (DTMC)

Stochastic process

Discrete events

Continuous event

Discrete time

Continuous time

Memoryless

A DTMC is a discrete time and memoriless discrete event stochastic process.

6

Example: a mouse in a maze (老鼠在迷宫 )

Which stochastic process can be used to represent the position of the mouse at time t?Under which assumptions, the system can be represented by a discrete time Markov chain?

1 2

3 45

0

start

exit

7

Example: a mouse in a maze

• Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited

• Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi-probably.

1 2

3 45

0

start

exit

1

1/2

1/2

1/2

11/4

1/4

1/2

1/41/4

1

8

Homogenuous DTMC

• A DTMC is said homogenuous iff its transitions probabilities do not depend on the time n, i.e.

P[Xn+1 = j Xn = i] = P[X1 = j X0 = i] = pij

• A homogenuous DTMC is then defined by its transition matrix P =[pij]i,jE

9

What is the transition matrix of the process?

• Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited

• Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi-probably.

1 2

3 45

0

start

exit

1

1/2

1/2

1/2

11/4

1/4

1/2

1/41/4

1

10

Stochastic Matrix

A square matrix is said stochastic iff

• all entries are non negative

• each line sums to 1

Properties:

• A transition matrix is a stochastic matrix

• If P is stochastic, then Pn is stochastic

• The eigenvalues of P are all smaller than 1, i.e. || ≤1

11

Assumptions

• In the remaining of the chapter, we limit ourselves to Markov chain

• of discrete time

• defined on a finite state space E

• homogeneous in time.

• Note that most results extend to countable state space.

12

Graphic representation of a DTMC

13

Classification of Discrete Time Markov Chains

14

Classification of states

• Let fjj be the probability of returning to state j after leaving j.

• A state j is said transient if fjj < 1

• A state j is said recurrent if fjj = 1

• A state j is said absorbing if pjj = 1.

• Let Tjj be the average reccurn time, i.e. time of returning to j

• A recurrent state j is positive recurrent if E[Tjj] is finite.

• A recurrent state j is null recurrent if E[Tjj] = .

15

Classify the states of the example

1 2

3 45

0

start

exit

1

1/2

1/2

1/2

11/4

1/4

1/2

1/41/4

1

16

Irreducible Markov chain

• A DTMC is said irreducible iff a state j can be reached in a finite number of steps from any other state i.

• An irreducible DTMC is a strongly connected graph.

17

Irreducble Markov chain

18

Periodic Markov chain

• A state j is said periodic if it is visited only in a number of steps which is multiple of an integer d > 1, called period.

• A state j is said aperiodic otherwise

• A state with a self-loop transition (i.e. pii > 0) is always aperiodic.

• All states of an irreducible Markov chain have the same period.

19

Partitionning a DTMC into irreducible sub-chains

• A DTMC can be partitionned into strongly connected components, each corresponding to an irreducible sub-chain.

20

Classification of irreducible sub-chains

• A sub-chain is said absorbing if there is no arc going out of it.

• Otherwise, the sub-chain is transient.

transcient sub-chain

absorbing sub-chain

absorbing sub-chain

21

Canonic form of transition matrix

• Q : transitions of transient sub-chains

• Pi : transititions between states of aborbing sub-chain i

• Ri: Transitions toward absorbing sub-chain i

22

Formal definitions

• A state j is said reachable from a state i if there is a path from i to j in the state transition diagram.

• A subset S of states is said closed if there is no transition leaving S.

• A closed set S is said irreducible if all states in S are mutually reachable.

• A Markov chain is said irreducible if its state space is irreducible.

23

Theorems

Th1. If a Markov chain has a finite state space, then at least one state is recurrent.

Th2. If i is a recurrent state and j is reachable from i, then state j is recurrent.

Th3. If S is a finite closed irreducible set of states, then every state in S is recurrent.

Th4. If i is a positive recurrent state and j is reachable from i, then state j is positive recurrent.

Th5. If S is a closed irreducible set of states, then every state in S is positive recurrent or every state in S is null recurrent or every state in S is transient.

Th6. If S is a finite closed irreducible set of states, then every state in S is positive recurrent.

24

Analysis of DTMC

25

Sojourn time in a state

• Let Ti be the temps spent in state i before jumping to other states.

• Ti is a random variable of geometric distribution.

1 1ni ii iiP T n p p

26

Properties of geometric distribution

• Let X be a random variable of geometric distribution with parameter p, i.e. P{X = n} = (1-p)n-1p.

• E[X] = 1/p

• Var(X) = 1/p2

• X = 1/p

• Coefficient of variation = X / E[X] = 1

• Memoryless (only discrete distribution of this property):

11

1

¨

11

1

n mm

n

P X n mP X n m X n

P X n

p pp p P X m

p

27

m-step transition probabilities

• The probability of going from i to j in m steps is

pij(m) = P{Xn+m = j|Xn=i} = P{Xm = j|X0=i}.

• Let P(m) = [pij(m)] be the m-step transition matrix

Properties (to prove):

• P(m) = Pm

• Chapman-Kolmogorov equation:

P(l+m) = P(l)P(m)

or l m l mij ik kj

k E

p p p

28

Example

• What is the probability that the mouse is still in room 2 at time 4? (p22

(4))

1 2

3 45

0

start

exit

1

1/2

1/2

1/2

11/4

1/4

1/2

1/41/4

1

29

Probability of going from i to j in exactly n steps

• fij(n) : probability of going from i to j in exactly n steps (without

passing j before)

• fij: probability of going from i to j in a finite number of steps

• Similar approach can be used to determine the average time Tij it takes for going from i to j

1

nij ij

n

ij ij ik kjk j

f f

f p p f

30

Probability distribution of states

• i(n) : probability of being in state i at time n

i(n) = P{Xn = i}

• (n) = (1(n), 2(n), ...) : vector of probability distribution over the state space at time n

• The probability distribution (n) depends on

─ the transition matrix P

─ the initial distribution (0)

• Remark: if the system is at state i for certainty, then i(0) = 1 and j(n) = 0, for j ≠i

• What is the relation between (n), (0), and P?

31

Transient state equations

• By conditioning on the state at time n,

Property:

Let P be the transition matrix of a markov chain and (0) the initial distribution, then over the state space at time n

(n+1) = (n)P

(n)= (0)Pn

32

Steady-state distribution

Key questions :

• Is the distribution (n) converges when n goes to infinity?

• If the distribution converges, does its limit = (1, 2, ...) depend on the initial distribution (0)?

• If a state is recurrent, what is the percentage of time spent in this state and what is the number of transitions between two successive visits to the state?

• If a state is absorbing, what is the probability of ending at this state? What is the average time to this state?

33

Steady state distribution

Theorem : For a irreducible and aperiodic DTMC with positive recurrent states, the distribution (n) converges to a limit vector which is independent of (0) and is the unique solution of the system:

• i are also called stationary probabilities (also called steady state or equilibrium distribution).

• For an irreducible and periodic DTMC, i are the percentage of time spent in state i

1ii E

P

,

1

j i iji E

ii E

p j E

Normalization equation

balance equationequilibrium equation

34

Flow balance equation

• Equation can be interpretated as balance equation of probability flow.

• A probability flow ipij is associated to each transition (i, j).

• is the sum of probability flow into node j

• is the sum of flow out of node j

• The flow balance equation : Outgoing flow = Incoming flow

j i iji E

p

i iji E

p

or j j jii E

p

35

A manufaturing system

• Consider a machine which can be either UP or DOWN.

• The state of the machine is checked every day.

• The average time to failure of an UP machine is 10 days.

• The average time for repair of a DOWN machine is 1.5 days.

• Determine the conditions for the state of the machine {Xn} at the begining of each day to be a Markov chain.

• Draw the Markov chain model.

• Find the transient distribution by starting from state UP and DOWN.

• Check whether the Markov chain is recurrent and aperiodic.

• Determine the steady state distribution.

• Determine the availability of the machine.

36

A telephone call process

• Discrete time model with time slots indexed by k = 0, 1, 2, ...

• At most one telephone call can occur in a single time slot, and there is a probability that a call occurs in any slot

• If the phone is busy, the call is lost; otherwise, the call is processed.

• There is a probability that a call in process completes in any time slot

• If both a call arrival and a call completion occur in the same time slot, the new call will be processed.

Issues to solve:

• Markov chain model

• Loss probability