S TOCHASTIC M ODELS L ECTURE 3 C ONTINUOUS - T IME M ARKOV P ROCESSES Nan Chen MSc Program in...
-
Upload
leona-atkinson -
Category
Documents
-
view
216 -
download
0
Transcript of S TOCHASTIC M ODELS L ECTURE 3 C ONTINUOUS - T IME M ARKOV P ROCESSES Nan Chen MSc Program in...
STOCHASTIC MODELS LECTURE 3
CONTINUOUS-TIME MARKOV PROCESSES
Nan ChenMSc Program in Financial EngineeringThe Chinese University of Hong Kong
(ShenZhen)Oct 14, 2015
Outline1. Introduction of Continuous-
Time Processes2. Limiting Probabilities
3.1 INTRODUCTION
A New Perspective on Poisson Process
• A Poisson process can be constructed as follows: – At each state it will stay for an exponential
time with mean – Then, it proceeds from to
Continuous-Time Markov Chains
• We may follow a similar way to construct a continuous-time Markov chains on a state space : – Each time when it enters a state , we select a
random sojourn time independently of its history;
– After a time it will exit the state . It will enter state with probability
(Remark: We have a discrete-time Markov chain embedded in the process. )
Transition Probability Function
• Let record the state of the above Markov chain over time. The following quantity
is known as the transition probability function of the process.
• Obviously, we have
Kolmogorov Backward/Forward Equations• Theorem (Kolmogorov Backward Equation) For states and times
where
Kolmogorov Backward/Forward Equations (Continued)• Theorem (Kolmogorov Forward Equation) For states and times
Example I: Two-State Chain
• Consider a machine that works for an exponential amount of time having mean before breaking down; and suppose that it takes an exponential amount of time having mean to repair the machine.
• If the machine is in working condition at time 0, then what is the probability that it will be working at time
Example I: Two-State Chain
• From the backward equations, we have
• From them, we can solve for
Computing Transition Probabilities
• For any pair of states and let
Then, we can rewrite the backward/forward equations as follows (backward) (forward)
Computing Transition Probabilities (Continued)• Introduce matrices by letting the
element in row , column of these matrices be, respectively,
The two equations can be written as (backward) (forward)• The solution should be
3.2 LIMITING PROBABILITIES
Limiting Probabilities
• In analogy with a basic result in discrete-time Markov chains, the probability that a continuous-time Markov chain will be in state at time often converges to a limiting value that is independent of the initial state.
• We intend to study
Limiting Probabilities (Continued)• To derive a set of equations for the , we
may let approach in the forward equation. Then,
In addition,
Limiting Probabilities (Continued)• It is easy to see that if the embedded Markov
chain has a limiting stationary distribution i.e.,
then
• The solution to the equations on the last slide should be
Example II: A Shoe Shine Shop• Consider a shoe shine establishment
consisting two chairs --- chair 1 and chair 2. A customer upon arrival goes initially to chair 1 where his shoes are cleaned and polish is applied. After this is done, the customer moves on to chair 2 where the polish is buffed.
• The service times at the two chairs are independent and exponentially distributed with respective rates and
Example II: A Shoe Shine Shop (Continued)• Suppose that potential customers arrive in
accordance with a Poisson process having rate , and that a potential customer will enter the system only if both chairs are empty.
• Let us define the state of the system as follows:– State 0: empty system– State 1: chair 1 is taken– State 2: chair 2 is taken
Example II: A Shoe Shine Shop (Continued)• What is the limiting probability for this
continuous-time Markov chain?
Homework Assignments
• Read Ross Chapter 6.1, 6.2, 6.4, 6.5, and 6.9.• Answer Questions:– Exercises 8 (Page 399, Ross)– Exercises 10, 13 (Page 400, Ross)– Exercises 14, 17 (Page 401, Ross)– Exercise 20 (Page 402, Ross)– (Optional, Extra Bonus) Exercise 48 (Page 407).