Post on 20-Jan-2016
description
A brief review
A continuous random variable X is said to have an
with parameter , 0, if its probability density
function is given by
0 ( )
0,
xe xf x
exponential
distribution
0
0
1 0 ( ) ( )
0, 0
1( ) ( )
xx
x
x
e xF x f y dy
x
E X xf x dx x e dx
The exponential distribution
A random variable is said to be memoryless if
( | ) ( ) for all , 0
( , ) ( )( )
( ) ( )
( ) ( ) ( )
P X s t X t P X s s t
P X s t X t P X s tP X s
P X t P X t
P X s t P X t P X s
The memoryless property
( )
If has the exponential distrbution, then
( ) ( ) ( )s t s t
X
P X s t e e e P X t P X s
Exponentially distributed random variables arememoryless.
The exponential distribution is the only distribution that has the memoryless property.
1 2
1 2 1 2
1 2
(min( , , ..., ) ) {( ), ( ), ..., ( )}
{( )} {( )} ... {( )}
...
n n
n
x x
P X X X x P X x X x X x
P X x P X x P X x
e e e
1 2( ... )
1 2
1 2
The distribution of the random variable (min( , , ..., )
1is exponentially distributed with mean .
...
N
N
x
x
n
n
e
P X X X
Suppose that X1, X2, ..., Xn are independent exponential random variables, with Xi having rate i, i=1, ..., n.What is P(min(X1, X2, ..., Xn )>x)?
The minimum of n exponentially distributed random variables
1
1 1
2 1 1 2
1 2 1 2 10
1 2 1 1 2 10 0
( )1 10 0
1
1 2
( ) ( | ) ( )
( | ) ( )
.
X
x x
x x x
P X X P X X X x f x dx
P X X X x e dx P x X e dx
e e dx e dx
Suppose that X1 and X2 are independent exponentially distributed random variables with rates 1 and 2.What is P(X1 < X2)?
Comparing two exponentially distributed random variables
The Poisson process
The counting process {N(t) t ≥ 0} is said to be a Poisson process having rate , > 0, if
(i) N(0) = 0.(ii) The process has independent increments.(iii) The number of events in any interval of length t is Poisson distributed with mean t. That is for all s, t ≥ 0
( ){ ( ) ( ) } , for 0,1,...
!
nt t
P N t s N t n e nn
Let Tn denote the inter-arrival time between the (n-1)th event and the nth event of a Poisson process, then the Tn (n=1, 2, ...) are independent, identically distributed exponential random variables having mean 1/.
The distribution of interarrival times for a Poisson process
Continuous Time Markov Chains (CTMC)
• A CTMC is a continuous time analog to a discrete time Markov chain
• A CTMC is defined with respect to a continuous time stochastic process {X(t): t ≥0}
• If X(t) = i the process is said to be in state i at time t
• {i: i=0, 1, 2, ...} is the state space
A stochastic process {X(t): t ≥0} is a continuous time Markov chain if for all s, t , u ≥0 and 0 ≤ u < s
P{X(t+s)=j|X(s)=i, X(u)=x(u), 0 ≤ u < s} = P{X(t+s)=j|X(s)=i}
A CTMC is said to have stationary transition probabilities if
P{X(t+s)=j|X(s)=i} is independent of s (and depends only on t).
A CTMC is said to have stationary transition probabilities if
P{X(t+s)=j|X(s)=i} is independent of s (and depends only on t).
Pij(t) = P{X(t+s)=j|X(s)=i}
A CTMC is said to have stationary transition probabilities if
P{X(t+s)=j|X(s)=i} is independent of s (and depends only on t).
Pij(t) = P{X(t+s)=j|X(s)=i}
Note: We shall always assume that the stationary property holds
• Ti: time the process spends in state i once it enters state i (the length of a visit to state i, a random variable).
Sojourn times
• Ti: time the process spends in state i once it enters state i (the length of a visit to state i, a random variable).
Example 1: X(0)=2, the first three transitions occur at t1 =3, t1 = 4.2 and t1 = 6.5 and X(3)=4, X(4.2) = 2 and X(6.5) =1.
Sojourn times
• Ti: time the process spends in state i once it enters state i (the length of a visit to state i, a random variable).
Example 1: X(0)=2, the first three transitions occur at t1 =3, t2 = 4.2 and t3 = 6.5 and X(3)=4, X(4.2) = 2 and X(6.5) =1.
The first sojourn time in state 4 = 4.2-3 = 1.2 The second sojourn time in state 2 = 6.5 – 4.2 = 1.3
Sojourn times
Example 2: Suppose the process has been in state 3 for 10 minutes, what is the probability that it will not leave state 3 in the next 5 minutes.
Example 2: Suppose the process has been in state 3 for 10 minutes, what is the probability that it will not leave state 3 in the next 5 minutes.
P(T3 > 15|T3> 10) = P(T3 > 5)
More generally,
P(Ti > s+t|Ti> s) = P(Ti > t)
Ti is memoryless and therefore has the exponential distribution
A CTMC is a stochastic process having the properties that each time it enters a state i
(i) the amount of time it spends in that state before making a transition into a different state is exponentially distributed with some mean 1/vi (or transition rate vi )
(ii) when the process leaves state i, it next enters state j with some probability Pij (Pii =0 and jPij =1, for all i)
An alternative definition of a CTMC
A CTMC is a stochastic process that moves from state to state according to a probability transition matrix (similar to a discrete time Markov chain) , but the amount of time it spends in each state is exponentially distributed.
To define a CTMC, we need to define a state space, a probability transition matrix, and a set of transition rates.
Example 1: Customers arrive to a store according to a Poisson process with rate . Let N(t) be the total number of customers that have arrived by time t.
Example 1: Customers arrive to a store according to a Poisson process with rate . Let N(t) be the total number of customers that have arrived by time t.
State space is {0, 1, 2, ...}; Ti is exponentially distributed with mean 1/ Pij =1 if j=i+1 and Pij = 0 otherwise
Example 2: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes the single agent at the counter to check-in a customer is exponentially distributed with mean 1/.
Example 2: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes the single agent at the counter to check-in a customer is exponentially distributed with mean 1/.
State space is {0, 1, 2, ...} P0,1 = 1; Pi,i+1 = /(+); Pi,i-1 = /(+) T0 =/v0 =; Ti = /(+) vi =+ for i =1, 2, ...
Example 2: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes the single agent at the counter to check-in a customer is exponentially distributed with mean 1/.
State space is {0, 1, 2, ...} P0,1 = 1; Pi,i+1 = /(+); Pi,i-1 = /(+) T0 =/v0 =; Ti = /(+) vi =+ for i =1, 2, ...
The above is an example of an M/M/1 queue.
The M/M/1 queue is an example of a birth and death process.
Example 2: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes the single agent at the counter to check-in a customer is exponentially distributed with mean 1/.
State space is {0, 1, 2, ...} P0,1 = 1; Pi,i+1 = /(+); Pi,i-1 = /(+) T0 =/v0 =; Ti = /(+) vi =+ for i =1, 2, ...
The above is an example of an M/M/1 queue.
The M/M/1 queue is an example of a birth and death process.
Example 3: Customers arrive to a service center according to a Poisson process with rate n when there are n customers in the system. Customers take an amount Tn that is exponentially distributed with mean 1/n when there are n customers in the system.
Birth and death process
Example 3: Customers arrive to a service center according to a Poisson process with rate n when there are n customers in the system. Customers take an amount Tn that is exponentially distributed with mean 1/n when there are n customers in the system.
State space is {0, 1, 2, ...} P0,1 = 1; Pn,n+1=n/(n+n); Pn,n-1=n/(n+n) T0 =/v0 =; Tn = /(n +n) vn =n +n for n =1, 2, ...
Birth and death process
State transition diagrams for a B&D process
10 2
3
• The Poisson process is a birth and death process with rate n= and n=.
• The M/M/1 queue is described by a birth and death process with rate n= and n=.
Example 4: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes one of the m agents at the counter to check-in a customer is exponentially distributed with mean 1/.
Example 4: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes one of the m agents at the counter to check-in a customer is exponentially distributed with mean 1/.
The system can be modeled as a birth and death process with transition rates n = ;n = n if 1 ≤ n < m n = m if n ≥ m
Example 4: Customers arrive to an airline check-in counter according to a Poisson process with rate . The time it takes one of the m agents at the counter to check-in a customer is exponentially distributed with mean 1/.
The system can be modeled as a birth and death process with transition rates n = ;n = n if 1 ≤ n < m n = m if n ≥ m
The above is an example of an M/M/m queue.
vi: rate with which the process leaves state i (once it enters state i)
qij: rate with which the process goes state j (once it enters state i) qij = Pijvi
(qij is also called the instantaneous transition rate from state i to j)
Transition rates
vi: rate with which the process leaves state i (once it enters state i)
qij: rate with which the process goes state j (once it enters state i) qij = Pijvi
(qij is also called the instantaneous transition rate from state i to j)
vij viPij = j qij
Pijqij/viqij/j qij
Transition rates
vi: rate with which the process leaves state i (once it enters state i)
qij: rate with which the process goes state j (once it enters state i) qij = Pijvi
(qij is also called the instantaneous transition rate from state i to j)
vij viPij = j qij
Pijqij/viqij/j qij
Specifying the instantaneous transition rates determines the parameters of the CTMC
Transition rates
State transition diagrams
10 2
q0,2
q2,1
q1,2q0,1
q2,0
q2,0
0
0
1 ( )lim
( )lim , for
iii
h
ijij
h
P hv
h
P hq i j
h
It can also be shown that
Properties
0 For all , 0, ( ) ( ) ( ).ij ik kjk
s t P t s P t P s
The Chapman-Kolmogrov equations
0 For all , 0, ( ) ( ) ( ).
( ) ( ( ) | (0) )
ij ik kjk
ij
s t P t s P t P s
P t s P X t s j X i
Proof :
The Chapman-Kolmogrov equations
0
0
For all , 0, ( ) ( ) ( ).
( ) ( ( ) | (0) )
= ( ( ) , ( ) | (0) )
ij ik kjk
ij
k
s t P t s P t P s
P t s P X t s j X i
P X t s j X t k X i
Proof :
The Chapman-Kolmogrov equations
0
0
0
For all , 0, ( ) ( ) ( ).
( ) ( ( ) | (0) )
= ( ( ) , ( ) | (0) )
= ( ( ) | ( ) , (0) ) (
ij ik kjk
ij
k
k
s t P t s P t P s
P t s P X t s j X i
P X t s j X t k X i
P X t s j X t k X i P
Proof :
( ) | (0) )
X t k X i
The Chapman-Kolmogrov equations
0
0
0
For all , 0, ( ) ( ) ( ).
( ) ( ( ) | (0) )
= ( ( ) , ( ) | (0) )
= ( ( ) | ( ) , (0) ) (
ij ik kjk
ij
k
k
s t P t s P t P s
P t s P X t s j X i
P X t s j X t k X i
P X t s j X t k X i P
Proof :
0
( ) | (0) )
= ( ( ) | ( ) ) ( ( ) | (0) )
k
X t k X i
P X t s j X t k P X t k X i
The Chapman-Kolmogrov equations
0
0
0
For all , 0, ( ) ( ) ( ).
( ) ( ( ) | (0) )
= ( ( ) , ( ) | (0) )
= ( ( ) | ( ) , (0) ) (
ij ik kjk
ij
k
k
s t P t s P t P s
P t s P X t s j X i
P X t s j X t k X i
P X t s j X t k X i P
Proof :
0
0
( ) | (0) )
= ( ( ) | ( ) ) ( ( ) | (0) )
= ( ) ( )
k
kj ikk
X t k X i
P X t s j X t k P X t k X i
P s P t
The Chapman-Kolmogrov equations
'
0
For all states , and time 0, ( ) ( ) ( )
( ) ( )= ( ) ( ) ( )
( ) ( ) [1 ( )] ( )
ij ik kj i ijk i
ij ij ik kj ijk
ik kj ii ijk i
i j t P t q P t v P t
P h t P t P h P t P t
P h P t P h P t
Proof :
Kolmogrov’s backward equations
'
0
0
For all states , and time 0, ( ) ( ) ( )
( ) ( )= ( ) ( ) ( )
( ) ( ) [1 ( )] ( )
( ) ( )lim
ij ik kj i ijk i
ij ij ik kj ijk
ik kj ii ijk i
ij ij
h
i j t P t q P t v P t
P h t P t P h P t P t
P h P t P h P t
P h t P t
h
Proof :
0
( ) 1 ( )lim ( ) [ ] ( )ik ii
kj ijk ih
P h P hP t P t
h h
Kolmogrov’s backward equations
'
0
0
For all states , and time 0, ( ) ( ) ( )
( ) ( )= ( ) ( ) ( )
( ) ( ) [1 ( )] ( )
( ) ( )lim
ij ik kj i ijk i
ij ij ik kj ijk
ik kj ii ijk i
ij ij
h
i j t P t q P t v P t
P h t P t P h P t P t
P h P t P h P t
P h t P t
h
Proof :
0
'
( ) 1 ( )lim ( ) [ ] ( )
( ) ( ) ( )
ik iikj ijk ih
ij ik kj i ijk i
P h P hP t P t
h h
P t q P t v P t
Kolmogrov’s backward equations
'Under suitable regularity conditions, ( ) ( ) ( )
for all , and time 0.
ij kj ik j ijk jP t q P t v P t
i j t
Kolmogrov’s forward equations
'
'
lim ( )
lim ( ) lim ( ) ( )
0
j ijt
ij kj ik j ijk jtt
kj k j jk j
kj k j j j j kj kk j k j
P P t
P t q P t v P t
q P v P
q P v P v P q P
Limiting probabilities
The limiting probabilities can be obtained by solving the following
system of equations:
0
1
kj k j jk j
jj
q P v P
P
rate at which the process leaves state
rate at which the process enters state
rate out of state rate into state
j j
kj kk j
j j kj kk j
v P j
q P j
v P q P
j j
The limiting probabilities Pj exist if
(a) all states of the Markov chain communicate (i.e., starting in state i, there is a positive probability of ever being in state j, for all i, j and
(b) the Markov is positive recurrent (i.e, starting in any state, the mean time to return to that state is finite).
When do the limiting probabilities exist?
The M/M/1 queue
10 2
3
0 1
1 2 0
State rate out of state rate into state
0
1 ( )
2
j j
P P
P P P
2 3 1
1 1
( )
1 ( ) n n n
P P P
n P P P
The M/M/1 queue
0 1
1 2
2 3
By adding to each equation the equation preceding it, we obtain
0
1
2
P P
P P
P P
n
11 n nP P
The M/M/1 queue
0
1 0
22 0
33 3
Solving in terms of yields
0 ( / )
1 ( / )
2 ( / )
1 ( /n
P
P P
P P
P P
n P
0)n P
The M/M/1 queue
0
0 0 1
0
1
Using the fact that 1,we obtain
( / ) 1
11
1 ( / )
/ (1 / )
Note that we must have / 1.
nn
n
n
n
n
n
n
P
P P
P
P