Sstt311 Sec3 2012 (Student)

7/27/2019 Sstt311 Sec3 2012 (Student)

http://slidepdf.com/reader/full/sstt311-sec3-2012-student 1/37

Section 3: Discrete-Time Markov Chains

3.1 Introduction

In this section we will discuss stochastic processes whose futures are conditionallyindependent of their pasts provided that their present values are known. Some definitions

needed are:

Definition (s) :

3.1. Vector - n-Tuple of numbers where the numbers are called components, e.g.

u u1, u2,...,un with component ui #

3.2. Scalar multiple - If u is a vector and k a scalar then k u is a scalar multiple of u, e.g.k u ku1, ku2,...,kun #

3.3. Matrix - Rectangular array of numbers, e.g.

A

a11 a12 ... a1n

a21 a22 ... a2n

... ... ... ...

am1 am2 ... amn

#

3.4. Probability vector - A vector u u1, u2,...,un where the components are non-negativeand 1 ࢣ

i1

nu i. #

3.5. Stochastic matrix - A square matrix P pij where each of its rows is a probabilityvector, i.e. m ࢣ

j1

m ࢣi1

n pij where ࢣ

i1

n pij 1 ; j 1,2,..., m #

3.2 Markov Chains

Markov chains are essentially the sequence of states entered by a system evolving in time,or the sequence of positions occupied by a moving particle.

Example (3.1) :

Consider a particle that moves along a set of m 1 nodes, labeled 0,1,,..., m, that are

arranged around a circle. At each step the particle is equally likely to move one position in

24

7/27/2019 Sstt311 Sec3 2012 (Student)


either the clockwise or counterclockwise direction. That is, if X n is the position of the particle

after its nth step then

P X n1 i 1 ࢯ X n i P X n1 i ࢤ 1 ࢯ X n i 12

where i

1 0 when i

m, and i ࢤ 1 m when i

0. Suppose now that the particle starts at0 and continuous to move around according to the preceding rules until all the nodes 1,2, ..., m

have been visited. What is the probability that node i, i 1,2,..., m, is the last node visited?

Solution :

Consider the first time that the particle is at one of the two neighbours of node i, that is,the first time that the particle is at one of the nodes i ࢤ 1 or i 1 (with m 1 0). Supposethat it is at node i ࢤ 1 the argument in the alternative situation is identical. Since neither

node i nor i 1 has yet been visited, it follows that i will be the last node visited if and only if i 1 is visited before i. This is so because in order to visit i 1 before i the particle will have

to visit all the nodes on the counterclockwise path from i ࢤ 1 to i 1 before it visits i. But theprobability that a particle at node i ࢤ 1 will visit i 1 before i is just the probability that aparticle will progress m ࢤ 1 steps in a specified direction before progressing one step in theother direction. Hence, because the preceding implies that the probability that node i is the last

node visited is the same for all i, and because these probabilities must sum to 1, we obtain

Pi is the last node visited 1m , i 1,2,..., m #

Definition :

3.6. Markov Chains - Consider a stochastic process X n, n 0,1,2,... that takes on a finiteor countable number of possible values. Unless otherwise mentioned, this set of possiblevalues of the process will be denoted by the set of nonnegative integers 0,1,2,.... If

X n i, then the process is said to be in state i at time n. We suppose that whenever theprocess is in state i, there is a fixed probability Pij that it will next be in state j. That is,we suppose that

P X n1 j ࢯ X n i, X nࢤ1 inࢤ1,..., X 1 i1, X 0 i0 Pij (3.1)

for all j ࢠ E and all n 0. Such a stochastic process is known as a Markov chain. #

Equation 3.1 may be interpreted as stating that, for a Markov chain, the conditionaldistribution of any future state X n1 given the past states X 0, X 1,..., X nࢤ1 and the present state X n, is independent of the past states and depends only on the present state.

The value Pij represents the probability that the process will, when in state i, make atransition into state j. Since probabilities are nonnegative and since the process must make atransition into some state, we have that

Pij 0, i, j 0

ࢣ j0

Ý

Pij 1, i 0,1,...

25

7/27/2019 Sstt311 Sec3 2012 (Student)


Definition :

3.7. One-step transition matrix - Let P denote the matrix of one-step transition probabilitiesPij, so that

P

P00 P01 P02 . .

P10 P11 P12 . .

Pi0 Pi1 P i2 . .

#

Example (3.2) : Transforming a Process into a Markov Chain

Suppose that whether or not it rains today depends on previous weather conditionsthrough the last two days. Specifically, suppose that if it has rained for the past two days, then

it will rain tomorrow with probability 0.7; if it rained today but not yesterday, then it will raintomorrow with probability 0.5; if it rained yesterday but not today, then it will rain tomorrowwith probability 0.4; if it has not rained in the past two days, then it will rain tomorrow withprobability 0.2.

If we let the state at time n depend only on whether or not it is raining at time n, then thepreceding model is not a Markov chain why not?. Transform this model into a Markov

chain.

Solution :

This model can be transformed into a Markov chain by saying that the state at any time is

determined by the weather conditions during both that day and the previous day. In otherwords, we can say that the process is in

state 0 : if it rained both today and yesterday

state 1 : if rained today but not yesterday

state 2 : if it rained yesterday but not today

state 3 : if it did not rain either yesterday or today

The preceding would then represent a four-state Markov chain having a transitionprobability matrix

P

0.7 0 0.3 0

0.5 0 0.5 0

0 0.4 0 0.6

0 0.2 0 0.8

#

EXAMPLES 3.3 - 3.12 ILLUSTRATE SOME STOCHASTIC MODELAPPLICATIONS

26

7/27/2019 Sstt311 Sec3 2012 (Student)


Example (3.3) : Random Walk Model

A Markov chain whose state space is given by the integers i 0,1,2, ... is said to be arandom walk if, for some number 0 p 1,

Pi,i1 p 1 ࢤ Pi,iࢤ1, i 0,1,...

The preceding Markov chain is called a random walk for we may think of it as being a

model for an individual walking on a straight line who at each point of time either takes onestep to the right with probability p or one step to the left with probability 1 ࢤ p. #

Example (3.4) : Gambling Model

Consider a gambler who, at each play of the game, either wins R1 with probability p orloses R1 with probability 1 ࢤ p. If we suppose that our gambler quits playing either when hegoes broke or he attains a fortune of R N , then the gambler’s fortune is a Markov chain havingtransition probabilities

Pi,i1 p 1 ࢤ Pi,iࢤ1, i 1,2,..., N ࢤ 1

P00 P NN 1

States 0 and N are called absorbing states since once entered they are never left. Note that

the preceding is a finite state random walk with absorbing barriers states 0 and N . #

Example (3.5) : Chain binomial models in epidemiology

Suppose at time t 0 there are S 0 susceptibles and I 0 infectives. After a certain latent

period of the infection, considered here as a unit of time, some of the susceptibles areinfected; thus, at time t 1, the initial S 0 susceptibles split into two groups: those who areinfected, I 1, and the remaining susceptibles, S 1. The process continues until there are nomore susceptibles in the population. The scheme is indicated in the figure below.

Note that, in general, S t S t 1 I t 1, t 0,1,2,.... Assuming that the probability

of a susceptible being infected is p we have, for the Greenwood Model,

PS t 1 st 1|S t st st

st ࢤ st 1pst ࢤst 11 ࢤ pst 1

The process S t , t 0,1,2,... is a Markov chain with the transition probabilities given

above. Given S 0 these transition probabilities specify the process S t completely. #

27

7/27/2019 Sstt311 Sec3 2012 (Student)


Example (3.6) : Population growth model

Suppose an organism produces a random number, say Y , of offspring with pk PY k ,k 0,1,2,..., ࢣ

k 0

Ý

pk 1. Each offspring in turn produces organisms independently

according to the same distribution pk . If Z t denotes the population size at the t th

generation, t 0,1,2,... it is seen that Z t is a Markov chain with transition probabilitiesgiven by P Z t j| Z t ࢤ 1 i PY 1 Y 2 ... Y i j where Y 1, Y 2, ... are independentidentically distributed random variables with probability distribution pk . The above modelin known as the Galton-Watson branching process. #

Example (3.7) : Spatial model for plant ecology

Consider a lattice of sites labelled by integer pairs i, j over some region. Let X ij denotethe random variable associated with the plant located at the site i, j. If we are interested in astudy of the pattern of infection in an array of plants we may set X ij 1 if the plant located ati, j is infected, and zero otherwise. It may be reasonable to assume that the conditional

distribution of X ij given all other site values depends only on the four nearest neighbour sites,namely, on X iࢤ1, j, X i1, j, X i, jࢤ1 and X i, j1. This is a natural extension of the Markovianassumption to the spatial process X ij. #

Example (3.8) : Model in population genetics

Consider a population of 2 N genes each of which belongs to one of the two genotypes, say A and B. Let X t denote the proportion of type A genes in the ith generation. Assuming thatthe total number of genes remains the same from generation to generation, and neglectingselection and mutation effects, the genes in the t 1th generation may be assumed to a

random sample of size 2 N of genes in the t th

generation. The sequence of random variables X t , t 1,2, ..., forms a Markov chain. Conditionally on X t ࢤ 1 x, 2 NX t will be a

binomial random variable with success probability x and index 2 N . #

Example (3.9) : Storage model

Let X t denote the annual random input during the years t , t 1, M the annualnon-random release at the end of each year, and Z t the content of the dam after the release.Then Z t 1 min Z t X t , K ࢤ min Z t X t , M where K is the capacity of the damand t 0,1,2,... If the inputs X t are assumed to be independent random variables, thesequence Z t will form a Markov chain. If, however, X t is assumed to follow a Markov

chain, X t , Z t can be viewed as a bivariate Markov chain. #

Example (3.10) : Compound Poisson model for insurance risk

Suppose an insurance company receives claims from its clients in accordance with aPoisson process with intensity . Assume that the amounts Y k , k 1,2,... of successive

claims are independent random variables with a common distribution function F y. Then, thetotal amount X t of claims arising in 0, t is given by X t Y 1 Y 2 ... Y N t where N t is a Poisson random variable with mean t . The process X t , t 0 is a compound Poissonprocess. #

28

7/27/2019 Sstt311 Sec3 2012 (Student)


Example (3.11) : Queueing models for telephone calls

Suppose calls arrive at a telephone exchange according to a Poisson process. Durations of successive calls may be assumed to be independent and to follow a negative exponential

distribution. The capacity of the exchange may be limited to say, K calls at any given time.Various questions such as the queue-size at any particular time, expected waiting time of a

call, etc. are of practical interest. #

Example (3.12) : Prediction of economic time series

Let X t denote the price of a certain commodity at time t , t 0,1,2,.... Suppose wewish to fit a linear model for X t , for example

X t 1 X t ࢤ 1 2 X t ࢤ 2 ... p X t ࢤ p

Z t 1 Z t ࢤ 1 2 Z t ࢤ 2 ... q Z t ࢤ q

where j j 1,2,..., p and k k 1,2,..., q are the real parameters to be estimated and the Z s are independent, identically distributed, unobservable random variables. Given the prices

at n time points, X 0, X 1,..., X n ࢤ 1, one may wish to predict future prices. #

29

7/27/2019 Sstt311 Sec3 2012 (Student)


Exercise (3.1) :

1. Which of the following vectors are probability vectors? Give reasons.

a. u 18

ࢤ,0, 16

, 12

, 13

b. v 1

3,0, 1

6, 1

2, 1

3c. w

13

,0,0, 16

, 12

2. Which of the following matrices are stochastic matrices? Give reasons.

A

13

13

13

12

0 12

B

1516

116

23

23

C

1 0

12

12

D

12

ࢤ 12

14

34

3. Prove: If A a ij is a stochastic matrix of order n and u u1, u2,...,un is a probabilityvector, then uA is also a probability vector.

4. Prove: If A and B are stochastic matrices, then the product AB is a stochastic matrix.Therefore all powers An are stochastic matrices.

5. Forecasting the Weather Suppose that the chance of rain tomorrow depends on theprevious weather conditions only through whether or not it is raining today and not onpast weather conditions. Suppose also that if it rains today, then it will rain tomorrow

with probability ; and if it does not rain today, then it will rain tomorrow withprobability . Find the transition probability matrix P.

6. Three white and three black balls are distributed in two urns in such a way that eachcontains three balls. We say that the system is in state i, i 0,1,2,3, if the first urncontains i white balls. At each step, we draw one ball from each urn and place the balldrawn from the first urn into the second and conversely with the ball from the second urn.

Let X n denote the state of the system after the n th step. Calculate its transition probabilitymatrix.

7. Consider a sequence of independent trials, at each of which there is success or failurewith probabilities p and q 1 ࢤ p respectively. Let E 0 occur at the nth trial if the result isa failure, and E k k 1,2,..., n if the result is a success with the previous failure at trialn ࢤ k , i.e. if there has been a sequence of k successes. Write down the transition matrix P.Here E i is what we have called state i, i 0,1,2,...

30

7/27/2019 Sstt311 Sec3 2012 (Student)


3.3 Chapman-Kolmogorov Equations

Definition :

3.8. n-Step transition matrix - We have already defined the one-step transition probabilitiesPij. We now define the n-step transition probabilities Pij

n to be the probability that a

process in state i will be in state j after n additional transitions. That is,

Pijn P X nk j ࢯ X k i, n, i, j 0

Pn

P00n P01

n P02n . .

P10n P11

n P12n . .

Pi0n Pi1

n Pi2n . .

#

Of course Pij1 Pij. The Chapman-Kolmogorov equations provide a method for

computing these n-step transition probabilities. These equations are

P ijnm

ࢣk 0

Ý

Pik n Pkj

m, for all n, m 0, all i, j (3.2)

and are most easily understood by noting that Pik

n Pkj

m represents the probability that starting in i

the process will go to state j in n m transitions through a path which takes it into state k atthe nth transition. Hence, summing over all intermediate states k yields the probability that theprocess will be in state j after n m transitions. Formally, we have

Pijnm

P X nm j ࢯ X 0 i

ࢣk 0

Ý

P X nm j, X n k ࢯ X 0 i

ࢣk 0

Ý

P X nm j ࢯ X n k , X 0 iP X n k ࢯ X 0 i

ࢣk 0

Ý

PkjmPik

n

If we let Pn denote the matrix of n-step transition probabilities P ijn , then equation 3.2

asserts that

Pnm Pn Pm

where the dot represents matrix multiplication. Hence,

P2 P11

P P P2

and by induction

31

7/27/2019 Sstt311 Sec3 2012 (Student)


Pn Pnࢤ11

Pnࢤ1.P Pn

That is, the n-step transition matrix may be obtained by multiplying the matrix P by itself n times.

Example : Chapman-Kolmogorov equations

General set of equations are:

Pijnm

ࢣk 0

Ý

Pik n Pkj

m

where m, n, i, j 0

n number of steps from state i to state k

m number of steps from state k to state jP ik

n P(going from state i to state k in n steps)

Pkjm P(going from state k to state j in m steps)

Consider the following graph G with

V G 1,2,3,4,5,6,7,8

E G 1,2, 1,7, 1,3, 1,8, 2,3, 3,4, 3,8, 4,5, 4,8, 5,6, 5,8, 6,7, 6,8, 7,8

For n 1 and m 2, find the probability of going from state 1 to state 5 in 3 steps.

The following are 3-step routes from state 1 to state 5

32

7/27/2019 Sstt311 Sec3 2012 (Student)


R1 : v1v3v4v5

R2 : v1v3v8v5

R3 : v1v8v4v5

R4 : v1v8v6v5

R5 : v1v7v6v5

R6 : v1v7v8v5

Each route is divided into two parts; the first part consisting of one step n 1 and thesecond part consisting of two steps m 2. Now

P131

14

Pgoing from state 1 to 3 in one step

P181 14 Pgoing from state 1 to 8 in one step

P171

14

Pgoing from state 1 to 7 in one step

Also

P352

14

. 13

112

Pgoing from state 3 to 5 through 4 in two steps

P352

14

. 16

124


P852

16

. 13

118


P852 1

6. 1

3 1

18 Pgoing from state 8 to 5 through 6 in two steps

P752

13

. 16

118


P752

13

. 13

19


Since

P R1 P131 P35

2

14

. 112

148

P R2 P131 P35

2

1

4

. 1

24

1

96P R3 P18

1 P852

14

. 118

172

P R4 P181 P85

2

14

. 118

172

P R5 P171 P75

2

14

. 118

172

P R6 P171 P75

2

14

. 19

136

we have

33

7/27/2019 Sstt311 Sec3 2012 (Student)


P153 ࢣ

k 3,7,8

P1k 1 Pk 5

2

148

196

172

172

172

136

29288

0.1007 #

Example (3.13) :

Consider Example 3.2 Given that it rained on Monday and Tuesday, what is theprobability that it will rain on Thursday?

Solution :

The two-step transition matrix is given by

P2 P2

0.7 0 0.3 0

0.5 0 0.5 0

0 0.4 0 0.6

0 0.2 0 0.8

.

0.7 0 0.3 0

0.5 0 0.5 0

0 0.4 0 0.6

0 0.2 0 0.8

0.49 0.12 0.21 0.180.35 0.20 0.15 0.30

0.20 0.12 0.20 0.48

0.10 0.16 0.10 0.64

Since rain on Thursday is equivalent to the process being in either state 0 or state 1 onThursday, the desired probability is given by P00

2 P01

2 0.49 0.12 0.61 #

So far, all of the probabilities we have considered are conditional probabilities. For

instance, Pijn is the probability that the state at time n is j given that the initial state at time 0 is

i. If the unconditional distribution of the state at time n is desired, it is necessary to specify theprobability distribution of the initial state. Let us denote this by

i P X 0 i, i 0 ࢣi0

Ý

i 1

All unconditional probabilities may be computed by conditioning on the initial state. That is,

34

7/27/2019 Sstt311 Sec3 2012 (Student)


P X n j ࢣi0

Ý

P X n j ࢯ X 0 iP X 0 i

ࢣi0

Ý

Pijni

35

7/27/2019 Sstt311 Sec3 2012 (Student)


Exercise (3.2) :

1. Consider Exercise 3.1.5 in which the weather is considered as a two-state Markov chain.If 0.7 and 0.4, then calculate the probability that it will rain four days from

today given that it is raining today.

2. A pensioner receives R2000 at the beginning of each month. The amount of money heneeds to spend during a month is independent of the amount he has and is equal to i with

probability Pi, i 1,2,3,4, ࢣi1

4Pi 1. If the pensioner has more than 3 at the end of a

month, he gives the amount greater than 3 to his son. If, after receiving his payment at thebeginning of a month, the pensioner has a capital of 5, what is the probability that hiscapital is ever 1 or less at any time within the following four months?

3. Remaining Lifetime: Consider some piece of equipment which is now in use. When itfails, it is replaced immediately by an identical one. When that one fails it is again

replaced by an identical one, and so on. Let pk denote the probability that a new itemlasts for k units of time, k 1,2,... Let X n be the remaining lifetime of the item in use attime n. Show that X n, n 0 is a Markov chain with state space 0,1,2,... andconstruct the transition matrix.

4. Three boys A, B and C are throwing a ball to each other. A always throws the ball to B

and B always throws the ball to C ; but C is just as likely to throw the ball to B as to A. Let X n denote the n th person to be thrown the ball. The state space of the system is A, B, C .

a. Is this a Markov chain? Why?

b. Construct the transition matrix P.

c. Suppose C was the first person with the ball, i.e. suppose

0,0,1 is the initialprobability distribution. Determine the probability that C will have the ball afterthree throws.

36

7/27/2019 Sstt311 Sec3 2012 (Student)


3.4 Classification of States

Definition :

3.7. Accessible - State j is said to be accessible from state i if Pijn 0 for some n 0. #

Note that this implies that state j is accessible from state i if and only if, starting in i, it ispossible that the process will ever enter state j. This is true since if j is not accessible from i,then

Pever enter j ࢯ start in i P n0

Ý

X n j ࢯ X 0 i

ࢣn0

Ý

P X n j ࢯ X 0 i

ࢣn0

Ý

Pijn

0

Definition :

3.8. Communicate - Two states i and j that are accessible to each other are said to

communicate, and we write i j. #

Note that any state communicates with itself since, by definition,

Pii0 P X 0 i ࢯ X 0 i 1

The relation of communication satisfies the following three properties

1. State i communicates with state i, all i 0.

2. If state i communicates with state j, then state j communicates with state i.

3. If state i communicates with state j, and state j communicates with state k , then state i

communicates with state k .

Definition (s) :

3.9. Class - Two states that communicate are said to be in the same class. The concept of communication divides the state space up into a number of separate classes. #

3.10. Irreducible - The Markov chain is said to be irreducible if there is only one class, that is,if all states communicate with each other. #

Example (3.14) :Consider the Markov chain consisting of the three states 0,1,2 and having transition

37

7/27/2019 Sstt311 Sec3 2012 (Student)


probability matrix

P

12

12

0

12

14

14

0 13

23

It is easy to verify that this Markov chain is irreducible. For example, it is possible to gofrom state 0 to state 2 since

0 1 2

That is, one way of getting from state 0 to state 2 is to go from state 0 to state 1

with probability 12

and then go from state 1 to state 2 with probability 14

. #

Definition (s) :

3.11. Closed - A set of states is said to be closed if no state outside it can be reached from any

state in it. A closed set is irreducible if no proper subset of it is closed. If the only closedset is the set of all states then the Markov chain is called irreducible. #

3.12. Absorbing - A state forming a closed set by itself is called an absorbing state. State j isabsorbing iff P j, j 1 #

Example (3.15) :Consider a Markov chain consisting of the four states 0,1,2,3 and having transition

probability matrix

P

12

12

0 0

12

12

0 0

14

14

14

14

0 0 0 1

The classes of this Markov chain are 0,1, 2 and 3. Note that while state 0 or 1 isaccessible from state 2, the reverse is not true. Since state 3 is an absorbing state, that is,P33 1, no other state is accessible from it. #

Let X be a Markov chain with state space E and transition matrix P. Let T be the time of the first visit to state i and let N i be the total number of visits to i. Also for any state i we let f idenote the probability that, starting in state i, the process will ever reenter state i.

Definition :

38

7/27/2019 Sstt311 Sec3 2012 (Student)


7/27/2019 Sstt311 Sec3 2012 (Student)


Then after a finite amount of time say, after time T 0 state 0 will never be visited, and after a

time say, T 1 state 1 will never be visited, and so on. Thus, after a finite timeT maxT 0, T 1,...,T M no states will be visited. But as the process must be in some stateafter time T we arrive at a contradiction, which shows that at least one of the states must berecurrent. #

Definition :

3.14. Null state - A recurrent state i is called null if E iT Ý. Otherwise it is called non-null.#

Corollary (3.1) :

If state i is recurrent, and state i communicates with state j, then state j is recurrent.

Proof :First, since state i communicates with state j, there exists integers k and m such that

P ijk 0, P ji

m 0. Now, for any integer n

P jjmnk P ji

mP iin Pij

k

This follows since the left side of the preceding is the probability of going from j to j inm n k steps, while the right side is the probability of going from j to j in m n k steps viaa path that goes from j to i in m steps, then from i to i in an additional n steps, then from i to j

in an additional k steps.From the preceding we obtain, by summing over n, that

ࢣn1

Ý

P jjmnk P ji

mPijk ࢣ

n1

Ý

Piin Ý

since P jimPij

k 0 and ࢣ

n1

Ý

Piin is finite since state i is recurrent. Thus, by the above proposition

it follows that state j is also recurrent. #

Remark (3.1) :

1. The above corollary also implies that transience is a class property. For if state i istransient and communicates with state j, then state j must also be transient. For if j wererecurrent then i would also be recurrent and hence could not be transient.

2. Also, along with the result that not all states in a finite Markov chain can be transient, weconclude that all states of a finite irreducible Markov chain are recurrent.

Example (3.16) :

Consider the Markov chain having states 0,1,2,3,4 and

40

7/27/2019 Sstt311 Sec3 2012 (Student)


P

12

12

0 0 0

12

12

0 0 0

0 0 1

2

1

20

0 0 12

12

0

14

14

0 0 12

Determine the recurrent state.

Solution :

This chain consists of the three classes 0,1, 2,3 and 4. The first two classes are

recurrent and the third transient. #

Example (3.17) : A Random Walk

Consider a Markov chain whose state space consists of the integers i 0,1,2,..., andhave transition probabilities given by

Pi,i1 p 1 ࢤ P i,iࢤ1, i 0,1,2,...

where 0 p 1. In other words, on each transition the process either moves one step to theright with probability p or one step to the left with probability 1 ࢤ p. One interpretation of this process is that it represents the winnings of a gambler who on each play of the gameeither wins or loses one rand.

Since all states clearly communicate, it follows that they are either all transient or all

recurrent. So let us consider state 0 and attempt to determine if ࢣn1

Ý

P00n is finite or infinite.

Since it is impossible to be even using the gambling model interpretation after an odd

number of plays we must, of course, have that

P002nࢤ1

0, n 1,2,...

On the other hand, we would be even after 2n trials if and only if we won n of these andlost n of these. Because each play of the game results in a win with probability p and loss with

probability 1 ࢤ p, the desired probability is thus the binomial probability

P002n

2nn pn1 ࢤ pn

2n!n!n!

p1 ࢤ pn, n 1,2,3,...

By using an approximation, due to Stirling, which asserts that

n! ß nn1/2eࢤn 2

where we say that an ß bn when limn Ý

an

bn 1, we obtain

P002nß

4 p1 ࢤ pn

n

Now, after we verify that an ß bn, then ࢣn

an Ý if and only if ࢣn

bn Ý. Hence,

41

7/27/2019 Sstt311 Sec3 2012 (Student)


ࢣn1

Ý

P00n will converge if and only if

ࢣn1

Ý

4 p1 ࢤ pn

n

does. However, 4 p1 ࢤ p 1 with equality holding if and only if p 12 . Hence,ࢣ

n1

Ý

P00n Ý if and only if p

12

. Thus, the chain is recurrent when p 12

and transient

when p 12

.

When p 12

, the preceding process is called a symmetric random walk . We can also look

at symmetric random walks in more than one dimension. For instance, in the two-dimensionalsymmetric random walk the process would, at each transition, either take one step to the left,

right, up or down, each having probability 14

. That is, the state is the pair of integers i, j and

the transition probabilities are given by

Pi, j,i

1, j P

i, j,iࢤ1, j P

i, j,i, j

1 P

i, j,i, jࢤ1

1

4

By using the same method as in the one-dimensional case, we now show that this Markov

chain is also recurrent. Since the preceding chain is irreducible, it follows that all states willbe recurrent if state 0 0,0 is recurrent. So consider P00

2n. Now after 2n steps, the chain willbe back in its original location if for some i, 0 i n, the 2n steps consist of i steps to theleft, i to the right, n ࢤ i up, and n ࢤ i down. Since each step will be either of these four types

with probability 14

, it follows that the desired probability is a multinomial probability. That

is,

P002n ࢣ

i0

n2n!

i!i!n ࢤ i!n ࢤ i!

1

4

2n

ࢣi0

n2n!n!n!

n!n ࢤ i!i!

n!n ࢤ i!i!

14

2n

14

2n 2nn ࢣ

i0

n

ni

nn ࢤ i

14

2n 2nn

2nn (3.3)

where the last equality uses the combinatorial identity

2nn ࢣ

i0

n

ni

nn ࢤ i

which follows upon noting that both sides represent the number of subgroups of size n one canselect from a set of n white and n black objects. Now,

2nn

2n!n!n!

ß2n2n1/2eࢤ2n 2

n2n1eࢤ2n2by Stirling s approximation

4n

n

Hence, from equation 3.3 we see that

42

7/27/2019 Sstt311 Sec3 2012 (Student)


P002nß

1n

which shows that ࢣn

P002n Ý, and thus all states are recurrent.

Interestingly enough, whereas the symmetric random walks in one and two dimensions areboth recurrent, all higher-dimensional symmetric random walks turn out to be transient. #

Remark (3.2) :

We can compute the probability that the one-dimensional random walk of Example 3.17

ever returns to state 0 when p 12

by conditioning on the initial transition:

Pever return Pever return ࢯ X 1 1 p Pever return ࢯ X 1 11ࢤ ࢤ p

Suppose that p 12

. Then it can be shown that P(ever returnࢯ X 1 1ࢤ 1, and thus

Pever return Pever return ࢯ X 1 1 p 1 ࢤ p (3.4)

Let Pever returnࢯ X 1 1. Conditioning on the next transition gives

Pever return ࢯ X 1 1, X 2 01 ࢤ p Pever return ࢯ X 1 1, X 2 2 p

1 ࢤ p Pever enter 0 ࢯ X 0 2 p

Now, if the chain is at state 2, then in order for it to enter state 0 it must first enter state 1 andthe probability that this ever occurs is why is that?. Also, if it does enter state 1 then theprobability that it ever enters state 0 is also . Thus, we see that the probability of everentering state 0 starting at state 2 is 2. Therefore, we have that

1 ࢤ p p2

The two roots of this equation are 1 and 1 ࢤ p

p . The first is impossible since we

know by transience that 1. Hence, 1 ࢤ p

p , and we obtain from equation 3.4 that

Pever return 1 ࢤ p 1 ࢤ p 21 ࢤ p

Similarly, when p 1/2 we can show that Pever return 2 p. Thus, in general we have that

Pever return 2min p, 1 ࢤ p #

Definition :

3.15. Period - State i is said to have period d if P iin 0 whenever n is not divisible by d , and d

is the largest integer with this property. For instance, starting in i, it may be possible for

the process to enter state i only at the times 2,4,6,8, ..., in which case state i has period 2.A state with period 1 is said to be aperiodic. It can be shown that periodicity is a classproperty. That is, if state i has period d , and states i and j communicate, then state j alsohas period d . #

3.16. Positive recurrent - If state i is recurrent, then it is said to be positive recurrent if, startingin i, the expected time until the process returns to state i is finite. It can be shown that

43

7/27/2019 Sstt311 Sec3 2012 (Student)


positive recurrence is a class property. While there exist recurrent states that are not

positive recurrent null recurrent, it can be shown that in a finite-state Markov chain allrecurrent states are positive recurrent. Positive recurrent, aperiodic states are calledergodic. #

44

7/27/2019 Sstt311 Sec3 2012 (Student)


Exercise (3.3) :

1. Prove the third property of communication: If state i communicates with state j, and state j communicates with state k , then state i communicates with state k .

2. Show that if state i is recurrent and state i does not communicate with state j, thenPij 0. This implies that once a process enters a recurrent class of states it can neverleave that class closed class.

3. Let X be a Markov chain with state space E a, b, c, transition matrix

P

0 13

23

14

34

0

25

0 35

and the initial distribution 25

, 15

, 25

. Compute the following

a. P X 1 b, X 2 b, X 3 b, X 4 a, X 5 c| X 0 a

b. P X 1 a, X 2 c, X 3 c, X 4 a, X 5 b| X 0 c

c. P X 1 b, X 3 a, X 4 c, X 6 b| X 0 a

d. P X 1 b, X 2 b, X 3 a

e. P X 2 b, X 5 b, X 6 b

4. Classify the states of the Markov chains with the following transition matrices

a. P

0.8 0 0.2 0

0 0 1 0

1 0 0 0

0.3 0.4 0 0.3

b. P

0 0 0.4 0.6 0

0 0.2 0 0.5 0.3

0.5 0 0.5 0 0

0 0 1 0 0

0.3 0 0.5 0 0.2

c. P

0.5 0 0 0.5 0

0 0.6 0 0 0.4

0.3 0 0.7 0 0

0 0 1 0 0

0 1 0 0 0

d. P

0 0 0 1

0 0 0 1

12

12 0 0

0 0 1 0

e. P

0.8 0 0 0 0 0.2 0

0 0 0 0 1 0 0

0.1 0 0.9 0 0 0 0

0 0 0 0.5 0 0 0.5

0 0.3 0 0 0.7 0 0

0 0 1 0 0 0 0

0 0.5 0 0 0 0.5 0

f . P

0 12

12

12

0 12

12

12

0

45

7/27/2019 Sstt311 Sec3 2012 (Student)


g. P

12

12

0 0 0

12

12

0 0 0

0 0 1

2

1

2

0

0 0 12

12

0

14

14

0 0 12

h. P

12

0 12

0 0

14

12

14

0 0

1

2

0 1

2

0 0

0 0 0 12

12

0 0 0 12

12

i. P

0 0 1 0 0 0 0

0 0.2 0 0 0.4 0.4 0

0.8 0 0 0 0.2 0 0

0 0 0 0 0 1 0

0 0 1 0 0 0 0

0 0 0 0.7 0 0.3 0

0 0 0 0 0 0 1

j. P

0 0 1 0 0 0 0 0

0 0 1 0 0 0 0 0

0 0 0 0.5 0 0.5 0 0

1 0 0 0 0 0 0 0

0 1 0 0 0 0 0 0

0.3 0.7 0 0 0 0 0 00.2 0.4 0 0 0.1 0 0.1 0.2

0 0 0.3 0 0 0.4 0 0.3

46

7/27/2019 Sstt311 Sec3 2012 (Student)


3.5 Limiting Probabilities

Definition :

3.17. Regular stochastic matrix - A stochastic matrix P is said to be regular if all the entries of some power Pm are positive, i.e. P ij 0. #

Example (3.18) :

The stochastic matrix

A

0 1

12

12

is regular since

A2

0 1

12

12

0 1

12

12

12

12

14

34

is positive in every entry.The stochastic matrix

B

1 0

1

2

1

2

is not regular since

B2

1 0

34

14

, B3

1 0

78

18

, B4

1 0

1516

116

In fact every power Bm will have 1 and 0 in the first row. #

Consider a two-state Markov chain where the one-step transition probability matrix isgiven by

P 0.7 0.3

0.4 0.6

From the above it follows that

P4

0.5749 0.4251

0.5668 0.4332

and

47

7/27/2019 Sstt311 Sec3 2012 (Student)


P8

0.572 0.428

0.570 0.430

Note that the matrix P8 is almost identical to the matrix P4, and secondly, that each of

the rows of P8 has almost identical entries. In fact it seems that P ijn is converging to somevalue as n Ý which is the same for all i. In other words, there seems to exist a limitingprobability that the process will be in state j after a large number of transitions, and this valueis independent of the initial state.

Theorem (3.1) :

1. For an irreducible ergodic Markov chain limn Ý

Pijn exists and is independent of i.

2. Letting

j

limn Ý Pij

n

, j 0then j is the unique nonnegative solution of

j ࢣi0

Ý

iPij, j 0

ࢣ j0

Ý

j 1 (3.5) #

Remark (3.3) :

1. An expression for P X n1 j can be derived by conditioning on the state at time n. Thatis,

P X n1 j ࢣi0

Ý

P X n1 j ࢯ X n iP X n i

ࢣi0

Ý

PijP X n i

Letting n Ý, and assuming that we can bring the limit inside the summation, leads to

j ࢣi0

Ý

Pij i

2. j, the limiting probability that the process will be in state j at time n, also equals thelong-run proportion of time that the process will be in state j. #

Example (3.19) : A model of class mobility

A problem of interest to sociologists is to determine the proportion of society that has an

upper- or lower-class occupation. One possible mathematical model would be to assume thattransitions between social classes of the successive generations in a family can be regarded astransitions of a Markov chain. That is, we assume that the occupation of a child depends onlyon his or her parent s occupation. Let us suppose that such a model is appropriate and that the

transition probability matrix is given by

48

7/27/2019 Sstt311 Sec3 2012 (Student)


P

0.45 0.48 0.07

0.05 0.70 0.25

0.01 0.50 0.49

(3.6)

That is, for instance, we suppose that the child of a middle-class worker will attain an upper-,middle or lower-class occupation with respective probabilities 0,05, 0.70 and 0.25.

The limiting probabilities i, thus satisfy

0 0.450 0.051 0.012

1 0.480 0.701 0.502

2 0.070 0.251 0.492

0 1 2 1

Hence,

0 0.07, 10.62, 2 0.31

In other words, a society in which social mobility between classes can be described by aMarkov chain with transition probability matrix given by 3.6 has, in the long run, 7 percentof its people in upper-class jobs, 62 percent of its people in middle-class jobs and 31 percentin lower-class jobs. #

Example (3.20) : The Hardy-Weinberg Law and a Markov chain in genetics

Consider a large population of individuals, each of whom possesses a particular pair of

genes, of which each individual gene is classified as being of type A or type a. Assume thatthe proportions of individuals whose gene pairs are AA, aa or Aa are, respectively, p0, q0 andr 0 p0 q0 r 0 1. When two individuals mate, each contributes one of his or her genes,chosen at random, to the resultant offspring. Assuming that the mating occurs at random, inthat each individual is equally likely to mate with any other individual, we are interested indetermining the proportions of individuals in the next generation whose genes are AA, aa or

Aa. Calling these proportions p, q and r , they are easily obtained by focusing attention on anindividual of the next generation and then determining the probabilities for the gene pair of

that individual.

Solution :

To begin, note that randomly choosing a parent and then randomly choosing one of its

genes is equivalent to just randomly choosing a gene from the total gene population. Byconditioning on the gene pair of the parent, we see that a randomly chosen gene will be type A

with probability

P A P A ࢯ AA p0 P A ࢯ aaq0 P A ࢯ Aar 0

p0 r 02

Similarly, it will be type a with probability

Pa q0 r 02

49

7/27/2019 Sstt311 Sec3 2012 (Student)


Thus, under random mating a randomly chosen member of the next generation will be type AA

with probability p, where

p P AP A p0 r 02

2

Similarly, the randomly chosen member will be type aa with probability

q PaPa q0 r 02

2

and will be type Aa with probability

r 2P APa 2 p0 r 02

q0 r 02

Since each member of the next generation will independently be of each of the three genetypes with probabilities p, q, r , it follows that the percentages of the members of the nextgeneration that are of type AA, aa or Aa are respectively p, q and r .

If we now consider the total gene pool of this next generation, then p r /2, the fraction of its genes that are A, will be unchanged from the previous generation. This follows either by

arguing that the total gene pool has not changed from generation to generation or by thefollowing algebra

p r 2

p0 r 02

2

p0 r 02

q0 r 02

p0 r 02

p0 r 02

q0 r 02

p0 r 02

since p0 r 0 q0 1

P A (3.7)

Thus, the fractions of the gene pool that are A and a are the same as in the initial generation.From this it follows that, under random mating, in all successive generations after the initialone the percentages of the population having gene pairs AA, aa and Aa will remain fixed atthe values p, q and r . This is known as the Hardy-Weinberg law.

Suppose now that the gene pair population has stabilized in the percentages p, q, r and let

us follow the genetic history of a single individual and her descendants. For simplicity,assume that each individual has exactly one offspring. So, for a given individual, let X ndenote the genetic state of her descendant in the n th generation. The transition probabilitymatrix of this Markov chain, namely,

AA aa Aa

AA p r 2

0 q r 2

aa 0 q r 2

p r 2

Aap

2

r 4

q

2

r 4

p

2

q

2

r 2

is easily verified by conditioning on the state of the randomly chosen mate. The limitingprobabilities for this Markov chain should just be p, q and r . To verify this we must show thatthey satisfy equation 3.5. Because one of the equations in equation 3.5 is redundant, itsuffices to show that

50

7/27/2019 Sstt311 Sec3 2012 (Student)


p p p r 2

r p

2

r 4

p r 2

2

q q q r 2

r q

2

r 4

q r 2

2

p q r 1

But this follows from equation 3.7, and thus the result is established. #

Definition (s) :

3.18. Stationary probabilities - The long run proportions j, j 0, are often called stationary

probabilities. The reason being that if the initial state is chosen according to theprobabilities j, j 0, then the probability of being in state j at any time n is also equal to j. That is, if

P X 0 j j j 0

then

P X n j j for all n, j 0

The preceding is easily proven by induction, for if we suppose it true for n ࢤ 1, thenwriting

P X n j ࢣi

P X n j ࢯ X nࢤ1 iP X nࢤ1 i

ࢣi

Pij i by the induction hypothesis

j by equation 3.5 #

3.19. Expected number of transitions - For state j, define m jj to be the expected number of

transitions until a Markov chain, starting in state j, returns to that state. Since, on theaverage, the chain will spend 1 unit of time in state j for every m jj units of time, it followsthat

j 1

m jj

In words, the proportion of time in state j equals the inverse of the mean time betweenvisits to j. #

Example (3.21) : The gambler s ruin problem

Consider a gambler who at each play of the game has probability p of winning one unit

and probability q 1 ࢤ p of losing one unit. Assuming that successive plays of the game areindependent, what is the probability that, starting with i units, the gambler s fortune will reach

N before reaching 0?

Solution :

If we let X n denote the player s fortune at time n, then the process X n, n 0,1,2,... is aMarkov chain with transition probabilities

51

7/27/2019 Sstt311 Sec3 2012 (Student)


P00 P NN 1

Pi,i1 p 1 ࢤ Pi,iࢤ1 i 1,2,..., N ࢤ 1

This Markov chain has three classes, namely, 0, 1,2,..., N ࢤ 1 and N ; the first and thirdclass being recurrent and the second transient. Since each transient state is visited only finitelyoften, it follows that, after some finite amount of time, the gambler will either attain his goalof N or go broke.

Let Pi, i 0,1,..., N , denote the probability that, starting with i, the gambler s fortune willeventually reach N . By conditioning on the outcome of the initial play of the game we obtain

Pi pPi1 qPiࢤ1 i 1,2,..., N ࢤ 1

or equivalently, since p q 1,

pPi qPi pPi1 qPiࢤ1

or

Pi1 ࢤ Pi q

p Pi ࢤ Piࢤ1 i 1,2,..., N ࢤ 1

Hence, since P0 0, we obtain from the preceding line that

P2 ࢤ P1 q

p P1 ࢤ P0 q

p P1

P3 ࢤ P2 q

p P2 ࢤ P1 q

p

2

P1

P i ࢤ P iࢤ1 q

p Piࢤ1 ࢤ Piࢤ2 q

p

iࢤ1P1

P N ࢤ P N ࢤ1 q

p P N ࢤ1 ࢤ P N ࢤ2 q

p

N ࢤ1P1

Adding the first i ࢤ 1 of these equations yields

Pi ࢤ P1 P1q

p q

p

2

... q

p

iࢤ1

or

Pi

1 ࢤ q / pi

1 ࢤ q / p

P1 if q

p 1

iP1 if q

p 1

Now, using the fact that P N 1, we obtain that

P1

1 ࢤ q / p

1 ࢤ q / p N if p 1

2

1 N

if p 12

and hence

52

7/27/2019 Sstt311 Sec3 2012 (Student)


P i

1 ࢤ q / pi

1 ࢤ q / p N if p 1

2

i N

if p 12

Note that, as N Ý,

P i

1 ࢤq

p

i

if p 12

0 if p 12

Thus, if p 12

, there is a positive probability that the gambler’s fortune will increase

indefinitely; while if p 12

, the gambler will, with probability 1, go broke against an

infinitely rich adversary. #

53

7/27/2019 Sstt311 Sec3 2012 (Student)


Exercise (3.4) :

1. Suppose a boy and a girl decide to flip coins; the one coming closest to the wall wins.The girl, being the better player, has a probability 0.6 of winning on each flip.

a. If the girl starts with 5 coins and the boy with ten, what is the probability that the girl

will wipe the boy out?b. What if the girl starts with 10 coins and the boy with 20?

2. A particle moves on a circle through points which have been marked 0,1,2,3,4 in

clockwise order. At each step it has a probability p of moving to the right clockwiseand 1 ࢤ p to the left counterclockwise. Let X n denote its location on the circle after thenth step. The process X n, n 0 is a Markov chain.

a. Find the transition probability matrix.

b. Calculate the limiting probabilities.

c. What is the expected number of steps the particle takes to return to the starting

position?d. What is the probability that all other positions are visited before the particle returns

to its starting state?

3. Each of two switches is either on or off during a day. On day n, each switch willindependently be on with probability

1 number of on switches during day n ࢤ 1

4

For instance, if both switches are on during day n ࢤ 1, then each will independently be on

during day n with probability 34

.

a. What fraction of days are both switches on? What fraction are both off?

b. Determine the limiting probabilities j for the model.

4. Let i denote the long-run proportion of time a given Markov chain is in state i.

a. Explain why i is also the proportion of transitions that are into state i as well asbeing the proportion of transition that are from state i.

b. iPij represents the proportion of transitions that satisfy what property?

c. ࢣi iPij represent the proportion of transitions that satisfy what property?

d. Using the preceding explain why

j ࢣiiPij

5. Suppose you use a food vending machine every day at lunchtime. Assume that themachine can be in two states

0 Working

1 Out of order

Suppose that, when the machine is working on a particular day, the probability is that itwill be out of order the next day; conversely, suppose that, when the machine is out of order on a particular day, the probability is that it will be back in working order thenext day.

54

7/27/2019 Sstt311 Sec3 2012 (Student)


a. Determine the one-step transition matrix P.

b. Calculate the limiting probabilities.

c. Find the mean recurrence times for states 0 and 1.

55

7/27/2019 Sstt311 Sec3 2012 (Student)


3.6 Branching Processes

In this section we consider a class of Markov chains, known as branching processes,

which have a wide variety of applications in the biological, sociological and engineeringsciences.Consider a population consisting of individuals able to produce offspring of the same

kind. Suppose that each individual will, by the end of its lifetime, have produced j newoffspring with probability P j, j 0, independently of the numbers produced by otherindividuals. We suppose that P j 1 for all j 0. The number of individuals initially present,denoted by X 0, is called the size of the zeroth generation. All offspring of the zeroth

generation constitute the first generation and their number is denoted by X 1. In general, let X ndenote the size of the n th generation. It follows that X n, n 0,1,... is a Markov chain havingas its state space the set of nonnegative integers.

Note that state 0 is a recurrent state, since clearly P00 1. Also, if P0 0, all other statesare transient. This follows since P i0 P0

i , which implies that starting with i individuals thereis a positive probability of at least P0

i that no later generation will ever consist of i individuals.Moreover, since any finite set of transient states 1,2,..., n will be visited only finitely often,

this leads to the important conclusion that, if P0 0, then the population will either die out orits size will converge to infinity.

Let

ࢣ j0

Ý

jP j

denote the mean number of offspring of a single individual, and let

2 ࢣ

j0

Ý

j ࢤ 2P j

be the variance of the number of offspring produced by a single individual.Let us suppose that X 0 1, that is, initially there is a single individual present. We

calculate E X n and V X n by first noting that we may write

X n ࢣi1

X nࢤ1

Z i

where Z i represents the number of offspring of the ith individual of the n ࢤ 1st generation. By

conditioning on X nࢤ1, we obtain

E X n E E X n ࢯ X nࢤ1

E E ࢣi1

X nࢤ1

Z i ࢯ X nࢤ1

E X nࢤ1

E X nࢤ1

where we have used the fact that E Z i . Since E X 0 1, the preceding yields

56

7/27/2019 Sstt311 Sec3 2012 (Student)


E X 1

E X 2 E X 1 2

E X

n

E

X

nࢤ1

n

Similarly, V X n may be obtained by using the conditional variance formula

V X n E V X n ࢯ X nࢤ1 V E X n ࢯ X nࢤ1

Now, given X nࢤ1, X n is just the sum of X nࢤ1 independent random variables each having thedistribution P j, j 0. Hence,

E X n ࢯ X nࢤ1 X nࢤ1 V X n ࢯ X nࢤ1 X nࢤ12

The conditional variance formula now yields

V X

n E

X

nࢤ12

V

X

nࢤ1

2nࢤ1

2V X nࢤ1

2nࢤ1 22nࢤ2

2V X nࢤ2

2nࢤ1 n 4V X nࢤ2

2nࢤ1 n 42nࢤ3

2V X nࢤ3

2nࢤ1 n

n1 6V X nࢤ3

...

...

2nࢤ1 n

... 2nࢤ2 2nV X 0

2nࢤ1 n

... 2nࢤ2

Therefore,

V X n 2nࢤ1 1 ࢤ n

1 ࢤ , 1

n2, 1

(3.8)

Let 0 denote the probability that the population will eventually die out under the assumptionthat X 0 1. More formally

0 limn Ý P X n 0 ࢯ X 0 1

We first note that 0 1 if 1. This follows since

n E X n

ࢣ j1

Ý

jP X n j

ࢣ j1

Ý

1.P X n j

P X n 1Since n 0 when 1, it follows that P X n 1 0, and hence P X n 0 1.

57

7/27/2019 Sstt311 Sec3 2012 (Student)


In fact, it can be shown that 0 1 even when 1. When 1, it turns out that0 1, and an equation determining 0 may be derived by conditioning on the number of offspring of the initial individual, as follows

0 Ppopulation dies out

ࢣ j0

Ý

Ppopulation dies out ࢯ X 1 jP j

Now, given that X 1 j, the population will eventually die out if and only if each of the j

families started by the members of the first generation eventually dies out. Since each familyis assumed to act independently and since the probability that any particular family dies out is just 0, this yields

Ppopulation dies out ࢯ X 1 j 0 j

and thus 0 satisfies

0 ࢣ j0

Ý

0 j

P j (3.9)

in fact when 1, it can be shown that 0 is the smallest positive number satisfyingEquation 3.9.

Example (3.22) :

If P0 12

, P1 14

, P2 14

, then determine 0.

Solution :

Since 34

1, it follows that 0 1. #

Example (3.23) :

If P0 14

, P1 14

, P2 12

, then determine 0.

Solution :

0 satisfies

0 14

140

120

2

or

202 ࢤ 30 1 0

The smallest positive solution of this quadratic equation is 0 12

. #

58

7/27/2019 Sstt311 Sec3 2012 (Student)


Example (3.24) :

In Examples 3.22 and 3.23, what is the probability that the population will die out if it

initially consists of n individuals?

Solution :

Since the population will die out if and only if the families of each of the members of theinitial generation die out, the desired probability is 0

2. For Example 3.22 this yields 0n 1

and for Example 3.23 0n

12

n

. #

59

7/27/2019 Sstt311 Sec3 2012 (Student)


Exercise (3.5) :

1. Consider a branching process having 1. If X 0 1, determine the expected number of individuals that ever exist in this population. What if X 0 n?

2. For a branching process, calculate 0 when

a. P0 14

, P2 34

b. P0 14

, P1 12

, P2 14

c. P0 16

, P1 12

, P3 13

Sstt311 Sec3 2012 (Student)

Documents

Transcript of Sstt311 Sec3 2012 (Student)