Polya urn

Pollyanna

May 1, 2014

1 Intro

In probability theory, statistics and combinatorics Urn Models have a long and colorful history. Clas-sical mathematicians Laplace and Bernoulis, amongst others, have made notable contributions to thisclass of problems. For example, Jacob Bernoulli in his Ars Conjectandi (1731) considered the problemof determining the proportions of di↵erent-colored pebbles inside an urn after a number of pebbles aredrawn from the urn. This problem also attracted the attention of Abraham de Moivre and Thomas Bayes.

Urn models can be used to represent many naturally occurring phenomena, such as atoms in a sys-tem, biological organisms, populations, economic processes, etc. In fact, in 1923 Eggenberger and Polyadevised an urn scheme (commonly known as Polya Urn) to model such (infinite) processes as the spreadof contagious diseases. In one of its simplest forms, the Polya Urn originally contains x white balls andy black balls. A ball is randomly drawn from the urn and replaced back to the urn along with additional(fixed) h balls of the same color as the drawn ball. This process continues forever. The fraction of whiteballs in the urn converges almost surely to a random limit, which has a beta distribution with parametersx/h and y/h.

There are a number of generalizations of the basic Polya Urn model. For example, when a ball isdrawn, not only it is returned to the urn, but if it is white, then A additional white balls and B additionalblack balls are returned to the urn, and if the drawn ball is black, then C additional white balls and Dadditional black balls are returned to the urn. This set-up involves the matrix [A B; C D], known as thereplacement matrix; a rule used for e↵ecting change in the composition and number of balls after eachdrawing.

Or, instead of only two di↵erently colored balls, there may be several di↵erently colored balls presentin the urn. Or, instead of the additional fixed h balls noted above, after every draw a di↵erent number,that may or may not be deterministic, may be added to the urn.

In the rest of the paper, Polya urn model is carefully studied in section 2. As an application of polyaurn, edge-reinforced random walk is studied in section 4. Finally, numerical simulation is presented insetction 5.

2 Polya Urn study

2.1 Definition and Notations

A Polya urn is an urn containing balls of up to K (K � 2) di↵erent colors. The urn evolves in discretetime steps - at each step, one ball is sampled uniformly at random; The color of the withdrawn ball isobserved, and two balls with that color are returned to the urn.

2.2 Count vector as a markov chain

We start with the markov property of Polya urn, which is not only of importance in itself, but also servesas an important building block in later analysis.

1

Lemma 2.1. Let Xjn

denote the number of balls with jth color in the urn at time step n, then the vector

{(X1n, ..., XKn

)}n�1 is a homogeneous Markov Chain.

To show it is a Markov Chain, observed that at each step one ball is sampled uniformly at random.Picking a ball of a particular color only relies on balls in the last step. Since the transition matrix doesnot depend on time index n, it is homogeneous.

Despite the simple proof, it is not clear how this result can be used since the state space of the vectoris infinite and it never goes back to the same state.

2.3 Ratio as a Martingale

Another quantity of interests is the ratio of number of balls with di↵erent colors. Statistically it is veryimportant and sometimes good enough to know this ratio.

Note it is not a markov chain. For example, at one time step this ratio is one half. Then the nextstep we don’t know what the ratio value would be since the ratio may come from the situation with 1white ball plus 1 black ball or the situation with 2 white balls plus 2 black balls, which, however, givedi↵erent next positions.

But it is a martingale as stated in the following lemma.

Lemma 2.2. Let ⇢i

= XinPj Xjn

, n � 0 be the ratio of ith color ball in the urn at the conslusion of step n.

Define Fn

= �{ci

}, where ci

is the color of ith ball.Then (⇢n

)n�0 is a Martingale w.r.t. F

n�1.

Proof.

E[⇢n+1|Fn

] = E[X

i,n+1PX

j,n+1|F

n

] = E[X

i,n+1PX

j,0 + n+ 1|F

n

] =1P

Xj,0 + n+ 1

E[Xi,n+1|Fn

]

=1P

Xj,0 + n+ 1

E[Xi,n

+ 1(cn+1 = i)|F

n

] =1P

Xj,0 + n+ 1

(Xi,n

+X

i,nPX

j,0 + n) = ⇢

n

Which implies that as the urn evolves the expected ratio of a particular color stays the same.Furthermore, when n ! 1, we have:

Lemma 2.3. Starting from 1 ball of each color, lim ⇢n

exists almost surely. y1 is uniform in [0, 1].

Proof. Since ⇢i

is a martingale and upper bounded by 1, by convergence theorem, we know the limitexists almost surely. The distribution of y1 is uniform is proved later in more general cases.

As time goes on su�ciently long, the proportion of picking balls of di↵erent colors stablizes. So, whatis the proportion’s distribution? In the next subsection we are going address this problem.

2.4 Proportion Limiting Distribution

Seeing that Polya urn is a Markov Chain, it is natural to seek its limiting discrete distributions underlyingthe process in closed form. Particularly, we are interested in how many balls of a color would be pickedwhen the process goes on for a su�ciently long time. As the first step, we give the closed form probabilityof picking a given number of balls starting from a particular configuration.

Lemma 2.4. Suppose a Polya urn starts with Xj0 balls with jth color (j = 1, ...,K) and ⌧0 =

Pj

Xj0.

Let Xjn

be the number of jth (j = 1, 2, ...,K) color balls drawn in the Polya urn after n draws, then

P (Xjn

= yj

, j = 1, ...,m) =

QK

j=1[Xj0(Xj0 + 1) ⇤ ... ⇤ (Xj0 + y

j

� 1)]

⌧0(⌧0 + 1) ⇤ ... ⇤ (⌧0 + n� 1)

✓n

y1, ..., ym

◆(1)

2

Proof makes use of the markov property by writing the probability as the product of probabiliy of nconsequtive draws and also notice there are

�n

y1,...,ym

�ways of picking n balls. Details are in the appendix.

While we see the transition probability is addative in time, Theorem 2.4 gives its closed form based onstarting and finishing numbers of balls. Starting from this, we could derive the expectation and varianceof X

jn

and also the Polya urn’s limiting distribution stated as follows.

Theorem 2.5. Suppose a Polya urn starts with Xj0 balls of jth color (j = 1, ...,K). Let X

jn

be the

number of jth (j = 1, 2, ...,K) color balls drawn in the Polya urn after n draws, when n ! 1,

(X1n

n, ..,

XKn

n)

D�! Dir(X10, ..., XK0). (2)

Proof follows Lemma 2.4 and Stirling approximation. Details are given in the appendix.

Remark Theorem 2.5 tells what proportion of colors of balls we are supposed to get if we start withsome known number of balls and draw infinitely many times. This depends on how many balls of eachcolor we start from. Asymptotically, we will have a fixed proportion to sample di↵erent colors of ballsand this proportion should follow Dirichlet distribution with parameters the initial numbers of balls.

2.5 An alternative perspective – Bayesian generative model

So far, we have reviewed Polya urn in its classical scheme where one repeatedly picks balls with di↵erentcolors from one urn. Each time picking a specific color relies on what we already have in the urn atthat time. In this section we provide another perspective reviewing Polya urn. It is more of a Bayesiantreatment and it’s easy to be extended to more general case, as we will show in section 2.6 whereK = +1.

We assume balls are drawn from one distribution G independently instead of drawn from one time-adaptive distribution. G = multi(p1, ..., pK) is a probability measure over all K colors and specifies amultinomial distribution. As standard Bayesian treatment, we treat G as a random variable and assigna prior distribution Dir(↵1, ...,↵K

) to it where ↵i

= Xi0 8i. Then the evolving process of this new urn

is described as follows,

G ⇠ Dir(↵1, ...,↵K

)

color(i) ⇠ G 8i(3)

Now it is no longer sequential picking, but all balls are generated independently. Since the truedistribution of G is unknown, We need to infer the conditional distribution of G given the observed balls.Whenever a new ball is observed, we update our belief on G and then make prediction on the upcomingball. Note that the underlying distribution of G is assumed to be unchanged.

So how does this relate to standard Polya urn model? The next Lemma shows that after observinga given number of balls, one has the same prediction on the color of the next ball as the standard Polyaurn model.

Theorem 2.6. By choosing prior Dir(X10, ..., XK0) where Xj0 denotes the initial number of balls with

jth color (j = 1, 2, ...,K), the posterior distribution of G given Xj

jth color balls, is Dir(X1, ..., XK

).The expected probability of picking an ith color ball is

XiPj Xj

, which is the same as given by Polya urn

model rule.

The proof follows the fact that Dirichlet and multinomial are conjugate and that it’s easy to computeexpectation of random variable from dirichlet distribtuion.

Remark While the probability of picking one color ball is changing during the process in both views ofPolya urn, the mechanisms are di↵erent. In the original view of polya urn, it relies on the assumptionthat we always pick balls uniformly in the urn. Since number of balls in the urn is changing, we havetime-adaptive probability. In the alternative view, balls are assumed to be drawn from one underlyingprobability distribution. What sequentially changes is the belief we have on its distribution.

3

Not surprisingly, as in non-Bayesian analysis, we could derive the probability of observing a givennumber of balls starting from some balls as stated in the following lemma.

Lemma 2.7. Let Xj

be the number of jth (j = 1, 2, ...,K) color balls observed, Xjn

be the number of

jth color balls appeared in the next n draws according to the conditional probability of G, then

p(Xjn

= yj

, 8j|X

j

yj

= n) =�(P

j

Xj

)Q

K

j=1 �(Xj

+ yj

)

�(P

j

Xj

+ yj

)Q

K

j=1 �(Xj

)

✓n

y1, ..., ym

◆(4)

Proof. For a fixed G = (p1, ..., pK), we know the distribution of (X1n,...,XKn

) is multinomial. Since G isa random variable with conditional probability Dir(X1, ..., XK

), we intergrate over G and get the resultin equation (4).

Remark Lemma 2.8 recovers the result in Lemma 2.4 but in a di↵erent way. The proof is morestraightforward since all subsequential samples are considered drawn independently from conditionaldistribution of G, which could be computed once given X

j

.

Corollary 2.8. Let Xj

be the number of jth (j = 1, 2, ...,K) color balls observed, Xjn

be the number of

jth color balls appeared in the next n draws according to the conditional probability of G, when n ! 1

p(Xjn

/n = yj

, 8j|X

j

yj

= 1)n!1����! Dir(X1, ..., XK

). (5)

Apparently this corollary recovers Theorem 2.5.

2.6 Extension: Polya urn with unbounded color numbers

We extend Polya urn with fixed number of color balls to Polya urn with possibly infinitely many colorsvia the generative model discussed in 2.5.

Our motivation is that, in real life data, we don’t know the total number of “colors” beforehand.Therefore it is unwise to set a fixed K before we observe the whole data population. Moreover, even ifwe observe the entire population, we have no clue whether the upcoming “ball” has a di↵erent “color”than the balls observed so far. For example, in Brain Image research, one models an unknown number ofspatial activation patterns in fMRI images. Another example, in text modeling, one models an unknownnumber of topics across several corpora of documents. Current Polya urn type model is limited in theseflexible situations. Is there a principled way to deal with this more flexible model?

It turns out to be highly non-trivial to do that due to the fact in equation (3) G is defined ina finite space, thus too restricted to deal with unbounded number of colors. One needs to extend itsspace to infinite space. Particularly, one defines a proper prior of G that has a support over infinite space.

Our solution resorts to Dirichlet Process (DP) [4, 3] . DP defines a distribution over distributionwhere the space could be infinite. We refer the formal definition of DP with parameter to [3] and givean equivalent one here,

Definition Let X be a set and A be a �-field of subsets of X . ↵ is a positive number and G0 is aprobability measure on (X ,A ). We say a Dirichlet Process has sample path G to be a probability on(X ,A ). Particularly, for any measureable partition of X , say {A

i

}ni=1, we have (G(A1), ..., G(A

n

)) ⇠Dirichlet(↵G0(A1), ...,↵G0(An

)).

To make sure this indeed defines a proper random process, we need to examine that the Kolmogorovconsistency conditions are satisfied. We refer to [3] for details of examination. In what follows we assumeit defines a proper random process.

Now we are ready to define the generalized Polya urn model

G ⇠ DP (↵, G0)

color(i) ⇠ G 8i(6)

4

where the G0 could be any distribution over colors, and scalar ↵ controls how likely one is to observe anew color. Dirichlet process has the following two good properties.

Lemma 2.9. A sample from Dirichlet Process is a discrete distribution, made of up countably infinite

number of point masses.

Lemma 2.10. Let G be a sample from Dirichlet process DP (↵, G0) on the color space and let cn

be the

color of nth ball observed, then the conditional probability of G is still Dirichlet Process with parameters

↵ and G0 +P

N

n=1 �cn .

Remark Lemma 2.9 makes it possible for us to use DP to model clustering problem. If not discrete,then the possibility of picking two identical points from G is zero. Lemma 2.10 tells that Dirichlet Processis conjugate with Multinomial distribution, also, making computation tractable.

Theorem 2.11. Let Xi

denote the number of balls with ith color drawn independently from generalized

Polya urn model defined in (6), then the probability of observing the next ball with ith color is

XiPXj+↵

;

the probability of observing the next ball with a new color is

↵PXj+↵

.

The proof follows Lemma 2.10 and expectation of random variable from Dirichlet Process.

Remark It is easy to see that (6) is an extension of standard Polya urn model since it allows us todraw a new color with non-zero probability. When ↵ goes to 0, we recover standard Polya urn model(3); when ↵ goes to infinity, (6) becomes drawing balls from stationary distribution G0. Meanwhile,it also explicitly allows flexible choice of G0, which is the base probability distribution over color space.For instance, it is easy to incorporate other attributes of colors in order to pick a new color.

3 Generalized Polya Urn with a Replacement Vector

We generalize the simple Polya urn containting two colors, black and white. Instead of replacing 2 ballsof the observed color, a deterministic number of balls of the observed color are returned to the urn.Define replacement vectors ~r = (r0, r1, . . . , ) and ~w = (w0, w1, . . . , ), for the n-th time of observing ablack ball, r

n

number of black balls are returned to the urn; similary, for the n-th time of observing awhite ball, w

n

number of whilte balls are returned.In the original Polya urn problem with replacement of 2(or increment of 1), it is easy to show that

there are infinitely many times we draw either ball.

Lemma 3.1. In the basic Polya’s urn problem with increment 1, there are infinitely many times we

choose a ball of either color.

Proof. Assume initially there are t balls in total. W.L.O.G. assume we identify one red ball initially inthe urn. Then in step n, p

n

, the probability of drawing the one we identified is 1/(t + n).P

n

pn

= 1and from Borel-Cantelli we conclude we draw the identified red balls infinitely many times. The restfollows.

Luckily we have a more general result for generalized Polya urn from Herman Rubin.

Theorem 3.2. Let ~r = (r0, r1, · · · ), ~w = (w0, w1, · · · ) be two sequences of nonnegative numbers such that

r0 > 0 and w0 > 0. Put Rk

=P

k

i=0 ri and Wk

=P

k

i=0 wi

. The first entry of the infinite sequence is

type r with probability R0/(R0 +W0), type w with probability W0/(R0 +W0). Given the first n entries

consist of x r’s and y = n � x w’s, in a given order, the probability that the n + 1st entry is r equals

Rx

/(Rx

+ Wy

), and the probability it is w equals Wy

/(Rx

+ Wy

). Let pr

= P (all but finitely many

elements of the sequence are red) and pw

= P (all but finitely many elements of the sequence are white).

Put �(r) =P1

i=0 R�1i

, �(w) =P1

i=0 W�1i

, then

1. If �(r) < 1 and �(w) < 1 then pr

> 0, pw

> 0 and pr

+ pw

= 1.

2. If �(r) < 1 and �(w) = 1, pr

= 1.

3. If �(r) = 1 and �(w) = 1, both pr

and pw

equal 0.

5

Example Assume the initial urn is (1,1). ri

= wi

= 1, 8i > 0. Then Rk

= 2k + 1,Wk

= 2k + 1. Wehave the same conclusion as above: there are infinitely many times we draw the balls of each color.

Example ai

= i, 8i > 0. Then Rk

= O(k2),Wk

= O(k2). �(r) < 1,�(w) < 1. So pr

+ pw

= 1, whichmeans either we draw only finitely many red balls, or only finitely many white balls.

4 Application on Edge-Reinforced Random Walk

Recall that a simple random walk ~X on Z is a process that takes values on the Z at any time i � 0, i.e.,X

i

2 Z, andP (X

i+1 = Xi

+ 1) = P (Xi+1 = X

i

� 1) =1

2.

Further, the edge-weighted random walk ~X on Z is a process that takes values on the Z at any timei � 0, and

P (Xi+1 = X

i

+ 1) =w(hi, i+ 1i)

w(hi, i+ 1i) + w(hi, i� 1i) ,

P (Xi+1 = X

i

� 1) =w(hi, i� 1i)

w(hi, i+ 1i) + w(hi, i� 1i) ,

where w(hi, i� 1i) is the weight of the edge hi, i� 1i.An Edge-Reinforced Random Walk(ERRW) on Z is an edge-weighted random walk on Z but every

time the edge is crossed, the weight of the crossed edge increases so that the process tends to revisit theedge. Thus, an ERRW process ~X is not a Markovian process because the probability of traversing anedge depends on the number of times it has traversed in its entire history.

Analogous with a weighted random walk, let w(n, i) be the weight of edge ei

= hi, i + 1i at time n.We say that ~X traverses e

i

at some time n > 0, if (Xn�1, Xn

) 2 {(i, i+ 1), (i+ 1, i)}. After the processtraverses edge e

i

at time n, we add a nonnegative increment a to w(n, i) so that the weight of the edgeei

at time (n+ 1) is w(n+ 1, i) = w(n, i) + a � w(n, i). Here are some notations.

Definition A walk is of sequence type if there is a sequence ~↵ = ↵1,↵2, . . . , of nonnegative numberssuch that at the k-th crossing of an edge it is reinforced with a

k

. Thus, if �(n, j) is the number of times(X0, X1, . . . , Xn

) crosses ej

then

w(n, j) = w(0, j) +

�(n,j)X

i=1

↵i

, a.s

Definition A walk has i.i.d reinforcement if there exists a sequence Z1, Z2, . . . , of i.i.d. nonnegativerandom variables such that

w(n, j) = w(0, j) +X

0in:(Xi,Xi+1)2{(j,j+1),(j+1,j)}

Zi

,

Definition Let v(l, j) denote the weight of edge j when it has been traversed l times. Thus

v(l, j) = w(0, j) +lX

i=1

↵i

, for sequence type reinforcement,

v(l, j) = w(0, j) +lX

i=1

Zi

, for i.i.d type reinforcement.

One key question for the ERRW is the recurrence of the process. Compared with the simple randomwalk on Z, it is not hard to imagine the ERRW is more likely to be recurrent or even stuck on a finitesubgraph of Z. Before showing the results on recurrence of the process, we first relate the ERRW processto the Polya’s urns.

6

4.1 Edge-Reinforced Random Walk via Polya’s urn

Imagine for each vertex j 2 Z we associate it with a Polya’s urn Uj

with two colors black and white.The black color indicates ‘to go up‘ and the white color indicates ‘to go down‘. The number of whiteand black balls in urn U

j

at time n keeps track of the weight of edge ej�1 = hj� 1, ji and e

j

= hj, j+1i,respectively.

For example, if we are to model a ERRW process with all initial weights and all increments equal to1(which is called Diaconis walk), we start at vertex 0 with 1 ball of each color in urn U0. We draw a ballfrom the urn U0 and walk in the direction indicated by the color of the ball. We then come to a newvertex with an associated urn, which will decide the next step. Once we reach a new vertex j for thefirst time, we visit e

j�1 but never ej

. So if j > 0, we add one more black ball to the urn Uj

at vertexj, and it should have 2 black balls and 1 white ball at the first time we reach the vertex j. Similarly, ifj < 0, then the urn U

j

at vertex j has 1 black balls and 2 white balls at the first time we reach vertex j.However, in the above setting, the urns associating with vertices are not independent of each other.

Consider the following scheme in the Diaconis walk. The first move is chosen by randomly drawing aball from the urn U0, and then we walk along the edge e1(or e0) in the direction corresponding to theresulting color. Once we leave the vertex 0, we put back the ball the drawn ball into the urn, alongwith TWO more balls of the same color. The idea behind this scheme is that at the time the walk everreturns to the vertex 0 and want to use urn U0 to make the next decision, it must have already traversedthe edge e1(or e0) twice, once from the vertex 0 to vertex 1 and once in the reversed direction back.Thus the weight of the chosen edge will have increased by exactly two. For the other vertex j 6= 0, theybehave in the same way as that for the vertex 0 once after the walk reaches it. Initially, we put 2 blackballs and 1 white ball into the urn U

j

for j > 0, and we put 1 black ball and 2 white balls into the urnUj

for j < 0. It is so because upon the first time the walk arrives at the vertex j, the edge ej�1 must

have already traversed once from vertex j � 1 to j. See Figure 4.1 for the demonstration of the initialsetting of the Diaconis walk via Polya’s urn. Under this scheme, the urns at each vertex are independentbecause the outcome of drawing from one urn cannot a↵ect any other, which is a very nice property.

Figure 1: The initial setting of the Diaconis walk via Polya’s urn [ref [2]]

For a general ERRW with reinforcement of a sequence ~↵ = ↵1,↵2, . . . ,. Define Tk;i = min{T

k;i >Tk�1;i;XTk;i = i}, i.e. the k-th time the walk stops at i. The setting of the Polya’s Urns model

corresponding to the ERRW is as follows.

1. Now first let’s look at the origin. T0;0 = 0, and the weight is (1, 1). If the first move is to theright, then the weight becomes (1, 1 + a1), and at time T1;0, the first time the walk comes back,the weight becomes (1, 1 + a1 + a2). Similarly, if the first move was to the left, the weight at T1;0

will be (1 + a1 + a2, 1). Actually this means the origin can be thought of as a Polya’s urn, with 1ball of each color. And the replacement vector will be (a1 + a2, a3 + a4, · · · ).

2. Now we extend the concept of associating an urn with each point to the rest. For the points to theright of the origin, they can also be thought of as Polya’s urn each. The replacement vector willbe exactly the same (a1 + a2, a3 + a4, · · · ) for the ball with color representing going to the right,the initial weight is (1 + a1, 1), and the replacement vector representing going to the left is now(a2 + a3, a4 + a5, · · · ).

3. For the points to the left of the origin, the situation just a mirror of the above.

7

4.2 Asymptotics of ERRW on ZFirst we gives the mathematical formulation of ERRW process.

Definition An ERRW on Z is a sequence ~X = {Xn

, n � 0} of integer valued r.v’s and a matrix[w] = w(n, j), n 2 N, j 2 Z of positive r.v’s all defined on the same probability space. Let G

n

be the�-field � ({X

i

, w(n, j), 0 i n, j 2 Z}). Then the following relations hold

1. w(n+ 1, j) � w(n, j) with equality if ~X does not traverse ej

at time n+1,

2. P (Xn+1 = j + 1|X

n

= j,Gn

) = 1� P (Xn+1 = j � 1|X

n

= j,Gn

) = w(n,j)w(n,j)+w(n,j�1)

A walk is initially fair if all the initial weights w(0, j) are 1. Then we have the following

Theorem 4.1. Let

~X be an initially fair ERRW. Then

P ( ~Xis recurrent) + P ( ~Xhas finite range) = 1

Proof. See [5]

Basically, the Theorem 4.1 states that for an initially fair ERRW process, the process will either berecurrent or stuck in a finite subgraph on Z. So the next interesting question is under which conditionsare we sure that the process will be recurrent?

Definition For the initially fair ERRW, if the reinforcement is a sequence type with a nonnegativesequence ~↵, then let

�(~vj

) =1X

n=0

v(n, j)�1 =1X

n=1

1 +

nX

i=1

↵i

!�1

;

and if the reinforcement if i.i.d, then

�(~vj

) =1X

n=0

v(n, j)�1 =1X

n=1

1 +

nX

i=1

Zi

!�1

The quantity �(~a) will have great e↵ect on the limit behavior of the ERRW process, determining whetherthe process will be recurrent or stuck in a finite subgraph.

Theorem 4.2. Let

~X be an initially fair ERRW, with reinforcing sequence ~↵, or i.i.d reinforcement

with associated r.v’s Z1, Z2, . . . ,. Then

1. �(~vj

) < 1 a.s. ) ~X has finite range a.s.

2. �(~vj

) = 1 a.s. ) ~X is recurrent a.s.

3. If

~X has finite range, there exist (random) integers N and j such that n > N ) Xn

2 j, j + 1.

Proof. See [5]

Given theorem 3.2 for the generalized Polya urn with a replacement vector ~a, we find the its connectionto theorem 4.2 for the ERRW with the reinforcement of the same vector ~a. Since the reinforced randomwalk always has a tendency to come back to origin initially (1 + a1 > 1) and the vertices on the Z aresymetric, we thus intuitively suggest that if we can draw infinitely many times balls of each color, weshould not be bounded in any range in the walk, since each position/urn is essentially similar to anyother. Based on this intuition, the relationship between �(a) and 1, where a = (a0, a1, a2, · · · ), shoulddetermine the recurrence of the reinforced random walk.

5 Simulation

We will use numerical simulation trying to validate our result.

8

5.1 Polya urn with two colors

We start one run with an urn with 1 red ball and 1 blue ball, and perform 100,000 draws and obtain theproportion of red balls in the urn. We performed 1000 runs to see the distribution of the final proportion.We then compare the empirical distribution of the final portion to the theoretical counterpart, whichshould be uniform. The data are summarized as a histogram after adjusting as probability (area enforcedto be 1). Empirical density function is a smooth estimate of the density of the data using density functionin R language.

Histogram of Proportion of Red Balls in the Urn (red=1, blue=1)

De

nsi

ty

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Empirical DensityUniform

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Empirical Cumluative Distribution (red=1, blue=1)

Proportion of Red Balls

F(x

)

Empirical DensityUniform

Now we change the scheme to urn initially with 6 red balls and 3 blue balls, with other settingunchanged. The following results are obtained.

Histogram of Proportion of Red Balls in the Urn (red=6, blue=3)

De

nsi

ty

0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Empirical DensityBeta(6,3)

0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Empirical Cumluative Distribution (red=6, blue=3)

Proportion of Red Balls

F(x

)

Empirical DensityBeta(6,3)

5.2 Recurrence of edge-reinforced random walk

Because of the di�culty of simulating Z numerically, we will restrict ourselves to the range |i| 200, i 2 Z.The simple random walk is clearly recurrent, since in this case a

i

= 0, i � 0, which is consistent withthe result in Theorem 3.2. We performed 500,000 steps in the walk and then show the number of timeseach point has been visited.

9

−200 −100 0 100 200

500

1000

1500

2000

Simple Random Walk (500,000 steps)

Tim

es v

isite

d

Now we change a1 = 0, ai

= 1pi

, 8i � 1. Clearly �(a) diverges, and the walk should be recurrent. We

still do 500,000 steps in the random walk, and get the following.

−200 −100 0 100 200

020

0040

0060

0080

00

Reinforced Random Walk (500,000 steps), phi=infty

Tim

es v

isite

d

It looks like the range is confined. But if we continue with more steps (5,000,000 steps), we see that thewalk goes all the way to the two extremes.

−200 −100 0 100 200

5000

1000

015

000

2000

025

000

3000

0

Reinforced Random Walk (5,000,000 steps), phi=infty

Tim

es v

isite

d

As ai

becomes bigger, more steps are needed to cover the interval [�200, 200], assuming the walkis recurrent. The transition point from recurrence to a local walk is determined by �(a). The walk is

10

restricted in finite range when �(a) < 1. Take a1 = 0, ai

= i, 8i � 2, then �(a) < 1, and use 5,000,000steps, we get the following walk (here we chose a1 = 0 to make the walk less origin centered)

−200 −100 0 100 200

0e+0

01e

+06

2e+0

63e

+06

4e+0

65e

+06

Reinforced Random Walk (5,000,000 steps), phi<infty

Tim

es v

isite

d

Further inspection shows that 4,999,995 out of 5,000,000 steps happened on the edge < 2, 3 >.

6 Acknowledgement

We sincerely thank Prof. Quentin Berger for his lecture and help throughout the project.

References

[1] David Blackwell and James B MacQueen. “Ferguson distributions via Polya urn schemes”. In: Theannals of statistics (1973), pp. 353–355.

[2] Elizabeth Deyoung and Jonathan Hanselman. “Multiple Particle Edge Reinforced Random Walkson Z”. In: (2008).

[3] Thomas S Ferguson. “A Bayesian analysis of some nonparametric problems”. In: The annals of

statistics (1973), pp. 209–230.

[4] Steven N MacEachern and Peter Muller. “Estimating mixture of Dirichlet process models”. In:Journal of Computational and Graphical Statistics 7.2 (1998), pp. 223–238.

[5] Henrik Renlund. “Reinforced random walk”. In: UUDM project report 1 (2005).

7 Appendix

Lemma 2.4Suppose a Polya urn starts with X

j0 balls with jth color (j = 1, ...,K) and ⌧0 =P

j

Xj0. Let X

jn

bethe number of jth (j = 1, 2, ...,K) color balls drawn in the Polya urn after n draws, then

P (Xjn

= yj

, j = 1, ...,m) =

QK

j=1[Xj0(Xj0 + 1) ⇤ ... ⇤ (Xj0 + y

j

� 1)]

⌧0(⌧0 + 1) ⇤ ... ⇤ (⌧0 + n� 1)

✓n

y1, ..., ym

◆

Proof. In a string of n draws achieving kj

jth-color ball drawings, kj

needs to sum to n. Suppose1 i

j1 ij2 ... i

jkj n are the time indexes of the j-th-color ball draws. The probability of thisparticular string is

KY

j=1

[X

j0

⌧ij1

⇥ Xj0 + 1

⌧ij2

⇥ ...⇥ Xj0 + k

j

� 1

⌧ijkj

] (7)

11

Note that the expression does not depend on the indexes. The time indexes can be chosen in�

n

y1,...,ym

�

ways.

Theorem 2.5Suppose a Polya urn starts with X

j0 balls with jth color (j = 1, ...,K). Let Xjn

be the number of jth(j = 1, 2, ...,K) color balls drawn in the Polya urn after n draws, when n ! 1,

(X1n

n, ..,

XKn

n)

D�! Dir(X10, ..., XK0).

Proof. By Lemma 2.4,

P (Xjn

= yj

, j = 1, ...,m) =

Qm

j=1[Xj0(Xj0 + 1) ⇤ ... ⇤ (Xj0 + y

j

� 1)]

⌧0(⌧0 + 1) ⇤ ... ⇤ (⌧0 + n� 1)

✓n

y1, ..., ym

◆

=

Qm

j=1 �(Xj0 + yj

)�(⌧0)Qm

j=1 �(Xj0)�(⌧0 + n)

�(n+ 1)Qm

j=1 �(yj + 1)= [

�(n+ 1)

�(⌧0 + n)

Qm

j=1 �(Xj0 + yj

)Q

m

j=1 �(yj + 1)]

�(⌧0)Qm

j=1 �(Xj0)

Applying Stirling approximation �(x+r)�(x+s) = xr�s as x ! 1 gives

P (X

jn

n z

j

, j = 1, 2, ...,m) ! �(⌧0)Qm

j=1 �(Xj0)

Z

z

dzmY

j=1

zxj0�1j

1(X

zi

= 1)

which is the c.d.f.of Dirichlet distribution with parameter(X10, ..., Xm0)

lemma 2.9A sample from Dirichlet Process is a discrete distribution, made of up countably infinite number of pointmasses.

lemma 2.10Let G be a sample from Dirichlet process DP (↵, G0) on the color space and let c

n

be the color of nthball observed, then the conditional probability of G is still Dirichlet Process with parameters ↵ andG0 +

PN

n=1 �cn .

We refer the proofs of Lemma 2.9 and 2.10 to paper [1, 3]

12