Bayesian Prior and Posterior

Study Guide for ES205

Yu-Chi HoJonathan T. LeeNov. 24, 2000

2

Outline Conditional Density Bayes Rule Conjugate Distribution Example Other Conjugate Distributions Application

3

Conditional Density The conditional probability density of

w happening given x has occurred, assume px(x) 0:

N

xp

xwpxwp

X

XWXW

,| ,

|

xpxwpxwp xXWXW |, |, wpwxp wWX ||

4

Bayes Rule Replace the joint probability density

function with the bottom equation from page 3:

nXX

WnWXX

nXXW

xxp

wpwxxp

xxwp

n

n

n

,,

|,,

,|

1,,

1|,,

1,,|

1

1

1

N

5

Conjugate Distribution W: parameter of interest in some

system X: the independent and identical

observation on the system Since we know the model of the

system, the conditional density of X|W could be easily computed, e.g.,

wxp WX ||N

6

Conjugate Distribution (cont.) If the prior distribution of W belong

to a family, for any size n and any values of the observations in the sample, the posterior distribution of W must also belong to the same family. This family is called a conjugate family of distributions.

N

7

Example An urn of white and red balls with

unknown w being the fraction of the balls that are red.

Assume we can take n sample, X1, …, Xn, from the urn, with replacement, e.g, n i.i.d. samples.This is a Bernoulli distribution.

N

8

Example (cont.) Total number of red ball out of n

trials, Y = X1 + … + Xn, has the binomial distribution

Assume the prior dist. of w is beta distribution with parameters and

1 , , | 1, , | 1

n

n yyX X W np x x w w w

N

11 1 wwWpW

9

Example (cont.) The posterior distribution of W is

which is also a beta distribution.

11

1|,,

1,,|

1

|,,

,|

1

1

yny

WnWXX

nXXW

ww

wpwxxp

xxwp

n

n

10

Example (cont.) Updating formula:

’ = + y Posterior (new) parameter =prior (old) parameter + # of red balls

’ = + (n – y) Posterior (new) parameter= prior (old) parameter + # of white balls

N

11

Other Conjugate Distributions The observations forms a Poisson

distribution with an unknown value of the mean w.

The prior distribution of w is a gamma distribution with parameters and .

The posterior is also a gamma distribution with parameters and + n.

Updating formula:’ = + y’ = + n

n

i ix1

N

12

Other Conjugate Distributions (cont.)

The observations forms a negative binomial distribution with a specified r value and an unknown value of the mean w.

The prior distribution of w is a beta distribution with parameters and .

The posterior is also a beta distribution with parameters + rn and .

Updating formula:’ = + rn’ = + y

N

n

i ix1

13

Other Conjugate Distributions (cont.) The observations forms a normal

distribution with an unknown value of the mean w and specified precision r.

The prior distribution of w is a normal distribution with mean and precision .

The posterior is also a normal distribution with mean and precision + nr.

Updating formula:

N

nr

rxn

nr

rxn

nr

14

Other Conjugate Distributions (cont.) The observations forms a normal

distribution with the specified mean m and unknown precision w.

The prior distribution of w is a gamma distribution with parameters and .

The posterior is also a gamma distribution with parameters and .

Updating formula:’ = + n/2’ = + ½

N

2

n

n

i i mx1

2

2

1

n

i i mx1

2

15

Summary of the Conjugate Distributions

Observations Prior Posterior

Bernoulli Beta Beta

Poisson Gamma Gamma

Negative binominal

Beta Beta

Normal Normal Normal

Normal Gamma Gamma

N

16

Application Estimate the state of the system

based on the observations: Kalman filter.

N

17

References:

• DeGroot, M. H., Optimal Statistical Decisions, McGraw-Hill, 1970.

• Ho, Y.-C., Lecture Notes, Harvard University, 1997.• Larsen, R. J. and M. L. Marx, An Introduction to

Mathematical Statistics and Its Applications, Prentice Hall, 1986.