MATH 643

Bayesian Statistics

2

Discrete Case

There are 3 suspects in a murder case

– Based on available information, the police think the following probabilities apply

2.0) HunterLocal( P

2.0)Guy SightedShort ( P

2.0)Shooter Sharp( P

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Local Hunter Short Sighted Guy Sharp Shooter

Pri

or

Pro

bab

ilit

y

3

Discrete Case

New evidence comes to light

– The shot came from 2000 feet

– The police assess the following probabilities

– These probabilities are called the likelihood

– In this case, the likelihood of making that shot

7.0) HunterLocal|ft 2000 fromShot ( P

1.0)Guy SightedShort |ft 2000 fromShot ( P

9.0)Shooter Sharp|ft 2000 fromShot ( P

4

Discrete Case

How can we change our prior probabilities to account for the new evidence?

– Bayes Theorem

47.01.09.07.01.02.07.0

2.07.0ft 2000 fromShot | HunterLocal

P

23.01.09.07.01.02.07.0

7.01.0ft 2000 fromShot |Guy SightedShort

P

30.01.09.07.01.02.07.0

1.09.0ft 2000 fromShot |Shooter Sharp

P

5

Discrete Case

What does this look like?

– The likelihoods of making the shot have increased or decreased the prior probabilities.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Local Hunter Short SightedGuy

Sharp Shooter

Pri

or

Pro

bab

ilit

y

Prior

Posterior

6

Continuous Case

You own a pretzel manufacturing company

– An important consideration is market share

– Y = # Customers out of N that buy your pretzel

– We are uncertain about Y so we express this in terms of probabilities

– Assume customers buy independently

– Y ~ Binomial(N,p)

yNy ppy

NP

)1(pN,|yY

7

Continuous Case

How can you estimate p?

– Assume that we know the total daily pretzel market (N)

– In one day suppose, y* people buy your brand

– Before we said for fixed p, Y=y is this likely

– Now, Y=y* is fixed and we wish to know p

yNy ppy

NypL

)1();( *

yNy ppy

NP

)1(pN,|yY

8

Continuous Case

What value of p would make observing Y=y* the most probable?

– What is the maximum of L(p;y*) with respect to p?

– This is the maximum likelihood estimate of p

0);( * ypLdp

d

N

yp

*

ˆ

9

Continuous Case

MLEs are only a best guess

– We can also say that we are 95% confident that p is somewhere in the interval

N

Ny

Ny

zN

y

N

Ny

Ny

zN

y

**

2/

*

**

2/

*1

,

1

10

Continuous Case

So really we are uncertain about the value of p

– We are trying to express this uncertainty through confidence levels that act like, but are not really probabilities

What do we do when we are uncertain about something?

We use probabilities!!

11

Continuous Case

What would be a good distribution to express uncertainty about p?

– p is a probability, lying between 0 and 1

– The beta distribution is very flexible for bounded variables like this

11

000

0 000 )1()()(

)()(

rnr pprnr

npf

yNy ppyNy

NP

)1(

)!(!

!pN,|yY

Prior distribution

Probability model

),( 0rnBetap o

12

Continuous Case

What does the prior distribution on p look like?

– This is for n0 = 4 and r0 = 1

0

0.5

1

1.5

2

2.5

3

0 0.2 0.4 0.6 0.8 1

p

f(p

)

13

Continuous Case

What does this mean we think our market looks like?

dppfPPp )(pN,|yYyYPrior

predictions

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0 2 4 6 8 10 12 14 16 18 20

y

P(Y

=y)

14

Continuous Case

How can you estimate p?

– Assume that we know the total daily pretzel market (N)

– In one day suppose, y* people buy your brand

– Update using Bayes Theorem

dppfP

pfPf

p

)(pN,|yY

)'(p'N,|yYyY|p'

*

**

15

Continuous Case

It turns out that the math works kind of nicely

– The beta distribution is called the natural conjugate distribution for the binomial probability model

– Remember that the p.d.f. for the beta and the p.m.f. for the binomial looked kind of similar

),( 0rnBetap o

),( *0 yrNnBetap o

Y* out of N

buy our pretzel

16

Continuous Case

What does this look like?

– The prior distribution has been modified by the likelihood function

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

0 0.2 0.4 0.6 0.8 1

p

f(p

) prior

posterior

17

Continuous Case

What does this mean we think our market looks like?

dppfPPp )y|(pN,|yYy|yY **Posterior

predictions

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0 2 4 6 8 10 12 14 16 18 20

y

P(Y

=y) prior

posterior