PRML 2.4-2.5: The Exponential Family & Nonparametric Methods

PRML 2.4-2.5

The exponential family &

Nonparametric methods June 11, 2014

by Shinichi TAMURA

NONPARAMETRIC METHODS THE EXPONENTIAL FAMILY

Today's topics

1. The exponential family 1.  What is exponential family? 2.  Maximum likelihood for EF 3.  How to decide priors for EF

2. Nonparametric methods 1.  What is the point of nonparametric methods ? 2.  Kernel density estimator 3.  Nearest-neighbour methods

June 11, 2014 PRML 2.4-2.5 Shinichi TAMURA


The Exponential Family

Almost all of the distributions we studied so far belong to a single class, namely the exponential family.

June 11, 2014 PRML 2.4-2.5

The exponential family

Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Bernoulli,


Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Bernoulli, multinomial,


Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Bernoulli, multinomial, Gaussian,


Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Bernoulli, multinomial, Gaussian, beta,


Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Bernoulli, multinomial, Gaussian, beta, gamma,


Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Bernoulli, multinomial, Gaussian, beta, gamma, von Mises...etc.


Shinichi TAMURA




June 11, 2014 PRML 2.4-2.5

Parametric distributions

Bernoulli, multinomial, Gaussian, beta, gamma, von Mises...etc.


Gaussian mixture...etc.

Shinichi TAMURA


p(x|!) = h(x)g(!) exp!!Tu(x)

"


The exponential family over x given is a class of distributions which form is

!



p(x|!) = h(x)g(!) exp!!Tu(x)

"



!

Natural parameter



p(x|!) = h(x)g(!) exp!!Tu(x)

"



!

Natural parameter Where and come across

x !



p(x|!) = h(x)g(!) exp!!Tu(x)

"



!

Natural parameter

Normalizing constant

Where and come across

x !




E.g. 1) The Bernoulli Distribution

p(x|!) = µx(1 ! µ)1!x

= "(!!) exp(!x)




E.g. 1) The Bernoulli Distribution

where

! = ln!

µ

1 ! µ

"

p(x|!) = µx(1 ! µ)1!x

= "(!!) exp(!x)u(x)

h(x) = 1

g(!)




E.g. 2) The Multinomial Distribution p(x|!) =

!µxk

k

= exp(!Tx)




E.g. 2) The Multinomial Distribution

where

! = (ln µ1, . . . , ln µM )T

!!

exp(!k) =!

µk = 1

p(x|!) =!

µxkk

= exp(!Tx)u(x)

h(x) = 1

g(!) = 1





where

! = (ln µ1, . . . , ln µM )T

!!

exp(!k) =!

µk = 1

p(x|!) =!

µxkk

= exp(!Tx)

It's inconvenient!

u(x)

h(x) = 1

g(!) = 1





Remove the constraint by

µM = 1 !!M!1

k=1 µk, xM = 1 !!M!1

k=1 xk

p(x|µ) = exp

!M!1"

k=1

xk ln µk +

#1 !

M!1"

k=1

xk

$ln

#1 !

M!1"

k=1

µk

$%

= exp

!M!1"

k=1

xk ln

#µk

1 !&M!1

k=1 µk

$+ ln

#1 !

M!1"

k=1

µk

$%

=

#1 !

M!1"

k=1

µk

$exp

!M!1"

k=1

xk ln

#µk

1 !&M!1

k=1 µk

$%.





Remove the constraint by Therefore...

µM = 1 !!M!1

k=1 µk, xM = 1 !!M!1

k=1 xk

p(x|µ) = exp

!M!1"

k=1

xk ln µk +

#1 !

M!1"

k=1

xk

$ln

#1 !

M!1"

k=1

µk

$%

= exp

!M!1"

k=1

xk ln

#µk

1 !&M!1

k=1 µk

$+ ln

#1 !

M!1"

k=1

µk

$%

=

#1 !

M!1"

k=1

µk

$exp

!M!1"

k=1

xk ln

#µk

1 !&M!1

k=1 µk

$%.




E.g. 2') The Multinomial Distribution w/o constraint

where

p(x|!) =!

µxkk

=

"1 +

M!1#

k=1

exp(!k)

$!1

exp(!Tx)

! =!ln

!µ1

1!P

j µj

", . . . , ln

!µM!1

1!P

j µj

", 0

"T

u(x)

h(x) = 1

g(!)




E.g. 3) The Gaussian Distribution

p(x|!) =1

(2"#2)1/2exp

!! 1

2#2(x ! µ)2

"

= (2")!1/2(!2!2)1/2 exp#

!21

4!2

$exp

!%!1 !2

& #xx2

$"




E.g. 3) The Gaussian Distribution

where

u(x)

h(x) = 1

g(!)p(x|!) =

1(2"#2)1/2

exp!! 1

2#2(x ! µ)2

"

= (2")!1/2(!2!2)1/2 exp#

!21

4!2

$exp

!%!1 !2

& #xx2

$"

! =!

µ

!2,! 1

2!2

"T



Today's topics





Maximum likelihood for EF

OK, we know what EF looks like. Then, how to estimate the parameter? Maximize likelihood! Frequentist way.




Suppose we have i.i.d. data , The log-likelihood of is

June 11, 2014 PRML 2.4-2.5

!X = {x1, . . . ,xN}

Shinichi TAMURA

ln p(X|!) = ln

!N"

n=1

p(xn|!)

#

= ln

!N"

n=1

h(xn)g(!) exp$!Tu(xn)

%#

=N&

n=1

ln h(xn) + N ln g(!) + !TN&

n=1

u(xn).

! !! ln p(X|!) = N!! ln g(!) +N&

n=1

u(xn). "# 0



Suppose we have i.i.d. data , The log-likelihood of is

June 11, 2014 PRML 2.4-2.5

!X = {x1, . . . ,xN}

Shinichi TAMURA

ln p(X|!) = ln

!N"

n=1

p(xn|!)

#

= ln

!N"

n=1

h(xn)g(!) exp$!Tu(xn)

%#

=N&

n=1

ln h(xn) + N ln g(!) + !TN&

n=1

u(xn).

! !! ln p(X|!) = N!! ln g(!) +N&

n=1

u(xn). "# 0By putting this to zero



Therefore Here, is determined only through , so it is called “sufficient statistics”. We need to store only for estimation.


!"! ln g(!ML) =1N

N!

n=1

u(xn).

!ML!

n u(xn)

!n u(xn)



E.g.) Gaussian distribution By and , That's what we already know. June 11, 2014 PRML 2.4-2.5 Shinichi TAMURA

g(!) = (!2!2)1/2 exp!!21/4!2

"u(x) = (x, x2)T

!" ln g(!) =

!! !1

2!2

! 12!2

+ !21

4!22

"=#

µ"2 + µ2

$.

! µML =1N

%

n

xn,

"2ML =

1N

%

n

x2n !

!1N

%

n

xn

"2

.



By the way, we want to know the relation between and .


!!ML



Gradient of by gives


!

!h(x)g(!) exp

"!Tu(x)

#dx = 1

!g(!)!

h(x) exp"!Tu(x)

#dx

+!

h(x)g(!) exp"!Tu(x)

#u(x)dx = 0.

" #! ln g(!) = E [u(x)] .



Gradient of by gives


!

!h(x)g(!) exp

"!Tu(x)

#dx = 1

!g(!)!

h(x) exp"!Tu(x)

#dx

+!

h(x)g(!) exp"!Tu(x)

#u(x)dx = 0.

" #! ln g(!) = E [u(x)] .

Similar to !"! ln g(!ML) =1N

N!

n=1

u(xn)



According to LLN, sample mean will converge to the expectation, so will converge to .


!!ML

!"! ln g(!ML) =1N

N!

n=1

u(xn)

!" ln g(!) = E [u(x)]





!!ML

!"! ln g(!ML) =1N

N!

n=1

u(xn)

!" ln g(!) = E [u(x)]

Converge





!!ML

!"! ln g(!ML) =1N

N!

n=1

u(xn)

!" ln g(!) = E [u(x)]

Converge Converge


Today's topics





Priors for EF

If you want to use the Bayesian inference, a prior distribution is needed. Then, how to decide it, if we don't know anything about the parameter?



Priors for EF

Three candidates: 1. Conjugate priors 2. Uniform distributions 3. Noninformative priors



Priors for EF

Three candidates: 1. Conjugate priors ... Easy to handle 2. Uniform distributions 3. Noninformative priors



Priors for EF

Three candidates: 1. Conjugate priors ... Easy to handle 2. Uniform distributions ... Principle of indifference 3. Noninformative priors



Priors for EF

Three candidates: 1. Conjugate priors ... Easy to handle 2. Uniform distributions ... Principle of indifference 3. Noninformative priors ... Make effects of priors little



Priors for EF – Conjugate priors





Distributions of EF has factors of , so conjugate priors is


g(!) exp(!Tu)

p(!|X , !) = f(X , !)!g(!) exp{!TX}

"!

= f(X , !)g(!)! exp{!!TX}.





g(!) exp(!Tu)

p(!|X , !) = f(X , !)!g(!) exp{!TX}

"!

= f(X , !)g(!)! exp{!!TX}.

Correspond





g(!) exp(!Tu)

p(!|X , !) = f(X , !)!g(!) exp{!TX}

"!

= f(X , !)g(!)! exp{!!TX}.

Normalizing constant





g(!) exp(!Tu)

p(!|X , !) = f(X , !)!g(!) exp{!TX}

"!

= f(X , !)g(!)! exp{!!TX}.




It will give posteriors as follows.


g(!) exp(!Tu)

p(!|X , !) = f(X , !)!g(!) exp{!TX}

"!

= f(X , !)g(!)! exp{!!TX}.

p(!|X,X , !) !N!

n=1

h(xn)g(!) exp"!Tu(xn)

#" g(!)! exp{!TX}

! g(!)N+! exp

$!T

%N&

n=1

u(xn) + !X'(




It will give posteriors as follows.


g(!) exp(!Tu)

p(!|X , !) = f(X , !)!g(!) exp{!TX}

"!

= f(X , !)g(!)! exp{!!TX}.

p(!|X,X , !) !N!

n=1

h(xn)g(!) exp"!Tu(xn)

#" g(!)! exp{!TX}

! g(!)N+! exp

$!T

%N&

n=1

u(xn) + !X'(

Correspond


Priors for EF – Uniform distributions






The uniform distribution is common choice for discrete bounded variable. C.f.: Principle of insufficient reason (or Principle of indifference)




The uniform distribution is common choice for discrete bounded variable. C.f.: Principle of insufficient reason (or Principle of indifference) But two problems arise when it is applied to continuous variables: 1.  The normalization problem 2.  The transformation problem




1. Normalization Problem If the parameter is unbounded These priors are called “improper”.

! !

"!p(!)d! =

! !

"!const d! ! "




1. Normalization Problem If the parameter is unbounded These priors are called “improper”. Note that these priors can give proper posteriors, because posteriors are proportional to likelihood, which can be normalized.

! !

"!p(!)d! =

! !

"!const d! ! "




2. Transformation problem Non-linear transformation gives non-constant priors. E.g.) (Sometimes, the posteriors are not sensitive to the difference.)

p(!) = 1!!"!=

!"

p(") = p(!)####d!

d"

#### = 2"





Not constant for !

p(!) = 1!!"!=

!"

p(") = p(!)####d!

d"

#### = 2"





Not constant for !Think "constant for what?"

p(!) = 1!!"!=

!"

p(") = p(!)####d!

d"

#### = 2"




Keep these problems in mind: 1.  The normalization problem 2.  The transformation problem


Priors for EF – Noninformative priors






Two examples of noninformative priors: 1. Priors for location parameters 2. Priors for scale parameters

These are constructed to make effects to posteriors as little as possible, so that the inference would be objective.




1. Priors for location parameters If the density form is p(x|µ) = f(x ! µ),




1. Priors for location parameters If the density form is the constant shift gives same density:

!x = x + c

p(x|µ) = f(x ! µ),

p(!x|!µ) = f(!x ! !µ).




1. Priors for location parameters If the density form is the constant shift gives same density: This property is “translation invariance” and these parameter is “location parameter”.

!x = x + c

p(x|µ) = f(x ! µ),

p(!x|!µ) = f(!x ! !µ).




1. Priors for location parameters To reflect the translation invariance, priors should be ! A

Bp(µ)dµ =

! A

Bp(µ ! c)dµ for"A,B.

#$ p(µ) = p(µ ! c).#$ p(µ) = constant.




1. Priors for location parameters To reflect the translation invariance, priors should be ! A

Bp(µ)dµ =

! A

Bp(µ ! c)dµ for"A,B.

#$ p(µ) = p(µ ! c).#$ p(µ) = constant.

We obtained uniform distributions after all. But unlike before, we know when to use it.




1. Priors for location parameters E.g.) The mean in Gaussian

p(x|µ) =1

(2!"2)1/2exp

!! 1

2"2(x ! µ)2

"




1. Priors for location parameters E.g.) The mean in Gaussian

p(x|µ) =1

(2!"2)1/2exp

!! 1

2"2(x ! µ)2

" f(x ! µ)This form is




1. Priors for location parameters E.g.) The mean in Gaussian This prior is also obtained as a limit of conjugates.

p(x|µ) =1

(2!"2)1/2exp

!! 1

2"2(x ! µ)2

" f(x ! µ)This form is

p(µ) = N (µ|µ0,!20)

!20!"!!!!"const.,

µN =!2

N!20 + !2

µ0 +N!2

0

N!20 + !2

µML "µML,

1!2

N

=1!2

0

+N

!2"N

!2.




2. Priors for scale parameters If the density form is p(x|!) =

1!

f!x

!

"




2. Priors for scale parameters If the density form is the constant scale gives same density:

p(x|!) =1!

f!x

!

"

p(!x|!!) =1!! f

"!x!!

#!x = cx




2. Priors for scale parameters If the density form is the constant scale gives same density: This property is “scale invariance” and these parameter is “scale parameter”.

p(x|!) =1!

f!x

!

"

p(!x|!!) =1!! f

"!x!!

#!x = cx




2. Priors for scale parameters To reflect the scale invariance, priors should be

! A

Bp(!)d! =

! A

Bp

"1c!

# $$$$d!

d(c!)

$$$$ d! for!A,B.

"# p(!) =1cp

"1c!

#.

"# p(!) $ 1!

.

"# p(ln !) = const.




2. Priors for scale parameters E.g.) The deviation in Gaussian p(!x|!) =

1(2"!2)1/2

exp"! 1

2!2!x2

#




2. Priors for scale parameters E.g.) The deviation in Gaussian

This form is 1! f!

x!

"

p(!x|!) =1

(2"!2)1/2exp

"! 1

2!2!x2

#




2. Priors for scale parameters E.g.) The deviation in Gaussian This prior is also obtained as a limit of conjugates.

This form is 1! f!

x!

"

p(!x|!) =1

(2"!2)1/2exp

"! 1

2!2!x2

#

p(!) = Gam(!|a0, b0)a0,b0!"!!!!!!"const

!,

aN = a0 +N

2"N

2,

bN = b0 +N

2"2

ML "N

2"2

ML,




Two examples of noninformative priors: 1. Priors for location parameters 2. Priors for scale parameters

p(x|µ) = f(x ! µ) =" p(µ) = const.

p(x|!) =1!

f!x

!

"=! p(!) " 1

!


Today's topics





Nonparametric methods

We learned “parametric approach”

June 11, 2014 PRML 2.4-2.5 Shinichi TAMURA June 11, 2014 PRML 2.4-2.5 Shinichi TAMURA



We learned “parametric approach” vs.

We will learn “nonparametric approach”




We learned “parametric approach” vs.

We will learn “nonparametric approach” What is the difference?





Parametric Nonparametric

Assume a specific form of the distribution

Put few assumption about the form of distribution

Simple Complex (depend on data size)

Poor Rich / Flexible Efficient Inefficient


Today's topics






We will learn: 1. Histogram methods 2. Kernel density estimators 3. Nearest-neighbour methods




1. Histogram methods Split the space into grids (or bins), and count data points.




1. Histogram methods Split the space into grids (or bins), and count data points. where


p(x) = pi =ni

N!i(x ! i-th bin),

!i = Width of ith bin (usually same for all i),

ni = # of observations which is assigned to ith bin,N = Total # of observations.



1. Histogram methods Split the space into grids (or bins), and count data points. where


p(x) = pi =ni

N!i(x ! i-th bin),

!i = Width of ith bin (usually same for all i),

ni = # of observations which is assigned to ith bin,N = Total # of observations.

This is piecewise constant, hence discontinuous.



1. Histogram methods – Example is...


! = 0.04

0 0.5 10

5

! = 0.08

0 0.5 10

5

! = 0.25

0 0.5 10

5

!





! = 0.04

0 0.5 10

5

! = 0.08

0 0.5 10

5

! = 0.25

0 0.5 10

5

Too narrow to catch enough points Too spiky (noisy)

!





! = 0.04

0 0.5 10

5

! = 0.08

0 0.5 10

5

! = 0.25

0 0.5 10

5


# of bins = MD (curse of dimensionality) !





! = 0.04

0 0.5 10

5

! = 0.08

0 0.5 10

5

! = 0.25

0 0.5 10

5


Good intermediate value






! = 0.04

0 0.5 10

5

! = 0.08

0 0.5 10

5

! = 0.25

0 0.5 10

5



Too wide to express the data Too smooth (less info)






! = 0.04

0 0.5 10

5

! = 0.08

0 0.5 10

5

! = 0.25

0 0.5 10

5



Too wide to express the data Too smooth (less info)

Find good value is very important!




Lessons from histogram methods


Estimate density at a particular point from data points of small local region.





Estimate density at a particular point from data points of small local region. The regions are defined by “smoothing parameter”, which control the complexity in relation with data size.





Estimate density at a particular point from data points of small local region. The regions are defined by “smoothing parameter”, which control the complexity in relation with data size.

Other problems •  Discontinuity •  Not scalable (curse of dimensionality)



Lessons from histogram methods Let's consider a small local region , then

where .


R

P =!R p(x)dx

Pr(K out of N data ! R) =N !

K!(N " K)!PK(1 " P )N!K ,




where . If 1.  K is large enough (smoother not too small) 2.  N is constant over (smoother small enough)


R

P =!R p(x)dx


K!(N " K)!PK(1 " P )N!K ,

R






R

P =!R p(x)dx


K!(N " K)!PK(1 " P )N!K ,

R

Contradictory






R

P =!R p(x)dx


K!(N " K)!PK(1 " P )N!K ,

R

Contradictory Depend on data size






R

P =!R p(x)dx


K!(N " K)!PK(1 " P )N!K ,

R

! p(x) =K

NV.


Today's topics





Kernel density estimators

Fix a region (e.g., hypercube centered on x, side is h) and count data by kernel function k(u) (Parzen window).


k(u) =

!1, |ui| ! 1/2, (i = 1, . . . D)0, otherwise.





Centered on origin, side is 1

k(u) =

!1, |ui| ! 1/2, (i = 1, . . . D)0, otherwise.





k(u) =

!1, |ui| ! 1/2, (i = 1, . . . D)0, otherwise.

Discontinuous kernel



Fix a region (e.g., hypercube centred on x, side is h) and count data by kernel function k(u) (Parzen window).


K =N!

n=1

k

"x ! xn

h

#,

V = hD,

! p(x) =1N

N!

n=1

1hD

k

"x ! xn

h

#.

k(u) =

!1, |ui| ! 1/2, (i = 1, . . . D)0, otherwise.



Symmetry of k(u) let us re-interpret the result.


N data points in the single cube centered on x



Symmetry of k(u) let us re-interpret the result.


N data points in the single cube centered on x

N cubes centered on xn around x



Other choice of k(u): Gaussian


k(u) =1

(2!)D/2exp

!!"u"2

2

".



Other choice of k(u): Gaussian


k(u) =1

(2!)D/2exp

!!"u"2

2

".

This kernel give continuous density.



Other choice of k(u): Gaussian You can use anything as long as it holds


k(u) ! 0,!

k(u)du = 1.

k(u) =1

(2!)D/2exp

!!"u"2

2

".



Example Again, we can see that smooth parameter h controls the outcome of estimations.


h = 0.005

0 0.5 10

5

h = 0.07

0 0.5 10

5

h = 0.2

0 0.5 10

5


Today's topics





Nearest-neighbour methods

Use a sphere as a region which centred on x and contains K (fixed number) data points.




Use a sphere as a region which centred on x and contains K (fixed number) data points. where V(x) denotes the volume of the sphere.


p(x) =K

NV (x),



Note that this density can not be normalized. From x* where faraway from all data points, the radius of the sphere is inversely proportional to x, thus integral diverge.


! !

"!

dx

r(x)!

! !

x!

dx

r(x)

!! !

x!

dx

x " x†

# $.

"!

RD

K

NV (x)dx %

!

RD

dxr(x)D

# $.


Nearest-neighbour estimators

Example Here again, smooth parameter K controls the outcome of estimations.


K = 1

0 0.5 10

5

K = 5

0 0.5 10

5

K = 30

0 0.5 10

5


Nearest-neighbour estimators

Example Here again, smooth parameter K controls the outcome of estimations. Furthermore, we can observe that in K=1 case.


K = 1

0 0.5 10

5

K = 5

0 0.5 10

5

K = 30

0 0.5 10

5

p(x) ! "



Another problem of Kernels and NNs These methods need all observed data for estimation, so both time and space complexity is O(N). It is very inefficient. On that point, parametric methods are quite efficient (c.f., sufficient statistics). Histograms are also efficient.





Histograms Kernels NNs K Not fixed Not fixed Fixed V Not fixed Fixed Not fixed Smoother h V Continuity No It depends Yes* Dimensionality Suffer Scalable Scalable Normalization Proper Proper Improper Data set Discard Keep Keep

!

* If K=1, not continuous



Use NNs as classifier To do this, use the sphere contains K points irrespective to the class.




Use NNs as classifier To do this, use the sphere contains K points irrespective to the class. where Kk is # in class k and sphere.


p(x|Ck) =Kk

NkV,

p(x) =K

NV,



Use NNs as classifier To do this, use the sphere contains K points irrespective to the class. where Kk is # in class k and sphere. Class priors are , so


p(x|Ck) =Kk

NkV,

p(x) =K

NV,

p(Ck|x) =p(x|Ck)p(Ck)

p(x)=

Kk

K.

p(Ck) = Nk/N



Use NNs as classifier Therefore, x will be classified to the greatest majority among x's K-nearest neighbours.




Use NNs as classifier Therefore, x will be classified to the greatest majority among x's K-nearest neighbours. If K=1, it is called “nearest-neighbour rule”.




Use NNs as classifier – Example Same as the discussion so far, here K acts as smooth parameter.


x6

x7

K = 1

0 1 20

1

2

x6

x7

K = 3

0 1 20

1

2

x6

x7

K = 31

0 1 20

1

2


Today's topics




PRML 2.4-2.5: The Exponential Family & Nonparametric Methods

Technology

Transcript of PRML 2.4-2.5: The Exponential Family & Nonparametric Methods