Regression

7/21/2019 Regression

http://slidepdf.com/reader/full/regression-56da9aa4a77a5 1/48

PATTERN RECOGNITIONAND MACHINE LEARNING

CHAPTER 3: LINEAR MODELS FOR

REGRESSION



Outline

• Discuss tutorial.• Regression Examples.

• The Gaussian distribution.• Linear Regression.• Maximum Likelihood estimation.



Polynomial ur!e "itting



#cademia Example

• Predict$ %nal percentage mark &or student.• "eatures$ ' assignment grades( midterm exam( %nal

exam( pro)ect( age.• *uestions +e could ask.

• , &orgot the +eights o& components. an youreco!er them &rom a spreadsheet o& the %nalgrades-

• , lost the %nal exam grades. o+ +ell can , still

predict the %nal mark-• o+ important is each component( actually-

ould , guess +ell someone/s %nal mark gi!entheir assignments- Gi!en their exams-



The Gaussian Distribution



entral Limit Theorem

The distribution o& the sum o& 0 i.i.d.random !ariables becomesincreasingly Gaussian as 0 gro+s.Example$ 0 uni&orm 12(34 random!ariables.



Reading exponential prob&ormulas• ,n in%nite space( cannot )ust &orm sum

5x p6x7 gro+s to in%nity.• ,nstead( use exponential( e.g.

p6n7 8 639:7n

• ;uppose there is a rele!ant &eature&6x7 and , +ant to express that <the

greater &6x7 is( the less probable xis=.• >se p6x7 8 exp6?&6x77.



Example$ exponential &ormsample si@e• "air oin$ The longer the sample

si@e( the less likely it is.•

p6n7 8 :?n.

ln[p(n)]

Sample size n



Exponential "orm$ Gaussianmean• The &urther x is &rom the mean( the

less likely it is.

ln[p(x)]

2(x-μ)



;maller !ariance decreasesprobability• The smaller the !ariance A:( the less

likely x is 6a+ay &rom the mean7. Or$ thegreater the precision( the less likely x is.

ln[p(x)]

1/σ2 = β



Minimal energy 8 maxprobability• The greater the energy 6o& the )oint

state7( the less probable the stateis.

ln[p(x)]

E(x)



Linear Basis "unction Models637Generally

+here C )6x7 are kno+n as basis functions.

Typically( C26x7 8 3( so that +2 acts as abias.

,n the simplest case( +e use linear basis&unctions $ Cd6x7 8 xd.



Linear Basis "unction Models6:7Polynomial basis&unctions$

These are global a smallchange in x aect allbasis &unctions.



Linear Basis "unction Models6F7Gaussian basis &unctions$

These are local a smallchange in x only aectnearby basis &unctions. ) and s control location and

scale 6+idth7.

Related to kernel methods.



Linear Basis "unction Models6H7;igmoidal basis &unctions$

+here

#lso these are local asmall change in x only

aect nearby basis&unctions. ) and s controllocation and scale6slope7.



ur!e "itting Iith 0oise



Maximum Likelihood and Least;Juares 637

#ssume obser!ations &rom a deterministic&unction +ith added Gaussian noise$

+hich is the same as saying(

Gi!en obser!ed inputs( (

and targets( ( +e obtain the likelihood&unction

+her

e



Maximum Likelihood and Least;Juares 6:7

Taking the logarithm( +e get

+here

is the sum?o&?sJuares error.



omputing the gradient and setting it to@ero yields

;ol!ing &or +( +e get

+here

Maximum Likelihood and Least;Juares 6F7

The Moore?Penrose pseudo?in!erse( .



Linear #lgebra9Geometry o& Least;Juaresonsider

; is spanned by.

+ML minimi@es thedistance bet+een andits orthogonal pro)ectionon ;( i.e. .

0?dimensional

M?dimensional



Maximum Likelihood and Least;Juares 6H7

Maximi@ing +ith respect to the bias( +2(alone( +e see that

Ie can also maximi@e +ith respect to K(gi!ing



2th Order Polynomial



Frd Order Polynomial



th Order Polynomial



O!er?%tting

Root?Mean?;Juare 6RM;7Error$



Polynomial oecients



Data ;et ;i@e$

th Order Polynomial



3st Order Polynomial



Data ;et ;i@e$

th Order Polynomial



*uadratic Regulari@ation

Penali@e large coecient !alues



Regulari@ation$



Regulari@ation$ !s.



Regulari@ed Least ;Juares 637

onsider the error &unction$

Iith the sum?o&?sJuares error &unction anda Juadratic regulari@er( +e get

+hich is minimi@ed by

Data term N Regulari@ation term

is calledtheregulari@ationcoecient.



Regulari@ed Least ;Juares 6:7

Iith a more general regulari@er( +e ha!e

Lasso *uadratic



Regulari@ed Least ;Juares 6F7

Lasso tends to generate sparser solutionsthan a Juadraticregulari@er.



ross?alidation &orRegulari@ation



Bayesian Linear Regression 637

• De%ne a con)ugate shrinkage priro!er +eight !ector !$

p6!Q7 8 06!Q2(?3I7

• ombining this +ith the likelihood&unction and using results &or marginaland conditional Gaussian distributions(gi!es a posterior distribution.

• Log o& the posterior 8 sum o& sJuarederrors N Juadratic regulari@ation.



Bayesian Linear Regression 6F7

2 data points obser!ed

Prior Data ;pace



Bayesian Linear Regression 6H7

3 data point obser!ed

Likelihood Posterior Data ;pace



Bayesian Linear Regression 6S7

: data points obser!ed




Bayesian Linear Regression 6'7

:2 data points obser!ed




Predicti!e Distribution 637

• Predict t &or ne+ !alues o& x byintegrating o!er +.

• an be sol!ed analytically.



Predicti!e Distribution 6:7

Example$ ;inusoidal data( Gaussian basis&unctions( 3 data point



Predicti!e Distribution 6F7

Example$ ;inusoidal data( Gaussian basis&unctions( : data points



Predicti!e Distribution 6H7

Example$ ;inusoidal data( Gaussian basis&unctions( H data points



Predicti!e Distribution 6S7

Example$ ;inusoidal data( Gaussian basis&unctions( :S data points



Limitations o& "ixed Basis"unctions• M basis &unction along each

dimension o& a D?dimensional inputspace reJuires MD basis &unctions$

the curse o& dimensionality.• ,n later chapters( +e shall see ho+

+e can get a+ay +ith &e+er basis

&unctions( by choosing these usingthe training data.

Regression

Documents

Transcript of Regression