# Linear Models for Classiﬁcation -...

### Transcript of Linear Models for Classiﬁcation -...

Discriminant Functions Generative Models Discriminative Models

Linear Models for ClassificationBishop PRML Ch. 4

Alireza Ghane

Discriminant Functions Generative Models Discriminative Models

Classification: Hand-written Digit Recognition

xi =

!"#$%&'"()#*$#+ !,- )( !.,$#/+ +%((/#/01/! 233456 334

7/#()#-! 8/#9 */:: )0 "'%! /$!9 +$"$;$!/ +,/ ") "'/ :$1< )(

8$#%$"%)0 %0 :%&'"%0& =>?@ 2ABC D,!" -$</! %" ($!"/#56E'/ 7#)")"97/ !/:/1"%)0 $:&)#%"'- %! %::,!"#$"/+ %0 F%&6 GH6

C! !//0I 8%/*! $#/ $::)1$"/+ -$%0:9 ()# -)#/ 1)-7:/J1$"/&)#%/! *%"' '%&' *%"'%0 1:$!! 8$#%$;%:%"96 E'/ 1,#8/-$#</+ 3BK7#)") %0 F%&6 L !')*! "'/ %-7#)8/+ 1:$!!%(%1$"%)07/#()#-$01/ ,!%0& "'%! 7#)")"97/ !/:/1"%)0 !"#$"/&9 %0!"/$+)( /.,$::9K!7$1/+ 8%/*!6 M)"/ "'$" */ );"$%0 $ >6? 7/#1/0"

/##)# #$"/ *%"' $0 $8/#$&/ )( )0:9 (),# "*)K+%-/0!%)0$:

8%/*! ()# /$1' "'#//K+%-/0!%)0$: );D/1"I "'$0<! ") "'/

(:/J%;%:%"9 7#)8%+/+ ;9 "'/ -$"1'%0& $:&)#%"'-6

!"# $%&'() *+,-. */0+12.33. 4,3,5,6.

N,# 0/J" /J7/#%-/0" %08):8/! "'/ OAPQKR !'$7/ !%:'),/""/

+$"$;$!/I !7/1%(%1$::9 B)#/ PJ7/#%-/0" BPK3'$7/KG 7$#" SI

*'%1' -/$!,#/! 7/#()#-$01/ )( !%-%:$#%"9K;$!/+ #/"#%/8$:

=>T@6 E'/ +$"$;$!/ 1)0!%!"! )( GI?HH %-$&/!U RH !'$7/

1$"/&)#%/!I >H %-$&/! 7/# 1$"/&)#96 E'/ 7/#()#-$01/ %!

-/$!,#/+ ,!%0& "'/ !)K1$::/+ V;,::!/9/ "/!"IW %0 *'%1' /$1'

!"# $%%% &'()*(+&$,)* ,) -(&&%') ()(./*$* ()0 1(+2$)% $)&%..$3%)+%4 5,.6 784 ),6 784 (-'$. 7997

:;<6 #6 (== >? @AB C;DE=FDD;?;BG 1)$*& @BD@ G;<;@D HD;I< >HJ CB@A>G KLM >H@ >? "94999N6 &AB @BO@ FP>QB BFEA G;<;@ ;IG;EF@BD @AB BOFCR=B IHCPBJ

?>==>SBG PT @AB @JHB =FPB= FIG @AB FDD;<IBG =FPB=6

:;<6 U6 M0 >PVBE@ JBE><I;@;>I HD;I< @AB +,$.W79 GF@F DB@6 +>CRFJ;D>I >?@BD@ DB@ BJJ>J ?>J **04 *AFRB 0;D@FIEB K*0N4 FIG *AFRB 0;D@FIEB S;@A!!"#$%&$' RJ>@>@TRBD K*0WRJ>@>N QBJDHD IHCPBJ >? RJ>@>@TRB Q;BSD6 :>J**0 FIG *04 SB QFJ;BG @AB IHCPBJ >? RJ>@>@TRBD HI;?>JC=T ?>J F==>PVBE@D6 :>J *0WRJ>@>4 @AB IHCPBJ >? RJ>@>@TRBD RBJ >PVBE@ GBRBIGBG >I@AB S;@A;IW>PVBE@ QFJ;F@;>I FD SB== FD @AB PB@SBBIW>PVBE@ D;C;=FJ;@T6

:;<6 "96 -J>@>@TRB Q;BSD DB=BE@BG ?>J @S> G;??BJBI@ M0 >PVBE@D ?J>C @AB+,$. GF@F DB@ HD;I< @AB F=<>J;@AC GBDEJ;PBG ;I *BE@;>I !676 X;@A @A;DFRRJ>FEA4 Q;BSD FJB F==>EF@BG FGFR@;QB=T GBRBIG;I< >I @AB Q;DHF=E>CR=BO;@T >? FI >PVBE@ S;@A JBDRBE@ @> Q;BS;I< FI<=B6

ti = (0, 0, 0, 1, 0, 0, 0, 0, 0, 0)

• Each input vector classified into one of K discrete classes• Denote classes by Ck

• Represent input image as a vector xi ∈ R784.• We have target vector ti ∈ {0, 1}10

• Given a training set {(x1, t1), . . . , (xN , tN )}, learning problemis to construct a “good” function y(x) from these.• y : R784 → R10

Linear Models for Classification Alireza Ghane / Greg Mori 1

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Models

• Similar to previous chapter on linear models for regression, wewill use a “linear” model for classification:

y(x) = f(wTx+ w0)

• This is called a generalized linear model• f(·) is a fixed non-linear function

• e.g.

f(u) =

{1 if u ≥ 00 otherwise

• Decision boundary between classes will be linear function of x• Can also apply non-linearity to x, as in φi(x) for regression

Linear Models for Classification Alireza Ghane / Greg Mori 2

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Models

• Similar to previous chapter on linear models for regression, wewill use a “linear” model for classification:

y(x) = f(wTx+ w0)

• This is called a generalized linear model• f(·) is a fixed non-linear function

• e.g.

f(u) =

{1 if u ≥ 00 otherwise

• Decision boundary between classes will be linear function of x• Can also apply non-linearity to x, as in φi(x) for regression

Linear Models for Classification Alireza Ghane / Greg Mori 3

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Models

• Similar to previous chapter on linear models for regression, wewill use a “linear” model for classification:

y(x) = f(wTx+ w0)

• This is called a generalized linear model• f(·) is a fixed non-linear function

• e.g.

f(u) =

{1 if u ≥ 00 otherwise

• Decision boundary between classes will be linear function of x• Can also apply non-linearity to x, as in φi(x) for regression

Linear Models for Classification Alireza Ghane / Greg Mori 4

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Model

���

���

−1 0 1

−1

0

1

���

���

0 0.5 1

0

0.5

1

y(x) = f(wTφ(x) + w0)

y

([x1

x2

])= f

([w1 w2

] [ φ1(x1, x2)φ2(x1, x2)

]+ w0

)Linear Models for Classification Alireza Ghane / Greg Mori 5

Discriminant Functions Generative Models Discriminative Models

Outline

Discriminant Functions

Generative Models

Discriminative Models

Linear Models for Classification Alireza Ghane / Greg Mori 6

Discriminant Functions Generative Models Discriminative Models

Discriminant Functions with Two Classes

x2

x1

wx

y(x)‖w‖

x⊥

−w0‖w‖

y = 0y < 0

y > 0

R2

R1

• Start with 2 class problem,ti ∈ {0, 1}

• Simple linear discriminant

y(x) = wTx+ w0

apply threshold function to getclassification

• Projection of x in w dir. is wTx||w||

Linear Models for Classification Alireza Ghane / Greg Mori 7

Discriminant Functions Generative Models Discriminative Models

Multiple Classes

R1

R2

R3

?

C1

not C1

C2

not C2

R1

R2

R3

?C1

C2

C1

C3

C2

C3

• A linear discriminant between two classes separates with ahyperplane

• How to use this for multiple classes?• One-versus-the-rest method: build K − 1 classifiers, between Ck

and all others• One-versus-one method: build K(K − 1)/2 classifiers, between

all pairs

Linear Models for Classification Alireza Ghane / Greg Mori 8

Discriminant Functions Generative Models Discriminative Models

Multiple Classes

R1

R2

R3

?

C1

not C1

C2

not C2

R1

R2

R3

?C1

C2

C1

C3

C2

C3

• A linear discriminant between two classes separates with ahyperplane

• How to use this for multiple classes?• One-versus-the-rest method: build K − 1 classifiers, between Ck

and all others• One-versus-one method: build K(K − 1)/2 classifiers, between

all pairs

Linear Models for Classification Alireza Ghane / Greg Mori 9

Discriminant Functions Generative Models Discriminative Models

Multiple Classes

R1

R2

R3

?

C1

not C1

C2

not C2

R1

R2

R3

?C1

C2

C1

C3

C2

C3

• A linear discriminant between two classes separates with ahyperplane

• How to use this for multiple classes?• One-versus-the-rest method: build K − 1 classifiers, between Ck

and all others• One-versus-one method: build K(K − 1)/2 classifiers, between

all pairs

Linear Models for Classification Alireza Ghane / Greg Mori 10

Discriminant Functions Generative Models Discriminative Models

Multiple Classes

R1

R2

R3

?

C1

not C1

C2

not C2

R1

R2

R3

?C1

C2

C1

C3

C2

C3

• A linear discriminant between two classes separates with ahyperplane

and all others• One-versus-one method: build K(K − 1)/2 classifiers, between

all pairs

Linear Models for Classification Alireza Ghane / Greg Mori 11

Discriminant Functions Generative Models Discriminative Models

Multiple Classes

Ri

Rj

Rk

xA

xB

x̂

• A solution is to build K linear functions:

yk(x) = wTk x+ wk0

assign x to class argmaxk yk(x)

• Gives connected, convex decision regions

x̂ = λxA + (1− λ)xByk(x̂) = λyk(xA) + (1− λ)yk(xB)

⇒ yk(x̂) > yj(x̂),∀j 6= kLinear Models for Classification Alireza Ghane / Greg Mori 12

Discriminant Functions Generative Models Discriminative Models

Multiple Classes

Ri

Rj

Rk

xA

xB

x̂

• A solution is to build K linear functions:

yk(x) = wTk x+ wk0

assign x to class argmaxk yk(x)

• Gives connected, convex decision regions

x̂ = λxA + (1− λ)xByk(x̂) = λyk(xA) + (1− λ)yk(xB)

⇒ yk(x̂) > yj(x̂),∀j 6= kLinear Models for Classification Alireza Ghane / Greg Mori 13

Discriminant Functions Generative Models Discriminative Models

Least Squares for Classification

• How do we learn the decision boundaries (wk, wk0)?• One approach is to use least squares, similar to regression• FindW to minimize squared error over all examples and all

components of the label vector:

E(W ) =1

2

N∑n=1

K∑k=1

(yk(xn)− tnk)2

• Some algebra, we get a solution using the pseudo-inverse as inregression

Linear Models for Classification Alireza Ghane / Greg Mori 14

Discriminant Functions Generative Models Discriminative Models

Least Squares for Classification

• How do we learn the decision boundaries (wk, wk0)?• One approach is to use least squares, similar to regression• FindW to minimize squared error over all examples and all

components of the label vector:

E(W ) =1

2

N∑n=1

K∑k=1

(yk(xn)− tnk)2

• Some algebra, we get a solution using the pseudo-inverse as inregression

Linear Models for Classification Alireza Ghane / Greg Mori 15

Discriminant Functions Generative Models Discriminative Models

Least Squares for Classification

• How do we learn the decision boundaries (wk, wk0)?• One approach is to use least squares, similar to regression• FindW to minimize squared error over all examples and all

components of the label vector:

E(W ) =1

2

N∑n=1

K∑k=1

(yk(xn)− tnk)2

• Some algebra, we get a solution using the pseudo-inverse as inregression

Linear Models for Classification Alireza Ghane / Greg Mori 16

Discriminant Functions Generative Models Discriminative Models

Problems with Least Squares

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

• Looks okay... least squares decisionboundary• Similar to logistic regression

decision boundary (more later)

• Gets worse by adding easy points?!• Why?

• If target value is 1, points far fromboundary will have high value,say 10; this is a large error so theboundary is moved

Linear Models for Classification Alireza Ghane / Greg Mori 17

Discriminant Functions Generative Models Discriminative Models

Problems with Least Squares

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

• Looks okay... least squares decisionboundary• Similar to logistic regression

decision boundary (more later)

• Gets worse by adding easy points?!

• Why?• If target value is 1, points far from

boundary will have high value,say 10; this is a large error so theboundary is moved

Linear Models for Classification Alireza Ghane / Greg Mori 18

Discriminant Functions Generative Models Discriminative Models

Problems with Least Squares

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

• Looks okay... least squares decisionboundary• Similar to logistic regression

decision boundary (more later)

• Gets worse by adding easy points?!• Why?

• If target value is 1, points far fromboundary will have high value,say 10; this is a large error so theboundary is moved

Linear Models for Classification Alireza Ghane / Greg Mori 19

Discriminant Functions Generative Models Discriminative Models

Problems with Least Squares

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

−4 −2 0 2 4 6 8

−8

−6

−4

−2

0

2

4

• Looks okay... least squares decisionboundary• Similar to logistic regression

decision boundary (more later)

• Gets worse by adding easy points?!• Why?

• If target value is 1, points far fromboundary will have high value,say 10; this is a large error so theboundary is moved

Linear Models for Classification Alireza Ghane / Greg Mori 20

Discriminant Functions Generative Models Discriminative Models

More Least Squares Problems

−6 −4 −2 0 2 4 6−6

−4

−2

0

2

4

6

−6 −4 −2 0 2 4 6−6

−4

−2

0

2

4

6

• Easily separated by hyperplanes, but not found using leastsquares!

• We’ll address these problems later with better models• First, a look at a different criterion for linear discriminant

Linear Models for Classification Alireza Ghane / Greg Mori 21

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

• The two-class linear discriminant acts as a projection:

y = wTx≥ −w0

followed by a threshold

• In which direction w should we project?• One which separates classes “well”

Linear Models for Classification Alireza Ghane / Greg Mori 22

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

• The two-class linear discriminant acts as a projection:

y = wTx≥ −w0

followed by a threshold

• In which direction w should we project?• One which separates classes “well”

Linear Models for Classification Alireza Ghane / Greg Mori 23

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

• The two-class linear discriminant acts as a projection:

y = wTx≥ −w0

followed by a threshold• In which direction w should we project?

• One which separates classes “well”

Linear Models for Classification Alireza Ghane / Greg Mori 24

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

• The two-class linear discriminant acts as a projection:

y = wTx≥ −w0

followed by a threshold• In which direction w should we project?• One which separates classes “well”

Linear Models for Classification Alireza Ghane / Greg Mori 25

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

−2 2 6

−2

0

2

4

−2 2 6

−2

0

2

4

• A natural idea would be to project in the direction of the lineconnecting class means

• However, problematic if classes have variance in this direction• Fisher criterion: maximize ratio of inter-class separation

(between) to intra-class variance (inside)

Linear Models for Classification Alireza Ghane / Greg Mori 26

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

−2 2 6

−2

0

2

4

−2 2 6

−2

0

2

4

• A natural idea would be to project in the direction of the lineconnecting class means

• However, problematic if classes have variance in this direction• Fisher criterion: maximize ratio of inter-class separation

(between) to intra-class variance (inside)

Linear Models for Classification Alireza Ghane / Greg Mori 27

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

−2 2 6

−2

0

2

4

−2 2 6

−2

0

2

4

• A natural idea would be to project in the direction of the lineconnecting class means

• However, problematic if classes have variance in this direction

• Fisher criterion: maximize ratio of inter-class separation(between) to intra-class variance (inside)

Linear Models for Classification Alireza Ghane / Greg Mori 28

Discriminant Functions Generative Models Discriminative Models

Fisher’s Linear Discriminant

−2 2 6

−2

0

2

4

−2 2 6

−2

0

2

4

• A natural idea would be to project in the direction of the lineconnecting class means

• However, problematic if classes have variance in this direction• Fisher criterion: maximize ratio of inter-class separation

(between) to intra-class variance (inside)

Linear Models for Classification Alireza Ghane / Greg Mori 29

Discriminant Functions Generative Models Discriminative Models

Math time - FLD

• Projection yn = wTxn

• Inter-class separation is distance between class means (good):

mk =1

Nk

∑n∈Ck

wTxn

• Intra-class variance (bad):

s2k =

∑n∈Ck

(yn −mk)2

• Fisher criterion:

J(w) =(m2 −m1)

2

s21 + s2

2

maximize wrt wLinear Models for Classification Alireza Ghane / Greg Mori 30

Discriminant Functions Generative Models Discriminative Models

Math time - FLD

• Projection yn = wTxn

• Inter-class separation is distance between class means (good):

mk =1

Nk

∑n∈Ck

wTxn

• Intra-class variance (bad):

s2k =

∑n∈Ck

(yn −mk)2

• Fisher criterion:

J(w) =(m2 −m1)

2

s21 + s2

2

maximize wrt wLinear Models for Classification Alireza Ghane / Greg Mori 31

Discriminant Functions Generative Models Discriminative Models

Math time - FLD

• Projection yn = wTxn

• Inter-class separation is distance between class means (good):

mk =1

Nk

∑n∈Ck

wTxn

• Intra-class variance (bad):

s2k =

∑n∈Ck

(yn −mk)2

• Fisher criterion:

J(w) =(m2 −m1)

2

s21 + s2

2

maximize wrt wLinear Models for Classification Alireza Ghane / Greg Mori 32

Discriminant Functions Generative Models Discriminative Models

Math time - FLD

• Projection yn = wTxn

• Inter-class separation is distance between class means (good):

mk =1

Nk

∑n∈Ck

wTxn

• Intra-class variance (bad):

s2k =

∑n∈Ck

(yn −mk)2

• Fisher criterion:

J(w) =(m2 −m1)

2

s21 + s2

2

maximize wrt wLinear Models for Classification Alireza Ghane / Greg Mori 33

Discriminant Functions Generative Models Discriminative Models

Math time - FLD

J(w) =(m2 −m1)

2

s21 + s2

2

=wTSBw

wTSWw

Between-class covariance:

SB = (m2 −m1)(m2 −m1)T

Within-class covariance:

SW =∑n∈C1

(xn −m1)(xn −m1)T +

∑n∈C2

(xn −m2)(xn −m2)T

Lots of math:

w ∝ S−1W (m2 −m1)

If covariance SW is isotropic, reduces to class mean difference vector

Linear Models for Classification Alireza Ghane / Greg Mori 34

Discriminant Functions Generative Models Discriminative Models

Math time - FLD

J(w) =(m2 −m1)

2

s21 + s2

2

=wTSBw

wTSWw

Between-class covariance:

SB = (m2 −m1)(m2 −m1)T

Within-class covariance:

SW =∑n∈C1

(xn −m1)(xn −m1)T +

∑n∈C2

(xn −m2)(xn −m2)T

Lots of math:

w ∝ S−1W (m2 −m1)

If covariance SW is isotropic, reduces to class mean difference vector

Linear Models for Classification Alireza Ghane / Greg Mori 35

Discriminant Functions Generative Models Discriminative Models

FLD Summary

• FLD is a dimensionality reduction technique (more later in thecourse)

• Criterion for choosing projection based on class labels• Still suffers from outliers (e.g. earlier least squares example)

Linear Models for Classification Alireza Ghane / Greg Mori 36

Discriminant Functions Generative Models Discriminative Models

Perceptrons

• Perceptrons is used to refer to many neural network structures(more in Chapter 5.)

• The classic type is a fixed non-linear transformation of input, onelayer of adaptive weights, and a threshold:

y(x) = f(wTφ(x))

• Developed by Rosenblatt in the 50s

• The main difference compared to the methods we’ve seen so faris the learning algorithm

Linear Models for Classification Alireza Ghane / Greg Mori 37

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning

• Two class problem• For ease of notation, we will use t = 1 for class C1 and t = −1

for class C2

• We saw that squared error was problematic• Instead, we’d like to minimize the number of misclassified

examples• An example is mis-classified if wTφ(xn)tn < 0• Perceptron criterion:

EP (w) = −∑n∈M

wTφ(xn)tn

sum over mis-classified examples only

Linear Models for Classification Alireza Ghane / Greg Mori 38

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning

• Two class problem• For ease of notation, we will use t = 1 for class C1 and t = −1

for class C2

• We saw that squared error was problematic• Instead, we’d like to minimize the number of misclassified

examples• An example is mis-classified if wTφ(xn)tn < 0• Perceptron criterion:

EP (w) = −∑n∈M

wTφ(xn)tn

sum over mis-classified examples only

Linear Models for Classification Alireza Ghane / Greg Mori 39

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning

• Two class problem• For ease of notation, we will use t = 1 for class C1 and t = −1

for class C2

• We saw that squared error was problematic• Instead, we’d like to minimize the number of misclassified

examples• An example is mis-classified if wTφ(xn)tn < 0• Perceptron criterion:

EP (w) = −∑n∈M

wTφ(xn)tn

sum over mis-classified examples only

Linear Models for Classification Alireza Ghane / Greg Mori 40

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning

• Two class problem• For ease of notation, we will use t = 1 for class C1 and t = −1

for class C2

examples• An example is mis-classified if wTφ(xn)tn < 0• Perceptron criterion:

EP (w) = −∑n∈M

wTφ(xn)tn

sum over mis-classified examples only

Linear Models for Classification Alireza Ghane / Greg Mori 41

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Algorithm

• Minimize the error function using stochastic gradient descent(gradient descent per example):

w(τ+1) = w(τ) − η∇EP (w)

= w(τ) + ηφ(xn)tn︸ ︷︷ ︸if incorrect

• Iterate over all training examples, only change w if the exampleis mis-classified

• Guaranteed to converge if data are linearly separable• Will not converge if not• May take many iterations• Sensitive to initialization

Linear Models for Classification Alireza Ghane / Greg Mori 42

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Algorithm

• Minimize the error function using stochastic gradient descent(gradient descent per example):

w(τ+1) = w(τ) − η∇EP (w) = w(τ) + ηφ(xn)tn︸ ︷︷ ︸if incorrect

• Iterate over all training examples, only change w if the exampleis mis-classified

• Guaranteed to converge if data are linearly separable• Will not converge if not• May take many iterations• Sensitive to initialization

Linear Models for Classification Alireza Ghane / Greg Mori 43

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Algorithm

• Minimize the error function using stochastic gradient descent(gradient descent per example):

w(τ+1) = w(τ) − η∇EP (w) = w(τ) + ηφ(xn)tn︸ ︷︷ ︸if incorrect

• Iterate over all training examples, only change w if the exampleis mis-classified

• Guaranteed to converge if data are linearly separable• Will not converge if not• May take many iterations• Sensitive to initialization

Linear Models for Classification Alireza Ghane / Greg Mori 44

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Illustration

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

• Note there are many hyperplanes with 0 error• Support vector machines (in a few weeks) have a nice way of

choosing one

Linear Models for Classification Alireza Ghane / Greg Mori 45

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Illustration

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

• Note there are many hyperplanes with 0 error• Support vector machines (in a few weeks) have a nice way of

choosing one

Linear Models for Classification Alireza Ghane / Greg Mori 46

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Illustration

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

• Note there are many hyperplanes with 0 error• Support vector machines (in a few weeks) have a nice way of

choosing one

Linear Models for Classification Alireza Ghane / Greg Mori 47

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Illustration

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

choosing one

Linear Models for Classification Alireza Ghane / Greg Mori 48

Discriminant Functions Generative Models Discriminative Models

Perceptron Learning Illustration

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

choosing oneLinear Models for Classification Alireza Ghane / Greg Mori 49

Discriminant Functions Generative Models Discriminative Models

Limitations of Perceptrons

• Perceptrons can only solve linearly separable problems in featurespace• Same as the other models in this chapter

• Canonical example of non-separable problem is X-OR• Real datasets can look like this too

Expressiveness of perceptrons

Consider a perceptron with g = step function (Rosenblatt, 1957, 1960)

Can represent AND, OR, NOT, majority, etc.

Represents a linear separator in input space:

!jWjxj > 0 or W · x > 0

I 1

I 2

I 1

I 2

I 1

I 2

?

(a) (b) (c)

0 1

0

1

0

1 1

0

0 1 0 1

xor I 2I 1orI 1 I 2and I 1 I 2

Chapter 20 11

Linear Models for Classification Alireza Ghane / Greg Mori 50

Discriminant Functions Generative Models Discriminative Models

Outline

Discriminant Functions

Generative Models

Discriminative Models

Linear Models for Classification Alireza Ghane / Greg Mori 51

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models

• Up to now we’ve looked at learning classification by choosingparameters to minimize an error function

• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck) whichgenerates the data for each class

Linear Models for Classification Alireza Ghane / Greg Mori 52

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models

• Up to now we’ve looked at learning classification by choosingparameters to minimize an error function

• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck) whichgenerates the data for each class

Linear Models for Classification Alireza Ghane / Greg Mori 53

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models

• Up to now we’ve looked at learning classification by choosingparameters to minimize an error function

• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck) whichgenerates the data for each class

Linear Models for Classification Alireza Ghane / Greg Mori 54

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models

• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck) whichgenerates the data for each class

Linear Models for Classification Alireza Ghane / Greg Mori 55

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models

• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck) whichgenerates the data for each class

Linear Models for Classification Alireza Ghane / Greg Mori 56

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models

• We’ll now develop a probabilistic approach• With 2 classes, C1 and C2:

p(C1|x) =p(x|C1)p(C1)

p(x)Bayes’ Rule

p(C1|x) =p(x|C1)p(C1)

p(x, C1) + p(x, C2)Sum rule

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)Product rule

• In generative models we specify the distribution p(x|Ck) whichgenerates the data for each class

Linear Models for Classification Alireza Ghane / Greg Mori 57

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models - Example

• Let’s say we observe x which is the current temperature• Determine if we are in Vancouver (C1) or Honolulu (C2)• Generative model:

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

• p(x|C1) is distribution over typical temperatures in Vancouver• e.g. p(x|C1) = N (x; 10, 5)

• p(x|C2) is distribution over typical temperatures in Honolulu• e.g. p(x|C2) = N (x; 25, 5)

• Class priors p(C1) = 0.1, p(C2) = 0.9

• p(C1|x = 15) = 0.0484·0.10.0484·0.1+0.0108·0.9 ≈ 0.33

Linear Models for Classification Alireza Ghane / Greg Mori 58

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models - Example

• Let’s say we observe x which is the current temperature• Determine if we are in Vancouver (C1) or Honolulu (C2)• Generative model:

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

• p(x|C1) is distribution over typical temperatures in Vancouver• e.g. p(x|C1) = N (x; 10, 5)

• p(x|C2) is distribution over typical temperatures in Honolulu• e.g. p(x|C2) = N (x; 25, 5)

• Class priors p(C1) = 0.1, p(C2) = 0.9

• p(C1|x = 15) = 0.0484·0.10.0484·0.1+0.0108·0.9 ≈ 0.33

Linear Models for Classification Alireza Ghane / Greg Mori 59

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Models

• We can write the classifier in another form

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

=1

1 + exp(−a)≡ σ(a)

where a = lnp(x|C1)p(C1)

p(x|C2)p(C2)

• This looks like gratuitous math, but if a takes a simple form thisis another generalized linear model which we have been studying

• Of course, we will see how such a simple form a = wTx+ w0

arises naturally

Linear Models for Classification Alireza Ghane / Greg Mori 60

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Models

• We can write the classifier in another form

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

=1

1 + exp(−a)≡ σ(a)

where a = lnp(x|C1)p(C1)

p(x|C2)p(C2)

• This looks like gratuitous math, but if a takes a simple form thisis another generalized linear model which we have been studying

• Of course, we will see how such a simple form a = wTx+ w0

arises naturally

Linear Models for Classification Alireza Ghane / Greg Mori 61

Discriminant Functions Generative Models Discriminative Models

Generalized Linear Models

• We can write the classifier in another form

p(C1|x) =p(x|C1)p(C1)

p(x|C1)p(C1) + p(x|C2)p(C2)

=1

1 + exp(−a)≡ σ(a)

where a = lnp(x|C1)p(C1)

p(x|C2)p(C2)

• This looks like gratuitous math, but if a takes a simple form thisis another generalized linear model which we have been studying

• Of course, we will see how such a simple form a = wTx+ w0

arises naturally

Linear Models for Classification Alireza Ghane / Greg Mori 62

Discriminant Functions Generative Models Discriminative Models

Logistic Sigmoid

−5 0 50

0.5

1

• The function σ(a) = 11+exp(−a) is known as the logistic sigmoid

• It squashes the real axis down to [0, 1]

• It is continuous and differentiable• It avoids the problems encountered with the too correct

least-squares error fitting (later)

Linear Models for Classification Alireza Ghane / Greg Mori 63

Discriminant Functions Generative Models Discriminative Models

Multi-class Extension

• There is a generalization of the logistic sigmoid to K > 2classes:

p(Ck|x) =p(x|Ck)p(Ck)∑j p(x|Cj)p(Cj)

=exp(ak)∑j exp(aj)

where ak = ln p(x|Ck)p(Ck)

• a. k. a. softmax function• If some ak � aj , p(Ck|x) goes to 1

Linear Models for Classification Alireza Ghane / Greg Mori 64

Discriminant Functions Generative Models Discriminative Models

Multi-class Extension

• There is a generalization of the logistic sigmoid to K > 2classes:

p(Ck|x) =p(x|Ck)p(Ck)∑j p(x|Cj)p(Cj)

=exp(ak)∑j exp(aj)

where ak = ln p(x|Ck)p(Ck)

• a. k. a. softmax function• If some ak � aj , p(Ck|x) goes to 1

Linear Models for Classification Alireza Ghane / Greg Mori 65

Discriminant Functions Generative Models Discriminative Models

Gaussian Class-Conditional Densities

• Back to that a in the logistic sigmoid for 2 classes• Let’s assume the class-conditional densities p(x|Ck) are Gaussians,

and have the same covariance matrix Σ:

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

{−1

2(x− µk)TΣ−1(x− µk)

}

• a takes a simple form:

a = lnp(x|C1)p(C1)

p(x|C2)p(C2)= wTx+ w0

• Note that quadratic terms xTΣ−1x cancelLinear Models for Classification Alireza Ghane / Greg Mori 66

Discriminant Functions Generative Models Discriminative Models

Gaussian Class-Conditional Densities

• Back to that a in the logistic sigmoid for 2 classes• Let’s assume the class-conditional densities p(x|Ck) are Gaussians,

and have the same covariance matrix Σ:

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

{−1

2(x− µk)TΣ−1(x− µk)

}

• a takes a simple form:

a = lnp(x|C1)p(C1)

p(x|C2)p(C2)= wTx+ w0

• Note that quadratic terms xTΣ−1x cancelLinear Models for Classification Alireza Ghane / Greg Mori 67

Discriminant Functions Generative Models Discriminative Models

Gaussian Class-Conditional Densities

• Back to that a in the logistic sigmoid for 2 classes• Let’s assume the class-conditional densities p(x|Ck) are Gaussians,

and have the same covariance matrix Σ:

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

{−1

2(x− µk)TΣ−1(x− µk)

}

• a takes a simple form:

a = lnp(x|C1)p(C1)

p(x|C2)p(C2)= wTx+ w0

• Note that quadratic terms xTΣ−1x cancelLinear Models for Classification Alireza Ghane / Greg Mori 68

Discriminant Functions Generative Models Discriminative Models

Gaussian Class-Conditional Densities

and have the same covariance matrix Σ:

p(x|Ck) =1

(2π)D/2|Σ|1/2exp

{−1

2(x− µk)TΣ−1(x− µk)

}

• a takes a simple form:

a = lnp(x|C1)p(C1)

p(x|C2)p(C2)= wTx+ w0

• Note that quadratic terms xTΣ−1x cancelLinear Models for Classification Alireza Ghane / Greg Mori 69

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning

• We can fit the parameters to this model using maximumlikelihood• Parameters are µ1, µ2, Σ−1, p(C1) ≡ π, p(C2) ≡ 1− π• Refer to as θ

• For a datapoint xn from class C1 (tn = 1):

p(xn, C1) = p(C1)p(xn|C1) = πN (xn|µ1,Σ)

• For a datapoint xn from class C2 (tn = 0):

p(xn, C2) = p(C2)p(xn|C2) = (1− π)N (xn|µ2,Σ)

Linear Models for Classification Alireza Ghane / Greg Mori 70

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning

• We can fit the parameters to this model using maximumlikelihood• Parameters are µ1, µ2, Σ−1, p(C1) ≡ π, p(C2) ≡ 1− π• Refer to as θ

• For a datapoint xn from class C1 (tn = 1):

p(xn, C1) = p(C1)p(xn|C1) = πN (xn|µ1,Σ)

• For a datapoint xn from class C2 (tn = 0):

p(xn, C2) = p(C2)p(xn|C2) = (1− π)N (xn|µ2,Σ)

Linear Models for Classification Alireza Ghane / Greg Mori 71

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning

• We can fit the parameters to this model using maximumlikelihood• Parameters are µ1, µ2, Σ−1, p(C1) ≡ π, p(C2) ≡ 1− π• Refer to as θ

• For a datapoint xn from class C1 (tn = 1):

p(xn, C1) = p(C1)p(xn|C1) = πN (xn|µ1,Σ)

• For a datapoint xn from class C2 (tn = 0):

p(xn, C2) = p(C2)p(xn|C2) = (1− π)N (xn|µ2,Σ)

Linear Models for Classification Alireza Ghane / Greg Mori 72

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning

• The likelihood of the training data is:

p(t|π,µ1,µ2,Σ) =

N∏n=1

[πN (xn|µ1,Σ)]tn [(1−π)N (xn|µ2,Σ)]1−tn

• As usual, ln is our friend:

`(t; θ) =

N∑n=1

tn lnπ + (1− tn) ln(1− π)︸ ︷︷ ︸π

+ tn lnN1 + (1− tn) lnN2︸ ︷︷ ︸µ1,µ2,Σ

• Maximize for each separately

Linear Models for Classification Alireza Ghane / Greg Mori 73

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning

• The likelihood of the training data is:

p(t|π,µ1,µ2,Σ) =

N∏n=1

[πN (xn|µ1,Σ)]tn [(1−π)N (xn|µ2,Σ)]1−tn

• As usual, ln is our friend:

`(t; θ) =

N∑n=1

tn lnπ + (1− tn) ln(1− π)︸ ︷︷ ︸π

+ tn lnN1 + (1− tn) lnN2︸ ︷︷ ︸µ1,µ2,Σ

• Maximize for each separately

Linear Models for Classification Alireza Ghane / Greg Mori 74

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Class Priors

• Maximization with respect to the class priors parameter π isstraightforward:

∂

∂π`(t; θ) =

N∑n=1

tnπ− 1− tn

1− π

⇒ π =N1

N1 +N2

• N1 and N2 are the number of training points in each class• Prior is simply the fraction of points in each class

Linear Models for Classification Alireza Ghane / Greg Mori 75

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Class Priors

• Maximization with respect to the class priors parameter π isstraightforward:

∂

∂π`(t; θ) =

N∑n=1

tnπ− 1− tn

1− π

⇒ π =N1

N1 +N2

• N1 and N2 are the number of training points in each class• Prior is simply the fraction of points in each class

Linear Models for Classification Alireza Ghane / Greg Mori 76

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Class Priors

• Maximization with respect to the class priors parameter π isstraightforward:

∂

∂π`(t; θ) =

N∑n=1

tnπ− 1− tn

1− π

⇒ π =N1

N1 +N2

• N1 and N2 are the number of training points in each class• Prior is simply the fraction of points in each class

Linear Models for Classification Alireza Ghane / Greg Mori 77

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Gaussian Parameters• The other parameters can also be found in the same fashion• Class means:

µ1 =1

N1

N∑n=1

tnxn

µ2 =1

N2

N∑n=1

(1− tn)xn

• Means of training examples from each class

• Shared covariance matrix:

Σ =N1

N

1

N1

∑n∈C1

(xn−µ1)(xn−µ1)T+

N2

N

1

N2

∑n∈C2

(xn−µ2)(xn−µ2)T

• Weighted average of class covariances

Linear Models for Classification Alireza Ghane / Greg Mori 78

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Gaussian Parameters• The other parameters can also be found in the same fashion• Class means:

µ1 =1

N1

N∑n=1

tnxn

µ2 =1

N2

N∑n=1

(1− tn)xn

• Means of training examples from each class

• Shared covariance matrix:

Σ =N1

N

1

N1

∑n∈C1

(xn−µ1)(xn−µ1)T+

N2

N

1

N2

∑n∈C2

(xn−µ2)(xn−µ2)T

• Weighted average of class covariances

Linear Models for Classification Alireza Ghane / Greg Mori 79

Discriminant Functions Generative Models Discriminative Models

Gaussian with Different Covariances

−2 −1 0 1 2−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

a = lnp(x|Cb)p(Cb)p(x|Cr)p(Cr)

a = ln(p(x|Cb))− ln(p(x|Cr))︸ ︷︷ ︸−(x−µb)T Σb(x−µb)+(x−µr)T Σr(x−µr)

+ ln(p(Cb))− ln(p(Cr))︸ ︷︷ ︸const.

Linear Models for Classification Alireza Ghane / Greg Mori 80

Discriminant Functions Generative Models Discriminative Models

Probabilistic Generative Models Summary

• Fitting Gaussian using ML criterion is sensitive to outliers• Simple linear form for a in logistic sigmoid occurs for more than

just Gaussian distributions• Arises for any distribution in the exponential family, a large class

of distributions

Linear Models for Classification Alireza Ghane / Greg Mori 81

Discriminant Functions Generative Models Discriminative Models

Outline

Discriminant Functions

Generative Models

Discriminative Models

Linear Models for Classification Alireza Ghane / Greg Mori 82

Discriminant Functions Generative Models Discriminative Models

Probabilistic Discriminative Models

• Generative model made assumptions about form ofclass-conditional distributions (e.g. Gaussian)• Resulted in logistic sigmoid of linear function of x

• Discriminative model - explicitly use functional form

p(C1|x) =1

1 + exp(−wTx+ w0)

and find w directly• For the generative model we had 2M +M(M + 1)/2 + 1

parameters• M is dimensionality of x

• Discriminative model will have M + 1 parameters

Linear Models for Classification Alireza Ghane / Greg Mori 83

Discriminant Functions Generative Models Discriminative Models

Probabilistic Discriminative Models

• Generative model made assumptions about form ofclass-conditional distributions (e.g. Gaussian)• Resulted in logistic sigmoid of linear function of x

• Discriminative model - explicitly use functional form

p(C1|x) =1

1 + exp(−wTx+ w0)

and find w directly• For the generative model we had 2M +M(M + 1)/2 + 1

parameters• M is dimensionality of x

• Discriminative model will have M + 1 parameters

Linear Models for Classification Alireza Ghane / Greg Mori 84

Discriminant Functions Generative Models Discriminative Models

Probabilistic Discriminative Models

• Generative model made assumptions about form ofclass-conditional distributions (e.g. Gaussian)• Resulted in logistic sigmoid of linear function of x

• Discriminative model - explicitly use functional form

p(C1|x) =1

1 + exp(−wTx+ w0)

and find w directly• For the generative model we had 2M +M(M + 1)/2 + 1

parameters• M is dimensionality of x

• Discriminative model will have M + 1 parameters

Linear Models for Classification Alireza Ghane / Greg Mori 85

Discriminant Functions Generative Models Discriminative Models

Generative vs. Discriminative

• Generative models• Can generate synthetic example

data• Perhaps accurate classification is

equivalent to accurate synthesis• e.g. vision and graphics

• Tend to have more parameters• Require good model of class

distributions

• Discriminative models• Only usable for classification• Don’t solve a harder problem than

you need to• Tend to have fewer parameters• Require good model of decision

boundary

Linear Models for Classification Alireza Ghane / Greg Mori 86

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Discriminative Model

• As usual we can use the maximum likelihood criterion forlearning

p(t|w) =

N∏n=1

ytnn {1− yn}1−tn ; where yn = p(C1|xn)

• Taking ln and derivative gives:

∇`(w) =

N∑n=1

(tn − yn)xn

• This time no closed-form solution since yn = σ(wTx)

• Could use (stochastic) gradient descent• But there’s a better iterative technique

Linear Models for Classification Alireza Ghane / Greg Mori 87

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Discriminative Model

• As usual we can use the maximum likelihood criterion forlearning

p(t|w) =

N∏n=1

ytnn {1− yn}1−tn ; where yn = p(C1|xn)

• Taking ln and derivative gives:

∇`(w) =

N∑n=1

(tn − yn)xn

• This time no closed-form solution since yn = σ(wTx)

• Could use (stochastic) gradient descent• But there’s a better iterative technique

Linear Models for Classification Alireza Ghane / Greg Mori 88

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Discriminative Model

• As usual we can use the maximum likelihood criterion forlearning

p(t|w) =

N∏n=1

ytnn {1− yn}1−tn ; where yn = p(C1|xn)

• Taking ln and derivative gives:

∇`(w) =

N∑n=1

(tn − yn)xn

• This time no closed-form solution since yn = σ(wTx)

• Could use (stochastic) gradient descent• But there’s a better iterative technique

Linear Models for Classification Alireza Ghane / Greg Mori 89

Discriminant Functions Generative Models Discriminative Models

Maximum Likelihood Learning - Discriminative Model

• As usual we can use the maximum likelihood criterion forlearning

p(t|w) =

N∏n=1

ytnn {1− yn}1−tn ; where yn = p(C1|xn)

• Taking ln and derivative gives:

∇`(w) =

N∑n=1

(tn − yn)xn

• This time no closed-form solution since yn = σ(wTx)

• Could use (stochastic) gradient descent• But there’s a better iterative technique

Linear Models for Classification Alireza Ghane / Greg Mori 90

Discriminant Functions Generative Models Discriminative Models

Iterative Reweighted Least Squares

• Iterative reweighted least squares (IRLS) is a descent method• As in gradient descent, start with an initial guess, improve it• Gradient descent - take a step (how large?) in the gradient

direction• IRLS is a special case of a Newton-Raphson method

• Approximate function using second-order Taylor expansion:

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

• Closed-form solution to minimize this is straight-forward:quadratic, derivatives linear

• In IRLS this second-order Taylor expansion ends up being aweighted least-squares problem, as in the regression case fromlast week• Hence the name IRLS

Linear Models for Classification Alireza Ghane / Greg Mori 91

Discriminant Functions Generative Models Discriminative Models

Iterative Reweighted Least Squares

• Iterative reweighted least squares (IRLS) is a descent method• As in gradient descent, start with an initial guess, improve it• Gradient descent - take a step (how large?) in the gradient

direction• IRLS is a special case of a Newton-Raphson method

• Approximate function using second-order Taylor expansion:

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

• Closed-form solution to minimize this is straight-forward:quadratic, derivatives linear

• In IRLS this second-order Taylor expansion ends up being aweighted least-squares problem, as in the regression case fromlast week• Hence the name IRLS

Linear Models for Classification Alireza Ghane / Greg Mori 92

Discriminant Functions Generative Models Discriminative Models

Iterative Reweighted Least Squares

• Iterative reweighted least squares (IRLS) is a descent method• As in gradient descent, start with an initial guess, improve it• Gradient descent - take a step (how large?) in the gradient

direction• IRLS is a special case of a Newton-Raphson method

• Approximate function using second-order Taylor expansion:

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

• Closed-form solution to minimize this is straight-forward:quadratic, derivatives linear

• In IRLS this second-order Taylor expansion ends up being aweighted least-squares problem, as in the regression case fromlast week• Hence the name IRLS

Linear Models for Classification Alireza Ghane / Greg Mori 93

Discriminant Functions Generative Models Discriminative Models

Iterative Reweighted Least Squares

direction• IRLS is a special case of a Newton-Raphson method

• Approximate function using second-order Taylor expansion:

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

• Closed-form solution to minimize this is straight-forward:quadratic, derivatives linear

Linear Models for Classification Alireza Ghane / Greg Mori 94

Discriminant Functions Generative Models Discriminative Models

Iterative Reweighted Least Squares

direction• IRLS is a special case of a Newton-Raphson method

• Approximate function using second-order Taylor expansion:

f̂(w+v) = f(w)+∇f(w)T (v−w)+1

2(v−w)T∇2f(w)(v−w)

• Closed-form solution to minimize this is straight-forward:quadratic, derivatives linear

Linear Models for Classification Alireza Ghane / Greg Mori 95

Discriminant Functions Generative Models Discriminative Models

Newton-Raphson

484 9 Unconstrained minimization

PSfrag replacements

f

!f

(x, f(x))

(x + !xnt, f(x + !xnt))

Figure 9.16 The function f (shown solid) and its second-order approximation!f at x (dashed). The Newton step !xnt is what must be added to x to give

the minimizer of !f .

9.5 Newton’s method

9.5.1 The Newton step

For x ! dom f , the vector

!xnt = "#2f(x)!1#f(x)

is called the Newton step (for f , at x). Positive definiteness of #2f(x) implies that

#f(x)T !xnt = "#f(x)T#2f(x)!1#f(x) < 0

unless #f(x) = 0, so the Newton step is a descent direction (unless x is optimal).The Newton step can be interpreted and motivated in several ways.

Minimizer of second-order approximation

The second-order Taylor approximation (or model) !f of f at x is

!f(x + v) = f(x) + #f(x)T v +1

2vT#2f(x)v, (9.28)

which is a convex quadratic function of v, and is minimized when v = !xnt. Thus,the Newton step !xnt is what should be added to the point x to minimize thesecond-order approximation of f at x. This is illustrated in figure 9.16.

This interpretation gives us some insight into the Newton step. If the functionf is quadratic, then x + !xnt is the exact minimizer of f . If the function f isnearly quadratic, intuition suggests that x + !xnt should be a very good estimateof the minimizer of f , i.e., x!. Since f is twice di"erentiable, the quadratic modelof f will be very accurate when x is near x!. It follows that when x is near x!,the point x + !xnt should be a very good estimate of x!. We will see that thisintuition is correct.

• Figure from Boyd and Vandenberghe, Convex Optimization• Excellent reference, free for download onlinehttp://www.stanford.edu/~boyd/cvxbook/

Linear Models for Classification Alireza Ghane / Greg Mori 96

Discriminant Functions Generative Models Discriminative Models

Conclusion

• Readings: Ch. 4.1.1-4.1.4, 4.1.7, 4.2.1-4.2.2, 4.3.1-4.3.3• Generalized linear models y(x) = f(wTx+ w0)

• Threshold/max function for f(·)• Minimize with least squares• Fisher criterion - class separation• Perceptron criterion - mis-classified examples

• Probabilistic models: logistic sigmoid / softmax for f(·)• Generative model - assume class conditional densities in

exponential family; obtain sigmoid• Discriminative model - directly model posterior using sigmoid

(a. k. a. logistic regression, though classification)• Can learn either using maximum likelihood

• All of these models are limited to linear decision boundaries infeature space

Linear Models for Classification Alireza Ghane / Greg Mori 97