Linear Models (I)

Rong Jin

Review of Information Theory What is information? What is entropy?

Average information Minimum coding length Important inequality

1( ) logi

ii

H P pp

1 1( ) log logi i

i ii i

H P p pp q

Distribution for Generating Symbols

Distribution for Coding Symbols

Review of Information Theory (cont’d) Mutual information

Measure the correlation between two random variables Symmetric

Kullback-Leibler distance

Difference between two distributions

,

( , )( ; ) ( ) ( | ) ( , ) log

( ) ( )x y

P x yI X Y H X H X Y P x y

P x P y

~( ) ( )

( , ) ( ) log [log ]( ) ( )D

D DD M D x Px

M M

P x P xKL P P P x E

P x P x

Outline Classification problems Information theory for text classification Gaussian generative Naïve Bayes Logistic regression

Classification ProblemsYXf :

XInput Y Output?

• Given input X={x1, x2, …, xm}

• Predict the class label y

• y{-1,1}, binary class classification problems

• y {1, 2, 3, …, c}, multiple class classification problems

• Goal: need to learn the function: YXf :

Examples of Classification Problems Text categorization:

Input features: words ‘campaigning’, ‘efforts’, ‘Iowa’, ‘Democrats’, … Class label: ‘politics’ and ‘non-politics’

Image Classification:

Input features: color histogram, texture distribution, edge distribution, … Class label: ‘bird image’ and ‘non-bird image’

Doc: Months of campaigning and weeks of round-the-clock efforts in Iowa all came down to a final push Sunday, …

Topic: politics

Which is a bird image?

Learning Setup for Classification Problems Training examples:

Identical Independent Distribution (i.i.d.)

Training examples are similar to testing examples Goal

Find a model or a function that is consistent with the training data

1 1 2 2{ , , , ,..., , }train n nD x y x y x y

Information Theory for Text Classification

If coding distribution is similar to the generating distribution short coding length good compression rate

1 1( ) log logi i

i ii i

H P p pp q

Distribution for Generating Symbols

Distribution for Coding Symbols

Compression Algorithm for TC

Compression Model M1

Compression Model M2

Politics

Sports

New Document

16K bits

10K bits

Topic:

Sports

Probabilistic Models for Classification Problems Apply statistical inference methods

Key: finding the best parameters Maximum likelihood (MLE) approach

Log-likelihood of data

Find the parameters that maximizes the log-likelihood

1( ) log ( | ; )

ntrain i ii

l D p y x

*1

max ( ) log ( | ; )n

train i iil D p y x

Training Examples

{ , }i ix y

Learning a Statistical Model

Prediction

p(y|x;)

Generative Models Not directly estimate p(y|x;) Using Bayes rule

Estimate p(xly;) instead of p(y|x;)

Why p(xly;)? Most well known distributions are p(xl). Allocate a separate set of parameters for each class

{1, 2,…, c}

p(xly;) p(xly) Describes the special input patterns for each class y

( ; ) ( | ; )( | ; )

( , ; )

p y p x yp y x

p y x

Gaussian Generative Model (I) Assume a Gaussian model for each class One dimension case

Results for MLE

2

22

1 2

( )1( | ; ) exp

22

{ , ,..., }, ={ , , ( )}

y

yy

c k k k

xp x y

p y k

2{ | } { | }

1 1, ( )

| { | } | | { | } |

| { | } |( )

i i

n nk i k i ki y k i y k

i i

i

x xi y k i y k

i y kp y k

n

( ; ) ( | ; )( | ; )

( , ; )

p y p x yp y x

p y x

Example

1.7, 0.1, 0.5

1.5, 0.2, 0.5male male male

female female female

p

p

• Height histogram for males and females.

• Using Gaussian generative model

• P(male|1.8) = ? , P(female|1.4) = ?

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 20

5

10

15

20

25

30

35

40

Empirical data for maleFitted distributionfor maleEmpirical data for femaleFitted distribution for female

Gaussian Generative Model (II) Consider multiple input features

X={x1, x2, …, xm} Multi-variate Gaussian distribution

y is a mm covariance matrix Results for MLE

Problem: Singularity of y : too many parameters

1

/ 2 1/ 2

1 1

1 1( | ; ) ~ ( , ) exp

22 | |

( , , ( 1),..., , , ( ))

T

y y y y ymy

c c

p x y N x x

p y p y k

{ | }

,{ , | , }

1,

| { | } |

1( )( )

|{ | } | | { | } |

i

s t

ny ii y y

i

ni j i i j jy s y s ys t y y y y

s t

xi y y

x xs y y t y y

Overfitting Issue Complex model Insufficient training

Consider a classification problem of multiple inputs 100 input features 5 classes 1000 training examples

Total number parameters for a full Gaussian model is 5 means 500 parameters 5 covariance matrices 50,000 parameters 50,500 parameters insufficient training data

Another Example of Overfitting

-6 -4 -2 0 2 4 6-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Another Example of Overfitting

-6 -4 -2 0 2 4 6-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Another Example of Overfitting

-6 -4 -2 0 2 4 6-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Another Example of Overfitting

-8 -6 -4 -2 0 2 4 6 8-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Naïve Bayes Simplify the model complexity

Diagonalize the covariance matrix y

Simplified Gaussian distribution

Feature independence assumption Naïve Bayes assumption

2

211/ 2 2

1

1 21 1

( )1( | ; ) exp ( | ; )

22

{ , ,..., }, { , ,..., , }

i i mm y i i

iimim y

ii

m i i i i ic c

xp x y p x y

Naïve Bayes A terrible estimator for But it is a very reasonable estimator for

Why?

The ratio of likelihood is more important

Naïve Bayes does a reasonable job on the estimation of ratio

( | ; )p x y

( | ; )p y x

' 1 ' 1

( ; ) ( | ; ) 1( | ; )

( '; ) ( | '; )( '; ) ( | '; )( '; ) ( | '; )

c cy y

p y p x yp y x

p y p x yp y p x yp y p x y

( | '; )

( | '; )

p x y

p x y

The Ratio of Likelihood Binary class

Both classes share the similar variance

2 2

2 21

21

2 2

12 221

( 1) ( | 1) ( 1)log log

( 1) ( | 1) ( 1)

2( 1)log

( 1)

2 ( ,..., )

i i i im

i ii

i i i i im

ii

i imc

ii

x xp y p x y p y

p y p x y p y

xp y

p y

x diag

( 1)log

( 1)

p y

p y

1{ ,..., }m

• A linear model !

Decision Boundary

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 20

5

10

15

20

25

30

35

40

Empirical data for maleFitted distributionfor maleEmpirical data for femaleFitted distribution for female

• Gaussian Generative Models == Finding a linear decision boundary

• Why not do it directly?