Linear Discriminant Analysis (LDA)shireen/pdfs/tutorials/Elhabian_LDA09.pdf · LDA Objective •The...

A Tutorial on Data Reduction

Linear Discriminant Analysis (LDA)

Shireen Elhabian and Aly A. FaragUniversity of Louisville, CVIP Lab

September 2009

Outline• LDA objective

• Recall … PCA

• Now … LDA

• LDA … Two Classes

– Counter example

• LDA … C Classes

– Illustrative Example

• LDA vs PCA Example

• Limitations of LDA

LDA Objective

• The objective of LDA is to performdimensionality reduction …

– So what, PCA does this L…

• However, we want to preserve as much of theclass discriminatory information as possible.

– OK, that’s new, let dwell deeper J …

Recall … PCA• In PCA, the main idea to re-express the available dataset to

extract the relevant information by reducing the redundancyand minimize the noise.

• We didn’t care about whether this dataset represent features from oneor more classes, i.e. the discrimination power was not taken intoconsideration while we were talking about PCA.

• In PCA, we had a dataset matrix X with dimensions mxn, wherecolumns represent different data samples.

• We first started by subtracting the mean to have a zero mean dataset,then we computed the covariance matrix Sx = XXT.

• Eigen values and eigen vectors were then computed for Sx. Hence thenew basis vectors are those eigen vectors with highest eigen values,where the number of those vectors was our choice.

• Thus, using the new basis, we can project the dataset onto a lessdimensional space with more powerful data representation.

n – feature vectors(data samples)

Now … LDA• Consider a pattern classification problem, where we have C-

classes, e.g. seabass, tuna, salmon …

• Each class has Ni m-dimensional samples, where i = 1,2, …, C.

• Hence we have a set of m-dimensional samples {x1, x2,…, xNi}belong to class ωi.

• Stacking these samples from different classes into one big fatmatrix X such that each column represents one sample.

• We seek to obtain a transformation of X to Y throughprojecting the samples in X onto a hyperplane withdimension C-1.

• Let’s see what does this mean?

LDA … Two Classes• Assume we have m-dimensional samples {x1,

x2,…, xN}, N1 of which belong to ω1 andN2 belong to ω2.

• We seek to obtain a scalar y by projectingthe samples x onto a line (C-1 space, C = 2).

– where w is the projection vectors used to project xto y.

• Of all the possible lines we would like toselect the one that maximizes theseparability of the scalars.

úúúú

êêêê

úúúú

êêêê

xwherexwy..

The two classes are not well separated when projected onto this line

This line succeeded in separating the two classes and in the meantime reducing the dimensionality of our problem from two features (x1,x2) to only a scalar value y.

LDA … Two Classes

• In order to find a good projection vector, we need to define ameasure of separation between the projections.

• The mean vector of each class in x and y feature space is:

– i.e. projecting x to y will lead to projecting the mean of x to the mean ofy.

• We could then choose the distance between the projected meansas our objective function

ååå

ÎÎÎ

( )222 111

~~)( µµµµµµ -=-=-= TTT wwwwJ

LDA … Two Classes

• However, the distance between the projected means is not a verygood measure since it does not take into account the standarddeviation within the classes.

This axis has a larger distance between means

This axis yields better class separability

LDA … Two Classes• The solution proposed by Fisher is to maximize a function that

represents the difference between the means, normalized by ameasure of the within-class variability, or the so-called scatter.

• For each class we define the scatter, an equivalent of thevariance, as; (sum of square differences between the projected samples and their classmean).

• measures the variability within class ωi after projecting it onthe y-space.

• Thus measures the variability within the twoclasses at hand after projection, hence it is called within-class scatterof the projected samples.

( )åÎ

ii ysw

µ 22 ~~

21~~ ss +

LDA … Two Classes• The Fisher linear discriminant is defined as

the linear function wTx that maximizes thecriterion function: (the distance between theprojected means normalized by the within-class scatter of the projected samples.

• Therefore, we will be looking for a projectionwhere examples from the same class areprojected very close to each other and, at thesame time, the projected means are as fartherapart as possible

LDA … Two Classes

• In order to find the optimum projection w*, we need to expressJ(w) as an explicit function of w.

• We will define a measure of the scatter in multivariate featurespace x which are denoted as scatter matrices;

• Where Si is the covariance matrix of class ωi, and Sw is called thewithin-class scatter matrix.

( )( )

21 SSS

--=åÎ

LDA … Two Classes• Now, the scatter of the projection y can then be expressed as a function of

the scatter matrix in feature space x.

Where is the within-class scatter matrix of the projected samples y.

( ) ( )

( )( )

( ) WWTTTT

SwSwwSSwwSwwSwss

wSwwxxw

212122

==+=+=+

=÷÷ø

öççè

LDA … Two Classes• Similarly, the difference between the projected means (in y-space) can be

expressed in terms of the means in the original feature space (x-space).

• The matrix SB is called the between-class scatter of the original samples/featurevectors, while is the between-class scatter of the projected samples y.

• Since SB is the outer product of two vectors, its rank is at most one.

( ) ( )( )( )

!!! "!!! #$ µµµµ

µµµµ

LDA … Two Classes

• We can finally express the Fisher criterion in terms of SW and SBas:

• Hence J(w) is a measure of the difference between classmeans (encoded in the between-class scatter matrix)normalized by a measure of the within-class scatter matrix.

wSwwSw

)( 1µµ

LDA … Two Classes

• To find the maximum of J(w), we differentiate and equate tozero.

( ) ( ) ( ) ( )( ) ( )

1 =-Þ

=÷÷ø

öççè

æ-÷÷

öççè

=÷÷ø

öççè

- wwJwSSwSwJwS

wSwSwwSwwS

wSwwSw

wSwbyDividingwSwSwwSwSw

wSwdwdwSwwSw

dwdwSw

wSwwSw

LDA … Two Classes• Solving the generalized eigen value problem

yields

• This is known as Fisher’s Linear Discriminant, although it is not adiscriminant but rather a specific choice of direction for the projectionof the data down to one dimension.

• Using the same notation as PCA, the solution will be the eigenvector(s) of

scalarwJwherewwSS BW ===- )(1 ll

( )211* maxarg)(maxarg µµ -=÷÷

öççè

æ== -

wSwwSwwJw

BWX SSS 1-=

LDA … Two Classes - Example• Compute the Linear Discriminant projection for the following two-

dimensional dataset.

– Samples for class ω1 : X1=(x1,x2)={(4,2),(2,4),(2,3),(3,6),(4,4)}

– Sample for class ω2 : X2=(x1,x2)={(9,10),(6,8),(9,5),(8,7),(10,8)}

0 1 2 3 4 5 6 7 8 9 100

LDA … Two Classes - Example

• The classes mean are :

÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ+÷÷ø

öççè

æ+÷÷ø

öççè

æ+÷÷ø

öççè

æ+÷÷ø

öççè

÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ+÷÷ø

öççè

æ+÷÷ø

öççè

æ+÷÷ø

öççè

æ+÷÷ø

öççè

6.74.8

• Covariance matrix of the first class:

( )( )

÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

æ=--=å

2.225.025.01

xxxS µµ

• Covariance matrix of the second class:

( )( )

÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

æ=--= å

3.305.005.03.2

6.74.8

xxxS µµ

• Within-class scatter matrix:

÷÷ø

öççè

÷÷ø

öççè

-+÷÷ø

öççè

5.53.03.03.3

3.305.005.03.2

2.225.025.01

21 SSSw

• Between-class scatter matrix:

( )( )

÷÷ø

öççè

--÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

æúû

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

44.1452.2052.2016.29

8.34.58.34.5

6.74.8

TBS µµµµ

LDA … Two Classes - Example• The LDA projection is then obtained as the solution of the generalized eigen

value problem

( )( )( )

2007.12,002007.1202007.1202339.4489.69794.22213.9

9794.22339.4489.62213.9

44.1452.2052.2016.29

1827.00166.00166.03045.0

44.1452.2052.2016.29

5.53.03.03.3

==Þ=-Þ=-Þ

=´---=

÷÷ø

öççè

=÷÷ø

öççè

æ-÷÷ø

öççè

æ÷÷ø

öççè

=÷÷ø

öççè

æ-÷÷ø

öççè

æ÷÷ø

öççè

llllll

• Hence

• The optimal projection is the one that given maximum λ = J(w)

4173.09088.0

8178.05755.0

2007.129794.22339.4489.62213.9

09794.22339.4489.62213.9

wwandw

=÷÷ø

öççè

æ=÷÷

öççè

÷÷ø

öççè

æ=÷÷

öççè

÷÷ø

öççè

æ=÷÷

öççè

"#"$%l

÷÷ø

öççè

÷÷ø

öççè

÷÷ø

öççè

ùêë

é÷÷ø

öççè

æ-÷÷ø

öççè

æ÷÷ø

öççè

4173.09088.0

8.34.5

1827.00166.00166.03045.0

6.74.8

5.53.03.03.3 1

211* µµWSw

Or directly;

LDA - Projection

-4 -3 -2 -1 0 1 2 3 4 5 60

Classes PDF : using the LDA projection vector with the other eigen value = 8.8818e-016

-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 100

LDA projection vector with the other eigen value = 8.8818e-016

The projection vector corresponding to the smallest eigen value

Using this vector leads to bad separability

between the two classes

LDA - Projection

0 5 10 150

Classes PDF : using the LDA projection vector with highest eigen value = 12.2007

0 1 2 3 4 5 6 7 8 9 100

LDA projection vector with the highest eigen value = 12.2007

The projection vector corresponding to the highest eigen value

Using this vector leads to good separability

between the two classes

LDA … C-Classes

• Now, we have C-classes instead of just two.

• We are now seeking (C-1) projections [y1, y2, …, yC-1] by meansof (C-1) projection vectors wi.

• wi can be arranged by columns into a projection matrix W =[w1|w2|…|wC-1] such that:

[ ]1211

|...||

úúúú

êêêê

úúúú

êêêê

wwwWandy

xwhere

xWyxwy TTii =Þ=

LDA … C-Classes

• If we have n-feature vectors, we can stack them into one matrixas follows;

[ ]1211

|...||.

.........

=úúúúú

êêêêê

úúúúú

êêêêê

wwwWandyyy

Xwhere

XWY T=

LDA – C-Classes

• Recall the two classes case, thewithin-class scatter was computed as:

• This can be generalized in the C-classes case as:

21 SSSw +=

( )( )

xxSwhere

Example of two-dimensional features (m = 2), with three classes C = 3.

Ni : number of data samples in class ωi.

LDA – C-Classes• Recall the two classes case, the between-

class scatter was computed as:

• For C-classes case, we will measure thebetween-class scatter with respect to themean of all classes as follows:

( )( )

µµµµ

( )( )TBS 2121 µµµµ --=

Example of two-dimensional features (m = 2), with three classes C = 3.

Ni : number of data samples in class ωi.

N: number of all data .

LDA – C-Classes

• Similarly,

– We can define the mean vectors for the projected samples y as:

– While the scatter matrices for the projected samples y will be:

åå"Î

1~1~ µµw

( )( )ååå= Î=

iiW yySS

~~~~ µµw

( )( )å=

TiiiB NS

~~~~~ µµµµ

LDA – C-Classes

• Recall in two-classes case, we have expressed the scatter matrices of theprojected samples in terms of those of the original samples as:

This still hold in C-classes case.

• Recall that we are looking for a projection that maximizes the ratio ofbetween-class to within-class scatter.

• Since the projection is no longer a scalar (it has C-1 dimensions), we then usethe determinant of the scatter matrices to obtain a scalar objective function:

• And we will seek the projection W* that maximizes this ratio.

LDA – C-Classes• To find the maximum of J(W), we differentiate with respect to W and equate

to zero.

• Recall in two-classes case, we solved the eigen value problem.

• For C-classes case, we have C-1 projection vectors, hence the eigen valueproblem can be generalized to the C-classes case as:

• Thus, It can be shown that the optimal projection matrix W* is the one whosecolumns are the eigenvectors corresponding to the largest eigen values of thefollowing generalized eigen value problem:

1,...2,1)(1 -====- CiandscalarwJwherewwSS iiiiiBW ll

scalarwJwherewwSS BW ===- )(1 ll

|...||)( -

wwwWandscalarWJwhereWWSSl

Illustration – 3 Classes• Let’s generate a dataset for each

class to simulate the three classesshown

• For each class do the following,

– Use the random number generatorto generate a uniform stream of500 samples that follows U(0,1).

– Using the Box-Muller approach,convert the generated uniformstream to N(0,1).

– Then use the method of eigen valuesand eigen vectors to manipulate thestandard normal to have the requiredmean vector and covariance matrix .

– Estimate the mean and covariancematrix of the resulted dataset.

• By visual inspection of the figure,classes parameters (means andcovariance matrices can be given asfollows:

Dataset Generation

÷÷ø

öççè

÷÷ø

öççè

÷÷ø

öççè

ùêë

é+=ú

ùêë

+=úû

ùêë

5.2315.3

,5.35.2

mean Overall

µµµµµµ

Zero covariance to lead to data samples distributed horizontally.

Positive covariance to lead to data samples distributed along the y = x line.

Negative covariance to lead to data samples distributed along the y = -x line.

In Matlab J

It’s Working … J

-5 0 5 10 15 20-5

X1 - the first feature

Computing LDA Projection Vectors

( )( )

µµµµ

( )( )

xxSwhere

Recall …

BW SS 1-

Let’s visualize the projection vectors W

-15 -10 -5 0 5 10 15 20 25-10

X1 - the first feature

Projection … y = WTxAlong first projection vector

-5 0 5 10 15 20 250

Classes PDF : using the first projection vector with eigen value = 4508.2089

Projection … y = WTxAlong second projection vector

-10 -5 0 5 10 15 200

Classes PDF : using the second projection vector with eigen value = 1878.8511

Which is Better?!!!• Apparently, the projection vector that has the highest eigen

value provides higher discrimination power between classes

-10 -5 0 5 10 15 200

Classes PDF : using the second projection vector with eigen value = 1878.8511

-5 0 5 10 15 20 250

Classes PDF : using the first projection vector with eigen value = 4508.2089

PCA vs LDA

Limitations of LDA L• LDA produces at most C-1 feature projections

– If the classification error estimates establish that more features are needed, some other method must beemployed to provide those additional features

• LDA is a parametric method since it assumes unimodal Gaussian likelihoods

– If the distributions are significantly non-Gaussian, the LDA projections will not be able to preserve anycomplex structure of the data, which may be needed for classification.

Limitations of LDA L• LDA will fail when the discriminatory information is not in the mean but

rather in the variance of the data

Thank You

Linear Discriminant Analysis (LDA)shireen/pdfs/tutorials/Elhabian_LDA09.pdf · LDA Objective •The...

Documents

Transcript of Linear Discriminant Analysis (LDA)shireen/pdfs/tutorials/Elhabian_LDA09.pdf · LDA Objective •The...

WT-LDA: User Tagging Augmented LDA for Web Service Clustering

Scaling up LDA

Realistic investigations of correlated electron systems ...scalettar.physics.ucdavis.edu/publications/lda+dmft.pdf · Article LDA LDA+DMFT LDA+U 0U=W∞ metal insulator Fig. 1 With

Lda Dca Final

LDA Topic Models

Facial Expression Analysis using Nonlinear Decomposable ...elgammal/pub/LeeAMFG05FacialExpres… · e.g. [17,19]. Nonlinear dimensionalityreduction has been recently exploited to

The World of ALDA - European Committee of the Regions · lda prijedor (bih) lda zavidoviĆi (bih) lda montenegro (mne) lda central and southern serbia (rs) lda subotica (rs) lda kosovo

Using LDA - Rutgers Universityyhung/HW586/LDA/Lecture LDA and R.pdf · 1 Using LDA Randy Julian Lilly Research Laboratories Linear Discriminant Analysis Used in Supervised Learning!

Face Recognition Using LDA Based Algorithms · 2002. 9. 26. · Face Recognition, Linear Discriminant Analysis (LDA), direct LDA, fractional-step LDA, principle component analysis

LDA Beginner's Tutorial

LDA - Spinea Oy · LDA LDA Air cooled water chillers with axial fans LDA water chillers are efficient, low-noise products designed for medium to large applications. They are suitable

Lda Risklab Madrid

LDA Steel Plant

Fast sampling for LDA William Cohen. MORE LDA SPEEDUPS FIRST - RECAP LDA DETAILS.

LDA Katalog

Lda Bylaws

Ace Lda Ma

- Sowhat 09.11.18

LDA Brochure

Twitter LDA