Eigen Decomposition
Based on the slides by Mani Thomas
Modified and extended by Longin Jan Latecki
Introduction Eigenvalue decomposition Physical interpretation of
eigenvalue/eigenvectors
What are eigenvalues? Given a matrix, A, x is the eigenvector
and is the corresponding eigenvalue if Ax = x A must be square and the determinant of A -
I must be equal to zeroAx - x = 0 iff (A - I) x = 0
Trivial solution is if x = 0 The non trivial solution occurs when det(A - I) = 0
Are eigenvectors unique? If x is an eigenvector, then x is also an
eigenvector and is an eigenvalueA(x) = (Ax) = (x) = (x)
Calculating the Eigenvectors/values Expand the det(A - I) = 0 for a 2 x 2 matrix
For a 2 x 2 matrix, this is a simple quadratic equation with two solutions (maybe complex)
This “characteristic equation” can be used to solve for x
0
00det
010
01detdet
2112221122112
211222112221
1211
2221
1211
aaaaaa
aaaaaa
aa
aa
aaIA
21122211
22211
2211 4 aaaa
aaaa
Eigenvalue example Consider,
The corresponding eigenvectors can be computed as
For = 0, one possible solution is x = (2, -1) For = 5, one possible solution is x = (1, 2)
5,0)41(
02241)41(
0
42
21
2
2211222112211
2
aaaaaa
A
0
0
12
24
12
240
50
05
42
215
0
0
42
21
42
210
00
00
42
210
yx
yx
y
x
y
x
yx
yx
y
x
y
x
For more information: Demos in Linear algebra by G. Strang, http://web.mit.edu/18.06/www/
Physical interpretation Consider a covariance matrix, A, i.e., A = 1/n S ST
for some S
Error ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue
25.0,75.1175.
75.121
A
Physical interpretation
Orthogonal directions of greatest variance in data Projections along PC1 (Principal Component) discriminate
the data most along any one axis
Original Variable A
Ori
gin
al V
ari
ab
le B
PC 1PC 2
Physical interpretation First principal component is the direction of
greatest variability (covariance) in the data Second is the next orthogonal
(uncorrelated) direction of greatest variability So first remove all the variability along the first
component, and then find the next direction of greatest variability
And so on … Thus each eigenvectors provides the
directions of data variances in decreasing order of eigenvalues
For more information: See Gram-Schmidt Orthogonalization in G. Strang’s lectures
Top Related