Week 6 Linear Algebra II
-
Upload
anh-tuan-nguyen -
Category
Documents
-
view
172 -
download
4
Transcript of Week 6 Linear Algebra II
AMATH 460: Mathematical Methodsfor Quantitative Finance
6. Linear Algebra II
Kjell KonisActing Assistant Professor, Applied Mathematics
University of Washington
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 1 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 2 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 3 / 53
Transposes
Let A be an m × n matrix, the transpose of A is denoted by AT
The columns of AT are the rows of AIf the dimension of A is m × n then the dimension of AT is n ×m
A =
[1 2 34 5 6
]AT =
1 42 53 6
Properties
(AT)T = AThe transpose of A + B is AT + BT
The transpose of AB is (AB)T = BTAT
The transpose of A−1 is (A−1)T = (AT)−1
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 4 / 53
Symmetric Matrices
A symmetric matrix satisfies A = AT
Implies that aij = aji
A =
1 2 32 5 43 4 9
= AT
A diagonal matrix is automatically symmetric
D =
1 0 00 5 00 0 9
= DT
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 5 / 53
Products RTR , RRT and LDLT
Let R be any m × n matrix
The matrix A = RTR is a symmetric matrix
AT = (RTR)T = RT(RT)T = RTR = A
The matrix A = RRT is also a symmetric matrix
AT = (RRT)T = (RT)TRT = RRT = A
Many problems that start with a rectangular matrix R end up withRTR or RRT or both!
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 6 / 53
Permutation Matrices
An n × n permutation matrix P has the rows of I in any orderThere are 6 possible 3× 3 permutation matrices
I =
11
1
P21 =
11
1
P32P21 =
11
1
P31 =
11
1
P32 =
11
1
P21P32 =
11
1
P−1 is the same as PT
Example
P32
123
=
1 0 00 0 10 1 0
1
23
=
132
P32
132
=
123
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 7 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 8 / 53
Spaces of Vectors
The space Rn consists of all column vectors with n components
For example, R3 all column vectors with 3 components and is called3-dimensional space
The space R2 is the xy plane: the two components of the vector arethe x and y coordinates and the tail starts at the origin (0, 0)
Two essential vector operations go on inside the vector space:
Add two vectors in Rn
Multiply a vector in Rn by a scalar
The result lies in the same vector space Rn
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 9 / 53
Subspaces
A subspace of a vector space is a set of vectors (including the zerovector) that satisfies two requirements:
If u and w are vectors in the subspace and c is any scalar, then(i) u + w is in the subspace
(ii) cw is in the subspace
Some subspaces of R3
L Any line through (0, 0, 0), e.g., the x axisP Any plane through (0, 0, 0), e.g., the xy plane
R3 The whole spaceZ The zero vector
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 10 / 53
The Column Space of A
The column space of a matrix A consists of all linear combinations ofits columns
The linear combinations are vectors that can be written as Ax
Ax = b is solvable if and only if b is in the column space of A
Let A be an m × n matrixThe columns of A have m componentsThe columns of A live in Rm
The column space of A is a subspace of Rm
The column space of A is denoted by R(A)
R stands for range
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 11 / 53
The Nullspace of A
The nullspace of A consists of all solutions to Ax = 0
The nullspace of A is denoted by N(A)
Example: elimination1 2 32 4 83 6 11
→1 2 3
0 0 20 0 2
→1 2 3
0 0 20 0 0
The pivot variables are x1 and x3
The free variable is x2 (column 2 has no pivot)
The number of pivots is called the rank of A
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 12 / 53
Linear Independence
A sequence of vectors v1, v2, . . . , vn is linearly independent if the onlylinear combination that gives the zero vector is 0v1 + 0v2 + . . .+ 0vn
The columns of A are independent if the only solution to Ax = 0 isthe zero vector
Elimination produces no free variablesThe rank of A is equal to nThe nullspace N(A) contains only the zero vector
A set of vectors spans a space if their linear combinations fill the space
A basis for a vector space is a sequence of vectors that(i) are linearly independent(ii) span the space
The dimension of a vector space is the number of vectors in everybasis
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 13 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 14 / 53
Sample Variance and Covariance
Sample mean of the elements of a vector x of length m
x =1m
m∑i=1
xi
Sample variance of the elements of x
Var(x) =1
m − 1
m∑i=1
(xi − x)2
Let y be vector of length m
Sample covariance of x and y
Cov(x , y) =1
m − 1
m∑i=1
(xi − x)(yi − y)
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 15 / 53
Sample Variance
Let e be a column vector of m oneseTxm =
1m
m∑i=1
1xi =1m
m∑i=1
xi = x
x is a scalar
Let e x be a column vector repeating x m times
Let x = x − e eTxm = x − e x
The i th element of x isxi = xi − x
Take another look at the sample variance
Var(x) =1
m − 1
m∑i=1
(xi − x)2 =1
m − 1
m∑i=1
x2i =
xTxm − 1 =
‖x‖2m − 1
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 16 / 53
Sample Covariance
A similar result holds for the sample covariance
Let y = y − e eTym = y − e y
The sample covariance becomes
Cov(x , y) =1
m − 1
m∑i=1
(xi − x)(yi − y) =1
m − 1
m∑i=1
xi yi =xTy
m − 1
Observe that Var(x) = Cov(x , x)
Proceed with Cov(x , x) and treat Var(x) as a special case
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 17 / 53
Variance-Covariance Matrix
Suppose x and y are the columns of a matrix R
R =
| |x y| |
R =
| |x y| |
The sample variance-covariance matrix is
Cov(R) =
Cov(x , x) Cov(x , y)
Cov(y , x) Cov(y , y)
=1
m − 1
xTx xTy
yTx yTy
=RTR
m − 1
The sample variance-covariance matrix is symmetric
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 18 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 19 / 53
Computing Covariance Matrices
Take a closer look at how x was computed
x = x − e x = x − e eTxm = x − eeT
m x =
(I − eeT
m
)︸ ︷︷ ︸
A
x
What is A?The outer product (eeT) is an m ×m matrix
I is the m ×m identity matrix
A is an m ×m matrix
Premultiplication by A turns x into x
Can think of matrix multiplication as one matrix acting on another
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 20 / 53
Computing Covariance Matrices
Next, consider what happens when we premultiply R by AThink of R in block structure where each column is a block
AR = A
| |x y| |
=
A|x|
A|y|
=
| |x y| |
= R
Expression for the variance-covariance matrix no longer needs R
Cov(R) =1
n − 1 RTR =1
m − 1(AR)T(AR)
Since R has 2 columns, Cov(R) is a 2× 2 matrixIn general, R may have n columns =⇒ Cov(R) is an n × n matrix
Still use the same m ×m matrix A
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 21 / 53
Computing Covariance MatricesTake another look at formula for variance-covariance matrixRule for transpose of a product
Cov(R) =1
m − 1(AR)T(AR) =1
m − 1RTATAR
Consider the product
ATA =
(I − eeT
m
)T (I − eeT
m
)
=
[IT −
(eeT
m
)T](
I − eeT
m
)
=
[IT − [eT]TeT
m
](I − eeT
m
)
=
(I − eeT
m
)(I − eeT
m
)Since AT = A, A is symmetric
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 22 / 53
Computing Covariance Matrices
Continuing . . .
ATA =
(I − eeT
m
)(I − eeT
m
)=
(I − eeT
m
)2= A2
= I2 − eeT
m − eeT
m +
(eeT
m
)(eeT
m
)
= I − 2eeT
m +e(eTe)eT
m2
= I − 2eeT
m +emeT
m2
=
(I − eeT
m
)= A
A matrix satisfying A2 = A is called idempotentKjell Konis (Copyright © 2013) 6. Linear Algebra II 23 / 53
Computing Covariance Matrices
Can simplify the expression for the sample variance-covariance matrix
Cov(R) =1
m − 1RT[ATA]R =
1m − 1︸ ︷︷ ︸scalar
RT︸︷︷︸n×m
A︸︷︷︸m×m
R︸︷︷︸m×n
How to order the operations . . .
RTAR = RT(
I − eeT
m
)R = RT
R − 1me
1×n︷ ︸︸ ︷eT︸︷︷︸
1×m
R︸︷︷︸m×n
= RT
R − 1m e︸︷︷︸
m×1M1︸︷︷︸1×n
= RT︸︷︷︸n×m
m×n︷ ︸︸ ︷ R︸︷︷︸m×n− 1
m M2︸︷︷︸m×n
= Cov(R)︸ ︷︷ ︸n×n
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 24 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 25 / 53
Orthogonal Matrices
Two vectors q,w ∈ Rm are orthogonal if their inner product is zero
〈q,w〉 = 0
Consider a set of m vectors {q1, . . . , qm} where qj ∈ Rm \ 0
Assume that the vectors {q1, . . . , qm} are pairwise orthogonal
Let qj =qj‖qj‖
be a unit vector in the same direction as qj
The vectors {qa, . . . , qm} are orthonormal
Let Q be a matrix with columns {qj}, consider the product QQT
QTQ =
— qT
1 —
· · · · · · · · ·
— qTm —
|
... |
q1... qm
|... |
=
qT
1 q1 qTi qj
. . .
qTi qj qT
mqm
= I
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 26 / 53
Orthogonal Matrices
A square matrix Q is orthogonal if QTQ = I and QQT = IOrthogonal matrices represent rotations and reflectionsExample:
Q =
cos(θ) − sin(θ)
sin(θ) cos(θ)
Rotates a vector in the xy plane through the angle θ
QTQ =
cos(θ) sin(θ)
− sin(θ) cos(θ)
cos(θ) − sin(θ)
sin(θ) cos(θ)
=
cos2(θ) + sin2(θ) − cos(θ) sin(θ) + sin(θ) cos(θ)
− sin(θ) cos(θ) + cos(θ) sin(θ) sin2(θ) + cos2(θ)
= I
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 27 / 53
Properties of Orthogonal Matrices
The definition QTQ = QQT = I implies Q−1 = QT
QT =
cos(θ) sin(θ)
− sin(θ) cos(θ)
=
cos(−θ) − sin(−θ)
sin(−θ) cos(−θ)
Cosine is an even function and sine is an odd function
Multiplication by an orthogonal matrix Q preserves dot products
(Qx) · (Qy) = (Qx)T(Qy) = xTQTQy = xTIy = xTy = x · y
Multiplication by an orthogonal matrix Q leaves lengths unchanged
‖Qx‖ = ‖x‖
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 28 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 29 / 53
Singular Value Factorization
So far . . .If Q is orthogonal then Q−1 = QT
if D is diagonal then D−1 is
D =
d1 0
. . .
0 dm
D−1 =
1/d1 0
. . .
0 1/dm
Orthogonal and diagonal matrices have nice properties
Wouldn’t it be nice if any matrix could be expressed as a product ofdiagonal and orthogonal matrices . . .
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 30 / 53
Singular Value Factorization
Every m × n matrix A can be factored into A = UΣV T where
Picture for m > nU is an m ×m orthogonal matrix whose columns are the left singularvectors of AΣ is an m × n diagonal matrix containing the singular values of A
Convention: σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0
V is an n × n orthogonal matrix whose columns contain the rightsingular vectors of A
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 31 / 53
Multiplication
Every invertible 2× 2 matrix transforms the unit circle into an ellipse328 7 Linear Transformations
GAGli
IFigure 76 U and V are rotations and reflections. is a stretching matrix,
This time it is V V I that disappears. Multiplying Z gives u and n as before.We have an ordinary factorization of the symmetric matrix A .4 T The columns of Uare the cigenvectom cIA 4 T
Example 4 Compute the eigenvectors u and u2 directly from AA’. The eigenvaluesare again u 2 and c-) = 8. The singular a1ues are still their square roots:
A/IT = 21 [2 l] [8 0111(2 Ij [0 21
This matrix happens to be diagonal. Its eigenvectors are (1,0) and (0, 1). This agreeswith n and u found earlier, but in the opposite order. Why should we take u1 to be(0. 1) instead of (I, 0)? Because we have to follow the order of the eigenvalues.We originally chose n = 2. The eigenvectors v (forATA) and u (for 44T) have
to stay with that choice. If you want the u’s in decreasing order, that is also possibleand gei elaib piJciiedi Then I’Ti = a ad u =‘ 2 This exchanges , and in Iand it exchanges u and a: in U. The new SVD is still correct:
7.3 Choice of Basis: Similarity and Diagonalization 329
A.v.1=*lOu, — —
rowspace _- — — —
/ -— A 0 = H
nuilspae of A i
Figure 7.7 The SVD chooses orthonormal bases so that As u
vector a1. We ea.n see those vectors in A. and make them into unit vectot.s:
multtple ci’ row—[Ij /2 W
diiple A u lurni 2 =- 2
L’l v5 [1JThen Av must equal o u. 11 does, with singular value e /Tii, The SVD couldstop there (it usually doesn’t
[2. 21 [2//51 rHU. l:V I
it is customary for U and V to be square. The matrices need a second column,The vector V: must be orthogonal to v . and u- must be orihoconal to u1
1 ii i r 1—i and u——l —2
The vector is in the nulIspace it is perpendicular to v1 in the row space. Multiplyby .4 to get Av = 0. We could say that the second singular value is u- = 0. but thisis against the rules. Stngular values are like pivots —-only the r nonzeros are counted.If A is 2 by 2 then all three matrices U, . V are 2 by 2 in the true SVD:
r -i -‘ 1 1 ii— V L. ‘—I -
[1 II v5 [I j L (t UJ /2 [
The matrices V and V contain orthonorma! bases for allfourfiindamental%Ilbspaces:
first r columns of V : row space of Alast n — r columns of V : nuuispace of Afirst r columns of L - : column space of’ .4last ni — r columns of U : nullspace of AH
A = UVT is now 2 Ii 01 2v 01 1/v Ijv1T1 [o 1] 0 ] l/2 l/j
Th thei mail b-t of freedom i to multiph an ci nv tot hs I The r suit, is stilli i n r I d hi o s lu Is d i R a cau e u U
It lie lens 01 ‘he e en eeto1 that k ep a, positi’ e. and it is the order ot thei ectois ti at pu a1 fir tin .
I’ i1d hd S\ I) of the 5ii guh r matri\ I The i ank is r IThe ‘ - oniT on ha ‘s \ector . The olumn pac ha oni one ha is
Av2 = UΣV Tv2 = UΣ
[−vT
1−−vT
2−
]v2 = U
[σ1 00 σ2
] [01
]=
[u1 u2
] [ 0σ2
]
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 32 / 53
Multiplication
Visualize: Ax = UΣV TxU rotate right 45◦
Σ stretch x coord by 1.5, stretch y coord by 2V rotate right 22.5◦
x :=1√2
[11
]x := V Tx x := Σx x := Ux
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 33 / 53
Example
> load("R.RData")> library(MASS)> eqscplot(R)
> svdR <- svd(R)> U <- svdR$u> S <- diag(svdR$d)> V <- svdR$v
> all.equal(U %*% S %*% t(V), R)[1] TRUE
> arrows(0, 0, V[1,1], V[2,1])> arrows(0, 0, V[1,2], V[2,2])
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
● ●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
−5 0 5−
50
5C Returns (%)
BA
C R
etur
ns (
%)
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 34 / 53
Example (continued)
> u <- V[, 1] * S[1, 1]> u <- u / sqrt(m - 1)> arrows(0, 0, u[1], u[2])
> w <- V[, 2] * S[2, 2]> w <- w / sqrt(m - 1)> arrows(0, 0, w[1], w[2])
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
● ●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
−5 0 5−
50
5C Returns (%)
BA
C R
etur
ns (
%)
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 35 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 36 / 53
Eigenvalues and Eigenvectors
Let R be an m × n matrix and let R =(I − eeT
m
)R
Let R = UΣV T be the singular value factorization of R
Recall that[Cov(R)
]= 1
m−1 RTR
= 1m−1
(UΣV T)T(UΣV T)
= 1m−1V ΣT(UTU
)ΣV T
= 1m−1V ΣTΣV T
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 37 / 53
Eigenvalues and Eigenvectors
Remember that Σ is a diagonal m × n matrix
Σ =
σ1
. . .σn
=⇒ ΣTΣ =
σ1. . .
σn
︸ ︷︷ ︸
n×m
σ1
. . .σn
︸ ︷︷ ︸
m×n
ΣTΣ is a diagonal matrix with σ21 ≥ · · · ≥ σ2
n along the diagonal
Let
Λ = 1m−1ΣTΣ =
λ1 =
σ21
m−1. . .
λn = σ2n
m−1
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 38 / 53
Eigenvalues and Eigenvectors
Substitute Λ into the expression for the covariance matrix of R[Cov(R)
]= 1
m−1V ΣTΣV T = V ΛV T
Let ej be a unit vector in the j th coordinate direction
Multiply a right singular vector vj by Cov(R)
[Cov(R)
]vj = V ΛV Tvj = V Λ
(V Tvj
)= V Λ ej
Recall that a matrix times a vector is a linear combination of thecolumns
Av = v1
|a1|
+ · · ·+ v1
|a1|
=⇒ Λej = 1
...λj...
= λjej
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 39 / 53
Eigenvalues and Eigenvectors
Substituting Λej = λjej . . .[Cov(R)
]vj = V ΛV Tvj = V Λ
(V Tvj
)= V Λ ej = Vλjej = λjVej = λjvj
In summaryvj is a right singular vector of R[Cov(R)
]= RTR[
Cov(R)]vj = λjvj[
Cov(R)]vj same direction as vj , length scaled by factor λj
In general: let A be a square matrix and consider the product AxCertain special vectors x are in the same direction as Ax
These vectors are called eigenvectors
Equation: Ax = λxx ; the number λx is the eigenvalueKjell Konis (Copyright © 2013) 6. Linear Algebra II 40 / 53
Diagonalizing a Matrix
Suppose an n × n matrix A has n linearly independent eigenvectors
Let S be a matrix whose columns are the n eigenvectors of A
S−1AS = Λ =
λ1. . .
λn
The matrix A is diagonalized
Useful representations of a diagonalized matrix
AS = SΛ S−1AS = Λ A = SΛS−1
Diagonalization requires that A have n eigenvectors
Side note: invertibility requires nonzero eigenvaluesKjell Konis (Copyright © 2013) 6. Linear Algebra II 41 / 53
The Spectral Theorem
Returning to the motivating example . . .
Let A = RTR where R is an m × n matrix
A is symmetric
Spectral Theorem Every symmetric matrix A = AT has thefactorization QΛQT with real diagonal Λ and orthogonal matrix Q:
A = QΛQ−1 = QΛQT with Q−1 = QT
Caveat A nonsymmetric matrix can easily produce λ and x that arecomplex
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 42 / 53
Positive Definite Matrices
The symmetric matrix A is positive definite if xTAx > 0 for everynonzero vector x
2× 2 case: xTAx = [ x1 x2 ]
[a bb c
] [x1x2
]= ax2
1 + 2bx1x2 + cx22 > 0
The scalar value xTAx is a quadratic function of x1 and x2
f (x1, x2) = ax21 + 2b x1x2 + cx2
2
f has a minimum of 0 at (0, 0) and is positive everywhere else
1× 1 a is a positive number
2× 2 A is a positive definite matrix
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 43 / 53
R Example
> eigR <- eigen(var(R))> S <- eigR$vectors> lambda <- eigR$values
> u <- sqrt(lambda[1]) * S[,1]> arrows(0, 0, u[1], u[2])
> w <- sqrt(lambda[2]) * S[,2]> arrows(0, 0, w[1], w[2])
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●●
●●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
● ●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
−5 0 5−
50
5
C Returns (%)
BA
C R
etur
ns (
%)
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 44 / 53
Outline
1 Transposes and Permutations
2 Vector Spaces and Subspaces
3 Variance-Covariance Matrices
4 Computing Covariance Matrices
5 Orthogonal Matrices
6 Singular Value Factorization
7 Eigenvalues and Eigenvectors
8 Solving Least Squares Problems
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 45 / 53
Least Squares
Citigroup Returns vs. S&P 500Returns (Monthly - 2010)
●
●
●
●
●
●
●
●
●
●
●
●
−0.05 0.00 0.05
−0.
100.
000.
050.
100.
150.
20
S&P 500 Returns
Citi
grou
p R
etur
ns
●
●
●
●
●
●
●
●
●
●
●
●
y~ = α + βx
Set of m points (xi , yi )
Want to find best-fit line
y = α + βx
Criterion:m∑
i=1
[yi − yi
]2 should
be minimumChoose α and β so that
m∑i=1
[yi − (α + βxi )
]2minimized when
α = αβ = β
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 46 / 53
Least Squares
What does the column picture look like?Let y = (y1, y2, . . . , ym)
Let x = (x1, x2, . . . , xm)
Let e be a column vector of m onesCan write y as a linear combination
y =
y1...
ym
= α
|e|
+ β
x1...
xm
=
1 x1...
...1 xm
[α
β
]= Xβ
Want to minimizem∑
i=1
[yi − yi
]2= ‖y − y‖2 = ‖y − Xβ‖2
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 47 / 53
QR Factorization
Let A be an m × n matrix with linearly independent columnsFull QR Factorization: A can be written as the product of
an m ×m orthogonal matrix Qan m × n upper triangular matrix R(upper triangular means rij = 0 when i > j)
A = QR
Want to minimize
‖y − Xβ‖2 = ‖y − QRβ‖2
Recall: orthogonal transformation leaves vector lengths unchanged
‖y − Xβ‖2 = ‖y − QRβ‖2 = ‖QT(y − QRβ)‖2 = ‖QTy − Rβ‖2
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 48 / 53
Least Squares
Let u = QTy
u − Rβ =
u1
u2
u3...
−
r11 r12
0 r22
0 0...
...
αβ
=
u1 − (r11α + r12β)
u2 − r22β
u3...
α and β effect only the first n elements of the vector
Want to minimize
‖u − Rβ‖2 =[u1 − (r11α + r12β)
]2+[u2 − r22β
]2+
m∑i=(n+1)
u2i
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 49 / 53
Least Squares
Can find α and β by solving the linear system Rβ = u r11 r12
0 r22
αβ
=
u1
u2
R first n rows of R, u first n elements of u
System is already upper triangular, solve using back substitution
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 50 / 53
R Example
First, get the data
> library(quantmod)> getSymbols(c("C", "ˆGSPC"))> citi <- c(coredata(monthlyReturn(C["2010"])))> sp500 <- c(coredata(monthlyReturn(GSPC["2010"])))
The x variable is sp500, bind a column of ones to get matrix X
> X <- cbind(1, sp500)
Compute QR factorization of X and extract the Q and R matrices
> qrX <- qr(X)> Q <- qr.Q(qrX, complete = TRUE)> R <- qr.R(qrX, complete = TRUE)
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 51 / 53
R Example
Compute u = QTy
> u <- t(Q) %*% citi
Solve for α and β
> backsolve(R[1:2,1:2], u[1:2])[1] 0.01708494 1.33208984
Compare with built-in least squaresfitting function
> coef(lsfit(sp500, citi))Intercept X
0.01708494 1.33208984
●
●
●
●
●
●
●
●
●
●
●
●
−0.05 0.00 0.05
−0.
100.
000.
050.
100.
150.
20S&P 500 Returns
Citi
grou
p R
etur
ns
●
●
●
●
●
●
●
●
●
●
●
●
y~ = α + βx
y = α + βx
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 52 / 53
http://computational-finance.uw.edu
Kjell Konis (Copyright © 2013) 6. Linear Algebra II 53 / 53