Tutorial 4
11
Math for CS Tutorial 4 1 Tutorial 4 Contents: 1. Least squares solution for overcomplete linear systems. 2. … via normal equations 3. … via A = QR factorization 4. … via SVD decomposition 5. SVD - Singular Value Decomposition, A = UΣV T
description
Tutorial 4. Contents: Least squares solution for overcomplete linear systems. … via normal equations … via A = QR factorization … via SVD decomposition SVD - Singular Value Decomposition , A = U Σ V T. Normal Equations. Consider the system - PowerPoint PPT Presentation
Transcript of Tutorial 4
Math for CS Tutorial 4 1
Tutorial 4
Contents:
1. Least squares solution for overcomplete linear systems.
2. … via normal equations
3. … via A = QR factorization
4. … via SVD decomposition
5. SVD - Singular Value Decomposition, A = UΣVT
Math for CS Tutorial 4 2
Normal Equations
Consider the system
It can be a result of some physical measurements, which usually incorporate
some errors. Since, we can not solve it exactly, we would like to minimize the
error:
r=b-Ax
r2=rTr=(b-Ax)T(b-Ax)=bTb-2xTATb+xTATAx
(r2)x=0 - zero derivative is a (necessary) minimum condition
-2ATb+2ATAx=0;
ATAx = ATb; – Normal Equations
5.3
5.2
2
21
12
11
x
Math for CS Tutorial 4 3
Normal Equations 2
ATAx = ATb – Normal Equations
5.3
5.2
2
21
12
11
x
65
56
21
12
11
211
121AAT
...,6
55.105.11
11
6
6
55.105.11
6
510
5.1056
5.1165
5.1056212
xxx
5.11
5.10
5.3
5.2
2
211
121bAT
Math for CS Tutorial 4 4
Least squares via A=QR decomposition
A(m,n)=Q(m,n)R(n,n), Q is orthogonal, therefore QTQ=I.
QRx=b
R(n,n)x=QT(n,m)b(m,1) -well defined linear system
x=R-1QTb
Q is found by Gram=Schmidt orthogonalization of A. How to find R?
QR=A
QTQR=QTA, but Q is orthogonal, therefore QTQ=I:
R=QTA
R is upper triangular, since in orthogonalization procedure only
a1,..ak (without ak+1,…) are used to produce qk
Math for CS Tutorial 4 5
Least squares via A=QR decomposition 2
Let us check the correctness:
QRx=b
Rx=QT
b
x=R-1
QT
b
10
11
01
A
3
20
6
1
2
16
1
2
1
Q
6
1
3
20
2
1
2
2
10
11
01
3
2
6
1
6
1
02
1
2
1
AQR T
10
11
01
18
2
3
20
18
2
6
1
2
11
18
2
6
1
2
11
6
1
3
20
2
1
2
2
3
20
6
1
2
16
1
2
1
QR
Math for CS Tutorial 4 6
Least squares via SVD
Ax=b;
A=UΣVT -singular value decomposition of A:
UΣVTx=b;
x= VΣ-1UTb
Math for CS Tutorial 4 7
Singular Value Decomposition 1
The SVD based on the fact that for any A there are orthonormal
bases v1,…vr for the row space and u1,…ur for the column space,
such, that
Avi=σiui, while σi>0
Thus, any matrix can be represented as
,where U and V are orthogonal, and Σ is diagonal.
Tnnnmmmnm VUA ,,,,
T
nr
nm
r
mm
nrnm
nn
VVVUUUA
,
......
.... 1
,
1
,
1,
Math for CS Tutorial 4 8
Singular Value Decomposition 2
First we find the matrix V:
ATA=(UΣVT)T(UΣVT)= VTΣT UTU ΣVT = VTΣTΣVT
This is an ordinary (eigenvector) factorization of a
symmetric matrix, therefore V is built of eigenvectors of
ATA. The eigenvectors of ATA are rows of VT.
In the same way one can prove, that
U is built from eigenvectors of AAT. However, an easier
way to find U and Σ is to use the equations: Avi=σiui
Math for CS Tutorial 4 9
SVD Example
Let us find SVD for the matrix
In order to find V, we are calculating eigenvectors of ATA:
(5-λ)2-9=0;
λ2-10 λ +16=0;
λ1,2=8,2
11
22A
53
35
11
22
12
12AAT
2
12
1
1V
2
12
1
2V
Math for CS Tutorial 4 10
SVD Example 2
Now, we obtain the U and Σ :
A=UΣVT:
2
0
2
12
1
11
22111 uAv ;2,
1
011
u
0
22
2
12
1
11
22222 uAv ;22,
0
112
u
2
1
2
12
1
2
1
220
02
01
10
11
22
Math for CS Tutorial 4 11
Appendix: derivative of xTATAx
xAAxAA
AAxxAAxAAxx
xAAxx
xAAxx
T
lklklkj
kjklkjj
lklklkj
lkjlklkjj
lkjlkljk
Tj
TT
22,
,,,,
,,
