Krylov-Subspace Methods - I
-
Upload
eugenia-lynch -
Category
Documents
-
view
67 -
download
1
description
Transcript of Krylov-Subspace Methods - I
Krylov-Subspace Methods - I
Lecture 6
Alessandra Nardi
Thanks to Prof. Jacob White, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy
Last lecture review
• Iterative Methods Overview– Stationary– Non Stationary
• QR factorization to solve Mx=b– Modified Gram-Schmidt Algorithm– QR Pivoting– Minimization View of QR
• Basic Minimization approach• Orthogonalized Search Directions• Pointer to Krylov Subspace Methods
4M
3M
2M
1M
12 13 14r r r
1Q
2Q
3Q
4Q
11r
22r 23 24r r
33r
44r
34r
Last lecture reminderQR Factorization – By picture
For i = 1 to N “For each Target Column”
For j = 1 to i-1 “For each Source Column left of target”
end
end
1
ii i
irp p
iii iMpr Mp
Normalize
i ix x v p
i ip e
Tjj
Tiir p M Mp
i ji i jp p pr Orthogonalize Search Direction
QR Factorization – Minimization ViewMinimization Algorithm
Iterative Methods
Solve Mx=b minimizing the residual r=b-Mx
Stationary: x(k+1) =Gx(k)+c• Jacobi
• Gauss-Seidel
• Successive Overrelaxation
Non Stationary: x(k+1) =x(k)+akpk
• CG (Conjugate Gradient) A symmetric and positive definite
• GCR (Generalized Conjugate Residual)
• GMRES, etc etc
Iterative Methods - CG
kkTk
kTkk
k
kT
k
kTk
k
kkkk
kkkk
drr
rrrd
Mdd
rr
Mdrr
dxx
)()(
)1()1()1(
1
)()(
)()1(
)()1(
)(
)(
)(
Convergence is related to:– Number of distinct eigenvalues– Ratio between max and min eigenvalue
Why ?How?
• General Subspace Minimization Algorithm– Review orthogonalization and projection formulas
• Generalized Conjugate Residual Algorithm– Krylov-subspace – Simplification in the symmetric case.– Convergence properties
• Eigenvalue and Eigenvector Review– Norms and Spectral Radius– Spectral Mapping Theorem
Outline
Approximately Solve Mx b
10 1Approximate as a weighted sum of , ,...,k
kx w w w
1 1 0
0
kk k
i ii
r b Mx r Mw
1
0
k
ki i
i
x w
Residual Minimizing idea: pick ' to minimizei s
2
21 1 1 0
22
kTk k k
i ii o
r r r r Mw
Arbitrary Subspace MethodsResidual Minimization
221 0
20 2
Minimizing is easy ifk
ki i
i
r r Mw
0 1 0 1, ,..., , ,...,k kspan p p p span w w w
0 1Create a set of vectors , ,..., such thatkp p p
0 or is orthogonal to T
i j i jMw Mw Mw Mw
and 0T
i jMp Mp
Use Gram-Schmidt on Mwi’s!
Arbitrary Subspace MethodsResidual Minimization
Arbitrary Subspace MethodsOrthogonalization
k
ii
iT
i
iT
kkk
TT
TTT
TT
TTT
TT
TTT
pMpMp
MpMwwp
MpMp
MpMwMpMp
MpMp
MpMw)(Mp)(Mpppwp
MpMp
MpMwMpMppwp
p
0
11
121212
00
02020211200222
00
01010100111
00
)()(
)()( :generalIn
0)()( and
0 and
0)()( and
1
0
For 1 to Tk
k ik k iT
i i i
Mw Mpi k p w p
Mp Mp
0
1
0
Tk
ikiT
i i i
r Mpx p
Mp Mp
Arbitrary Subspace Solution Algorithm
1. Given M, b and a set of search directions: {w0,…,wk}
2. Make wi’s MMT orthogonal and get new search directions: {p0,…,pk}
3. Minimize the residual:
For i = 0 to k
For j = 1 to i-1
end
end
1
ii iMp
p p Normalize
i ip w
Tj jii
Tip p M pMpp Orthogonalize Search Direction
01ii
T
ii rx x pMp Update Solution
Arbitrary Subspace Solution Algorithm
Krylov Subspace
• How about the initial set of search directions {w0,…,wk} ?
• A particular choice that is commonly used is: {w0,
…,wk} {b, Mb, M2b…}
m(A,v) span{v, Av, A2v, …, Am-1v} is called Krylov Subspace
1 0 0
0
kk i
i ki
x M r M r
1 0 1 0 0
0
kk i
i ki
r r M r I M M r
kth order polynomial
Krylov Subspace Methods
k
iii
k wx0
1
},...,,{},..., 121 bMMbbspanw,wspan{w k
k
Krylov Subspace MethodsSubspace Generation
0Note: for any 0
0 1 0 0 0 00span , = span ,r r r Mr r Mr
The set of residuals also can be used as a representation of the Krylov-Subspace
Generalized Conjugate Residual AlgorithmNice because the residuals generate next search directions
1
11
0
Tkkjk
k jTj
j j
Mr Mpp r p
Mp Mp
Tkk
k T
k k
r Mp
Mp Mp
1k kk kx x p
1k kk kr r Mp
Determine optimal stepsize in kth search direction
Update the solution (trying to
minimize residual) and the residual
Compute the new orthogonalized search direction (by using the most
recent residual)
Krylov-Subspace MethodsGeneralized Conjugate Residual Method
(k-th step)
1
11
0
Tkkjk
k jTj
j j
Mr Mpp r p
Mp Mp
Tkk
k T
k k
r Mp
Mp Mp
1k kk kx x p
1k kk kr r Mp
Vector inner products, O(n)Matrix-vector product, O(n) if sparse
Vector Adds, O(n)
O(k) inner products, total cost O(nk)
If M is sparse, as k (# of iters) approaches n,3total cost ( ) (2 ) .... ( ) ( )O n O n O kn O n
Better Converge Fast!
Krylov-Subspace MethodsGeneralized Conjugate Residual Method
(Computational Complexity for k-th step)
1 1
1 11 1
0
T Tk kkj kk k
k j k kT Tj k kj j
Mr Mp Mr Mpp r p p r p
Mp MpMp Mp
If k (# of iters ) n, then symmetric, sparse, GCR is O(n2 )
1If then T k jM M r Mp j k
Better Converge Fast!
An Amazing fact that will not be derived
Orthogonalization in one step
Krylov-Subspace MethodsGeneralized Conjugate Residual Method
(Symmetric Case – Conjugate Gradient Method)
Summary
• What is an iterative non stationary method: x(k+1) =x(k)+akpk
• How search to calculate:– Search directions (pk)– Step along search directions (ak)
• Krylov Subspace GCR • GCR is O(k2n)
– Better converge fast!
Now look at convergence properties of GCR
0 jIf for all in GCR, thenj k 0 0
0 1span , ,..., span , ,...,1) kkp p p r Mr Mr
th1 02) is the k ( ) , o rderkk kx M r
21
2polynomial which minimizes kr
1 1 0 03) ( )k kkr b Mx r M M r
0 01( )k kI M M r M r
01where is the order poly 1
th
k M r k 21
12minimizing subject to 0 = 1 k
kr
Krylov Methods Convergence AnalysisBasic properties
Krylov Methods Convergence AnalysisOptimality of GCR poly
• GCR optimality property (key property of the
algorithm): GCR picks the best (k+1)-th order
polynomial minimizing and subject to:
)(1 Mk2
2
1kr
1)0(1 k
GCR Optimality Property
k+1 polynomial such that 0 =1
ThereforeAny polynomial which satisfies the
constraints can be used to get an upper bound on
1
0
kr
r
1 0k+1 k+1( ) r where is any ordertk hkr M
Krylov Methods Convergence AnalysisOptimality of GCR poly
Eigenvalues and eigenvectors of a matrix M satisfy
eigenvector
eigenvalue
Or, is an eigenvalue of ifi M is singulariM I
= 0ii uM I
ii iu uM
is an eigenvector of ifiu M
Eigenvalues and eigenvectors reviewBasic definitions
Almost all NxN matrices have N linearly independent Eigenvectors
1 2 3 Nu u u u
M
1 1 2 2 3 3 N Nu u u u
The set of all eigenvalues of M is known as the Spectrum of M
Eigenvalues and eigenvectors reviewA symplifying assumption
Almost all NxN matrices have N linearly independent Eigenvectors
MU U1 0 0
0 0
0 0 N
1 1orM MU U U U Does NOT imply distinct eigenvalues, can equal i j Does NOT imply is nonsingular M
Eigenvalues and eigenvectors reviewA symplifying assumption
Re
Im
i
The spectral Radius of M is the radius of the smallest circle, centered at the origin, which encloses all of
M’s eigenvalues
Eigenvalues and eigenvectors reviewSpectral radius
L2 (Euclidean) norm :
n
iixx
1
2
2
L1 norm :
L norm :
n
iixx
11
ii
xx max
12x
Unit circle
Unit square
1
1 11x
1
x
Eigenvalues and eigenvectors reviewVector norms
Vector induced norm : Axx
AxA
xx 1maxmax
n
iij
jAA
11
max
Induced norm of A is the maximum “magnification” of by
= max abs column sum
n
jij
iAA
1
max = max abs row sum
2A = (largest eigenvalue of ATA)1/2
x A
Eigenvalues and eigenvectors reviewMatrix norms
Theorem: Any induced norm is a bound on the spectral radius
max1 ll
l
Mxx
M
Proof:First pick , 1i i l
x u u
i i i i i il l lMu u u
Eigenvalues and eigenvectors reviewInduced norms
Given a polynomial
0 1p
pf x a a x a x
Apply the polynomial to a matrix
0 1p
pf M a a M a M
Then
spectrum f M f spectrum M
Useful Eigenproperties Spectral Mapping Theorem
Krylov Methods Convergence AnalysisOverview
where is any (k+1)-th order polynomial
subject to:
may be used to get an upper bound on
)(~1 Mk
0
1
r
r k
1)0(~1 k
01
01
101
1 )(~)()( rMrMrrMr kkk
kk
)(~1 Mk
Matrix norm property GCR optimality property
• Review on eigenvalues and eigenvectors– Induced norms: relate matrix eigenvalues to the
matrix norms– Spectral mapping theorem: relate matrix eigenvalues
to matrix polynomials
• Now ready to relate the convergence properties of Krylov Subspace methods to eigenvalues of M
Krylov Methods Convergence AnalysisOverview