Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini...
Transcript of Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini...
![Page 1: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/1.jpg)
0.5setgray0
0.5setgray1
Krylov subspace methods:
a versatile tool in Scientific Computing
V. Simoncini
Dipartimento di Matematica
Universita di [email protected]
http://www.dm.unibo.it/˜simoncin
Krylov subspace methods – p. 1
![Page 2: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/2.jpg)
The framework
Given the large n× n linear system
Ax = b
Find xm such that xm ≈ x
x0 initial guess r0 = b−Ax0 (if no info, take x0 = 0)
Krylov subspace approximation: xm = x0 + zm
zm ∈ Km(A, r0) := span{r0, Ar0, A2r0, . . . , A
m−1r0}
? Projection onto a much smaller space m¿ n
Krylov subspace methods – p. 2
![Page 3: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/3.jpg)
The framework
Given the large n× n linear system
Ax = b
Find xm such that xm ≈ x
x0 initial guess r0 = b−Ax0 (if no info, take x0 = 0)
Krylov subspace approximation: xm = x0 + zm
zm ∈ Km(A, r0) := span{r0, Ar0, A2r0, . . . , A
m−1r0}
? Projection onto a much smaller space m¿ n
Krylov subspace methods – p. 2
![Page 4: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/4.jpg)
Basic Idea of Projection
Assume x0 = 0.
Let {v1, . . . , vm} be a basis of
Km(A, r0) = span{r0, Ar0, A2r0, . . . , A
m−1r0}
and Vm := [v1, . . . , vm]
Thenxm = Vmym ym ∈ R
m
ym coefficients of linear combination
Krylov subspace methods – p. 3
![Page 5: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/5.jpg)
Some popular Krylov subspace methods
rm = b−Axm = b−AVmym
Mostly theoretical (for nonsymmetric A):
GMRES (Generalized Minimum RESidual)
ym : miny∈Rm
‖rm‖2
FOM (Full Orthogonalization Method)
ym : rm ⊥ Km
Note:for A symmetric pos. def., FOM becomes CG (Conjugate Gradients)
Krylov subspace methods – p. 4
![Page 6: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/6.jpg)
Some popular Krylov subspace methods
More Practical (for nonsymmetric A):
GMRES(m): Restarted GMRES
FOM(m): Restarted FOM (far less popular)
BiCGStab(`): short-term recurrence
Restarted Procedure: Given x0, r0
do until convergence∗ Run m steps of “Method” to get xm
∗ Compute rm = b−Axm
∗ Set x0 ← xm, r0 ← rm
Characteristics:
? Economy-versions
? “Good” properties are lost or preserved only locally
Krylov subspace methods – p. 5
![Page 7: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/7.jpg)
Some popular Krylov subspace methods
More Practical (for nonsymmetric A):
GMRES(m): Restarted GMRES
FOM(m): Restarted FOM (far less popular)
BiCGStab(`): short-term recurrence
Restarted Procedure: Given x0, r0
do until convergence∗ Run m steps of “Method” to get xm
∗ Compute rm = b−Axm
∗ Set x0 ← xm, r0 ← rm
Characteristics:
? Economy-versions
? “Good” properties are lost or preserved only locally
Krylov subspace methods – p. 5
![Page 8: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/8.jpg)
Some popular Krylov subspace methods
More Practical (for nonsymmetric A):
GMRES(m): Restarted GMRES
FOM(m): Restarted FOM (far less popular)
BiCGStab(`): short-term recurrence
Restarted Procedure: Given x0, r0
do until convergence∗ Run m steps of “Method” to get xm
∗ Compute rm = b−Axm
∗ Set x0 ← xm, r0 ← rm
Characteristics:
? Economy-versions
? “Good” properties are lost or preserved only locally
Krylov subspace methods – p. 5
![Page 9: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/9.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods. a versatile Tool for complex problems:
many requirements may be relaxed
Krylov subspace methods – p. 6
![Page 10: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/10.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods. a versatile Tool for complex problems:
many requirements may be relaxed
Krylov subspace methods – p. 6
![Page 11: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/11.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods. a versatile Tool for complex problems:
many requirements may be relaxed
Krylov subspace methods – p. 6
![Page 12: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/12.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods. a versatile Tool for complex problems:
many requirements may be relaxed
Krylov subspace methods – p. 6
![Page 13: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/13.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods. a versatile Tool for complex problems:
many requirements may be relaxed
Krylov subspace methods – p. 6
![Page 14: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/14.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods. a versatile Tool for complex problems:
many requirements may be relaxed
Krylov subspace methods – p. 6
![Page 15: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/15.jpg)
Restarted Methods
Convergence strongly depends on choice of m ...
0 100 200 300 400 500 600 700 80010−10
10−8
10−6
10−4
10−2
100
102
GMRES(15)
Number of Matrix−vector multiplies
norm
of re
lative r
esid
ual
GMRES(20)
Krylov subspace methods – p. 7
![Page 16: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/16.jpg)
Restarted Methods
Convergence strongly depends on choice of m ...
0 100 200 300 400 500 600 700 80010−10
10−8
10−6
10−4
10−2
100
102
GMRES(15)
Number of Matrix−vector multiplies
norm
of re
lative r
esid
ual
GMRES(20)
Krylov subspace methods – p. 7
![Page 17: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/17.jpg)
Restarted Methods
Convergence strongly depends on choice of m ... true?
0 100 200 300 400 500 600 700 80010−10
10−8
10−6
10−4
10−2
100
102
GMRES(15)
FOM(15)
Global subspace generated
norm
of re
lative r
esid
ual
GMRES(20)
Krylov subspace methods – p. 8
![Page 18: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/18.jpg)
Restarted Methods
Convergence strongly depends on choice of m
0 50 100 150 200 250 300 35010−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
GMRES(10)
FOM(10)
Number of Matrix−vector multiplies
norm
of re
lative r
esid
ual
Krylov subspace methods – p. 9
![Page 19: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/19.jpg)
Restarted Methods
Switch to FOM residual vector at the very first restart
0 50 100 150 200 250 300 350 400 45010−10
10−8
10−6
10−4
10−2
100
102
Number of Matrix−vector multiplies
no
rm o
f re
lative
re
sid
ua
l
GMRES(15)
GMRES(15)+FOM
Pictures from Simoncini, SIMAX 2000.
Krylov subspace methods – p. 10
![Page 20: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/20.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methodsIndefinite inner products
Inexact methods
Krylov subspace methods – p. 11
![Page 21: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/21.jpg)
Enhanced Restarted Methods
Warning: Large m not always means faster convergence
Current research:
Deflated Methods (originally used for A s.p.d.)Mansfield, Nicolaides, Erhel etal., Saad etal., Nabben, Vuik, ...
Augmented Methods ( information saved from previous restarts)De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...
See also Eiermann, Ernst, Schneider (JCAM 2000)
Truncated Methods (only local information maintained)Golub, Ye, Notay, Szyld, ...
Krylov subspace methods – p. 12
![Page 22: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/22.jpg)
Enhanced Restarted Methods
Warning: Large m not always means faster convergence
Current research:
Deflated Methods (originally used for A s.p.d.)Mansfield, Nicolaides, Erhel etal., Saad etal., Nabben, Vuik, ...
Augmented Methods ( information saved from previous restarts)De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...
See also Eiermann, Ernst, Schneider (JCAM 2000)
Truncated Methods (only local information maintained)Golub, Ye, Notay, Szyld, ...
Krylov subspace methods – p. 12
![Page 23: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/23.jpg)
Enhanced Restarted Methods
Warning: Large m not always means faster convergence
Current research:
Deflated Methods (originally used for A s.p.d.)Mansfield, Nicolaides, Erhel etal., Saad etal., Nabben, Vuik, ...
Augmented Methods ( information saved from previous restarts)De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...
See also Eiermann, Ernst, Schneider (JCAM 2000)
Truncated Methods (only local information maintained)Golub, Ye, Notay, Szyld, ...
Krylov subspace methods – p. 12
![Page 24: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/24.jpg)
Enhanced Restarted Methods
Warning: Large m not always means faster convergence
Current research:
Deflated Methods (originally used for A s.p.d.)Mansfield, Nicolaides, Erhel etal., Saad etal., Nabben, Vuik, ...
Augmented Methods ( information saved from previous restarts)De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...
See also Eiermann, Ernst, Schneider (JCAM 2000)
Truncated Methods (only local information maintained)Golub, Ye, Notay, Szyld, ...
Krylov subspace methods – p. 12
![Page 25: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/25.jpg)
De Sturler’s method
Tricky way to enhance approximation space
0 200 400 600 800 1000 120010−10
10−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
Iteration index
Rela
tive r
esid
ual norm
GMRES(50)
GMRES(150)
Krylov subspace methods – p. 13
![Page 26: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/26.jpg)
De Sturler’s method
Trickly way to enhance approximation space
0 200 400 600 800 1000 120010−10
10−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
Iteration index
Rela
tive r
esid
ual norm
GMRES(50)
GMRES(150)
GCROT(7,23,23,4,1,0)
Code: courtesy of Oliver Ernst.
Krylov subspace methods – p. 14
![Page 27: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/27.jpg)
Truncated methods and “Quasi-optimality”. I
? A truncated method discards “older” vectors
{v1, v2, . . . , vm−k, vm−k+1, . . . , vm︸ ︷︷ ︸orthogonal
, vm+1, . . . , }
(local optimality properties)
Limited memory requirements
Optimality is lost
1553.gif How “old” is old?
Krylov subspace methods – p. 15
![Page 28: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/28.jpg)
Truncated methods and “Quasi-optimality’. II
Example: A is non-normal, spectrum on circle |1− z| = 0.5
0 5 10 15 20 25 30 3510−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
number of iterations
norm
of re
sid
ual
k=5
k=10
k=∞
0 10 20 30 40 50 60 70 8010−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
101
number of iterations
norm
of
resid
ual
k = ∞
k = 30
k = 20
k = 10
Low eigenvector cond. number
high eigenvector cond. number
Krylov subspace methods – p. 16
![Page 29: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/29.jpg)
Truncated methods and “Quasi-optimality’. II
Example: A is non-normal, spectrum on circle |1− z| = 0.5
0 5 10 15 20 25 30 3510−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
number of iterations
norm
of re
sid
ual
k=5
k=10
k=∞
0 10 20 30 40 50 60 70 8010−9
10−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
101
number of iterations
norm
of re
sid
ual
k = ∞
k = 30
k = 20
k = 10
Low eigenvector cond. numberhigh eigenvector cond. number
Krylov subspace methods – p. 16
![Page 30: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/30.jpg)
Truncated methods and “Quasi-optimality’. III
? If A is nonsymmetric, but harmless modification of a symmetric matrixthen short truncation suffices
Ax = b A symmetric ⇒ P−1Ax = P−1b
P−1v = L−T L−1v + ε1, ε = 10−5, L Incomplete Cholesky of A
0 2 4 6 8 10 12 14 16 18 20
10−10
10−8
10−6
10−4
10−2
100
number of iterations
norm
of re
sid
ual
PCG
k = ∞ k = 2, 3
Krylov subspace methods – p. 17
![Page 31: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/31.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner productsInexact methods
Krylov subspace methods – p. 18
![Page 32: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/32.jpg)
Need for a “different” inner product ?
Typical orthogonality: rm ⊥ Km
Common alternative:
? Given M Hermitian and positive definite,
rm ⊥M Km
i.e., for Range(Vm)=Km it holds V ∗
mMrm = 0
may lead to minimization of ‖rm‖M or ‖em‖M
♣ In many cases, use of M−1
2 AM−1
2 hpd
What about different alternatives?
Krylov subspace methods – p. 19
![Page 33: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/33.jpg)
Need for a “different” inner product ?
Typical orthogonality: rm ⊥ Km
Common alternative:
? Given M Hermitian and positive definite,
rm ⊥M Km
i.e., for Range(Vm)=Km it holds V ∗
mMrm = 0
may lead to minimization of ‖rm‖M or ‖em‖M
♣ In many cases, use of M−1
2 AM−1
2 hpd
What about different alternatives?
Krylov subspace methods – p. 19
![Page 34: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/34.jpg)
Motivations for an indefinite inner product
Exploit inherent properties of the problem. For instance,
A complex symmetric
Exploit matrix structure
A =
(H B
BT 0
)
(or, say, A Hamiltonian)
... to gain in efficiency with (hopefully) no loss in reliability
Krylov subspace methods – p. 20
![Page 35: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/35.jpg)
Motivations for an indefinite inner product
Exploit inherent properties of the problem. For instance,
A complex symmetric
Exploit matrix structure
A =
(H B
BT 0
)
(or, say, A Hamiltonian)
... to gain in efficiency with (hopefully) no loss in reliability
Krylov subspace methods – p. 20
![Page 36: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/36.jpg)
Motivations for an indefinite inner product
Exploit inherent properties of the problem. For instance,
A complex symmetric
Exploit matrix structure
A =
(H B
BT 0
)
(or, say, A Hamiltonian)
... to gain in efficiency with (hopefully) no loss in reliability
Krylov subspace methods – p. 20
![Page 37: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/37.jpg)
An example. Indefinite (Constraint) Preconditioner
Ax = b A =
(H B
BT 0
), H = HT , H ≥ 0
Preconditioning: AP−1x = b
Block indefinite Preconditioner:
P =
(H B
BT 0
)H ≈ H
* AP−1 not symmetrizable!* However: AP−1 is P−1-Hermitian (Hermitian wrto P−1)
⇒ Cheap short-term recurrence(Simplified Lanczos - Freund & Nachtigal ’95)
Krylov subspace methods – p. 21
![Page 38: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/38.jpg)
An example. Indefinite (Constraint) Preconditioner
Ax = b A =
(H B
BT 0
), H = HT , H ≥ 0
Preconditioning: AP−1x = b
Block indefinite Preconditioner:
P =
(H B
BT 0
)H ≈ H
* AP−1 not symmetrizable!* However: AP−1 is P−1-Hermitian (Hermitian wrto P−1)
⇒ Cheap short-term recurrence(Simplified Lanczos - Freund & Nachtigal ’95)
Krylov subspace methods – p. 21
![Page 39: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/39.jpg)
An example. Indefinite (Constraint) Preconditioner
Ax = b A =
(H B
BT 0
), H = HT , H ≥ 0
Preconditioning: AP−1x = b
Block indefinite Preconditioner:
P =
(H B
BT 0
)H ≈ H
* AP−1 not symmetrizable!* However: AP−1 is P−1-Hermitian (Hermitian wrto P−1)
⇒ Cheap short-term recurrence(Simplified Lanczos - Freund & Nachtigal ’95)
Krylov subspace methods – p. 21
![Page 40: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/40.jpg)
Preconditioner Performance
P−1 =
(H B
BT 0
)−1
=
(I −B
O I
) (I O
O −(BTB)−1
) (I O
−BT I
)
( eH = I if prescaling used)
3D Magnetostatic problem. Number of iterations
size QMR QMR(Pdef ) QMR(P )1119 2368 40 152208 2825 36 134371 5191 43 178622 >10000 49 16
22675 >10000 81 25
Pdef = diag (I, BT B) hpd
In practice: BT B ≈ S Incomplete Cholesky fact. ⇒ bP
Krylov subspace methods – p. 22
![Page 41: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/41.jpg)
Preconditioner Performance
P−1 =
(H B
BT 0
)−1
=
(I −B
O I
) (I O
O −(BTB)−1
) (I O
−BT I
)
( eH = I if prescaling used)
3D Magnetostatic problem. Number of iterations
size QMR QMR(Pdef ) QMR(P )1119 2368 40 152208 2825 36 134371 5191 43 178622 >10000 49 16
22675 >10000 81 25
Pdef = diag (I, BT B) hpd
In practice: BT B ≈ S Incomplete Cholesky fact. ⇒ bP
Krylov subspace methods – p. 22
![Page 42: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/42.jpg)
Preconditioner Performance
P−1 =
(H B
BT 0
)−1
=
(I −B
O I
) (I O
O −(BTB)−1
) (I O
−BT I
)
( eH = I if prescaling used)
3D Magnetostatic problem. Number of iterations
size QMR QMR(Pdef ) QMR(P )1119 2368 40 152208 2825 36 134371 5191 43 178622 >10000 49 16
22675 >10000 81 25
Pdef = diag (I, BT B) hpd
In practice: BT B ≈ S Incomplete Cholesky fact. ⇒ bP
Krylov subspace methods – p. 22
![Page 43: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/43.jpg)
Outline
Application-driven practical issues:
Basic considerations on restarted methods
“Quasi-optimal” methods
Indefinite inner products
Inexact methods
Krylov subspace methods – p. 23
![Page 44: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/44.jpg)
Inexact methods
It is given an operator v → Aε(v).
Efficiently solve the given problem in the approximation space
Km = span{v,Aε1(v),Aε2(Aε1(v)), . . .}, v ∈ Cn
with dim(Km) = m, where Aε → A for ε→ 0 (ε may be tuned)
? for A = A, ε = 0⇒ Km = span{v, Av, A2v, . . . , Am−1v}
? Analysis also possible for eigenproblem
Krylov subspace methods – p. 24
![Page 45: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/45.jpg)
Inexact methods
It is given an operator v → Aε(v).
Efficiently solve the given problem in the approximation space
Km = span{v,Aε1(v),Aε2(Aε1(v)), . . .}, v ∈ Cn
with dim(Km) = m, where Aε → A for ε→ 0 (ε may be tuned)
? for A = A, ε = 0⇒ Km = span{v, Av, A2v, . . . , Am−1v}
? Analysis also possible for eigenproblem
Krylov subspace methods – p. 24
![Page 46: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/46.jpg)
Some typical situations
A(v) function (linear in v):
A result of a complex functional application
Schur complement: A = BT S−1B S expensive to invert
Flexible preconditioned system: AP−1x = b, where
P−1vi ≈ P−1i vi
etc.
In the eigenvalue context: shift-and-invert strategy
Ax = λMx A(v) = (A− σM)−1v
Krylov subspace methods – p. 25
![Page 47: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/47.jpg)
Some typical situations
A(v) function (linear in v):
A result of a complex functional application
Schur complement: A = BT S−1B S expensive to invert
Flexible preconditioned system: AP−1x = b, where
P−1vi ≈ P−1i vi
etc.
In the eigenvalue context: shift-and-invert strategy
Ax = λMx A(v) = (A− σM)−1v
Krylov subspace methods – p. 25
![Page 48: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/48.jpg)
Questions
? Do we need to have ε small to get good approximation?
good approximation: ‖rm‖ ≤ ε0 (fixed tolerance)
? Do we need to have ε fixed throughout?
? Do we still converge to a meaningful solution if ε varies?
? What happens to convergence rate when ε varies?
Krylov subspace methods – p. 26
![Page 49: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/49.jpg)
Questions
? Do we need to have ε small to get good approximation?
good approximation: ‖rm‖ ≤ ε0 (fixed tolerance)
? Do we need to have ε fixed throughout?
? Do we still converge to a meaningful solution if ε varies?
? What happens to convergence rate when ε varies?
Krylov subspace methods – p. 26
![Page 50: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/50.jpg)
Assuming A is exact...
Km Krylov subspace Vm = [v1, . . . , vm] orthogonal basis
Arnoldi relation:
AVm = VmHm + vm+1hm+1,meTm = Vm+1Hm
with v = Vme1‖v‖
Krylov subspace methods – p. 27
![Page 51: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/51.jpg)
Working with an inaccurate A
A = A → Aε(v) = Av + f
AVm = Vm+1Hm + Fm︸︷︷︸[f1,f2,...,fm]
Fm error matrix, ‖fj‖ = O(εj)
——————————
How large is Fm allowed to be?
xm = Vmym
rm = b−AVmym = b− Vm+1Hmym − Fmym
= Vm+1(e1β −Hmym)︸ ︷︷ ︸computed residual =:rm
−Fmym
where Fmym =
m∑
i=1
fi(ym)i
Krylov subspace methods – p. 28
![Page 52: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/52.jpg)
Working with an inaccurate A
A = A → Aε(v) = Av + f
AVm = Vm+1Hm + Fm︸︷︷︸[f1,f2,...,fm]
Fm error matrix, ‖fj‖ = O(εj)
——————————
How large is Fm allowed to be? xm = Vmym
rm = b−AVmym = b− Vm+1Hmym − Fmym
= Vm+1(e1β −Hmym)︸ ︷︷ ︸computed residual =:rm
−Fmym
where Fmym =
m∑
i=1
fi(ym)i
Krylov subspace methods – p. 28
![Page 53: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/53.jpg)
Relaxed methods
Fmym =
m∑
i=1
fi(ym)i
In fact, for several methods there exists `m such that
| (ym)i | ≤ `m‖ri−1‖
Therefore, ‖fi‖ is allowed to be large!
More precisely,
If ‖fi‖ ≤`m
m
1
‖ri−1‖ε i = 1, . . . , m
then ‖Fmym‖ ≤ ε ⇒ ‖rm − rm‖ ≤ ε
Bouras, Frayssè, Giraud, Simoncini, Szyld, Sleijpen, Van den Eshof, Gratton ...
Krylov subspace methods – p. 29
![Page 54: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/54.jpg)
Relaxed methods
Fmym =
m∑
i=1
fi(ym)i
In fact, for several methods there exists `m such that
| (ym)i | ≤ `m‖ri−1‖
Therefore, ‖fi‖ is allowed to be large!
More precisely,
If ‖fi‖ ≤`m
m
1
‖ri−1‖ε i = 1, . . . , m
then ‖Fmym‖ ≤ ε ⇒ ‖rm − rm‖ ≤ ε
Bouras, Frayssè, Giraud, Simoncini, Szyld, Sleijpen, Van den Eshof, Gratton ...
Krylov subspace methods – p. 29
![Page 55: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/55.jpg)
Numerical experiment: Schur complement
BT S−1B︸ ︷︷ ︸A
x = b at each it. i solve Swi = Bvi
Inexact FOM
δm = ‖rm − (b − Vm+1Hm
ym)‖
0 20 40 60 80 100 12010−8
10−7
10−6
10−5
10−4
10−3
10−2
10−1
100
101
number of iterations
magnitude
δm
||rm
||
||rm
||
εinner
~
Krylov subspace methods – p. 30
![Page 56: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/56.jpg)
Eigenproblem
Inverted Arnoldi: Ax = λx Find min |λ| y ← A(v) = A−1v
Matrix SHERMAN5
−200 −100 0 100 200 300 400 500 600−0.04
−0.03
−0.02
−0.01
0
0.01
0.02
0.03
0.04
real part
imagin
ary
part
0 2 4 6 8 10 1210−14
10−12
10−10
10−8
10−6
10−4
10−2
100
102
number of outer iterations
magnitude
ε
inner residual norm
exact res.
inexact residual
Krylov subspace methods – p. 31
![Page 57: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/57.jpg)
Structural Dynamics
(A+ σB)x = b
Solve for many σ’s simultaneously ⇒ (AB−1 + σI)x = b
(Perotti & Simoncini 2002)
Inexact solutions with B at each iteration:
Prec. Fill-in 5 Prec. Fill-in 10e-time [s] # outer its e-time [s] # outer its
Tol 10−6 14066 296 13344 289Dynamic Tol 11579 301 11365 293
20 % enhancement with tiny change in the code
(Preconditioned CG-type iteration for B)
Krylov subspace methods – p. 32
![Page 58: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/58.jpg)
Relaxed procedure
? A may be replaced by Aεiwith increasing εi and still converge
? Stable procedure for not too sensitive (e.g. non-normal) problems
Property inherent of Krylov approximation
⇓
Many more applications for this general setting
Krylov subspace methods – p. 33
![Page 59: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/59.jpg)
Conclusions
Often, enough confidence to tailor methods to problems
Ability to relax many of the classical requirements
Further enhancements are ahead!
Recent Survey:Recent computational developments in Krylov Subspace Methods for linear systemsSimoncini & Szyld, 200559 pp., 352 referencesto appear in Numer. Linear Algebra w/Appl.
http://www.dm.unibo.it/˜simoncin
Krylov subspace methods – p. 34
![Page 60: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/60.jpg)
Conclusions
Often, enough confidence to tailor methods to problems
Ability to relax many of the classical requirements
Further enhancements are ahead!
Recent Survey:Recent computational developments in Krylov Subspace Methods for linear systemsSimoncini & Szyld, 200559 pp., 352 referencesto appear in Numer. Linear Algebra w/Appl.
http://www.dm.unibo.it/˜simoncin
Krylov subspace methods – p. 34
![Page 61: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/61.jpg)
Conclusions
Often, enough confidence to tailor methods to problems
Ability to relax many of the classical requirements
Further enhancements are ahead!
Recent Survey:Recent computational developments in Krylov Subspace Methods for linear systemsSimoncini & Szyld, 200559 pp., 352 referencesto appear in Numer. Linear Algebra w/Appl.
http://www.dm.unibo.it/˜simoncin
Krylov subspace methods – p. 34
![Page 62: Krylov subspace methods: a versatile tool in Scientic Computing V. Simoncini …simoncin/iccam06.pdf · 2006. 7. 14. · De Sturler, Morgan, Sorensen, Baglama etal, Baker etal, ...](https://reader035.fdocuments.in/reader035/viewer/2022071501/612110d8fac5682eda074cec/html5/thumbnails/62.jpg)
Conclusions
Often, enough confidence to tailor methods to problems
Ability to relax many of the classical requirements
Further enhancements are ahead!
Recent Survey:Recent computational developments in Krylov Subspace Methods for linear systemsSimoncini & Szyld, 200559 pp., 352 referencesto appear in Numer. Linear Algebra w/Appl.
http://www.dm.unibo.it/˜simoncin
Krylov subspace methods – p. 34