Genetics Using Punnett Squares Genetics Using Punnett Squares.
LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for...
Transcript of LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for...
![Page 1: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/1.jpg)
LSMR:An iterative algorithm forleast-squares problems
David Fong Michael Saunders
Institute for Computational and Mathematical Engineering (iCME)Stanford University
Copper Mountain Conference on Iterative MethodsCopper Mountain, Colorado Apr 5–9, 2010
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 1/31
![Page 2: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/2.jpg)
LSMR in one slide
solve Ax = b
min∥∥∥∥(AλI
)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
![Page 3: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/3.jpg)
LSMR in one slide
solve Ax = bmin
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
![Page 4: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/4.jpg)
LSMR in one slide
solve Ax = bmin
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
![Page 5: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/5.jpg)
LSMR in one slide
solve Ax = bmin
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
min ‖Ax− b‖2
LSQR ≡ CG on the normal equationLSMR ≡ MINRES on the normal equation
Almost same complexity as LSQRBetter convergence properties
for inexact solves
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 2/31
![Page 6: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/6.jpg)
LSQR
Iterative algorithm for
min
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
Properties
• A is rectangular (m× n) and often sparse• A can be an operator• CG on the normal equation (ATA+ λ2I)x = ATb
• Av, ATu plus O(m+ n) operations per iteration
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 3/31
![Page 7: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/7.jpg)
LSQR
Iterative algorithm for
min
∥∥∥∥(AλI)x−
(b0
)∥∥∥∥2
Properties
• A is rectangular (m× n) and often sparse• A can be an operator• CG on the normal equation (ATA+ λ2I)x = ATb
• Av, ATu plus O(m+ n) operations per iteration
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 3/31
![Page 8: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/8.jpg)
Monotone convergence of residualMeasure of Convergence• rk = b−Axk• ‖rk‖ → ‖r̂‖, ‖ATrk‖ → 0
— LSQR— LSMR
LSQR ‖rk‖
0 50 100 150 200 250 300 350 400 450 5001.69
1.695
1.7
1.705
1.71
1.715
1.72
1.725
1.73
1.735Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
||r||
LSQR log ‖ATrk‖
0 50 100 150 200 250 300 350 400 450 500−5
−4
−3
−2
−1
0
1
2Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
log|
|ATr|
|
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 4/31
![Page 9: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/9.jpg)
Monotone convergence of residualMeasure of Convergence• rk = b−Axk• ‖rk‖ → ‖r̂‖, ‖ATrk‖ → 0
— LSQR— LSMR
LSQR ‖rk‖
0 50 100 150 200 250 300 350 400 450 5001.69
1.695
1.7
1.705
1.71
1.715
1.72
1.725
1.73
1.735Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
||r||
LSQR log ‖ATrk‖
0 50 100 150 200 250 300 350 400 450 500−5
−4
−3
−2
−1
0
1
2Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
iteration count
log|
|ATr|
|
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 4/31
![Page 10: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/10.jpg)
Monotone convergence of residualMeasure of Convergence• rk = b−Axk• ‖rk‖ → ‖r̂‖, ‖ATrk‖ → 0
— LSQR— LSMR
‖rk‖
0 50 100 150 200 250 300 350 400 450 5001.69
1.695
1.7
1.705
1.71
1.715
1.72
1.725
1.73
1.735
iteration count
||r||
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
lsqrlsmr
log ‖ATrk‖
0 50 100 150 200 250 300 350 400 450 500−5
−4
−3
−2
−1
0
1
2
iteration count
log|
|ATr|
|
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
lsqrlsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 4/31
![Page 11: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/11.jpg)
LSMR Algorithm
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 5/31
![Page 12: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/12.jpg)
Golub-Kahan bidiagonalization
Given A (m× n) and b (m× 1)
Direct bidiagonalization
UTAV = B
Iterative bidiagonalization
1 β1u1 = b, α1v1 = ATu1
2 for k = 1, 2, . . . , setβk+1uk+1 = Avk − αkuk
αk+1vk+1 = ATuk+1 − βk+1vk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 6/31
![Page 13: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/13.jpg)
Golub-Kahan bidiagonalization
Given A (m× n) and b (m× 1)
Direct bidiagonalization
UTAV = B
Iterative bidiagonalization
1 β1u1 = b, α1v1 = ATu1
2 for k = 1, 2, . . . , setβk+1uk+1 = Avk − αkuk
αk+1vk+1 = ATuk+1 − βk+1vk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 6/31
![Page 14: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/14.jpg)
Golub-Kahan bidiagonalization (2)
The process can be summarized by
AVk = Uk+1Bk
ATUk+1 = Vk+1LTk+1
where
Lk =
α1
β2 α2
. . . . . .βk αk
, Bk =
α1
β2 α2
. . . . . .βk αk
βk+1
=
(Lk
βk+1eTk
)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 7/31
![Page 15: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/15.jpg)
Golub-Kahan bidiagonalization (3)
Vk spans the Krylov subspace:
span{v1, . . . , vk} = span{ATb, (ATA)ATb, . . . , (ATA)k−1ATb}
Define xk = Vkyk
Subproblem to solve
minyk‖rk‖ = min
yk‖β1e1 −Bkyk‖ (LSQR)
minyk‖AT rk‖ = min
yk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥ (LSMR)
where rk = b−Axk, β̄k = αkβk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 8/31
![Page 16: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/16.jpg)
Golub-Kahan bidiagonalization (3)
Vk spans the Krylov subspace:
span{v1, . . . , vk} = span{ATb, (ATA)ATb, . . . , (ATA)k−1ATb}
Define xk = Vkyk
Subproblem to solve
minyk‖rk‖ = min
yk‖β1e1 −Bkyk‖ (LSQR)
minyk‖AT rk‖ = min
yk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥ (LSMR)
where rk = b−Axk, β̄k = αkβk
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 8/31
![Page 17: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/17.jpg)
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥
= minyk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
![Page 18: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/18.jpg)
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
![Page 19: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/19.jpg)
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
![Page 20: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/20.jpg)
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
![Page 21: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/21.jpg)
Least squares subproblem
minyk
∥∥∥∥∥β̄1e1 −(BT
k Bk
β̄k+1eTk
)yk
∥∥∥∥∥= min
yk
∥∥∥∥∥β̄1e1 −(RT
kRk
qTk Rk
)yk
∥∥∥∥∥ Qk+1Bk =
(Rk
0
), RT
k qk = β̄k+1ek
= mintk
∥∥∥∥∥β̄1e1 −(RT
k
ϕkeTk
)tk
∥∥∥∥∥ tk = Rkyk, qk = (β̄k+1/(Rk)k,k)ek = ϕkek
= mintk
∥∥∥∥( zkζ̄k+1
)−(R̄k
0
)tk
∥∥∥∥ Q̄k+1
(RT
k β̄1e1
ϕkeTk 0
)=
(R̄k zk
0 ζ̃k+1
)
Things to notexk = Vkyk, tk = Rkyk, zk = R̄ktk, two cheap QRs
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 9/31
![Page 22: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/22.jpg)
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
![Page 23: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/23.jpg)
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
![Page 24: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/24.jpg)
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
![Page 25: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/25.jpg)
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
![Page 26: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/26.jpg)
Least squares subproblem (2)
Remember xk = Vkyk, tk = Rkyk, zk = R̄ktk
Key steps to compute xkxk = Vkyk
= Wktk RTkW
Tk = V T
k
= W̄kzk R̄Tk W̄
Tk = W T
k
= xk−1 + ζkw̄k
where zk =(ζ1 ζ2 · · · ζk
)T
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 10/31
![Page 27: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/27.jpg)
Flow chart of LSMR
Computational cost
Av,ATv, 3m+ 3n
O(1)
3n
O(1)
A,�b
Golub-Kahanbidiagonalization
QR decompositions
Estimate backward error
large
Solution x
small
Update x
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 11/31
![Page 28: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/28.jpg)
Flow chart of LSMRComputational cost
Av,ATv, 3m+ 3n
O(1)
3n
O(1)
A,�b
Golub-Kahanbidiagonalization
QR decompositions
Estimate backward error
large
Solution x
small
Update x
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 11/31
![Page 29: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/29.jpg)
Computational and storage requirement
Storage Workm n m n
MINRES on ATAx = ATb Av1 x, v1, v2, w1, w2 8LSQR Av, u x, v, w 3 5LSMR Av, u x, v, h, h̄ 3 6
where hk, h̄k are scalar multiples of wk, w̄k
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 12/31
![Page 30: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/30.jpg)
Numerical experiments
Test Data• University of Florida Sparse Matrix Collection• LPnetlib: Linear Programming Problems• A = (Problem.A)’ b = Problem.c (127 problems)
Solve min ‖Ax− b‖2with LSQR and LSMR
• Backward error tests: nnz(A) ≤ 63220
• Reorthogonalization: nnz(A) ≤ 15977
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 13/31
![Page 31: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/31.jpg)
Numerical experiments
Test Data• University of Florida Sparse Matrix Collection• LPnetlib: Linear Programming Problems• A = (Problem.A)’ b = Problem.c (127 problems)
Solve min ‖Ax− b‖2with LSQR and LSMR
• Backward error tests: nnz(A) ≤ 63220
• Reorthogonalization: nnz(A) ≤ 15977
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 13/31
![Page 32: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/32.jpg)
Backward errors - optimalOptimal backward error
µ(x) ≡ minE‖E‖ s.t. (A+ E)T (A+ E)x = (A+ E)T b
Higham 1995:
µ(x) = σmin(C), C ≡[A ‖r‖
‖x‖
(I − rrT
‖r‖2
)].
Cheaper estimate (Grcar, Saunders, & Su 2007):
K =
(A‖r‖‖x‖
)v =
(r0
)y = argmin
y‖Ky − v‖
µ̃(x) =‖Ky‖‖x‖
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 14/31
![Page 33: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/33.jpg)
Backward errors - optimalOptimal backward error
µ(x) ≡ minE‖E‖ s.t. (A+ E)T (A+ E)x = (A+ E)T b
Higham 1995:
µ(x) = σmin(C), C ≡[A ‖r‖
‖x‖
(I − rrT
‖r‖2
)].
Cheaper estimate (Grcar, Saunders, & Su 2007):
K =
(A‖r‖‖x‖
)v =
(r0
)y = argmin
y‖Ky − v‖
µ̃(x) =‖Ky‖‖x‖
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 14/31
![Page 34: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/34.jpg)
Backward error estimates - E1, E2
Again, (A+ Ei)T (A+ Ei)x = (A+ Ei)
T b
Two estimates given by Stewart (1975 and 1977):
E1 =exT
‖x‖2, ‖E1‖ =
‖e‖‖x‖
, e = r̂ − r
E2 = −rrTA
‖r‖2, ‖E2‖ =
‖ATr‖‖r‖
where r̂ is the residual for the exact solution
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 15/31
![Page 35: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/35.jpg)
‖rk‖ for LSQR and LSMR
0 50 100 150 200 250 3002.75
2.8
2.85
2.9
2.95
3
3.05
3.1
3.15
iteration count
||r||
Name:lp greenbeb, Dim:5598x2392, nnz:31070, id=100
lsqrlsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 16/31
![Page 36: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/36.jpg)
log10(‖E2‖) for LSQR and LSMR
0 200 400 600 800 1000 1200 1400 1600 1800−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
E2)
Name:lp pilot ja, Dim:2267x940, nnz:14977, id=88
E2 LSQRE2 LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 17/31
![Page 37: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/37.jpg)
Backward errors for LSQR
0 200 400 600 800 1000 1200 1400 1600 1800−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
Bac
kwar
d E
rror
)
Name:lp cre a, Dim:7248x3516, nnz:18168, id=93
E1 LSQRE2 LSQROptimal LSQR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 18/31
![Page 38: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/38.jpg)
Backward errors for LSMR
0 200 400 600 800 1000 1200 1400 1600 1800−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
Bac
kwar
d E
rror
)
Name:lp cre a, Dim:7248x3516, nnz:18168, id=93
E1 LSMRE2 LSMROptimal LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 19/31
![Page 39: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/39.jpg)
Optimal backward errors for LSQR and LSMR
0 100 200 300 400 500 600−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
||ATr|
|/||r
||)
Name:lp pilot, Dim:4860x1441, nnz:44375, id=107
Optimal LSQROptimal LSMR
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 20/31
![Page 40: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/40.jpg)
Space-time trade-offs
LSMR is well-suited for limited memory computations.
What if we have• more memory• Av expensive
Can we speed things up?
Some ideas:• Reorthogonalization• Restarting• Local reorthogonalization
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 21/31
![Page 41: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/41.jpg)
Space-time trade-offs
LSMR is well-suited for limited memory computations.
What if we have• more memory• Av expensive
Can we speed things up?
Some ideas:• Reorthogonalization• Restarting• Local reorthogonalization
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 21/31
![Page 42: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/42.jpg)
Reorthogonalization
Golub-Kahan processInfinite precision Finite precisionUk, Vk orthonormal Lose orthogonalityAt most min(m,n) iterations Could take 10n or more
Apply modified Gram-Schmidt to uk+1 and/or vk+1:
u← u− (uTj u)uj j = k, k−1, k−2, . . .
(similarly for v)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 22/31
![Page 43: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/43.jpg)
Reorthogonalization
Golub-Kahan processInfinite precision Finite precisionUk, Vk orthonormal Lose orthogonalityAt most min(m,n) iterations Could take 10n or more
Apply modified Gram-Schmidt to uk+1 and/or vk+1:
u← u− (uTj u)uj j = k, k−1, k−2, . . .
(similarly for v)
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 22/31
![Page 44: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/44.jpg)
Effects of reorthogonalization on various problems
0 10 20 30 40 50 60 70 80 90−8
−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp ship12l, Dim:5533x1151, nnz:16276, id=91
NoOrthoOrthoUOrthoVOrthoUV
0 100 200 300 400 500 600 700 800 900−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp scfxm2, Dim:1200x660, nnz:5469, id=62
NoOrthoOrthoUOrthoVOrthoUV
0 20 40 60 80 100 120 140 160 180−6
−5
−4
−3
−2
−1
0
1
iteration count
log(
E2)
Name:lp agg2, Dim:758x516, nnz:4740, id=58
NoOrthoOrthoUOrthoVOrthoUV
0 2000 4000 6000 8000 10000 12000−7
−6
−5
−4
−3
−2
−1
0
iteration count
log(
E2)
Name:lpi gran, Dim:2525x2658, nnz:20111, id=94
NoOrthoOrthoUOrthoVOrthoUV
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 23/31
![Page 45: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/45.jpg)
Orthogonality of Uk
100
200
300
400
500
600
5
6
Original
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on V
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U ,V
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 24/31
![Page 46: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/46.jpg)
Orthogonality of Vk
100
200
300
400
500
600
5
6
Original
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on V
100
200
300
400
500
600
5
6
7
8
910
11
1213
141516
MGS on U ,V
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 25/31
![Page 47: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/47.jpg)
What we learnt so far
• Reorthogonalizing Vk (only) is sufficient• Reorthogonalizing Uk (only) is nearly as good• xk converges the same for all options
What can be improved• May still use too much memory• Need more flexibility for space-time trade-off
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 26/31
![Page 48: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/48.jpg)
Reorthogonalization with Restarting
Restarting LSMR
rk = b−Axk min ‖A∆x− rk‖
Restarting leads to stagnation
0 500 1000 1500 2000 2500 3000 3500 4000 4500−7
−6
−5
−4
−3
−2
−1
0
iteration count
Bac
kwar
d E
rror
Name:lp maros, Dim:1966x846, nnz:10137, id=81
NoOrthoRestart5Restart10Restart50NoRestart
0 2000 4000 6000 8000 10000 12000 14000 16000−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
Bac
kwar
d E
rror
Name:lp cre c, Dim:6411x3068, nnz:15977, id=90
NoOrthoRestart5Restart10Restart50NoRestart
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 27/31
![Page 49: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/49.jpg)
Reorthogonalization with Restarting
Restarting LSMR
rk = b−Axk min ‖A∆x− rk‖
Restarting leads to stagnation
0 500 1000 1500 2000 2500 3000 3500 4000 4500−7
−6
−5
−4
−3
−2
−1
0
iteration count
Bac
kwar
d E
rror
Name:lp maros, Dim:1966x846, nnz:10137, id=81
NoOrthoRestart5Restart10Restart50NoRestart
0 2000 4000 6000 8000 10000 12000 14000 16000−7
−6
−5
−4
−3
−2
−1
0
1
iteration count
Bac
kwar
d E
rror
Name:lp cre c, Dim:6411x3068, nnz:15977, id=90
NoOrthoRestart5Restart10Restart50NoRestart
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 27/31
![Page 50: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/50.jpg)
Local reorthogonalization• Reorthogonalize wrto only the last l vectors• Partial speed-up• Less memory• Depends on efficiency of Av and ATu
Speed up with local reorthogonalization
0 50 100 150 200 250 300 350 400 450−7
−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
−2
iteration count
Bac
kwar
d E
rror
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
NoOrthoLocal5Local10Local50FullOrtho
0 200 400 600 800 1000 1200 1400−6
−5
−4
−3
−2
−1
0
1
2
iteration count
Bac
kwar
d E
rror
Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89
NoOrthoLocal5Local10Local50FullOrtho
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 28/31
![Page 51: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/51.jpg)
Local reorthogonalization• Reorthogonalize wrto only the last l vectors• Partial speed-up• Less memory• Depends on efficiency of Av and ATu
Speed up with local reorthogonalization
0 50 100 150 200 250 300 350 400 450−7
−6.5
−6
−5.5
−5
−4.5
−4
−3.5
−3
−2.5
−2
iteration count
Bac
kwar
d E
rror
Name:lp fit1p, Dim:1677x627, nnz:9868, id=80
NoOrthoLocal5Local10Local50FullOrtho
0 200 400 600 800 1000 1200 1400−6
−5
−4
−3
−2
−1
0
1
2
iteration count
Bac
kwar
d E
rror
Name:lp bnl2, Dim:4486x2324, nnz:14996, id=89
NoOrthoLocal5Local10Local50FullOrtho
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 28/31
![Page 52: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/52.jpg)
Conclusions
Advantages of LSMR• All the good properties of LSQR• Monotone convergence of ‖ATrk‖• ‖rk‖ still monotonic• Cheap near-optimal backward error estimate⇒ reliable stopping rule
• Backward error almost surely monotonic
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 29/31
![Page 53: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/53.jpg)
Acknowledgement
• Michael Saunders• Chris Paige• Stanford Graduate Fellowship
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 30/31
![Page 54: LSMR: An iterative algorithm for least-squares problemsLSMR: An iterative algorithm for least-squares problems David Fong Michael Saunders Institute for Computational and Mathematical](https://reader030.fdocuments.in/reader030/viewer/2022040516/5e749031468fdb22e748eab0/html5/thumbnails/54.jpg)
Questions
LSMR in MATLAB andslides for this talk are
downloadable athttp://zi.ma/lsmr
David Fong, Michael Saunders LSMR Algorithm http://zi.ma/lsmr 31/31