Numerical Methods Rafał Zdunek Iterative...
65
Numerical Numerical Methods Methods Rafał Zdunek Rafał Zdunek Iterative Methods Iterative Methods (4h.) (4h.)
Transcript of Numerical Methods Rafał Zdunek Iterative...
NumericalNumerical
MethodsMethods
Rafał ZdunekRafał Zdunek
Iterative MethodsIterative Methods
(4h.)(4h.)
Introduction
• Stationary basic iterative methods, • Krylov subspace methods,• Nonnegative matrix factorization,• Multi-dimensional array decomposition
methods (tensor decompositions).
Bibliography[1] A. Bjorck, Numerical Methods for Least Squares Problems, SIAM,
Philadelphia, 1996, [2] G. Golub, C. F. Van Loan, Matrix Computations, The John Hopkins
University Press, (Third Edition), 1996, [3] J. Stoer
R. Bulirsch, Introduction to Numerical Analysis (Second Edition),
Springer-Verlag, 1993, [4] C. D. Meyer, Matrix Analysis and Applied Linear Algebra, SIAM, 2000,[5] Ch. Zarowski, An Introduction to Numerical Analysis for Electrical and
Computer Engineers, Wiley, 2004,[6] Cichocki, R. Zdunek, A. H. Phan, S.-I. Amari, Nonnegative Matrix and
Tensor Factorization: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation, Wiley and Sons, UK, 2009
DefinitionIterative linear solvers attempt to iteratively approximate a solution [ ] N
jx ℑ∈=*x to a system of linear equations:
bAx = ,
where [ ] NMija ×ℑ∈=A is a coefficient matrix and [ ] M
ib ℑ∈=b is a data vector, using following updates:
( )( 1) ( ) , ,k kf+ =x x A b ,
where ( )kx is an approximation to the solution x in the k-th iterative step, and ( ), ,f ⋅ ⋅ ⋅ is an
update function determined by an underlying iterative method, where ( )( ) *lim , ,k
kf
→∞→x A b x .
Stationary basic iterative methods
Let ,=Ax b ,n n×∈ℜAwhere (nonsingular) ,n∈ℜx n∈ℜb
,= −A S TAssume the splitting: thus 1 ,k k+ = +Sx Tx b (Basic Iterative Methods)
We have 1 ,k k+ = +x Gx c where 1 1 ,− −= = −G S T I S A 1−=c S b
Theorem: The iterations {xk } are convergent to x = A-1b for any starting guess x0 if and only if every eigenvalue λ
of S-1T satisfies | λ
| <1. Its convergence rate depends
on the maximum value of | λ
|, which is known as a spectral radius of S-1T:
( ) .max1iiλρ =− TS
.
.
Stationary basic iterative methods
Proof: Let ek =xk -x* denote an error in the k-th iteration. Since Sxk+1 = Txk +b, we have S(xk+1 – x*) = T(xk – x*) +b, and the error in xk+1 is given by ek+1 =S-1T ek
= (S-1T)k+1 e0 . If ρ(S-1T)<1, (S-1T)k → 0.
The matrix S can be regarded as a preconditioner. There are several choices for splitting A. For example:
1. Jacobi method: S = diagonal part of A;
2. Gauss-Seidel method: S = lower triangular part of A;
3. Sucessive Over-Relaxation (SOR) method: combination of 1 and 2.
Jacobi methodLet ,n n×∈ℜA
11
22
0 00
,0
0 0
n n
nn
aa
a
×
⎡ ⎤⎢ ⎥⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎣ ⎦
S
,= −A S T
12 1
21
( 1)
1 ( 1)
00
.
0
n
n n
n n
n n n
a aa
aa a
×
−
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
T
Thus ( )11 ,k k
−+ = +x S Tx b or ( 1) ( )1 , 1, 2, , .
nk k
i i ij jj iii
x b a x i na
+
≠
⎛ ⎞= − =⎜ ⎟
⎝ ⎠∑ …
Remark: if ∃i: aii = 0, a permutation of rows or columns is necessary.
Gauss-Seidel methodAssume
11
21 22
1 ( 1)
0 0
,0
n n
n n n nn
aa a
a a a
×
−
⎡ ⎤⎢ ⎥⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
S
12 1
( 1)
00 0
.
0 0 0
n
n n
n n
a a
a×
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎣ ⎦
T
Thus ( )11 ,k k
−+ = +x S Tx b or
1( 1) ( 1) ( )
1 1
1 . i n
k k ki i ij j ij j
j j iii
x b a x a xa
−+ +
= = +
⎛ ⎞= − −⎜ ⎟
⎝ ⎠∑ ∑
Remark: Note that the Gauss-Seidel method is a cyclic method and uses also the elements xi
(k+1) that have been already updated in the current iterative cycle.However, unlike the Jacobi method, the computation for each element cannot be done in parallel.
Successive Over-Relaxation (SOR)
Let ,n n×∈ℜA ,= + +A L D U
11
22
0 00
,0
0 0 nn
aa
a
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦
D
12 1
( 1)
00 0
.
0 0 0
n
n n
a a
a −
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦
U21
1 ( 1)
0 0 00
,00n n n
a
a a −
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
L
Assuming1 ,ω
= +S L D 1 .ωω−⎛ ⎞= − +⎜ ⎟
⎝ ⎠T U D ( )1
1 .k k−
+ = +x S Tx bUsing
we have ( ) ( )( )( )11 1 ,k kω ω ω ω−+ = + − + −x L D b U D x 0 < ω
< 2 - relaxation
parameter
Successive Over-Relaxation (SOR)
Finally( )
1( 1) ( ) ( 1) ( )
1 11 .
i nk k k k
i i i ij j ij jj j iii
x x b a x a xaωω
−+ +
= = +
⎛ ⎞= − + − −⎜ ⎟
⎝ ⎠∑ ∑
For ω
=1, the SOR simplifies to the Gauss-Seidel method.
If :T n n×= ∈ℜA A (symmetric) ,T=U L11 ,
2
T
ω ω ω ω
−⎛ ⎞⎛ ⎞ ⎛ ⎞= + +⎜ ⎟⎜ ⎟ ⎜ ⎟− ⎝ ⎠⎝ ⎠ ⎝ ⎠
D D DP L L
( )11 ,k k kγ −+ = − −x x P Ax b (SSOR method)
Stationary basic iterative methods for LS problems
Let ,m n×∈ℜA
,T = −A A S T
where ,m n≥
Splitting:
We have 1 ,k k+ = +x Gx c where 1 ,Tn
−= −G I S A A 1 T−=c S A b
and consider the normal equations of the first kind:
,T T=A Ax A b
In general: ( )1 ,k k k+ = + −x x B b Ax
The matrix B can take various forms, depending on the underlying method.
Landweber iterations,T = −A A S TLet 1 ,nα
=S I 1 ,Tnα
= −T I A A for the splitting
and is a relaxation parameter.0α >
We have 1 ,k k+ = +x Gx c where ,Tn α= −G I A A .Tα=c A b
Finally ( )1 ,Tk k kα+ = + −x x A b Ax where .Tα=B A
The Landweber iterations are also known as the Richardson’s first-order method.
If (range of AT), then the Landweber iterations are convergent to x = A+b if
( )0 ,TR∈x A
( ) 1
max0 2 .Tα λ−
< < A A
Jacobi method for LS problemsLet [ ]1 2, , , ,m n
n×= ∈ℜA a a a…
21 2
22 2
2
2
0 0
0,
0
0 0
n n
n
×
⎡ ⎤⎢ ⎥⎢ ⎥
= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a
aS
a
,T = −A A S T
1 2 1
2 1
1
1 1
00
.
0
T Tn
Tn n
Tn n
T Tn n n
×
−
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a a a aa a
Ta a
a a a a
From ( )11 ,k k k
−+ = + = +x S Tx b Gx c where
( )( 1) ( )2
2
, j 1, 2, , .Tjk k
j j k
j
x x n+ = + − =a
b Axa
…
1 ,Tn
−= −G I S A A 1 T−=c S A b
we have:
Jacobi method for LS problemsIn the parallel mode:
( ) ( )11 ,T
k k k k k−
+ = + − = + −x x B b Ax x S A b Ax where 1 T−=B S A
and assuming that all columns in A are non-zero, we have : 0.jjj s∀ >
The Jacobi method is symmetrizable since:
( )1/2 1/2 1/2 1/2.Tn
− − −− =S I G S S A AS
Gauss-Seidel method for LS problems
Assume 21 2
22 1 2 2
21 1 2
0 0
,0
Tn n
T Tn n n n
×
−
⎡ ⎤⎢ ⎥⎢ ⎥
= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a
a a aS
a a a a a
1 2 1
1
00 0
.
0 0 0
T Tn
n nTn n
×
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a a a a
Ta a
Thus ( )11 ,k k
−+ = +x S Tx b leads to
( )( )11 1 .T T
k k k k−
+ += + − + +x x S A b Tx S T x
Kaczmarz algorithm
1 2
2
, Ti i kk k i
i
bα+
−= +
a xx x aa
In the sequential mode the Kaczmarz algorithm (devised by a Polish mathematician: Stefan Kaczmarz, and published in 1937) is given by:
Let1
,m n
m
×
⎡ ⎤⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎣ ⎦
aA
awhere is the i-th row vector of A.1 n
i×∈ℜa
mod 1i k m= −
In image reconstruction, this algorithm is known as the unconstrained Algebraic Reconstruction Technique (ART).
0 2α< < - relaxation
Kaczmarz algorithmHyperplane defined by i-th equation:
The algorithm converges to:
1 2 mH H H∩ ∩ ∩…
Kaczmarz algorithm
For consistent case:
Limit point:
where is an initial guess and G is a generalized inverse of A.
( ) ( )( ) LSNkkNP xAGbxxxx A +=+==
∞→)(lim 00
*
0x
(K. Tanabe, Numerische Mathematik, 1971)
( )R∈b A
(nullspace) (minimal norm LS solution)
Kaczmarz algorithm
For inconsistent case:Limit point:
where
⇒
where:
( ) ( )( ) bGxxxx A~lim 00
* +==∞→ Nkk
P
,r δ= +b b b ,r nδ δ δ= +b b b ( ) ,r Rδ ∈b A ( ) ,Tn Nδ ∈b A
( ) ( )( )220
* ;, nrLSSd bGbGbGbAxx δδδ +==
( ) 2, .d = −u v u v
(C. Popa, R. Zdunek, Mathematics and Computers in Simulations, 2004)
( )R∉b A
Kaczmarz algorithm (example from image reconstruction)
Krylov subspace methods1k k kα+ = +x x r
k k= −r b Ax
The Richardson method: - solution update
- residual update
For α
= 1, we have ( )1 0 0 1 2 01
.k
ik k
i+
=
= + + + + + = − ⋅∑x x r r r r I A r…
Thus { } ( )11 0 0 0 0 0span , , , ; .k k
k K ++ − ∈ ⋅ ⋅ =x x r A r A r A r… - Krylov subspace
1
1 0 01
.k
ik i
iα
+
+=
= +∑r r A r
Assuming 1 1,k k+ += −r b Ax we have
11
1 0 01
,k
ik i
iα
+−
+=
= +∑x x A r
(Linear combinationcoefficients)
Krylov subspace methodsA family of nonparametric polynomial iterative methods based on the general formula:
( )1 1 0p ,k k+ += ⋅r A r
where ( ) 11p ++ Π∈ kk A (space of polynomials with (k+1) degrees ) and ( ) 10p 1 =+k
(Residual polynomial)
( )1 0 0q ,k k+ − = ⋅x x A r
(Iterative polynomial)
( ) ( )1p 1 q .k k+ = − ⋅A A A
Assuming * :k k= −ε x x ( ) ( )1 10 0p p .k k k k
− −= = =ε A r A A r A ε
Krylov subspace methodsThe classes of nonparametric polynomial iterative methods:
• Orthogonal Residual (OR) polynomials:
( ) ( ) 0, for p ,p
0, for OR ORk l
k lt t
k l> =⎧
= ⎨ ≠⎩where ( ) ( )p 0 p 0 1.OR OR
k l= =
Methods: CG, Lanczos, GENCG and FOM.
Minimal error criterion: ( ){ }* * 0 0min : , .OR kk K− = − ∈ +
AAx x x x x x A r
The Ritz-Galerkin condition: ( )0 , .OR kk K⊥r A r
• Minimal Residual (MR) polynomials: ( ) ( )p , p 0,MR MRk lt t t = .k l≠for
Minimal error criterion: ( ){ }0 0min : , .MR kk K− = − ∈ +b Ax b Ax x x A r
The Petrov-Galerkin condition: ( )0,rAAr kMRk K ⊥ Methods: CR, GMRES, MINRES
Krylov subspace methods• Minimal Error (ME) polynomials: ( ) ( ) 0p,p =ttME
k for ( ) 1p ,kt −∀ ∈Π
( )p 0 1,MEk = ( ) ( )p 0 1.ME
k
′=where
( ){ }* * 0 0min : , .ME kk K− = − ∈ +x x x x x x A A rMinimal error criterion:
These methods satisfy the Ritz-Galerkin condition: ( )0 , .ME kk K⊥r A r
• Minimal Residual (MR) polynomials for normal equations: 2 ,=A x Ab
( ) ( )2 2 2p , p 0MR MRk l =A A A for .k l≠
Minimal error criterion: ( ){ }20 0min : , .LSQR k
k K− = − ∈ +b Ax b Ax x x A A r
Methods: LSQR
Stieltjes algorithmThe OR polynomials satisfy the 3-term recurrences and can be computed by the Stieltjes algorithm:
( ) 0p 1 =− t( )0p 1,t =
( ) ( )( ) ( )( ) ( )( ) ( )
( )( ) ( ) ( ) ( )
1 1
1 1
1 2
2 2
1 2 1,2, ,
, p,
p , p
p ,p,
p , p
,
p p p ,
k kk
k k
k kk
k k
k k k
k k k k k k k K
t t tt t
t t tt t
t t t t
η
ξ
γ η ξ
γ η ξ
− −
− −
− −
− −
− − =
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪⎪ ⎪
=⎨ ⎬⎪ ⎪⎪ ⎪= − +⎪ ⎪⎪ ⎪= − −⎩ ⎭ …
( ) ( ) ( ){ }ttt kp,,p,p 10 …
Conjugate Gradients (CG) algorithm
Let be an additional polynomial for determining a search direction for ( )tk 1φ − ( )p .k t
The search direction is tangential in the point to the hyperplane spanned by
( ){ }tk k 1p,1 −−
( ) ( ) ( ) ( ){ }0 1 2 1 p , p , p , p .k kt t t t− −… Thus
( ) ( ) ( ) ( )1 21 1 1
p pφ p .k k
k k k k
t tt t
tξ − −
− − −
⎛ ⎞−= + ∈Π⎜ ⎟⎝ ⎠
( ) ( )( ) ( )( ) ( )( ) ( )
( ) ( ) ( )( ) ( ) ( )
1 1
1 1
1 1
1 2 2
1 2 1
1 1 1,2, ,
p , p1 ,φ ,φ
p ,p,
p ,p
φ φ p ,
p p φ ,
k kk
k k k
k kkk
k k k
k k k k
k k k k k K
t tt t t
t tt t
t t t
t t t t
αγ
ξβγ
β
α
− −
− −
− −
− − −
− − −
− − =
⎧ ⎫= =⎪ ⎪
⎪ ⎪⎪ ⎪⎪ ⎪
= =⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪= −⎩ ⎭ …
From the orthogonality of OR polynomials:
( )( ) ( )p p 0,T
l kt t = for kl ≠
( ) ( ) ( ) ( ){ }tttt kk 1101 p,,p,pspanφ −− ∈ …
( ) ( ) ( ){ }1 1φ span p , pk k kt t t t− −∈
Conjugate Gradients (CG) algorithm
1 1
2 2
1 2 1
1 1
1 1
1 1
1 1
1,2,
,
,
,
,,
Tk k
k Tk k
k k k kTk k
k Tk k
k k k k
k k k k
k
β
β
α
αα
− −
− −
− − −
− −
− −
− −
− −
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪
= −⎪ ⎪⎪ ⎪⎩ ⎭
r rr r
z z r
r rz Az
x x zr r Az
…
[ ]1, ,1 ,T n= ∈ℜe … 1 ,− =r e 0z =−1
- initial guess,0x 0 0= −r b Ax n n×∈ℜA- initial residual vector, where is a symmetric and positive-definite matrix.
Remark: ( )1 1p ,k kt− −r ( )1 1φk kt− −z
From the orthogonality of OR polynomials:
( )( ) ( )p p 0,T
l kt t = for we have:,l k≠ 0.Tl k =r r
From the t-orthogonality of the search direction polynomials:
for we have:,l k≠( )( ) ( )φ φ 0T
l kt t t = 0.Tl k =z Az
Conjugate Gradients (CG) algorithm
Theorem:
For any the sequence generated by the conjugate direction algorithm converges to the solution x* of the linear system in at most n iterations.
0n∈ℜx { }kx
Convergence rate: ( )( )
* *0
1,
1
k
k
κ
κ
⎛ ⎞−⎜ ⎟− ≤ −⎜ ⎟+⎝ ⎠
A A
Ax x x x
A
where is a condition number of A. ( )κ A
CG algorithm for normal equations
( )( )( )
( )
1
1 1
1
10,1,
,
,
,
,
,
Tk k
Tk k
k Tk k
k k k kTk k
k Tk k
k k k kk
β
β
α
α
−
− −
−
+=
⎧ ⎫= −⎪ ⎪⎪ ⎪
=⎪ ⎪⎪ ⎪⎪ ⎪= −⎨ ⎬⎪ ⎪⎪ ⎪=⎪ ⎪⎪ ⎪
= +⎪ ⎪⎩ ⎭
q A b Ax
Aq Aw
Aw Aww q w
w qAw Aw
x x w…
( )1 0 ,T− = −w A b Ax
- initial guess,0x
1 0,k k− =q w
0Tl k =w A Aw
kzThe search direction is replaced with the correction vector that satisfies the following orthogonality conditions:
kw
1.
2. for .l k≠
CGNR algorithmConjugate Gradient Normal Residual (CGNR) algorithm for normal equations:
( )
( )
1 1
2 2
1 2 1
1 1
1 1
1 1
1,2,
,
,
,
,
,
Tk k
k Tk k
k k k kTk k
k Tk k
k k k kT
k k
k
β
β
α
α
− −
− −
− − −
− −
− −
− −
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪= −⎪ ⎪⎩ ⎭
r rr r
z z r
r rAz Az
x x z
r A y Ax…
[ ]1, ,1 ,T n= ∈ℜe … 1 ,− =r e 0z =−1
- initial guess,0x ( )0 0 ,T= −r A b Ax - initial residual vector,
Preconditioning• Left preconditioning: 1 1 .− −=M Ax M b
• Right preconditioning: 1 ,− =AM y b 1 .−=x M y
In general, a good preconditioner M should meet the following requirements:• the preconditioned system should be easy to solve,• the inverse of the preconditioner should be cheap to construct and apply,• the condition number of the preconditioned matrix is considerably lower,• eigenvalues of of the preconditioned matrix are clustered,• the computational cost of calculating M-1 is much lower than for A-1.
Usually the incomplete Cholesky factorization or LU factorization are used to compute M from A.
PCG algorithm
11 1
1 1
2 2
1 2 1
1 1
1 1
1 1
1 1 1,2,
ˆ ,ˆ
,ˆ
ˆ ,ˆ
,
,,
k kTk k
k Tk k
k k k kTk k
k Tk k
k k k k
k k k k k
β
β
α
αα
−− −
− −
− −
− − −
− −
− −
− −
− − =
⎧ ⎫=⎪ ⎪⎪ ⎪=⎪ ⎪⎪ ⎪
= +⎪ ⎪⎨ ⎬⎪ ⎪=⎪ ⎪⎪ ⎪
= +⎪ ⎪⎪ ⎪= −⎩ ⎭
r M r
r rr r
z z r
r rz Az
x x zr r Az
…
( ){ }1 1* 0 0min : , .kK − −− ∈ +
Ax x x x M A M r
[ ]1, ,1 ,T n= ∈ℜe … 1 ,− =r e
1 ,− =z 0
- initial guess,0x 0 0= −r b Ax n n×∈ℜA- initial residual vector, where is a symmetric and positive-definite matrix.
M - preconditioner1ˆ .k k
−=r M rPreconditioning for the residual vector:
Minimal error criterion for the preconditioned vectors:
Lanczos algorithmThe Lanczos algorithm generates the orthogonal vectors {v1 , v2 , ..., vK } that span the Krylov subspace for the symmetrix matrix A and the residual vector r0 :
( ) { } { }10 0 0 0 1 2, span , , , span , , , .K K
KK −= =A r r Ar A r v v v… …
( )1 1
1 1
11
1 1, ,
, ,
,
,
.
k k k
k k n k k k
k k
kk
k k K
α
α β
β
β
+ −
+ +
++
+ =
⎧ ⎫=⎪ ⎪
= − −⎪ ⎪⎪ ⎪⎨ ⎬=⎪ ⎪⎪ ⎪=⎪ ⎪⎩ ⎭
Av v
v A I v v
vvv
…
0 ,=v 01 0 ,β = r 01
1
,β
=rv n n
n×∈ℜI
0 0= −r b AxAlgorithm:
Remark: Note that the Lanczos algorithm can be easily obtained from the 3-term recurrence Stieltjes algorithm by replacing t with A and the orthogonal polynomials
, ...., with the vectors:( )t0p ( )tkp 1, , .kv v…
Lanczos algorithmThe vectors satisfy the recurrence formula:
11
,KK K T
K Kβ++
⎡ ⎤= ⎢ ⎥⋅⎣ ⎦
JAV V
e
1 2
2 2
0 0
.0 0
0 0
K
K
K K
α ββ α
ββ α
⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
J
[ ]0 0 1 T KK = ∈ℜe …
[ ]KK vvvV ,,, 21 …=
where and the tridiagonal Jacobi matrix:
Exact solution can be obtained for K ≤
n.
Lanczos algorithmAssuming the Ritz-Galerkin condition: T
k k =V r 0 for k = 0, 1, ..., K.
Considering and the orthogonality of the Lanczos vectors, we have0
01 r
rv =
0 1Tk k =V Ax r e where [ ]1 1,0, ,0 .T k= ∈ℜe …
From the minimal error criterion for the Ritz-Galerkin condition and
( ) { }0 1 2span , , , ,k k− ∈x x v v v… we have 0 ,k k k= +x x V ζ
where ξk is a vector of the coefficients of a linear combination of the Lanczos vectors. For one obtains0 0,=x
0 1.k k =J ζ r e
Lanczos algorithm
( )1 1
1 1
11
1 1, ,
, ,
,
,
.
k k k
k k n k k k
k k
kk
k k K
α
α β
β
β
+ −
+ +
++
+ =
⎧ ⎫=⎪ ⎪
= − −⎪ ⎪⎪ ⎪⎨ ⎬=⎪ ⎪⎪ ⎪=⎪ ⎪⎩ ⎭
Av v
v A I v v
vvv
…
0 ,=v 01 0 ,β = r 01
1
,β
=rv n n
n×∈ℜI0 0= −r b Ax
Lanczos algorithm for the system Ax = b:
[ ]1, , ,K K=V v v…
1 1
1 2
1
1
0 00
00 0
KK
K K
α ββ α
ββ α
−
−
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦
J[ ] KT ℜ∈= 0,,0,11 …e1
0 1K K−=ζ J r e
0 K K= +x x V ζ
Arnoldi algorithmThe Arnoldi algorithm computes orthogonal vectors for the Krylov subspace for the non-symmetrix matrix A and the residual vector r0 with the modified Gram-Schmidt orthogonalization.
,
, 1, ,
1,
11, 1, ,
,
,,
,
.
k
Tn k n
n k n n k
k k
kk k k K
hh
h
h
=
+
++ =
=⎧ ⎫⎪ ⎪⎧ ⎫=⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪← −⎪ ⎪⎪ ⎪⎩ ⎭⎨ ⎬
=⎪ ⎪⎪ ⎪⎪ ⎪=⎪ ⎪⎩ ⎭
w Av
v ww w v
wwv
…
…
0
01 r
rv =
[ ]KK vvV ,,1 …=The orthogonal Arnoldi vectors:
satisfy the formula: 1 1,K K K K+ +=AV V H
11 12 13 1,
21 22 23 2,( 1)
32 331,
,
( 1),( 1) ( 1),
0
0 0
K
KK K
K K
K K
K K K K
h h h hh h h h
h hh
h h
+ ×+
+ − +
⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
H
where HK+1,K is an upper Hessenberg matrix:
GMRES algorithmIf A is a symmetric matrix, the Arnoldi algorithm simplifies to the Lanczos algorithm.
In GMRES algorithm, the Petrov-Galerkin condition is used, and from the Minimal Residual (MR) polynomial criterion, we have:
{ }02 2min min ,
KK K− = −
x ζb Ax r AV ζ
where 0 .K K= +x x V ζ
( )0 1 0 1 1, 0 1 1,2 2 2 22.K K K K K K K K K+ + +− = − = −r AV ζ V r e H ζ r e H ζ
From the relation 1 1, :K K K K+ +=AV V H
Finally0 1 1,2 2
min .K
K K K+−ζ
r e H ζ
GMRES algorithmIn GMRES, the (K+1)-th row of the upper Hessenberg matrix H(K+1),K is neglected, and the square matrix HK,K is transformed to the upper triangular one using the QR factorization with the Givens rotations:
, .K K K K=H Q R
Since the upper triangular system can be easily solved with
the Gausian eliminations.2
1,K =Q
GMRES algorithm1
01 1, ,
1 1 ,1
1k k
11 1
22
,,
,
, , , ,
, ,
If: : Elseif: : 11 , ,, ,11
Elseif:
Tn n n
kn n n n n k
Tnk n k k k k
k
k
kk k
k k k kk k k k
kk
hh
h rh
hh
h hc s cs c s
ρ
ρθ θρ
ρ ρθθ
θθ
−
− =
+ ++
+
+
+ +
⎧ ⎫=⎪ ⎪= ⎨ ⎬= −⎪ ⎪⎩ ⎭
= = = =
⎧ ⎧= =⎪ ⎪⎪ ⎪> ≤⎨ ⎨⎪ ⎪ = = −= = −⎪ ⎪ ++ ⎩⎩
v uu Av
u u v
uu v r Q h
…
{1
, , 1
, , , 1 ,
, 1 1, 1
1
1
,1
0,
1
0: 1, 0,,
, ,, ,
, ,
,If: : Elseif: :
,
k k k
k k k k k k k
k k k k k k k k k k
k k k k k k
k k k k k k
k
k n k nn
k k kk k
k k k k
h c sr r c h sq c q q s qq s q cf c s
rk m k mr
f
ϕ ϕ ϕ
+
+
+
+ + +
+
−
=
−
= = =
← −
← ←
← − ←
= =
⎧−⎪⎪ =< = = +⎨
⎪⎪ = +⎩
∑v ww x x V R
x x w
,k
⎧ ⎫⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎩ ⎭
f
Initialization:
0 0= −r b Ax
1 0 ,h = r1
01 hr
v =
1,1 1,q = 11 h=φ
(Saad and Schultz, 1986)
Bi-Conjugate GradientsFor a non-symmetrix matrix A, the 3-term Stieltjes recurrences are applied to generate 2 Krylov bases:
( ) { }0 0 1; , , ,kkK span −=A r r r…
( ) { }0 0 1; , , .k TkK span −=A r r r…
Tk k k=R R D
[ ]0 1, , .k k−=R r r…[ ]0 1, , ,k k−=R r r…
k kk
×∈ℜD
- the shifted Krylov base
Let
The bases are mutually orthogonal: - Petrov-Galerkin condition,
- diagonal matrix.
BiCG algorithm1 1
2 2
1 2 1
1 2 1
1 1
1 1
1 1
1 1
1 1
1,2,
,
,,
,
,,
,
Tk k
k Tk k
k k k k
k k k kTk k
k Tk k
k k k k
k k k kT
k k k k
k
β
ββ
α
αα
α
− −
− −
− − −
− − −
− −
− −
− −
− −
− −
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪
= +⎪ ⎪⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪
= −⎪ ⎪⎪ ⎪= −⎪ ⎪⎪ ⎪⎩ ⎭
r rr r
z z rz z r
r rz Az
x x zr r Az
r r A z
…
Initialization:(Fletcher, 1975)
0 0= −r b Ax
0 0 ,=r r [ ]1, ,1 ,T n= ∈ℜe …
1 ,− =r e 1 ,− =r e
1 ,− =z 0 0z =−1~
A – non-singular (n x n) matrix
If the denominator equals to zero, a serious breakdown occurs.
CGS algorithmIn the BiCG method, we have: ( ) 0p ,k k=r A r ( ) 0p .T
k k=r A r
From the orthogonality condition:
( ) ( ) ( ) ( ) ( )20 0 0 0 0 0, p , p , p p , p 0.T
k k k k k k k= = = =r r A r A r r A A r r A r
Equivalently, in the CGS (CG Square) method the residual vector is defined as follows:
( )20p .k k=r A r
Starting from the orthogonality condition:0 0 ,=r r 0 , 0.k =r r
CGS algorithm
( )
( )( )
0 1
0 2
1 1
1 1
0 1
0
1
1
1,2,
,
,,
,
,,
.
Tk
k Tk
k k k k
k k k k k k
Tk
k Tk
k k k k
k k k k k
k k k k k
k
β
β
β β
α
αα
α
−
−
− −
− −
−
−
−
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪
= + +⎪ ⎪⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= −⎪ ⎪
= + +⎪ ⎪⎪ ⎪
= − +⎪ ⎪⎪ ⎪⎩ ⎭
r rr r
u z rq u z q
r rr Aq
z u Aqx x u z
r r A u z
…
Initialization:
0 0 ,= −r b Ax [ ]1, ,1 ,T n= ∈ℜe …
1 ,− =r e 0 ,=z 0 0 0.=q r
A – non-singular (n x n) matrix,
(Sonneveld, 1989)
BiCGSTAB algorithmThe residual vector is defined as follows: ( ) ( ) 0q pk k k=r A A r
where ( ) ( )( ) ( )1 2q 1 1 1 .k kω ω ω= − − −A A A A…
( )( )( )
( )( )
0 11
0
1
0
0 1
1 1 1,2,
, , ,
, ,
,
.
TTk kk
k k k k k kT Tk k k
k k k k k k k k k k
Tk k
k Tkk
k k k k k k k
α α ω
α ω ω
αβω
β ω
−−
−
−
− − =
⎧ ⎫= = − =⎪ ⎪
⎪ ⎪⎪ ⎪
= + + = −⎪ ⎪⎨ ⎬⎛ ⎞⎛ ⎞⎪ ⎪⎜ ⎟= ⎜ ⎟⎪ ⎪⎜ ⎟⎝ ⎠⎝ ⎠⎪ ⎪⎪ ⎪= + −⎩ ⎭
As sr r s r AzAr Az As As
x x z s r s As
r rAr Ar
z r z Az…
Initialization:
0 0 ,= −r b Ax
0 0.=z r
A – non-singular (n x n) matrix,
If ωk = 0, a serious breakdown
(Van der Vorst, 1992)
QMR methodSimilarly to the GMRES algorithm, the Quasi-Minimal Residual (QMR) method uses the Petrov-Galerkin condition but the Minimal Residual (MR) polynomial criterion takes the form:
{ }0min mink
k k− = −x ζb Ax r AR ζ
where [ ]10 ,, −= kk rrR … contains the successive residual vectors , and
1 1, .k k k k+ +=AR R H
Thus the MR criterion can be simplified to 0 1 1,mink
k k k+−ζ
r e H ζ and
0 .k k= +x x R ζ
QMR method
( )
1 1
1 1
1 1
1 1 1 1
21 1
221 1
1
, ,
,
, ,
,
, ,
1, , ,1
k kk k
k k
Tk k
k T Tk k
k k k k k k k k k k
T Tk k
k Tk k
Tk k k k k k k k
k k k kk k k
k k k kk
k k k k k
α
α α
β
β β
ε γϑ γ ε
γ β β γϑ
ε ϑ γ
− −
− −
− −
− − − −
− −
− −
−
= =
=
= − = −
=
= − = −
= = = −+
= +
v wv wv w
w vq A Ap
p v w p q w v q
q A Apw v
v Ap v w A q w
v v
d p ( )2 21 1 1
1 1 1,2,
, ,, .
k k k k k k k
k k k k k k k
ε ϑ γ− − −
− − =
⎧ ⎫⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪= + = −⎩ ⎭
d s Ap sx x d r r s
…
Initialization:
0 0 ,= −r b AxA – non-singular (n x n) matrix,
(Freund and Nachtigal, 1991)
0 0 ,=v r 0 0 ,=w r
0 ,=p 0 0 ,=q 0 0 ,=d 0 0 .=s 0
Remark: numerically stable
LSQR method
{ }
1
11 1
1 1
11 1 1
1 Bi diagonalization
2 21k 1 1 1 1 2 1
QR factorization
1 1k
1 1
,
, ,
,
, .
, , ,
c , ,
k k k k
kk k k
k
k k k k
kk k k
k
k k k k k k k k
k kk
k k
r c r r r s
r sr r
β
αα
α
ββ
α β α
β
+
++ +
+ +
++ + +
+ −
− + −
+
= −⎧ ⎫⎪ ⎪⎪ ⎪= =⎪ ⎪⎨ ⎬= −⎪ ⎪⎪ ⎪
= =⎪ ⎪⎩ ⎭
= = + =
= =
v Au vvv v
u Av uuu u
( )Givens rotations
2 11 1
1
1 1 Updates 1, ,
, ,
, ,
k k kk k k k k k
k
k k k k k kk K
rc
rs s
η
η η
−− −
− −=
⎧ ⎫⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎧ ⎫⎪ ⎪⎨ ⎬⎪ ⎪⎩ ⎭⎪ ⎪⎧ ⎫⎪ ⎪− ⋅
= = + ⋅ ⋅⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎪ ⎪= − ⋅ = ⋅⎩ ⎭⎩ ⎭
v ww x x w
r r…
Initialization:
0 0 ,= −r b Ax
A – non-singular (m x n) matrix
1 0 ,=u r
1 1,=v Au 1 1 ,α = v
1 1,β = u11
1
,β
=uu
0 1,c = 0 0,s =
0 ,=w 0 0 1.η β=
(Paige and Saunders, 1982)
(normal equations),
Nonnegative Matrix Factorization (NMF)
Nonnegative matrix factorization (NMF) solves the following problem: estimate (up to scale and permutation ambiguity) such nonnegative matrices and that
given only the assigned lower-rank J, and possibly the prior knowledge on the estimated factors or noise distribution.
I J×+∈ℜA T J×
+∈ℜB
,I T×+∈ℜX
T≅X AB
J I T≤Usually
Rank-1 matrix factorization
1 1
J JT
j j j jj j= =
= + = +∑ ∑X a b V a b V
(Outer product)(Noise or error matrix)
Feature extraction
Published in Nature, 401, (1999): D.D. Lee from Bell Lab, H.S. Seung from MIT
(X)
(A)
(BT)
NMF for BSSy1 y2 y3
y4 y5 y6
y7 y8 y9
s1 s2
s3 s4
Original images Mixed images
Alternating minimization• First step (Update for B): • Second step (Update for A):
Iterative algorithm:
T =AB XT T=BA X
For 1, 2,t = … doInitialize randomly: (0) (0),A B
( )( )( ) ( 1) ( 1)
0arg min , ,
Tt t tD − −
≥←
BB X A B
( )( )( ) ( 1) ( )
0arg min , ,
Tt t tD −
≥←
AA X A B
End
( )|| TD X AB - objective function
Objective functions• Square Euclidean distance (square Frobenius norm):
• Generalized Kullback-Leibler divergence (I-divergence):
( )22
1 1
1 1( || ) || || [ ]2 2
I TT T T
F F it iti t
D x= =
= − = −∑∑X AB X AB AB
s t 0 0 .ij tja b i j t. . ≥ , ≥ ∀ , ,
( || ) log [ ][ ]
T TitKL it it itT
it it
xD x x⎛ ⎞⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠
= + −∑X AB ABAB
11
s t 0 0 || || 1I
jtj ij iji
b a a=
. . ≥ , ≥ , = = .∑a
Multiplicative algorithms• Euclidean distance ⇒ ISRA
• I-divergence ⇒ EMML
[ ] [ ],
[ ] [ ]
Tij tj
ij ij tj tjT Tij tj
a a b b← , ←XB X AAB B BA A
1 1
1 1
( [ ] ) ( [ ] )
I TT Tij it it tj it iti t
tj tj ij ijI Tqj pjq p
a x b xb b a a
a b= =
= =
/ /← , ← ,∑ ∑
∑ ∑AB AB
(Image Space Reconstruction Algorithm)
(Expectation-Maximization Maximum Likelihood)
ALS algorithmEuclidean distance ⇒ ALS (Alternating Least-Squares)
( ) ( )|| 0,T
T T TFD∇ = − ≡
BX AB A AB X ( ) ( )|| 0,T T
FD∇ = − ≡A
X AB AB X B
Stationary points:
ALS algorithm:
( ) ( )1,
TT T − +← =B X A A A A X
( ) ( )1,
TT T − +← =A XB B B X B
( ) ,T
P +Ω⎡ ⎤← ⎢ ⎥⎣ ⎦
B A X
( ) ,T
P +Ω⎡ ⎤← ⎢ ⎥⎣ ⎦
A X B
Projected ALS :
( )22
1 1
1 1( || ) || || [ ]2 2
I TT T T
F F it iti t
D x= =
= − = −∑∑X AB X AB AB
TensorsDefinition: Let
I1
, I2
, . . . , IN
∈ N denote index upper bounds. A tensor of
order N is an
N-way array where elements
are indexed
by
in
∈ {1, 2, . . . , In
} for 1 ≤ n ≤ N.
1 2 NI I I× × ×∈ℜX …1 2, , , ni i ix
Tensors are obviously generalizations of vectors and matrixes, for example, a third-order tensor (or 3-way array) has 3 modes (or indices or dimensions). A zero-order tensor is a scalar, a first-order tensor is a vector, a second-order tensor is a matrix, and tensors of order three and higher are called higher- order tensors.
Tensor fibers
Tensor slices
Unfolding
Mode-1 unfolding
Mode-2 unfolding Mode-3 unfolding
Unfolding
PARAFAC
1
J
j j jj=
= +∑X a b c V
PARAFAC
1 2 31
J
j j jj
I=
= + = × × × +∑X a b c V A B C V
Harshman’s PARAFAC
( )1
J
j j j jjλ
=
= +∑X a b c V
TUCKER decomposition
( )1 1 1
J R P
jrp j r pj r p
g= = =
= +∑∑∑X a b c V
