Model-assisted effective large scale

matrix computations

V. Simoncini

Dipartimento di Matematica, Universita di Bologna

valeria.simoncini@unibo.it

1

An intuitive relation

−1−0.5

00.5

1

−1

−0.5

0

0.5

10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

−∆u = f, (x, y) ∈ (−1, 1)2

u = 0 on ∂[−1, 1]2

GivenA ∈ Rn×n, f ∈ R

n

Findu ∈ Rn s.t. Au = f

2

A slow start

Solution of a (square) algebraic linear system of large dimension.

Find u ∈ Rn such that

Au = f

with A symmetric and positive definite: 〈x,Ax〉 > 0 ∀x 6= 0

Very large dimension ⇒ iterative procedure

Given an initial guess u0 ∈ Rn (u0 = 0 in the following), generate

sequence

u1,u2,u3, . . . ,um, . . . → u

3

“Projection” methods (or, reduction methods)

• Approximation vector space Km. At each iteration m

um such that um ∈ Km

Km: dimensiona m, with the “expansion” property:

Km ⊆ Km+1

• Computation of iterate. Galerkin condition:

residual rm := f −Aum ⊥ Km

⇒ This condition uniquely defines um ∈ Km

(Conjugate Gradients, Hestenes & Stiefel, ’52)

aAt most

4

“Projection” methods (or, reduction methods)

• Approximation vector space Km. At each iteration m

um such that um ∈ Km

Km: dimensiona m, with the “expansion” property:

Km ⊆ Km+1

• Computation of iterate. Galerkin condition:

residual rm := f −Aum ⊥ Km

⇒ This condition uniquely defines um ∈ Km

(Conjugate Gradients, Hestenes & Stiefel, ’52)

aAt most

5

Optimality property of Galerkin projection method

Let u⋆ be the true solution. Galerkin property:

residual rm := f −Aum ⊥ Km

is equivalent to:

um solution to minu∈Km

‖u⋆ − u‖A

where ‖ · ‖A is the energy norm, namely ‖x‖2A := 〈x,Ax〉

Note: ‖u⋆ − u‖A cannot be computed!

But, it can be easily estimated - particularly well if convergence is fast

(Strakos, Tichy, Meurant, Golub, Arioli, ...)

⇒ Use estimate of ‖u⋆ − um‖A to monitor when to stop iterating

6

Optimality property of Galerkin projection method

Let u⋆ be the true solution. Galerkin property:

residual rm := f −Aum ⊥ Km

is equivalent to:

um solution to minu∈Km

‖u⋆ − u‖A

where ‖ · ‖A is the energy norm, namely ‖x‖2A := 〈x,Ax〉

Note: ‖u⋆ − u‖A cannot be computed!

But, it can be easily estimated - particularly well if convergence is fast

(Strakos, Tichy, Meurant, Golub, Arioli, ...)

⇒ Use estimate of ‖u⋆ − um‖A to monitor when to stop iterating

7

Optimality property of Galerkin projection method

Let u⋆ be the true solution. Galerkin property:

residual rm := f −Aum ⊥ Km

is equivalent to:

um solution to minu∈Km

‖u⋆ − u‖A

where ‖ · ‖A is the energy norm, namely ‖x‖2A := 〈x,Ax〉

Note: ‖u⋆ − u‖A cannot be computed!

But, it can be easily estimated - particularly well if convergence is fast

(Strakos, Tichy, Meurant, Golub, Arioli, ...)

⇒ Use estimate of ‖u⋆ − um‖A as stopping criterion

8

A typical example

0 20 40 60 80 100 120 140 160 18010

−6

10−5

10−4

10−3

10−2

10−1

100

number of iterations

no

rm

residual 2−norm

energy norm of error

9

Convergence and spectral properties

• In exact arithmetic (i.e., in theory), finite termination property

• A-priori bound for energy norm of the error:

If Km = spanf ,Af , . . . ,Am−1f, then

‖u⋆ − um‖A ≤ 2

(√κ− 1√κ+ 1

)m

‖u⋆ − u0‖A

where κ = λmax(A)λmin(A)

Consequences:

• Convergence: The closer κ to 1 the faster

• Convergence depends on spectral properties, not directly on

problem size!

10

Convergence and spectral properties

• In exact arithmetic (i.e., in theory), finite termination property

• A-priori bound for energy norm of the error:

If Km = spanf ,Af , . . . ,Am−1f, then

‖u⋆ − um‖A ≤ 2

(√κ− 1√κ+ 1

)m

‖u⋆ − u0‖A

where κ = λmax(A)λmin(A)

Consequences:

• Convergence: The closer κ to 1 the faster

• Convergence depends on spectral properties, not directly on

problem size!

11

Exploring the intuitive relation. −∆u = f

V space of continuous solns, Vh discrete space of approximate solns

Vh ≈ V, Vh e.g., space of piecewise low degree polynomials

−1.5 −1 −0.5 0 0.5 1 1.5−1.5

−1

−0.5

0

0.5

1

1.5square domain Ω

⇒ −1.5 −1 −0.5 0 0.5 1 1.5−1.5

−1

−0.5

0

0.5

1

1.5Q1 finite element subdivision

12

Exploring the intuitive relation. −∆u = f

V space of continuous solns, Vh discrete space of approximate solns

Continuous weak formulation

Findu ∈ V s.t. (∇u,∇v) = (f, v), ∀v ∈ V0 (x, y) =∫Ω xy

Discrete weak formulation

Finduh ∈ Vh s.t. (∇uh,∇vh) = (f, vh), ∀vh ∈ Vh,0 ⇔ Au = f

φk basis for Vh, then uh =∑

φku(k)

Error minimization property

minwh∈Vh

‖∇u−∇wh‖ ⇔ minw∈Km

‖u−w‖A

So what?

13

Exploring the intuitive relation. −∆u = f

V space of continuous solns, Vh discrete space of approximate solns

Continuous weak formulation

Findu ∈ V s.t. (∇u,∇v) = (f, v), ∀v ∈ V0 (x, y) =∫Ω xy

Discrete weak formulation

Finduh ∈ Vh s.t. (∇uh,∇vh) = (f, vh), ∀vh ∈ Vh,0 ⇔ Au = f

φk basis for Vh, then uh =∑

φku(k)

Error minimization property

minwh∈Vh

‖∇u−∇wh‖ ⇔ minw∈Km

‖u−w‖A

So what?

14

Exploring the intuitive relation. −∆u = f

V space of continuous solns, Vh discrete space of approximate solns

Continuous weak formulation

Findu ∈ V s.t. (∇u,∇v) = (f, v), ∀v ∈ V0 (x, y) =∫Ω xy

Discrete weak formulation

Finduh ∈ Vh s.t. (∇uh,∇vh) = (f, vh), ∀vh ∈ Vh,0 ⇔ Au = f

φk basis for Vh, then uh =∑

φku(k)

Error minimization property

minwh∈Vh

‖∇u−∇wh‖ ⇔ minw∈Km

‖u−w‖A

So what?

15

Exploring the intuitive relation. −∆u = f

V space of continuous solns, Vh discrete space of approximate solns

Continuous weak formulation

Findu ∈ V s.t. (∇u,∇v) = (f, v), ∀v ∈ V0 (x, y) =∫Ω xy

Discrete weak formulation

Finduh ∈ Vh s.t. (∇uh,∇vh) = (f, vh), ∀vh ∈ Vh,0 ⇔ Au = f

φk basis for Vh, then uh =∑

φku(k)

Error minimization property

minwh∈Vh

‖∇u−∇wh‖ ⇔ minw∈Km

‖u−w‖A

So what?

16

From intuition to practice

1. Spectral properties of A depend on problem

2. The error energy norms (functional and algebraic) are related

————————–

1. Spectral properties of A depend on problem. Crucial in

‖u⋆ − um‖A ≤(√

κ− 1√κ+ 1

)m

‖u⋆ − u0‖A

and κ = O( 1h2 )

h κ n # iterations

2−1 3.33 25 3

2−3 51.71 289 24

2−5 829.86 4225 101

17

From intuition to practice

1. Spectral properties of A depend on problem

2. The error energy norms (functional and algebraic) are related

————————–

1. Spectral properties of A depend on problem. Crucial in

‖u⋆ − um‖A ≤(√

κ− 1√κ+ 1

)m

‖u⋆ − u0‖A

and κ = O( 1h2 )

h κ n # iterations

2−1 3.33 25 3

2−3 51.71 289 24

2−5 829.86 4225 101

18

Model-based algebraic acceleration strategies

Spectrally equivalent matrices: A ∼ A if ∃ positive c1, c2 such that

c1〈x, Ax〉 ≤ 〈x,Ax〉 ≤ c2〈x, Ax〉 ∀x ∈ Rn

... usually, c1, c2 do not depend on h

A Realization: Geometric/Algebraic Multigrid

(Brandt, Bramble, Ruge, Stuben, Hackbusch, Trottenberg, Briggs, Henson, McCormick, ...)

Formally, if A = LLT and M = L−1AL−T , then κ(M) = O( c2c1) and

Au = f ⇒ Mu = f , u = LT u, f = Lf

h n # iterations

2−3 289 7

2−5 4225 8

2−8 66049 9

19

Model-based algebraic acceleration strategies

Spectrally equivalent matrices: A ∼ A if ∃ positive c1, c2 such that

c1〈x, Ax〉 ≤ 〈x,Ax〉 ≤ c2〈x, Ax〉 ∀x ∈ Rn

... usually, c1, c2 do not depend on h

A Realization: Geometric/Algebraic Multigrid

(Brandt, Bramble, Ruge, Stuben, Hackbusch, Trottenberg, Briggs, Henson, McCormick, ...)

Formally, if A = LLT and M = L−1AL−T , then κ(M) = O( c2c1) and

Au = f ⇒ Mu = f , u = LT u, f = Lf

h n # iterations

2−3 289 7

2−5 4225 8

2−8 66049 9

20

Exploring the intuitive relation

2. The energy norm of the error:

‖∇u−∇uh‖ ⇔ ‖u⋆ − um‖ALet uh,m ∈ Vh, with coefficients um (similarly for uh). Then

‖∇(u− uh,m)‖2 = ‖∇(u− uh)‖2 + ‖u⋆ − um‖2Awith

‖∇(u− uh)‖ = O(h)

(other constants depend, e.g., on the regularity of the solution)

⇒ Stopping criterion for the iterative method:

‖u⋆ − um‖A = O(h)

• Very loose linear system accuracy (cheap...)

• Different perspective from Gauss elimination type solvers

(Arioli, Noulard, Russo, Loghin, Wathen, Jiranek, Strakos, Vohralık, ...)

21

Exploring the intuitive relation

2. The energy norm of the error:

‖∇u−∇uh‖ ⇔ ‖u⋆ − um‖ALet uh,m ∈ Vh, with coefficients um (similarly for uh). Then

‖∇(u− uh,m)‖2 = ‖∇(u− uh)‖2 + ‖u⋆ − um‖2Awith

‖∇(u− uh)‖ = O(h)

(other constants depend, e.g., on the regularity of the solution)

⇒ Stopping criterion for the iterative method:

‖u⋆ − um‖A = O(h)

• Very loose linear system accuracy (cheap...)

• Different perspective from Gauss elimination type solvers

(Arioli, Noulard, Russo, Loghin, Wathen, Jiranek, Strakos, Vohralık, ...)

22

An application to the Stokes equations

−∇2~u+∇p = ~0,

∇ · ~u = 0,

on some domain Ω ⊂ Rn, with b.c. ~u = ~w on ∂Ω

With the appropriate choices of bilinear forms and approximation

spaces, we obtain: A BT

B 0

u

p

=

f

g

(A sym.pos.def) MMMx = b

with MMM large, real indefinite symmetric matrix

Saddle point algebraic linear system

23

An application to the Stokes equations

−∇2~u+∇p = ~0,

∇ · ~u = 0,

on some domain Ω ⊂ Rn, with b.c. ~u = ~w on ∂Ω

With the appropriate choices of bilinear forms and approximation

spaces, we obtain: A BT

B 0

u

p

=

f

g

⇔ MMMx = b

with MMM large, real indefinite symmetric matrix

Saddle point algebraic linear system (cf. Benzi & Golub & Liesen, ’05)

24

Iterative solver and accelerator

MMMx = b

MMM is symmetric and indefinite. Petrov-Galerkin condition:

xm ∈ Km, s.t. min ‖b−MMMxm‖2rm = b−MMMxm, m = 0, 1, . . . (Paige & Saunders, ’75)

If PPP = HHT is an accelerator (sym. pos.def.),

xm ∈ Km, s.t. min ‖b−MMMxm‖PPP

where Km = H−1spanb,MMMPPP−1b, . . . , (MMMPPP−1)m−1b

25

Iterative solver and accelerator

MMMx = b

MMM is symmetric and indefinite. Petrov-Galerkin condition:

xm ∈ Km, s.t. min ‖b−MMMxm‖2rm = b−MMMxm, m = 0, 1, . . . (Paige & Saunders, ’75)

If PPP = HHT is an accelerator (sym. pos.def.),

xm ∈ Km, s.t. min ‖b−MMMxm‖PPP−1

where Km = H−1spanb,MMMPPP−1b, . . . , (MMMPPP−1)m−1b

26

An accelerator for Stokes mixed approximation

MMM =

A BT

B 0

, PPP =

A 0

0 S

A ∼ A, S ∼ BA

−1B

T

An example: from IFISS 3.1 (Elman, Ramage, Silvester): Lid driven cavity; Q2-Q1

approximation

I → mass matrix

−∆ → Algebraic MG

(spectrally equivalent matrix)

(Mardal & Winther, ’11)

2D. Final residual norm < 10−6

size(M) its Time (secs)

578 26 0.04

2178 26 0.14

8450 26 0.50

132098 26 11.17

27

An accelerator for Stokes mixed approximation

MMM =

A BT

B 0

, PPP =

A 0

0 S

A ∼ A, S ∼ BA

−1B

T

An example: from IFISS 3.1 (Elman, Ramage, Silvester): Lid driven cavity; Q2-Q1

approximation

S → pressure mass matrix

A → Algebraic MG

(spectrally equivalent matrix)

(Mardal & Winther, ’11)

2D. Final residual norm < 10−6

size(M) its Time (secs)

578 26 0.04

2178 26 0.14

8450 26 0.50

132098 26 11.17

28

An accelerator for Stokes mixed approximation

MMM =

A BT

B 0

, PPP =

A 0

0 S

A ∼ A, S ∼ BA

−1B

T

An example: from IFISS 3.1 (Elman, Ramage, Silvester): Lid driven cavity; Q2-Q1

approximation

S → pressure mass matrix

A → Algebraic MG

(spectrally equivalent matrix)

(Mardal & Winther, ’11)

2D. Final residual norm < 10−6

size(MMM) its Time (secs)

578 26 0.04

2178 26 0.14

8450 26 0.50

132098 26 11.17

29

A stopping criterion for Stokes mixed approximation

MMM =

A BT

B 0

, PPP =

A 0

0 S

A ∼ A, S ∼ BA−1BT

Ideal case: A = A, S = BA−1BT . Then a “natural” norm:

energy norm in u-space and L2 norm in p-space:

‖x− xm‖2PPPideal= ‖u− um‖2A + ‖p− pm‖2BA−1BT

For stable discretization, heuristic relation between error and residual:

‖x− xm‖PPPideal≤

√2

γ2‖b−MMMxm‖

PPP−1ideal

∼√2

γ2‖b−MMMxm‖PPP−1 < tol

γ inf-sup constant: γ2 ≤ qTBA−1BTq

qTQq, ∀q 6= 0

30

A stopping criterion for Stokes mixed approximation

MMM =

A BT

B 0

, PPP =

A 0

0 S

A ∼ A, S ∼ BA−1BT

Ideal case: A = A, S = BA−1BT . Then a “natural” norm:

energy norm in u-space and L2 norm in p-space:

‖x− xm‖2PPPideal= ‖u− um‖2A + ‖p− pm‖2BA−1BT

For stable discretization, heuristic relation between error and residual:

‖x− xm‖PPPideal≤

√2

γ2‖b−MMMxm‖

PPP−1ideal

∼√2

γ2‖b−MMMxm‖PPP−1 < tol

γ inf-sup constant: γ2 ≤ qTBA−1BTq

qTQq, ∀q 6= 0

31

A stopping criterion for Stokes mixed approximation

MMM =

A BT

B 0

, PPP =

A 0

0 S

A ∼ A, S ∼ BA−1BT

Ideal case: A = A, S = BA−1BT . Then a “natural” norm:

energy norm in u-space and L2 norm in p-space:

‖x− xm‖2PPPideal= ‖u− um‖2A + ‖p− pm‖2BA−1BT

For stable discretization, heuristic relation between error and residual:

‖x− xm‖PPPideal≤

√2

γ2‖b−MMMxm‖

PPP−1ideal

∼√2

γ2‖b−MMMxm‖PPP−1 < tol

γ inf-sup constant: γ2 ≤ qTBA−1BTq

qTQq, ∀q 6= 0

32

Estimating the inf-sup constant

Spectrum of transformed coefficient matrix:

spec(MMMPPP−1) ⊆ [Λ−, λ−] ∪ [λ+,Λ+]

with (Elman-Silvester-Wathen, ’05)

λ− ≤ 1

2(δ −

√δ2 + 4δγ2) δ ≤ λ+

where δ = λmin(AA−1).

If these bounds were tight (equalities), then

γ2 =λ2− − λ−λ+

λ+

33

Estimating the inf-sup constant

Spectrum of transformed coefficient matrix:

spec(MMMPPP−1) ⊆ [Λ−, λ−] ∪ [λ+,Λ+]

with (Elman-Silvester-Wathen, ’05)

λ− ≤ 1

2(δ −

√δ2 + 4δγ2) δ ≤ λ+

where δ = λmin(AA−1).

Were these bounds tight (equalities), then

γ2 =λ2− − λ−λ+

λ+

34

Estimating the inf-sup constant

Assume that these bounds are indeed tight (equalities). Then

γ2 =λ2− − λ−λ+

λ+

In practice, adaptive estimate with Harmonic Ritz values:

γ2 ≈ γ2m =

(θ(m)− )2 − θ

(m)− θ

(m)+

θ(m)+

, mth MINRES iteration

(Silvester & Simoncini, ’11)

35

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

36

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

37

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

38

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

39

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

40

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

41

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

42

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

43

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

44

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

45

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

46

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

47

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

48

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

49

Typical convergence pattern

0 10 20 30 40 50 60 70−60

−40

−20

0

20

40

60

number of MINRES iterations

Ha

rmo

nic

Ritz v

alu

es

Harmonic Ritz values as iterations proceed

50

Another IFISS example. Flow over a step.

em: error at iteration m rm: residual at iteration m

Optimal γ Adaptive γm

E = Pideal, M∗ = P−1

51

A model-based stopping criterion

If an a-posteriori discretization error estimate η(m) is available, namely

c η(m) ≤ ‖∇(~u− ~u(m)h )‖+ ‖p− p

(m)h ‖ ≤ C η(m), m = 1, 2, 3 . . .

with Cc= O(1)

(Ainsworth, Oden, Kay, Silvester, Elman, Wathen, Liao, Jiranek, Strakos, Vohralık, ...)

We can devise a stopping criterion:

‖x− xm‖PPPideal∼

√2

γ2m

‖b−MMMxm‖PPP−1 < η(m)

52

A model-based stopping criterion

Another IFISS example. Smooth colliding flow.

rm: residual at iteration m

0 10 20 3010

−6

10−4

10−2

100

102

iteration number m

h=1/4 (grid level 4)

√2/γ 2

m| |rm| |M∗

ηm

| |rm| |M∗

0 10 20 3010

−6

10−4

10−2

100

102

iteration number m

h=1/16 (grid level 6)

√2/γ 2

m| |rm| |M∗

ηm

| |rm| |M∗

M∗ = PPP−1

53

Final consideration

Opening the black box may be rewarding, with a mutual gain

References:

* M. Benzi, G.H. Golub and J. Liesen, Numerical Solution of Saddle Point

Problems, Acta Numerica, 14, 1-137 (2005)

* H. Elman, D. Silvester and A. Wathen, Finite Elements and Fast Iterative

Solvers, with applications in incompressible fluid dynamics, Oxford University

Press, (2005)

* K.A. Mardal and R. Winther, Preconditioning discretizations of systems of

partial differential equations, Numer. Linear Algebra with Appl., 18, 1-40 (2011)

* D. Silvester and V. Simoncini An Optimal Iterative Solver for Symmetric

Indefinite Systems stemming from Mixed Approximation. ACM TOMS, (2011)

* Multigrid Website MG-Net, http://www.mgnet.org (up to 2010)

54