Alex Townsend Cornell University

townsend@cornell.edu

Joint work with:

Continuous analogues of Krylov methods for differential operators

Marc Aurèle Gilles

Based on “Continuous analogues of Krylov methods for differential operator”, submitted last week.

Computing trends

1960

Punchcards DOS

1980 2000 2050

Windows95 Device oblivious

interaction

Tabulations IEEE 754 Software Discretizationoblivioussoftware

Discretize-then-solve“Discretize-then-solve”

Lu = f on ⌦u|@⌦ = g

Ax = bMatrix

Discretesolution x

uApprox.solution

Linear,elliptic,

2nd order, PDE

Finite diff. and pseudospectral

L• is an unbounded operator• is a bounded operatorA

• Want to be well-conditionedA

• Want to capture A L2nd order diff matrices

(see [Weideman & Trefethen, 1988] and [Fornberg & Flyer, 2015])

Solve-then-discretize

Matrix Krylov subspace methods

Kk(A, b) = Span�b, Ab,A2b, . . . , Ak�1b

Task: Solve Ax = b for x

Krylov subspace methods are iterative solvers, computing iterates from:

A is largefast mat-vecs

CG + Legendre-Galerkin disc.PAx = PbPreconditioning:

Popular strategies for spectral methods: 1. Low-order methods

(same DE, crude disc.)

2. Equivalent preconditioning(crude DE, same disc.)

e.g. [Canuto & Quarteroni, 1985], [Orszag, 1980]

e,g, [Axelsson & Karatson, 2009], [Hiptmair, 2006]

[Shewchuk,1994] and [Liesen & Strakoš, 2013]

For details, see [Shen, Tang, and Wang, 2011].

“For Krylov methods you apply the operator in forward mode, so imposing bcs does not make sense.”

Marcus Webb

Operator Krylov methods

“One needs to consider the weak formulation and define the right spaces for the domain and range.”

Folkmar BornemannNick Trefethen

“I find it remarkable that on these fundamental issues we are so much in the fog.”

A discussion in Sept 2012, involving ≈120 numerical analysts:

Task: Solve Lu = f with u|@⌦ = 0fast op-funs

Qu: What is a continuous analogue of Krylov subspaces for diff ops?

The first SIAM spotlight book is on this topic. [Malèk & Strakoš, 2014]

[Daniel, 1965]

L = 2nd order di↵ op

Continuous analogues

Krylov method

MINRES[Paige & Saunders, 1975]

Matrix properties Operator properties

A = symm.

conjugate gradient[Hestenes & Stiefel, 1952]

A = pos. def. L has pos. eigs

is self-adjointL

GMRES[Saad & Schultz, 1986] A = invertible Ellipticity & some

regularity

Task: Solve Lu = f with u|@⌦ = 0fast op-funs

L = 2nd order di↵ op

B[v, w] =dX

i,j=1

Z

⌦aijvxiwxj +

dX

i=1

Z

⌦bivxiw +

Z

⌦cvw

[Evans, 2010]

Lu = �dX

i,j=1

(aij(x)uxi)xj +dX

i=1

bi(x)uxi + c(x)uL : H20 7! L2,

Naive attempt:

Four problems

Task: Solve Lu = f with u|@⌦ = 0L = di↵ opfast op-funs

�uxx = 1� x2

Kk(L, f) = Span�1� x2, 2

u(±1) = 0 Doesn't contain sol.Example:Kk(L, f) = Span

�f,Lf, . . . ,Lk�1f

Problem 4 f |@⌦ 6= 0 Imposing bc in ? Kk

Problem 2 L destroys smoothness Repeated op-funs?Problem 3 L is an unbounded op Convergence?

Problem 1 L destroys bcs Repeated op-funs?

Zdenek Strakoš Josef Málek

Ideas abound:

Sheehan Olver Jared Aurentz Ralf Hiptmair Jie Shen

Idea 1: Orthogonal projections“Orthogonal projections allow us to do repeated op-fun products.”

Related to projected Krylov methods in quadratic programming.[Gould, Orban, & Rees, 2014]

Pick an approximation space V0 Lu = ffor u in .f 2 V0Assume for now.

V0 ⇢ H20(⌦)

f

LfL(Lf)

Without projections*If repeated op-funs are possible.

Set V0 ⇢ H

20(⌦)(Need closed -subspace.)L2

⇧V0� = arg minq2V0

k�� qkL2

V0 ⇢ H20(⌦)

f

Lf

With projections

L(⇧V0Lf)

⇧V0Lf

(⇧V0L)2f

Idea 1: Orthogonal projectionsWhy orthogonal projection?Self-adjoint(!):

Example, again: �uxx = 1� x2 u(±1) = 0

K3(⇧⇤L⇧, f) = Span

�1� x2, (�21x4 + 14x2 + 7)/4,�147x4 + 161x2 � 14

V0 = {p(x) =4X

j=0

↵jxj : p(±1) = 0} ⇧V0� = arg min

q2V0

k�� qkL2

⇧⇤V0

= ⇧V0

V0 ⇢ H20(⌦)

f

f 2 V0L⇧V0f

⇧⇤V0L⇧V0f

L⇧V0(⇧⇤V0L⇧V0f)

(⇧⇤V0L⇧V0)

2f

Krylov subspace:Kk(⇧

⇤V0L⇧V0 , f) = Span{f,⇧⇤

V0L⇧V0f, . . . , (⇧

⇤V0L⇧V0)

k�1f}

Respects bilinear form:

v, w 2 V0

B[⇧V0v,⇧V0w] = B[v, w]for

Operator CG methodMatrix CG Operator CG

= symm. pos. def. matrixA f 2 V0L = self-adj., pos. eigvals.,

The convergence of this operator

CG method is poor!

This op. CG method is equiv. to matrix CG on specific Galerkin disc.

Not yet a continuous analogue

V0 = H20(⌦)…so need to take !

But, we want a discretization oblivious algorithm…

Idea 2: Operator preconditioners“Symmetric operator preconditioners improve convergence.”

For this example, the preconditioner is similar to integral reformulation.[Zebib, 1981], [Greengard, 1991], & [Driscoll, 2014]

RTARy = RT bx = Ry

Symm. preconditioning: Symm. operator preconditioning:Rv|@⌦ = 0R⇤LRv = R⇤f,

u = Rv

Example: BVP on (�1, 1)

Less art in the science: (Preconditioner only depends on .)L

R� is easyB[R�,R ] ⇡ h�, i

[Hiptmair, 2006] and [Axelsson & Karátson, 2009]

⇧� = �� 1

2

Z 1

�1�(s)dsW0 = {v 2 L2(⌦) :

Z 1

�1v(s)ds = 0}

R⇤� =

Z 1

x�(s)dsR� =

Z x

�1�(s)dsPick ,

Operator PCG method

T = ⇧⇤R⇤LR⇧

L = self-adj., pos. eigvals.,

Combine ortho. projectionswith op. preconditioners:

E1:

E2:

E3:

Operator PCG method

�((2 + cos(⇡x))u0)0 = (1 + x2)�1, u(±1) = 0

Convergence of op. PCG method

�((1 + x2)u0)0 + (⇡4 cos(⇡x))2u = (1 + x2)�1, u(±1) = 0

�u00 + 2(⇡4 )2u = (1 + x2)�1, u(±1) = 0

R⇤f 2 W0

Kk(T ,R⇤f) = Span{R⇤f, T R⇤f, . . . , T k�1R⇤f}

PCG theorems

CG iterates:

Matrix CG Operator PCGCG iterates: u0 = 0, u1, . . . ,x0 = 0, x1, . . . ,

Fact 2: pTj Apk = 0, j 6= kpk = search direction

Fact 2:B[Rpj ,Rpk] = 0, j 6= k

pk = search function

rk = kb�Axkk2Fact 1: rTj rk = 0, j 6= k hrj , rki = 0, j 6= kFact 1:

rk = kR⇤f �R⇤LukkL2

Fact 3: xk = arg miny2Kk(A,b)

kx� ykA Fact 3: uk = argminq2Xk

ku� qkLXk = {p 2 H

10(⌦) : p = Rq, q 2 Kk(T ,R⇤f)}

Fact 4:kx� xkkA 2

p(A)� 1p(A) + 1

!k

kxkA

Fact 4:ku� ukkL 2

p(T )� 1p(T ) + 1

!k

kukL

(see [Kirby, 2010] for a def. of .) (T )

Idea 3: An ancillary problem for righthand side“The ancillary problem allows us to deal with general righthand sides.”

T = ⇧⇤R⇤LR⇧Kk(T ,R⇤f) = Span{R⇤f, T R⇤f, . . . , T k�1R⇤f}

Rv|@⌦ = 0R⇤LRv = R⇤f,We solve

using the Krylov subspace

We need R(R⇤f)|@⌦ = 0 to apply the operator PCG method.

1. Modify RHS before solving:

Find a RR⇤LRv1|@⌦ = RR⇤f |@⌦v1 so that

2. Solve R⇤LRv2 = R⇤g v2 g = f �R⇤LRv1for with

3. Set v = v1 + v2

Practical realization of the PCG method for BVPs

ApproxFun

We let Chebfun deal with the discretization of functions:

Numerical experiment: smooth functions

Oscillatory sol., simple var. coeffs

!1 = 10

Simple sol., oscillatory var. coeffs

!2 = 10

u(±1) = 0

Family of BVPs:

f(x)• chosen so that the exact solution is u(x) = sin(!2⇡x)

� d

dx

✓(2 + cos(!1⇡x))

du

dx

◆= f(x) on (�1, 1)

Numerical experiment: piecewise smooth

u(±1) = 0

BVP with piecewise smooth functions:

�((1 + 2| cos(⇡x)|)u0)0 = sign(cos(30⇡x)) on (�1, 1)

chebfunpref.setDefaults(’splitting’,true)L = chebop(@(x,u) -diff((1+2*abs(cos(pi*x))).*diff(u)));f = chebfun(@(x) sign(cos(3*pi*x)));L.lbc = 0; L.rbc = 0;u = pcg(L, f)

u0(x)

u(x)

Similar to [Lee & Greengard, 1997]

Operator GMRES

Matrix GMRES solves via the following approx. problems:Ax = b

Operator GMRES solves via

T = ⇧⇤R⇤LR⇧Kk(T ,R⇤f) = Span{R⇤f, T R⇤f, . . . , T k�1R⇤f}

where

We hope that uk = Rvk converges to the solution of

R⇤LRv = R⇤f

Lu = f

Numerical experiment: operator GMRES

BVP with smooth functions:�(exu0)0 + u0 � 10u = sin(30⇡x) on (�1, 1) u(±1) = 0

GMRES with restarts

L = chebop(@(x,u) -diff(exp(x).*diff(u)) + diff(u) - 10*u);

f = chebfun(@(x) sin(30*pi*x));

L.lbc = 0; L.rbc = 0;

u = gmres(L, f);

plot(u)

From experience, the behavior of operator GMRES is similar to matrix GMRES.

u(x)

Thank you

u = pcg(L, f)

u = minres(L, f)

u = gmres(L, f)

u = pcg(A, b)

u = minres(A, b)

u = gmres(A, b)

Matrix Krylov methods Operator Krylov methods

Dr. Gilles

What’s next: Elliptic PDEs with inv. sqrt Laplacianpreconditioners

AND

Idea 1: Orthogonal projectionsIdea 2: Operator preconditionersIdea 3: Ancillary problem for rhs