Performance Imrovement of Algebraic Multigrid Solver by...

LESExtrapolation Methods

Computational ExamplesConclusions

Performance Imrovement of Algebraic MultigridSolver by Vector Sequence Extrapolation

A. Jemcov1 J.P. Maruszewski1 H. Jasak2

1ANSYS-Fluent Inc. 2Wikki Ltd.

May 30, 2007

A. Jemcov, J.P. Maruszewski, H. Jasak Performance Improvement by Extrapolation



Outline

1 IntroductionGoals of PresentationMotivation

2 Extrapolation MethodsFixed-Point Methods for Linear Systems of EquationsProjective Forward ExtrapolationMinimal Polynomial ExtrapolationReduced Rank Extrapolation

3 Computational ExamplesLES of Forward Facing StepFree-Surface Simulation of Droplet Impact

4 ConclusionsSummary




MotivationMotivation

Outline









Goals

Performance improvement of Agglomerative AlgebraicMultigrid Method (AAMG) for large matrices is examined

Performance improvement must be achieved through codenon-intrusive techniques that can be easily parallelized

Hybrid method that uses Vector Sequence Extrapolationand AAMG solver is proposedIn particular, three hybrid methods are considered:

Projective Forward Extrapolation (PFE)Minimal Polynomial Extrapolation (MPE)Reduced Rank Extrapolation (RRE)

Compare performance of three methods and makerecommendations





Outline









Motivation

Implicit segregated pressure based solvers used in thisstudy (PISO, SIMPLE, etc.)

Implicit nature of the solvers leads to linear systems ofequations for pressure and momentum that need to besolved

Momentum equations are mostly dominated by localstructures, traditional stationary iterative methods are(mostly) effective

Pressure equation has a global nature dominated by manyscales, convergence of traditional stationary iterativemethods can be slow

Geometric complexity and low quality mesh are anothersource of difficulties





AMG Method

Implicit discretization of pressure equation results insparse linear problem

Ax = b

Matrix A stored in arrow format with sparsity pattern storedin compressed row format

Stationary iterative methods obtained by decomposingmatrix A

A = M − N

Depending on choice of M and N, Gauss-Seidel, Jacobi,SSOR, etc. are obtained





AMG Method

In multigrid methods, stationary iterative methods are usedas smoothers rather than solversAcceleration is obtained through application of smootherson hierarchy of matrix levels obtained by coarseningprocedureRestriction Rn+1

n and prolongation operators Pnn+1 are

obtained through process of agglomerationCoarse matrices are obtained through projection

An+1 = Rn+1n AnPn

n+1

Restriction and prolongation operators are also used totransfer solution and residual to and coarse levels

xn+1 = Rn+1n xn

xn = Pnn+1xn+1





AMG Coarsening

Agglomeration process





µ-Cycle(xn, rn)

Create multigrid levels:An, Rn+1

n , Pnn+1, n = 0, 1, 2, ..., N − 1

for n = 0 to N − 1ν1 pre-smoothing sweeps:solve Anxn = bn

, rn = bn − Anxn

if n ! = N − 1bn+1 = Rn+1

n rn

xn+1 = 0xn+1 = µ-Cycle(xn+1

, rn+1) µ timesCorrect xn

new = xn + Pnn+1xn+1

endν2 post-smoothing sweeps:solve Anxn

new = bn

endA. Jemcov, J.P. Maruszewski, H. Jasak Performance Improvement by Extrapolation




AMG Method

Agglomerative Algebraic Multigrid Method is favored incomputationally intensive calculations that involve matriceswith M-matrix properties

AAMG Algorithm is very cheap to implement - doesn’trequire storage of prolongation and restriction operators(matrices)

Every equation in fine level has a coarse equationrepresentation

Assumption is that the error is homogeneous -prolongation coefficients are all equal

Convergence of AAMG can be improved by using scalingof corrections (variational formulation)




Fixed-Point MethodsPFEMPERRE

Outline









Fixed-Point Methods

Consider again linear problem

Ax = b

With splitting A = M − N stationary iterative method isobtained

x(ν+1) = Rx(ν) + M−1b

Here R is the iteration matrix

R = M−1N





Fixed-Point Methods

Define error as the difference between current iterate x(ν)

and the fixed-point x(∗)

e(ν) = x(ν)− x∗

Error propagation equation

e(ν+1) = Re(ν)

The iteration matrix possesses the recursive property:

e(ν+1) = Rνe(0)





Fixed-Point Methods

Define corrections

△x(ν) = x(ν)− x(ν−1)

The recursive property is also applicable to corrections

△x(ν+1) = Rν△x(0)

The importance of the recursive property is that generatesKrylov subspace

K = span(△x(0),△x(1)

, · · · ,△x(n))

Therefore, solution can be sought in that subspace in thefollowing form:

x∗ = x(ν) +∑

ν

αν△x(ν)





Fixed-Point Methods

The fundamental question is: ”How αν can be determinedto accelerate iterative process

αν are obtained by introducing additional conditions(constraints) in the Krylov subspace

Three methods of selecting αν presented here

In practice, finding the solution is implemented as arestarted procedure

xn = x(n−k) +k

∑

ν=0

αν△x(ν)

It is important to determine the proper dimension k of therestart space to get the best performance





Outline









Projective Forward Extrapolation

The simplest approach to determining alpha(ν) is to use aconstant value and restrict Krylov subspace to twodimensions

x∗ = x(ν) + α△x(ν)

If α is set to 1, linear extrapolation based on first orderdifference is obtained

This is the PFE-AAMG algorithm

To make this algorithm effective, number of AAMGsmoothing steps must be performed before extrapolation

In applications we use 10 AAMG smoothing steps

Method is attractive from the efficiency point of view due tolow storage requirements





Projective Forward Extrapolation

PFE-AAMG method was inspired by work of Gear at al.(2005)

Fixed-Point iteration is treated as discrete dynamicalsystem with prescribed mapping

x(ν+1) = Rx(ν) + M−1b

The idea is to remove fast transients and then to projectonto a slow manifold

This method works well for ODEs and it assumesdifferentiability of the smooth manifold

PFE method can also be used for nonlinear problems withfast and slow time-scales





Outline









Minimal Polynomial Extrapolation

If higher Krylov subspace of higher dimensionality is used,PFE is not effective due to the assumptions ofdifferentiabilityInstead, natural condition of the restarted method atconvergence is

x∗ = x∗ +k

∑

ν=0

ανeν

In other words, α(ν) must satisfy the following equation

k∑

ν=0

αkν=0eν = 0

Ideally, this condition should be connected to the iterativeprocess so that αν are determined from that process






One way to connect extrapolating coefficients and iterativeprocess is to use property of Krylov subspace methodsthat are known to implicitly define minimal polynomial Pk

related to iteration matrix R

Pk (R)△x0 =

k∑

ν=0

cνRν

△x(0) = 0

Further manipulation leads tok

∑

ν=0

cνRν△x(0) = (I − R)

k∑

ν=0

cνe(ν) = 0

Since I − R has a full rank, we obtaink

∑

ν=0

cνe(ν) = 0.






Coefficients cν are obtained by requiring that the followingL2 norm is minimized is

argminαν

∣

∣

∣

∣

∣

∣

∣

∣

∣

∣

x(ν) +

k∑

ν=0

αν△x(ν)

∣

∣

∣

∣

∣

∣

∣

∣

∣

∣

2

In practical terms we collect k correction vectors △x(ν) andform rectangular matrix

Uk−1 =[

△x(0),△x(1)

, · · · ,△x(k−1)]






The following over-determined problems is solved

Uk−1c̃ = △x(k)

MPE method is obtained when the following constraint isused

ck = 1

αν coefficients are recovered by a simple scaling

αν =cν

∑k−1v=0 cν

MPE method is equivalent to Full Orthogonal Method inLinear Algebra





Outline









Reduced Rank Extrapolation

Reduced Rank Extrapolation is recovered by selecting theconstraints to be

k∑

ν

αν = 1

This leads to the following over-determined problem

Uk α̃ = 0

αν coefficients are obtained from

αν = cν

RRE method is equivalent to GMRES method in LinearAlgebra




LES ProblemFree-Surface Problem

Outline









LES

Consider turbulent flow over forward facing step atRe = 10000





Computational Examples

Second order accurate in space and time scheme used

CFL number held at unity

2 PISO correctors

Computational mesh size 660000 cells

Mesh aggressively graded towards the wall

Computational hardware: 2.16G Hz Intel CoreDuo CPUwith 2Gb

Convergence tolerance for pressure equation set to10E − 08





LES

W-Cycle, Group Size 4, ILU(0) smoother, 0 pre-sweepsand 2 post-sweeps

0 10 20 30 40 50Cycles

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

||R||

AMGPFE-AMGMPE-AMGRRE-AMG





LES


Method AMG Cycles Time [s]AMG 47 27.95PFE-AMG 40 24.20MPE-AMG 36 22.90RRE-AMG 39 24.58





LES: MPE-AMG


0 10 20 30 40 50Cycles

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

||R||

AMGMPE-AMG, KDIM = 5MPE-AMG, KDIM = 10MPE-AMG, KDIM = 15





LES: MPE-AMG


Dimension AMG Cycles Time [s]0 47 27.955 36 22.9

10 45 29.2415 46 29.56





LES-RRE-AMG


0 10 20 30 40 50Cycles

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

||R||

AMGRRE-AMG, KDIM = 5RRE-AMG, KDIM = 10RRE-AMG, KDIM = 15





LES: RRE-AMG


Dimension AMG Cycles Time [s]0 47 27.955 39 24.58

10 33 21.8615 32 22.27





Outline









Free Surface

Droplet free surface at time t = 2.77325E − 05 s





Free Surface






Free Surface


0 10 20 30 40 50Cycles

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

||R||

AMGPFE-AMGMPE-AMGRRE-AMG





Free Surface: MPE-AMG


Method AMG Cycles Time [s]AMG 47 63.39PFE-AMG 37 48.21MPE-AMG 41 58.90RRE-AMG 24 36.13







0 10 20 30 40 50Cycles

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

||R||

AMGMPE-AMG, KDIM = 5MPE-AMG, KDIM = 10MPE-AMG, KDIM = 15







Dimension AMG Cycles Time [s]0 47 63.395 41 58.9010 41 59.9515 43 63.60





Free Surface: RRE-AMG


0 10 20 30 40 50Cycles

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

||R||

AMGRRE-AMG, KDIM = 5RR-AMG, KDIM = 10RRE-AMG, KDIM = 15





Free Surface: RRE-AMG


Dimension AMG Cycles Time [s]0 47 63.395 41 58.9010 22 33.1815 21 32.56




Summary

Outline








Summary

Conclusions

All three algorithms show significant improvements of AMGalgorithm

PFE-AMG cheap and simple, not as effective as MPE andRRE

RRE superior to MPE with the same implementationcomplexity

Dimension of restart space is important

All three algorithms are implemented as wrappers to AMGand easily extend to any linear and nonlinear solver




Summary

RRE Acceleration of Explicit Density Density BasedSolver

Baseline run NACA 0012 - no acceleration

Scaled ResidualsExplicit airfoil: No Extrapolation

FLUENT 6.3 (2d, dp, dbns exp, ske)May 23, 2007

Iterations

300025002000150010005000

1e+01

1e+00

1e-01

1e-02

1e-03

1e-04

1e-05

1e-06

1e-07

epsilonkenergyy-velocityx-velocitycontinuityResiduals




Summary

RRE Acceleration of Explicit Density Density BasedSolver

Accelerated run NACA 0012 - RRE acceleration

Scaled ResidualsExplicit airfoil: With Extrapolation

FLUENT 6.3 (2d, dp, dbns exp, ske)May 23, 2007

Iterations

25002250200017501500125010007505002500

1e+02

1e+00

1e-02

1e-04

1e-06

1e-08

1e-10

1e-12

1e-14

epsilonkenergyy-velocityx-velocitycontinuityResiduals


Performance Imrovement of Algebraic Multigrid Solver by...

Documents

Transcript of Performance Imrovement of Algebraic Multigrid Solver by...