14th EuroAd Workshop - Markus Towara - An effective discrete adjoint model for OpenFOAM.pdf

STCE

An (more) effective Discrete Adjoint Model for OpenFOAM

Markus Towara, Uwe NaumannSoftware and Tools for Computational Engineering ScienceRWTH Aachen University

EuroAD 2013, Oxford, 10. December 2013

STCECFD @ STCE

I A. Sen (aboutFLOW Project, discrete OpenFOAM)I A. Daztouri (aboutFLOW Project, discrete ACE+)I M. Foerster (fortissiFOAM Project, OpenFOAM in the cloud)I me (discrete OpenFOAM)

,

STCE RWTH-Aachen, Discrete OpenFOAM, EuroAD 2013 2/ 35

STCEOutline

OpenFOAM Introduction

Discrete OpenFOAM

Improvements

Verification and Results

Work in Progress - PISO Foam

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEOpenFOAM

I Open Field Operation and ManipulationI Open-Source (GPLv3) CFD solverI developed by OpenCFD Ltd., currently at version 2.2.xI > 1 Mio. Lines of CodeI includes tools for meshing, pre-, post-processingI rising adoption in industry and academia due to lack of licence costs well suited for parallel

architectures

,


STCECFD Basics

I Navier Stokes equations for incompressible steady flow:

v v = 2v 1p momentum conservation

v = 0 mass conservation

I Or in three Dimensions:

vxvx

x+ vy

vx

y+ vz

vx

z=

p

x+

2vxx2

+2vx

y2+2vx

z2

vxvy

x+ vy

vy

y+ vz

vy

z=

p

y+

2vyx2

+2vy

y2+2vy

z2

vxvz

x+ vy

vz

y+ vz

vz

z=

p

z+

2vzx2

+2vz

y2+2vz

z2

u

x+v

y+w

z= 0

,


STCECFD Basics

(Decoupled) Solution of the partial differential equations (SIMPLE-Algorithm):

I discretize / linearize momentum conservation equationsI solve momentum equations for velocity v, assume pressure p as knownI obtained velocity field fulfills momentum equation but not mass conservation equation as the

pressure field was guessed and not correct

I discretize mass conservation equationI use mass conservation equation to correct pressure fieldI use new pressure field to correct velocity fieldI Loop...

,


STCETopology Optimization

no optimization added material reconstruction

Add penalty term to Navier-Stokes equation1 :

(v ) v = 2vp v1C. Othmer: A continuous adjoint formulation for the computation of topological and surface sensitivities of ducted

flows. Intern. J. f. Num. Meth. in Fluids. p. 861877, 2008.,


STCEHow to find appropriate

I Define Cost Function J, e.g. total pressure loss between inlet and outlet:

J =

p+1

2v2n d

I Calculate sensitivity of the Cost function w.r.t. parameters iJ

i=???

I Calculate updated porosity field n+1, e.g.:

n+1i = ni J

n

ni, while insuring 0 < i < max

I Loop until converged...

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEMotivation for ADin Context of Shape Optimization

I AD promises:I greater flexibility w.r.t new objectivesI easy adaption to new solver types / generationsI calculated derivatives are exact w.r.t the used discretizationI higher order derivatives available

I Problems:I memory requirementsI how to retain parallelism?

,


STCEHow to implement

n m, so Adjoint Mode is method of choiceI source code transformationI operator overloading

I due to complexity of OpenFOAM code (templates, massively object oriented) the only suitableoption

I operator overloading tool dco/c++I every floating point value in OpenFOAM is replaced with custom adjoint datatype dco::a1s::typeI needed intermediate values are stored in a structure called tape

,


STCEHow to implement

I put dco into src/OpenFOAMI include dco.hppI replace doubles with active datatype from dcoI OpenFOAM has own typedef for scalar floating point values just one substitution

I in theory we now just need to recompile OpenFOAM and are ready to go

,


STCEOpenFOAM a1s mode

in src/OpenFOAM/primitives/Scalar/doubleScalar/doubleScalar.h: replace:

namespace Foam{

typedef double doubleScalar;...

}with:

#include dco.hppnamespace Foam{

typedef dco::a1s::type doubleScalar;...

}

,


STCEOpenFOAM t1s mode

in src/OpenFOAM/primitives/Scalar/doubleScalar/doubleScalar.h: replace:

namespace Foam{

typedef double doubleScalar;...

}with:

#include dco.hppnamespace Foam{

typedef dco::t1s::type doubleScalar;...

}

,


STCEin Reality

I some minor changes have to be made in the OpenFOAM code:I unions dont support active datatypesI some casts are missing and have to be done by handI some functions (pow,max,min) have to be adapted

,


STCEusage of discrete OpenFOAM - t1s

Black-Box tangent-linear Version of simpleFoam, calculates Ji :

double sens = 0;dco::t1s::set(alpha[i],1,1);

for (runTime++; !runTime.end(); runTime++){... // solve for U,p}// Sum pressure over inlet faces scaled with face areadoubleScalar J = gSum( p.boundaryField()patch.magSf() );

dco::t1s::get(J,sens,1);

Need to do this N-times to get full sensitivity field!

,


STCEusage of discrete OpenFOAM - a1s

Black-Box adjoint Version of simpleFoam, calculates gradient of J :

dco::a1s::static tape tape(tapeSize);double sens = new double[alpha.size()];

for(int i=0; i

STCEOutline


Discrete OpenFOAM

Improvements



,


STCEFirst Results

I Black-Box approach severely bound by memory bandwith! (Need to allocate around 20 GB)I By doing Checkpointing we can actually get faster...

,


STCEHow to improve?

I Knowledge and Profiling reveals that most of calculation time and tape memory is spent in(iterative) linear solvers

I Analytical insight allows us to treat the adjoints of linear solvers analytically

Lemma

For a Linear Equation Sytem Ax = b we can calculate the adjoints for A and b by: 2

b = AT x AT b = xA = b xTI This gives us an additional Linear Equation System which we have to solve during the gathering

of the adjoints

2M. Giles, Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation, Advancesin Automatic Differentiation 2008

,


STCETreatment of Linear Solvers

Black-Box Continuous Linear Solver

I when we encounter an linear solver during the augmented forward run we can stop tapingI when we encounter the gap in the tape during the interpretation we have to fill in the gap by

hand

I have to remember A and x!

,


STCEApplication to OpenFOAM

I Abstract Base Class for all Linear Solvers exists can tackle all implemented solvers at onceI Discretization matrices are stored in a coordinate sparse matrix format transposing matrix

straightforward

I take care not to mess up symmetry of matrices

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEResults

I we are seeing a nice improvement in both runtime and memory usage

,


STCEVerification

I We calculate a reference solution with solvers GAMG for U and smoothSolver for p, both withtolerance 1e 10

I After that we calculate solutions with different tolerances for the forward and adjoint linearsolvers and compare to the reference solution

I 11k Cells in mesh

Velocity Sensitivity

,


STCEVerification

Runtime Tape-Memory

,


STCEError

,


STCERuntime

,


STCEOutline


Discrete OpenFOAM

Improvements



,


STCEWork in Progress - Unsteady Adjoints

Unsteady PISO solver with unsteady cost function (potentially depends on every timestep)

,



,


STCESummary and Outlook

I Fully discrete mode uses loads of memoryI Treatment of linear solvers greatly reduces thisI Its also fasterI Application to unsteady flow, heat transfer, turbulence in progress

,


STCE

Thank you!Questions?

,


OpenFOAM IntroductionDiscrete OpenFOAMImprovementsVerification and ResultsWork in Progress - PISO Foam

14th EuroAd Workshop - Markus Towara - An effective discrete adjoint model for OpenFOAM.pdf

Documents

Transcript of 14th EuroAd Workshop - Markus Towara - An effective discrete adjoint model for OpenFOAM.pdf