HPC and combustion

Thierry POINSOT

IMFT, Université de Toulouse, CNRS and CERFACS

Contributions from G. Staffelbach, L. Gicquel, B. Cuenot, O. Vermorel, F. Duchaine, E. Riber (CERFACS), L. Selle (IMFT), V. Moureau (CORIA), S. Mendez (UM2)

1

OUTLINE

• COMBUSTION RESEARCH AND HPC

• PERFORMING HPC COMBUSTION SIMULATION ON TIER0/TIER1 SYSTEMS FOR INDUSTRY:

!HOW CAN THIS BE DONE ? WHAT DID THE FRENCH COMMUNITY DO ?

!WHAT ARE THE WEAKEST PARTS OF OUR ORGANIZATION TO DO THIS ?

2

COMBUSTION OVERVIEW

Just remember two equations:

3

ENERGY ON EARTH TODAY = COMBUSTION

ENERGY ON EARTH = COMBUSTIONCOMBUSTION IS PRODUCING MORE THAN 90 PERCENT OF THE ENERGY TODAY. THIS WILL DECREASE... BUT NOT TOMORROW

4

COMBUSTION OVERVIEW

Just remember two equations:

5

ENERGY ON EARTH TODAY = COMBUSTION

ENERGY ON EARTH TOMORROW = COMBUSTION

CLIMATE CHANGE AND ENERGY MARKET: 2010/2030

•COMBUSTION SCIENCE MUST ALLOW THIS WITHOUT INCREASING EMISSIONS, WASTING FOSSIL FUELS OR MAKING CLIMATE CHANGE WORSE (!...)

6

OPTIMIZATION AND SIMULATION NEEDED

SIMULATION OF COMBUSTION: MULTISCALE -MULTIPHYSICS

7

Contrails

Engines!

Heat transfer, radiation, flow, turbulence, chemistry, fatigue, vibration, acoustics

COMBUSTION: MULTISCALE -MULTIPHYSICS

8

Within the combustion chamber: nanoseconds and nanometers

1 cm

1 cm

Fields of density in a H2-O2 engine

COMBUSTION: MULTISCALE -MULTIPHYSICS

9

Outside the engine: miles and days !

2 miles

WHICH EQUATIONS ?

•The Navier Stokes equations (5 + N unknowns: density, velocities and energy, N species). Partial differential equations -> non local, intense communication required

•Kinetics : N = 10 to 300 species reacting through 3000 reactions (everything local)

•Heat transfer through the walls, radiation, noise, soot

•All these flows are turbulent10

WHAT DO WE COMPUTE ?

•FINITE VOLUME CODES USING DOMAIN DECOMPOSITION AND MPI.

•Typically 100 Mcells with 100 variables at each cell (3 velocities, density, energy + 5 to 90 species) over 1000000 time steps. -> 10^16 unknowns

11

WHAT ABOUT PARALLELISM ?

•THE NAVIER STOKES EQUATIONS ARE ‘EMBARRASINGLY DIFFICULT’ TO PARALLELIZE -> THIS PROBLEM HAS BEEN IDENTIFIED AND IS REMODELING OUR COMMUNITY IN LARGER COLLABORATIVE TEAMS.

•THIS IS NOT A ‘ONE-PROFESSOR ONE-CODE’ SHOW ANY MORE BECAUSE THE CODES ARE USED FOR INDUSTRY APPLICATIONS ON A DAILY BASIS

12

13

ok

A joint initiative of French labs for the promotion of SUper-Computing for the modeling of Combustion, mixing and complex fluids in rEal SyStems.

The SUCCESS scientific groupSUCCESS was created in 2012 to help the promotion of super-computing in the area of Computational Fluid Dynamics (CFD) for complex geometries. It is coordinated by the CORIA lab and is composed of 8 French public labs

Our objectivesDistribute in the labs research HPC codes for CFD in complex geometriesEnsure the training of usersManage the development roadmapShare databases of high-resolution simulationsPromote super-computing

Some facts8 French public labsAround 120 researchers and students2 PRACE proposals accepted over the recent yearsSeveral prizes related to !"##$!! codes: Bull-Joseph Fourier prize, IBM faculty award, ...

The codes

A massively-parallel finite-volume and finite-element 3D code for the simulation of

compressible turbulent reactive and two-phase flows.

A massively-parallel finite-volume 3D code for the simulation of turbulent reactive and two-

phase flows at low-Mach number.

The labsCNRS labs: CORIA, EM2C, I3M, LEGI, IMFT, LMAEPIC labs: CERFACS, IFP-EN

!"##$!!%%http://success.coria-cfd.fr

LMA

AVBP Strong scaling examples

14

65536

57344

49152

40960

32768

24576

16384

8192

0

equi

vale

nt p

erfo

rman

ce

6553657344491524096032768245761638481920cores

ANL INTREPID, Bluegene P(1)

PRACE/TGCC, CURIE, BullX(2)

GENCI/CINES, JADE, SGI Altix ICE(3)

PRACE/JSC, JUQUEEN, Bluegene Q(4)

INCITE/ARNL, INTREPID, Bluegene P(5)

HLRS/PRACE, HERMIT, CRAY XE6 (5)

Ideal

(1) 93M Tetrahedra case - 1 step Chemistry - 2 tasks per node(2) 200M Tetrahedra case - 2 step Chemistry(3) 29M Tetrahedra case - 7 step Chemistry(4) 75M Tetrahedra case - No chemistry - 64 tasks per nod(5) 75M Tetrahedra case - No chemistry - 4 tasks per node

1 billion cells (BG/Q)

V. Moureau, P. Domingo, L. Vervisch, D. VeynanteDNS analysis of a Re = 40,000 swirl burner

0 2048 4096 6144 8192 10240 12288

Number of cores

0 0

2048 2048

4096 4096

6144 6144

8192 8192

10240 10240

12288 12288

Scal

e-up

linearYALES2

YALES2 scale-up on Babel @ IDRIS (Blue Gene/P)Up to 12288 cores and 2.6 billion tetrahedrons

2.6B tets

878M tets329M tets

41M tets

14M tets

2008

YALES2 solver(CORIA Rouen)

Juelich Workshop 2010

ORGANIZATION OF THE FRENCH NUMERICAL COMBUSTION COMMUNITY

16

SNECMA SNECMA DMSTURBOMECA

RHODIAALSTOM

AIR LIQUIDEHERAKLES

AIRBUSSIEMENSANSALDO

17

!"#$%&'()*()+,#-./#0)-1(23'(4567(89&:;<

CER

FAC

SIn

stitu

t Fra

nçai

s du

Pétro

leG6S'564'58+*Q!"#$%&$'()'(*+,-./!0%&/'(+1,

2"3-%&-+1456)+%7658*,!/792%&"65*+8))+,$:"%&"(18;<,$:%281=<'>+1

$?2@$2"0A/!A.@2/0:18>B%76(

"'=+)*

-'=+:*+5*Q

C@2-"0%&D8))65';<+,$:/E."2-0%&E'5=+*,C@2-"0%&D+51'1,C7C%&E'5=+6(F,

/2@0:G$7C0

0GC$."0@C0GA.C!2"2@C#2//0/!

0!/%G!H:!A2I0J%A2%#/0@-2

K

-'=+

@++=*

-'LM(4+5%;'LM618+*%N-'LM(O1P%;+14+5*Q%-20R%E65;+)'16R%I2@-!=>?=@!

A4+,)--'B

C$0@#./A

=>@!C!D@=>E(@F(GFCH(G=@I(@I!(JD4(@FF1GFCH=>E(6F@I(=>(7C4?!(4>K(=>?=@!(7CFL!?@D

MN

!"#$%&'()$*+,-.*/$01#2)&34$#5$67*6!+$85$9:;<

THIS ORGANIZATION ALLOWS TO DO OTHER THINGS IN OTHER FIELDS:

19

S. Mendez, F. NicoudYALES 2

Université de Montpellier

TWO TYPICAL PRACE EXAMPLES

• PRECCINSTA: A laboratory burner using a Turbomeca injection system. Developed in EC project PRECCINSTA and used as the classical benchmark for LES codes in the world. Most important problems in the GT industry can be reproduced on the PRECCINSTA burner. CH4 + Air at 1 bar.

• MASCOTTE: a high pressure experiment corresponding to rocket combustion (60 to 100 bars). Very powerful flames: H2/O2. Very extreme conditions (flames of the order or 10 microns)

20

21

PRECCINTSA (2005): THE FIRST LES OF SWIRLED BURNERS

3 millions cells

Roux, Lartigue, Poinsot, Meier and Bérat Comb and Flame 141, 2005

22

COLD FLOW - PRECCINSTA

23

All LES are compared with measurements DLR (LDA):• Velocity profiles are compared at five stations along the burner. • Comparison for axial, tangential and radial velocities (mean and RMS)

X= 5 mm

X= 1,5 mm

X= 25 mm

X= 15 mm

X= 35 mm

FROM SWIRLER

TO OUTLET

U

W

Comparison of mean velocity fields

24

-40

-20

0

20

40

Dis

tance

fro

m a

xis

[mm

]

151050

-40

-20

0

20

40

151050

-40

-20

0

20

40

151050

-40

-20

0

20

40

151050

-40

-20

0

20

40

151050

x=1.5 mm x=5 mm x=15 mm x=25 mm x=35 mm

Uxp Profiles: Red solid: CDP - Black solid: TTGC - Black dotted: TTGC_SSS - Circles: Exp.RMS velocity profiles: Stanford code (red), CERFACS code (black)

and DLR experiments (symbols)

PRECCINSTA:•First computed with LES in 2004: 1

Million cells (AVBP)

•Repeated in 2007 with 10 Mcells (AVBP)

•In 2009, repeated with 100 Mcells, 500 Mcells (YALES)

•In 2010 and 2011, on PRACE machines: JUGENE: 2 billion and CURIE 12 billion cells (YALES)

25

3T

DLR PRECCINSTA BURNER, Experiments: W. Meier et al., Combust. Flame, 150(1/2):2–26, 2007

=84>%?84%'#5$#@$(#1'&)4$>485A$3B)$C$&183)18#5$85$D,+**67/!.$EF1$G#>1)%>H$*I,6.J

3U

More details in:Roux et al, Combustion and Flame (2005)Moureau et al, Journal of Computational Physics (2007) (2 papers)Galpin et al, Combustion and Flame (2008)Moureau et al, Combustion and Flame (2011)Franzelli et al, Combustion and Flame (2011)

IS THIS ENOUGH ?:•At 20 billion cells, we are almost reaching

what we need in terms of resolution for the large flow structures but near walls and within the flame front, this is not enough

• This is a single burner at atmospheric pressure with gaseous fuel (no liquid phase). Real combustors have 16 to 24 burners, working at 20 to 100 bars, with liquid fuels.

•We still miss a 10^6 factor in power28

ROCKET COMBUSTION:

• PRACE Project by EM2C and CERFACS

• ROCKET COMBUSTION IS A VERY SPECIFIC AND NARROW FIELD: IT IS ALSO DIFFICULT SINCE FLUIDS IN THESE ENGINES ARE IN SUPERCRITICAL CONDITIONS (100 bars)

• CERFACS AND EM2C HAVE DEVELOPED SUPERCRITICAL CAPABILITIES IN AVBP IN THE LAST FIVE YEARS

• PRACE WORK PERFORMED IN 2012: ACOUSTIC EXCITATION OF SUPERCRITICAL FLAMES IN THE CONFIGURATION MASCOTTE OF ONERA

29

! T. Schmitt, H. Layal, M. Boileau, S. Ducruix, S.Candel (EM2C), ! A. Ruiz, G. Staffelbach, B. Cuenot and T. Poinsot (CERFACS)Large-Eddy Simulation of high-frequency instabilities under transcritical conditionsObjective: Observe and understand the flame behavior in transcritical flows of an experimental setup submitted to artificial acoustic transverse perturbation.

!

CEA - CERFACS 30

8.5M hours at TGCC

CEA - CERFACS

WITH FORCING:

31

WHAT IS GOING ON IN THE USA IN THE FUELD OF COMBUSTION AND HPC ?

32

In the last 15 years: the ASCI projects. Most ASCI projects in the USA were actually combustion projects: gas turbines, scramjets, rockets, fires, galaxies...

Now: the CODESIGN Centers. Example: Sandia CODESIGN Center. Main idea: «Compute combustors before you build them»

FOR THE FUTURE IN EUROPE

• EESI and PRACE have opened the path for a fast HPC evolution in Europe.

• Machines are already here (Tier1 and Tier0).

• The main questions today are:

(1) the codes: see GIS SUCCESS lead by CNRS

(2) the money: from industry and the EC

(3) the users !

34

Money: the ERC (European Research Council) INTECOCIS advanced grant at Institut de

Mécanique des Fluides and CERFACS 2013-2018 (intecocis.inp-toulouse.fr)

35

• Five year, 2.5 Meuros project on HPC tools for combustion instabilities. Coordinator: IMFT

• Ten researchers on numerical combustion, 5 years

• Collaboration with GENCI, SAFRAN, ANSALDO, SIEMENS

USERS / FORMATION: ‘ignored’ problem ?

36

The last ten years using AVBP at CERFACS and the las t two years suppor t ing an exp los ive development of AVBP and YALES at SAFRAN have shown that building an efficient HPC CFD team in combustion required multiple experts:

HPC experts- develop the code- port it on new machines- do the first demos

Combustion experts- know the physics- write the models- use the code but do not address HPC

CERFACS

37

HPC experts- develop the code- port it on PRACE/INCITE machines- do the first demos

Combustion experts- know the physics- write the models- use the code but do not address HPC

CERFACS

Industry experts:- know combustors- know ‘old style’ CFD and learn HPC

Students:- know nothing

Sous traitants:- know a little bit on CFD and the combustors

Combustion labs:- know theory and experiments- dont know the code details or HPC

38

HPC experts- develop the code- port it on PRACE machines- do the first demos

Combustion experts- know the physics- write the models- use the code but do not address HPC

CERFACS today

Industry experts:- know the engine- know ‘old style’ CFD and learn HPC

Students:- know nothing

Sous traitants:- know a little bit on CFD and the engine

Combustion labs:- know theory and experiments- dont know the code details or HPC

Interface / formation- interfaces to access the code- formation to learn the basics

INTERFACES: DISTRIBUTING HPC CODES TO INDUSTRY

39

C’est bientôt

fini?Mmmh...

La taille de ce tube est le vrai point critique aujourd’hui

A . Dauptain Copyright

DISTRIBUTING LES CODES:

40

00

• Codes CANNOT be used in industry without additional interfaces (it works with labs but not with industry).

• These interfaces must integrate a very large set of information coming from industrial needs

• The time required to write these interfaces is VERY large: the codes MUST be modified and simplified to adjust to the interfaces

INTERFACES: THE C3S EXAMPLE

• In the last 4 years, CERFACS has developed an interface called C3S for AVBP users in labs and industry

41

00

• C3S installed and working at CERFACS, IMFT, EM2C, IFPEN, SNECMA, Villaroche, SNECMA Vernon, TURBOMECA Bordes, etc

EXAMPLE :

42

BUT THIS IS NOT ENOUGH!CREATING ONE INTERFACE FOR ONE CODE IS NOT

ENOUGH:

• Keeping up with AVBP developments in C3S is difficult:-> must coordinate code and interface evolutions. The interface MUST evolve with the code (or vice versa ?)

• If you write a piece of code, you should also, at the same time, write the interface

!In 2011: a new initiative -> C3Sm (A. Dauptain)

43

C3SM • Is an engine... which writes interfaces:

!developed by CERFACS

!used by all french groups developing AVBP or any other code linked to AVBP, to automatically produce and update the interface and the documentation: YALES, PRISSMA (radiation), AVTP (heat transfer), N3S (RANS), Coolant (SAFRAN dedicated software for cooling)...

44

!"#$%&'&(&)*#+,-"+./'%-0)-/0'-&"1#2345

2345

2&"'*".#657 5&'*.% 895

2*#$%&'&(&)*#%*"+#$&//-:)*#1• #)0#(%;0'-&"#+,895#$0%#)*/#+;<*)&$$*.%/#+*#(&+*• #)0#(%;0'-&"#+*##=&'*.%/#895#$*%>&%=0"'/#*'#(&=$0'-:)*/

?<0"'#1#2&"'%-:.'-&"#=&"&)-'@-A.*

B;<*)&$$*.%/#+*#(&+*/#/(-*"'-CA.*/

!'-)-/0'*.%/#+*#(&+*/#/(-*"'-CA.*/

2%;0'*.%/#+,-"'*%>0(*/

7*#(%;0'*.%#+,-"'*%>0(*/#*/'#0.#=-)-*.#+.#+-/$&/-'->D#-)#*/'#-"(&"'&.%"0:)*D#*'#(%;*#+*#),0''*"'*E

!"#$% &'%

?$%F/#1#2&"'%-:.'-&"#%;$0%'-*

B;<*)&$$*.%/#+*#(&+*/#/(-*"'-CA.*/

!'-)-/0'*.%/#+*#(&+*/#/(-*"'-CA.*/

2%;0'*.%/#+,-"'*%>0(*/

7*/#+;<*)&$$*.%/#*'#.'-)-/0'*.%/#$*.<*"'#>0-%*#;<&).*%#)*.%#&.'-)#(@*G#*.HD#/0"/#$0//*%#$0%#),-"'*%=;+-0-%*#+.#(%;0'*.%#

+,-"'*%>0(*E

2345

&()*"&%

WHAT ABOUT FORMATION ?

48

• PhDs are of course a good solution:

!EM2C, CORIA, IMFT, IFP, CERFACS produce more than 20 PhD per year. They go to industry and they know HPC.

!But this is not fast enough

• Systematic courses on HPC/CFD/Combustion are required: in 2011, formation cycles at CERFACS

49

!"#$%&'()*#$+,-./*0-%*1(%2$*3..4*5&#")(6-7*"/&72*89:;*

!"#!$%&'(&)*+%,"!!%,#-%./+01%,"!,%

<-.$*'-"=)&72*"/&72*>=$7;81?*%!2#!3%&'(&)*+%,"!!%!4#,"%5.67.+8%,"!,%!9#!:%576*%,"!,%

8.@(7'$.*#$/,*2$7$%(6-7*"/&72*<3!A8BC* ,-#,2%5.67.+8%,"!,%

D(+(*(//&#&)(6-7*"/&72*>=$7;81?*9"%5.67.+8#!;(%<*)+7.+8%

,"!,%

;%-2%(##&72*&7*E-%+%(7*FG*** !9#!$%=.+'>%,"!,%

<-#="+$%*(%',&+$'+"%$/*(7.*-=6#&/(6-7* ,,%=.+'>%,"!,%

;(%())$)*=%-2%(##&72*+--)/* ,9#%,$%./+01%,"!,%

D-H7/'()&72*')&#(+$*/'$7(%&-/*H&+,*+,$*H$(+,$%I+4=&72*J(/$.*#$+,-.-)-24*D5<1K?*

,:#,-%./+01%,"!,%

5-)"6-7*#$+,-./*0-%*-=6#&/(6-7*=%-J)$#/* 3#!!%=.8%,"!,%

E"7.(#$7+()/*+-*"7.$%/+(7.*(7.*(7()4L$*,&2,*M.$)&+4*<ED*/&#")(6-7/*

,!#,:%=.8%,"!,%

<-.$*'-"=)&72*"/&72*>85K5* %9"#9!%=.8#!;(%576*%,"!,%

!"#$%&'()*#$+,-./*0-%*+,$*$)$'+%-#(27$6'*=%-J)$#*"/&72*<35<*

-#4%%576*%,"!,%

5-)"6-7*#$+,-./*0-%*)&7$(%*/4/+$#/* !!#!,%576*%,"!,%

*K7+%-."'6-7*+-*.(+(*(//&#&)(6-7* !4#!3%576*%,"!,%

5-)"6-7/*-0*=(%6()*.&N$%$76()*$O"(6-7/P*(==)&'(6-7/*+-*+,$*H(@$*=%-=(2(6-7*=%-J)$#*

,:%#%,-%%576*%,"!,%

8.@(7'$.*()2-%&+,#/*0-%*.(+(*(//&#&)(6-7* ,2#,3%576*%,"!,%

!"#$%%&&&'()*+,(-'+*%.*,/0/01%

%%%%(+.0606?@'*+<.';A<+%

B6% ,"!!C% DEFGHDI% 060J.(*;% .% (+.0606?%/+&?+.=%<&+%'&=/7(*+%;'0*6'*%(.+?*J6?%;(7K*6(;C% L>M;C% *6?06**+;% .6K%+*;*.+'>*+;C% 06% .'.K*=0'% .6K% .//10*K%+*;*.+'>A%%N>*% (+.0606?% ;*;;0&6;% .+*% (.7?>(% )8%DEFGHDI%+*;*.+'>*+;%.6K%*6?06**+;A%%L+.'J'.1% (+.0606?% ;*;;0&6;%K*K0'.(*K% (&%(>*% '&K*;% K*O*1&/*K% &+% '&#K*O*1&/*K%)8% DEFGHDI% .+*% /.+(% &<% (>*% /+&?+.=A%DEFGHDI% 0;% .7(>&+0P*K% (&% ?0O*% (+.0606?%;*;;0&6;% Q0(>06% (>*% B6K0O0K7.1% +0?>(% (&%(+.0606?%RMBGSA%

50

www.cerfacs.fr/elearning/combustion/

51

CONCLUSIONS• DANS 10 ANS, LES GROUPES QUI AURONT CONTINUE A

DEVELOPPER DES CODES DE PETITE TAILLE EN COMBUSTION, AURONT BEAUCOUP DE MAL A PUBLIER MAIS L’EFFORT POUR CONSTRUIRE DES GRANDS CODES EST ENORME

• SUCCESS: REPONSE ‘LOGICIEL’ DES COMBUSTIONNISTES FRANCAIS. COUVRE LES LABOS ET LES INDUSTRIELS

• L’ARRIVEE DE GENCI A PERMIS A LA COMMUNAUTE ‘COMBUSTION NUMERIQUE’ DE DISPOSER DES MOYENS NÉCESSAIRES POUR DEMONTRER SON SAVOIR FAIRE

• PRACE ET INCITE COMPLETENT CE DISPOSITIF

• LA COMBUSTION EST UN EXEMPLE OU CES OUTILS HPC SONT UTILISES AUJOURD’HUI PAR L’INDUSTRIE QUI SUPPORTE LEUR DEVELOPPEMENT DE FACON TRES FORTE

52