Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large...

79
1 Experiences with Experiences with Large Scale Numerical Simulation Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) www10.informatik.uni-erlangen.de Dundee, June 28, 2005 Zur Anzeige wird der QuickTime Dekompressor Cinepak bentigt. Ulrich Rüde ([email protected])

Transcript of Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large...

Page 1: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

1

Experiences withExperiences withLarge Scale Numerical Simulation Large Scale Numerical Simulation

Lehrstuhl für Informatik 10

(Systemsimulation)

www10.informatik.uni-erlangen.de

Dundee, June 28, 2005

Zur Anzeige wird der QuickTime Dekompressor Cinepak

bentigt.

Ulrich Rüde

([email protected])

Page 2: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

2

OverviewOverview

MotivationThree examples

Material science and process technology:Metal FoamsNano TechnologyBiomedical Technology:The Inverse EEG problem

High End ComputingTrends in High End Numerical ComputingParallel Hierarchical Hybrid Grids (HHG) for FE simulationsParallel Lattice Boltzmann Methods for Free Surface Flow

Conclusions

Page 3: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

3

Part I

Motivation:Motivation:Computational ScienceComputational Science and Engineering (CSE) and Engineering (CSE)

Page 4: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

4

MotivationMotivation

“The panels overarching finding is that a new age has dawned in scientific and engineering

research …”

(from the “NSF report on Cyberinfrastructure”, Feb. 2003)

…. this revolution is driven by

Simulation for Technology and ScienceSimulation for Technology and Science

Page 5: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

5

PITAC Report to the US President on PITAC Report to the US President on

Computational ScienceComputational ScienceJune 2005June 2005

PRINCIPAL FINDING

Computational science is now indispensable to the solution of complex problems in every sector, from traditional science and engineering domains to such key areas as national security, public health, and economic innovation. Advances in computing and connectivity make it possible to develop computational models and capture and analyze unprecedented amounts of experimental and observational data to address problems previously deemed intractable or

beyond imagination.

Page 6: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

6

The Two Principles of ScienceThe Two Principles of Science

TheoryTheoryMathematical Mathematical Models, Differential Models, Differential Equations, NewtonEquations, Newton

ExperimentsExperimentsObservation and Observation and prototypesprototypes

empirical Sciencesempirical Sciences

Computational ScienceComputational Science

Simulation, OptimizationSimulation, Optimization

(quantitative) virtual Reality(quantitative) virtual Reality

ThreeThree

Page 7: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

7

ComputationalComputational ScienceScience and and EngineeringEngineering

ScienceScienceTechnologyTechnology

TheorTheoryy Observation Observation Experiment Experiment PrototypesPrototypes

ComputationComputationComputer SimulationComputer Simulation

VirtuVirtuaal l ExperimentsExperimentsVirtual PrototypesVirtual Prototypes

Virtual Virtual RealitRealityy

AlgorithmicAlgorithmic Modelling forModelling for PPhysihysicscs,, C Chemihemistry;stry;

Electrical Mechanical, Chemical Electrical Mechanical, Chemical Engineering; Material SciencesEngineering; Material Sciences

Bio- and Medical SciencesBio- and Medical Sciences,,……

Page 8: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

8

CSE is a broad CSE is a broad multidisciplinarymultidisciplinary area that encompasses area that encompasses applicationsapplications in science/engineering, applied mathematics, in science/engineering, applied mathematics, numerical analysis, and computer science. numerical analysis, and computer science. Computer models Computer models and computer simulationsand computer simulations have become an important part of the have become an important part of the research repertoire, supplementing (and in some cases research repertoire, supplementing (and in some cases replacing) experimentation. Going from application area to replacing) experimentation. Going from application area to computational resultscomputational results requires domain expertise, requires domain expertise, mathematical mathematical modeling, numerical analysis, algorithm development, software modeling, numerical analysis, algorithm development, software implementation, program execution, analysis, validation and implementation, program execution, analysis, validation and visualization of resultsvisualization of results. CSE involves all of this. CSE involves all of this..

SIAM’SIAM’ss Definition Definition ofof CSE CSEhttp://www.siam.org/cse/report.htmhttp://www.siam.org/cse/report.htm

Page 9: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

9

CSE makes use of the techniques of applied mathematics and computer CSE makes use of the techniques of applied mathematics and computer science for the science for the development of problem-solving methodologiesdevelopment of problem-solving methodologies and and robust tools which will be the building blocks for solutions to scientific robust tools which will be the building blocks for solutions to scientific and engineering problems of ever-increasing complexity. It and engineering problems of ever-increasing complexity. It differs from differs from mathematics or computer sciencemathematics or computer science in that analysis and methodologies are in that analysis and methodologies are directed directed specificallyspecifically at the solution of problem classes from at the solution of problem classes from science and science and engineeringengineering, and will generally require a detailed knowledge or , and will generally require a detailed knowledge or substantial substantial collaborationcollaboration from those disciplines. The computing and from those disciplines. The computing and mathematical techniques used may be more domain specific, and the mathematical techniques used may be more domain specific, and the computer science and mathematics skills needed will be broader.computer science and mathematics skills needed will be broader. It is It is more thanmore than a scientist or engineer a scientist or engineer using a canned codeusing a canned code to generate and visualize results (skipping all of the to generate and visualize results (skipping all of the intermediate steps).intermediate steps).

SIAM's Definition of CSESIAM's Definition of CSE (2) (2)What is it NOT!What is it NOT!

Page 10: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

10

Part IIa

Metal Foams Metal Foams

In collaboration with theIn collaboration with theInstitut für Werkstoffwissenschaften Institut für Werkstoffwissenschaften

Lehrstuhl Werkstoffkunde und Technologie der Metalle Lehrstuhl Werkstoffkunde und Technologie der Metalle WTM (R.F. Singer, WTM (R.F. Singer, C. KörnerC. Körner))

Page 11: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

11

GlassCeramics

MetalsPolymers

Structural Properties stiffness

energy absorption damping

Functional Properties burner, shock absorber,

heat exchanger, batteries

large, dynamic surface expansion

Examples of FoamsExamples of Foams

Page 12: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

12

Towards Simulating Metal FoamsTowards Simulating Metal Foams

Bubble growth, Bubble growth, coalescence, collapse, coalescence, collapse, drainage,drainage, rheology, etc. are rheology, etc. are still poorly understoodstill poorly understood

• Simulation as a tool to Simulation as a tool to better understand, control better understand, control and optimize the processand optimize the process

Page 13: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

13

The Lattice-Boltzmann MethodThe Lattice-Boltzmann Method

Based on cellular automataIntroduced by von Neumann around 1940

Famous: Conway’s Game of Life

Complex system with simple rulesRegular grid

Local rules specifying time evolution

Intrinsically parallel for model & simulation, similar to elliptic PDE solvers

Page 14: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

14

The Lattice-Boltzmann MethodThe Lattice-Boltzmann Method

Weakly compressible approximation of the Navier-Stokes equations

Easy implementation

Applicable for small Mach numbers (< 0.1)

Easy to adapt, e.g. forComplicated or time-varying geometries

Free surfaces

Additional physical and chemical effects

Page 15: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

15

The Lattice-Boltzmann MethodThe Lattice-Boltzmann MethodReal valued representation of particles

Discrete velocities and positions

Algorithm consists of two steps:

Stream

Collide

Page 16: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

16

The Stream StepThe Stream Step

Move particle distribution functions along corresponding velocity vector

Normalized time step, cell size, and particle speed

Page 17: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

17

The Collide StepThe Collide Step

“Computes collisions” of particles in cell

Weigh equilibrium velocities and velocities from streaming depending on fluid viscosity

Page 18: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

18

LBM DemonstrationLBM Demonstration(Java applet)

file:///Users/ruede/doc/lehr/vorles/ws03/hppt/lbm/jlb-comp/start.html

Page 19: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

19

Free surfaces with LBMFree surfaces with LBM

Metal Foams – large gas volume

Only simulate and track fluid motion

Compute boundary conditions at free surface

Page 20: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

20

Free surface implementationFree surface implementation

Before stream step, compute mass exchange across cell boundaries for interface cells

Calculate bubble volumes and pressure

Surface curvature for surface tension

Change topology if interface cells become full or empty – keep layer of interface cells closed

Page 21: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

21

Curvature calculation (version I)Curvature calculation (version I)

Alternative approaches:

Integrate normals over surface (weighted triangles)

Level set methods (track surface as implicit function)

Page 22: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

22

Boundary ConditionsBoundary Conditions

Gas

Liquid

Problem: Missing distribution functions at interface cells after streaming!

Reconstruction such that macroscopic boundary conditions are satisfied.

Körner et al. Lattice Boltzmann Model for Free Surface Flow, to be published in Journal of Computational Physics

Page 23: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

23

Surface Tension (Vers. 2)Surface Tension (Vers. 2)

ΑΑΑ −=δ

Α

Α

1ν_3n

_

2n_

Marching-cube surface triangulationCompute a curvature for each triangle

Associate with each LBM cell the average curvature of its triangles

Complicated Beats level sets for our applications (mass conservation).

κ = 12

dA

dV

Page 24: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

24

Free surface flow: Breaking DamFree surface flow: Breaking Dam

Zur Anzeige wird der QuickTime Dekompressor YUV420 codec

bentigt.

Page 25: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

25

Rising BubblesRising Bubbles

Zur Anzeige wird der QuickTime Dekompressor YUV420 codec

bentigt.

Page 26: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

26

More Rising BubblesMore Rising Bubbles

Zur Anzeige wird der QuickTime Dekompressor YUV420 codec

bentigt.

Page 27: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

27

Simulation VerificationSimulation Verificationby Experimentby Experiment

Simulation and Experiment: Simulation and Experiment: N. ThüreyN. Thürey

Zur Anzeige wird der QuickTime Dekompressor YUV420 codec

bentigt.

Page 28: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

28

0,000

0,001

0,002

0,003

0,004

0,005

0,006

0,007

0,008

0 50 100

Distance in l.u.

Velocity

Numerical result

Analytical result

Stokes´ Law: Climbing rate of a bubble exposed to gravity

Climb rate

Ideal bubble No boundaries Equilibrium state

R = 8, τ = 0.74, g = 10-4, σ = 2*10-2100 x 100 x 140 cellsExample:

Rel. error: 2 %

Error = function of the system size

Verification for bubble dynamicsVerification for bubble dynamics(C. Körner)(C. Körner)

Zur Anzeige wird der QuickTime Dekompressor Cinepak

bentigt.

Page 29: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

29

True Foams with Disjoining PressureTrue Foams with Disjoining Pressure

Zur Anzeige wird der QuickTime Dekompressor Cinepak

bentigt.

Zur Anzeige wird der QuickTime Dekompressor

bentigt.

Page 30: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

30

VisualizationVisualization

Ray-tracingRefractionReflectionCausticsAbout 15 Min per frame

= 1 day for 4 secsAbout same compute time as flow simulation

Page 31: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

31

Part II b

Nanotechnology:Nanotechnology:Interacting Particles in a FluidInteracting Particles in a Fluid

Cooperation withCooperation withW. Peukert, H.J. Schmid W. Peukert, H.J. Schmid

(Chemical Engineering, Particle Technology)(Chemical Engineering, Particle Technology)

Page 32: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

32

Nano TechnologyNano TechnologyProperties of materials and products determined by structure of the nano-scale particles

Possible applications of the LBM:

Simulate the behavior of particles and particle agglomerates in solutions (e.g. breaking up or further agglomeration)

On a larger scale simulate segregation processes

Page 33: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

33

Nano TechnologyNano Technology

Curved Boundaries:Particles approximated with spheresImprove accuracy of LBM simulations by using curved boundary conditions

Standard No-SlipReflect DFs at cell boundary

More accurate:Take distance to boundary surface into account, then interpolate DFs accordingly

Page 34: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

34

Nanotechnology ApplicationsNanotechnology Applications

Fluid-Body Interaction:

Compute the forces acting upon a body due to the fluid flow around it

Integrate DFs towards the body for all cells on its surface

Body-Fluid Interaction:Bodies moving in the fluid

Modify outgoing DFs at the boundary with the surface velocity of the body

C. Feichtinger: Studienarbeit

Page 35: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

35

Nanotechnology ApplicationsNanotechnology ApplicationsMoving particle agglomerate in the flow

K. Iglberger - just completed Master Thesis Projectand Chr. Feichtinger - Studienarbeit

Zur Anzeige wird der QuickTime Dekompressor YUV420 codec

bentigt.

Zur Anzeige wird der QuickTime Dekompressor YUV420 codec

bentigt.

Page 36: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

36

Part II c

Biomedical EngineeringBiomedical Engineering

The Inverse EEG ProblemThe Inverse EEG Problem

with M. Mohr and C. Freundlwith M. Mohr and C. Freundl

Page 37: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

37

Bio-electric Field ComputationsBio-electric Field Computations

Reconstruction of electromagnetic fieldsReconstruction of electromagnetic fields from EEG-Measurements:from EEG-Measurements:

Source LocalizationSource Localization

NeurosurgeryNeurosurgeryKopfklinikum ErlangenKopfklinikum Erlangen

View throughView throughoperation microscopeoperation microscope

Page 38: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

38

Why simulate and not experiment?Why simulate and not experiment?

Open brain EEG-Open brain EEG-Measurements for Measurements for

LocalizingLocalizing functional brain regions functional brain regions

Simulation basedSimulation basedVirtual operation planningVirtual operation planning

Image: Chr. Johnson, Salt Lake City

Page 39: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

39

Problem of inverse EEG/MEGProblem of inverse EEG/MEGDirect Problem: Direct Problem:

• Known:Known: Sources (strength, Sources (strength, position, orientation)position, orientation)

• Wanted:Wanted: Potentials on the Potentials on the head surfacehead surface

• Inverse ProblemInverse Problem• Known:Known: Potentials on the Potentials on the

head surfacehead surface• Wanted:Wanted: Sources (strength, Sources (strength,

position, orientation)position, orientation)

Page 40: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

40

Source LocalizationSource Localization3D MRI data3D MRI data

512 x 512 x 256512 x 512 x 256voxelsvoxels

segmentationsegmentation4 compartmens4 compartmens

Localized Localized Epileptic focusEpileptic focus

Dipole localizationDipole localizationsearch algorithmsearch algorithm = optimization= optimization

Collaborators: Univ. of Utah (Chris Johnson), Ovidius Univ. Constanta (C. Popa)Collaborators: Univ. of Utah (Chris Johnson), Ovidius Univ. Constanta (C. Popa)Bart Vanrumste (Gent, Univ. of Canterbury, New Zealand), G. Greiner, F. FahlbuschBart Vanrumste (Gent, Univ. of Canterbury, New Zealand), G. Greiner, F. Fahlbusch

Page 41: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

41

Part IIIa

High Performance ComputingHigh Performance ComputingTrends in High End ComputingTrends in High End Computing

Page 42: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

42

Simulation isSimulation is

Performance hungry andMemory intensiveParallel Supercomputing required

Page 43: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

43

Supercomputer Performance: TOP 500 ListSupercomputer Performance: TOP 500 List

Zur Anzeige wird der QuickTime Dekompressor TIFF (Unkomprimiert)

bentigt.

Page 44: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

44

Zur Anzeige wird der QuickTime Dekompressor TIFF (Unkomprimiert)

bentigt.

Earth Simulator (Japan)36 TFlop

Zur Anzeige wird der QuickTime Dekompressor TIFF (Unkomprimiert)

bentigt.

Columbia Supercomputer(NASA)52 TFlops

IBM Blue Gene/L and its CompetitorsIBM Blue Gene/L and its Competitors

Zur Anzeige wird der QuickTime Dekompressor TIFF (Unkomprimiert)

bentigt.RochesterRochester71 TFlop71 TFlop

Page 45: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

45

Architecture example: Our Pet DinosaurArchitecture example: Our Pet Dinosaur

8 Proc and 8 GB per node8 Proc and 8 GB per node

Performance:Performance: 1344 CPUs (168*8) 1344 CPUs (168*8) 12 GFlop/node12 GFlop/node 2016 GFlop total2016 GFlop total Linpack: 1645 Gflop Linpack: 1645 Gflop

(82% of theoretical peak) (82% of theoretical peak) Very sensitive to data Very sensitive to data

structuresstructures To be replaced by a 6000 Proc. To be replaced by a 6000 Proc.

SGI in 1Q 2006;SGI in 1Q 2006; upgrade to >70 Tflop in 2007upgrade to >70 Tflop in 2007

Hitachi SR 8000Hitachi SR 8000 at the Leibniz-Rechenzentrum derat the Leibniz-Rechenzentrum der

Bayerischen Akademie der Bayerischen Akademie der WissenschaftenWissenschaften

(#5 at time of installation in 2000, now #273)(#5 at time of installation in 2000, now #273)

Page 46: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

46

LSS Cluster-ComputerLSS Cluster-Computer

Fujitsu-Siemens HPC LineFujitsu-Siemens HPC Line

Programming MethodsProgramming MethodsCache Optimization Cache Optimization

C++ Expression TemplatesC++ Expression Templates

(Parallel) Algorithms(Parallel) Algorithms

Cooperations inCooperations inMaterial SciencesMaterial Sciences

EngineeringEngineering• MechanicalMechanical• ElectricalElectrical• ChemicalChemical

Medical TechnologyMedical Technology

. . .. . .

Page 47: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

47

LSSLSS-Cluster-ClusterCompute Nodes (8x4 CPUs)Compute Nodes (8x4 CPUs)

CPU: AMD Opteron 848 2.2 GHz, max. 4.4 GFlops

RAM:16 GByte

Interactive Nodes (9x2 CPUs)Interactive Nodes (9x2 CPUs) CPU:

AMD Opteron 248

High-Speed Network InfiniBandHigh-Speed Network InfiniBand

10 GBit/s

Fujitsu-SiemensFujitsu-Siemens

Page 48: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

48

V10.000.000.000

1.000.000.000

100.000.000

10.000.000

1.000.000

100.000

1.000

10.000

100

4004

80368

Pentium

Merced

1K

64K

256K

1M4M

64M

1G

4G

1970 1975 1980 1985 1990 1995 2000 2005

Year

Tra

ns

isto

rs/D

ie

Microprocessor(Intel)

DRAM

Growth:42% per year

Growth:52% per year

Moore's Law in Semiconductor Technology(F. Hossfeld)

80468

Pentium Pro

Page 49: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

49

1021

1018

1015

1012

109

1024

103

106

1

1950 1960 1970 1980 1990 2000 2010 2020

Year

Ato

ms/

Bit

Information Density & Energy Dissipation(adapted by F. Hossfeld from C. P. Williams et al., 1998)

10 -9

10 -6

10 -3

1

10 3

10 6

10 9

1012

1015

En

erg

y/lo

gic

Op

era

tion

[p

ico

-Jou

les]

kT

Semiconductor Technology

≈ 2017

Page 50: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

50

Current Challenge:Current Challenge:Parallelism on all levels andParallelism on all levels and

The Memory WallThe Memory Wall

Parallel computing is easy, good (single) processor performance is Parallel computing is easy, good (single) processor performance is difficult (B. Gropp, Argonne)difficult (B. Gropp, Argonne)

There has been no significant progress in High Performance Computing There has been no significant progress in High Performance Computing

over the past 5 years (H. Simon, NERSC)over the past 5 years (H. Simon, NERSC)

Instruction level parallelismInstruction level parallelism

Memory bandwidth and latencyMemory bandwidth and latency are the limiting factors are the limiting factors

Cache-aware algorithmsCache-aware algorithms

Conventional Conventional complexity measurescomplexity measures (based on operation count) are (based on operation count) are

becoming increasingly becoming increasingly unrealisticunrealistic..

Page 51: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

51

Part IIIb

High Performance ComputingHigh Performance ComputingHierarchical Hybrid Grids (HHG)Hierarchical Hybrid Grids (HHG)

with B. Bergen and F. Hülsemannwith B. Bergen and F. Hülsemann

Page 52: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

52

Structured vs. Unstructured Grids(on Hitachi SR 8000)

gridlib/HHG MFlops rates for matrix-vector multiplication on one node on the Hitachi

compared with highly tuned JDS results for sparse matrices (courtesy of G. Wellein, RRZE Erlangen)

0

1000

2000

3000

4000

5000

6000

7000

8000

729 35,937 2,146,689 # unknowns

JDSStencils

Page 53: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

53

What are hierarchical hybrid grids?What are hierarchical hybrid grids?

Standard geometric multigrid approach:Standard geometric multigrid approach:Purely unstructured input gridPurely unstructured input grid

resolves geometry of problem domainresolves geometry of problem domainPatch-wise regular refinementPatch-wise regular refinement

applied repeatedly to every cell of the coarse gridapplied repeatedly to every cell of the coarse gridgenerates nested grid hierarchies naturally suitable generates nested grid hierarchies naturally suitable for geometric multigrid algorithmsfor geometric multigrid algorithms

New: New: Modify storage formats and operations on the grid to Modify storage formats and operations on the grid to exploit the exploit the regular substructuresregular substructures

Page 54: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

54

Common misconceptionsCommon misconceptions

Hierarchical hybrid grids (HHG) Hierarchical hybrid grids (HHG) are not yet another block structured gridare not yet another block structured grid

HHG are more flexible (HHG are more flexible (unstructured, hybrid unstructured, hybrid input gridsinput grids))

are not yet another unstructured geometric multigrid are not yet another unstructured geometric multigrid packagepackage

HHG achieve better performance -- HHG achieve better performance -- unstructured treatment of regular regions does unstructured treatment of regular regions does not improve performancenot improve performance

Page 55: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

55

Refinement example

Input Grid

Page 56: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

56

Refinement example

Refinement Level one

Page 57: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

57

Refinement example

Refinement Level Two

Page 58: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

58

Refinement example

Structured InteriorStructured Interior

Page 59: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

59

Refinement example

Structured Interior

Page 60: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

60

Refinement example

Edge Interior

Page 61: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

61

Refinement example

Edge Interior

Page 62: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

62

Problems and Solutions

Problems with C++ on HitachiOnly alpha version quality of C++

Excessive compile times

Poor code quality

Solution for gridlib-HHGconservative C++

resorting to mixed language programming C++/F77 (after painful experience)

Page 63: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

63

Results, Scaling, EfficiencyResults, Scaling, Efficiency(results by F. Hülsemann, Ben Bergen)(results by F. Hülsemann, Ben Bergen)

Brick-shaped Finite elementsBrick-shaped Finite elements

Poisson equationDirichlet boundary conditionsMultigrid FMG(2,2) cycle27 point stencil9 cubes/processrefinement level 7 (h=1/128)

Speedup for the same problem

(6 times regularly refined)

4810139.49550

459438.94512

444719.47256

442359.74128

441179.4864

Time (s)Dof x 106#CPU

0

5

10

15

20

25

30

35

0 20 40 60 80Number of processes

Linear Observed

Page 64: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

64

Performance on SGI Altix forPerformance on SGI Altix for Tetrahedral HHG-Mesh Tetrahedral HHG-Mesh

Scale-up test: Compute time vs. Proc #Problem size

2 Proc: 66 Mio unknowns512 & 1024 Proc: 17,000 Mio. unknowns

Bergen, Hülsemann, UR: Is 1.7×1010 unknowns the largest Finite Element system that can be solved today? accepted to Supercomputing in Nov ´05.

Per Proc. Performance

Smoothing aloneComplete V(3,3) MG cycle

Page 65: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

65

Part IIIc

High Performance ComputingHigh Performance ComputingParallel Free Surface LBM-MethodsParallel Free Surface LBM-Methods

Page 66: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

66

Parallelization of LBM CodeParallelization of LBM CodeStandard LBM-Code in C (1-D Partitioning):

- excellent performance on single SR8000 node- almost linear speed-up- large partitions favorable

Performance on SR8000

Ca. 30% of Peak Performance

Page 67: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

67

ParallelisationParallelisation

Standard LBM-Code: Scalability

Largest Simulation:1,08*109 cells

370 GByte memory

Communication Cost because of large data volume (64 MByte)

Efficiency ~ 75%

Page 68: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

68

ParallelisationParallelisation

Free surface LBM-Code

Standard LBM Free surface LBM

1 sweep through grid 5 sweeps through grid

Cell type changes, Closed boundary for bubbles, Initialization of modified cells, Mass balance correction

Page 69: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

69

ParallelisationParallelisation

Free surface LBM-Code:

Standard LBM Free surface LBM

1 sweep through grid 5 sweeps through grid

1 row of ghost nodes 4 rows of ghost nodes

Page 70: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

70

PerformancePerformanceFree surface LBM-Code Free surface LBM-Code

Standard LBM-CodeStandard LBM-Code

Performance lousy on a single node! Conditionals: 2,9 SLBM 51 free surface LBMPentium 4: almost no degradation ~ 10%SR 8000: enormous degradation (pseudo-vector, predictable jumps)

Page 71: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

71

Part IV

ConclusionsConclusions

Page 72: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

72

Conclusions (1)Conclusions (1)High performance simulation still requires

“heroic programming”Parallel Programming is easy, node performance is difficult (B. Gropp, Argonne)Which architecture ?

ASCI-type: custom CPU, massively parallel cluster of SMPs• nobody has been able to show that these machines scale efficiently,

except on a few very special applications and using enormous human effort

Earth-simulator-type: Vector CPU, as many CPUs as affordable• impressive performance on vectorizable code, but need to check with

more demanding data and algorithm structuresHitachi Class: modified custom CPU, cluster of SMPs

• excellent performance on some codes, but unexpected slowdowns on others, too exotic to have a sufficiently large software base

Others: BlueGene, Cray X1, Multithreading, PIM, reconfigurable …, quantum computing, …

Page 73: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

73

Conclusions (2)Conclusions (2)Which data structures?

structured (inflexible) unstructured (slow)HHG (high development effort, even prototype 50 K lines of code)meshless … (useful in niches)

Where are we going?the end of Moore’s lawnobody builds CPUs with numerical simulation requirements high on the list of priorities.petaflops: 100,000 processors and we can hardly handle 1000It’s the locality - stupid!the memory wall

• latency• bandwidth

Distinguish between algorithms where control flow is• data independent: latency hiding techniques(pipelining, prefetching, etc)

can help• data dependent

Page 74: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

74

In the Future?In the Future?

What’s beyond Moore’s Law?What’s beyond Moore’s Law?

Page 75: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

75

Part VI

Outlook: Other applicationsOutlook: Other applications

Computational Steering andComputational Steering andReal-Time simulationReal-Time simulation

Page 76: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

76

Near-Real-Time Free-Surface LBMNear-Real-Time Free-Surface LBM

Zur Anzeige wird der QuickTime Dekompressor

bentigt.

Page 77: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

77

AcknowledgementsAcknowledgementsCollaborators

In Erlangen: WTM, LSE, LSTM, LGDV, LFG, RRZE, etc.Especially for foams: C. Körner (WTM)International: Utah, Technion, Constanta, Ghent, Boulder, ...

Dissertationen ProjectsB. Bergen (HHG)

C. Freundl (Parallel Expression Templates for PDE-solver)

J. Härtlein (Expression Templates for FE-Applications)

N. Thürey (LBM, free surfaces)

... and 6 more

16 Diplom- /Master- ThesisStudien- /Bachelor- Thesis

Especially for Performance-Analysis/ Optimization of the LBM• J. Wilke, K. Iglberger, S. Donath

... and 24 more

KONWIHR, DFG, NATO, BMBFKONWIHR, DFG, NATO, BMBFElitenetzwerk BayernElitenetzwerk Bayern

Bavarian Graduate School in Computational Engineering (with TUM, Jan. 2005)Bavarian Graduate School in Computational Engineering (with TUM, Jan. 2005)Special International PhD program: Identifikation, Optimization and Optimal Control Special International PhD program: Identifikation, Optimization and Optimal Control for Engineering Applications (with Bayreuth and Würzburg) starting for Engineering Applications (with Bayreuth and Würzburg) starting Jan. 06Jan. 06

Page 78: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

78

Talk is OverTalk is Over

Please wake up!Please wake up!

Page 79: Experiences with Large Scale Numerical Simulation · 2015. 5. 13. · 1 Experiences with Large Scale Numerical Simulation Lehrstuhl für Informatik 10 (Systemsimulation) Dundee, June

79

ASIMASIMAnnual ConferenceAnnual Conference

18. Symposium18. Symposium

Simulation Simulation TechniquesTechniques

12. - 15. September 200512. - 15. September 2005

in in ErlangenErlangen

www10.informatik.uni-erlangen.de/asim2005/

Zur Anzeige wird der QuickTime Dekompressor

bentigt.

Near-Real-Time Free-Surface LBMNear-Real-Time Free-Surface LBM