Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra...
Transcript of Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra...
![Page 1: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/1.jpg)
Challenges in fluid flow simulations using
Exa-scale computingMahendra Verma
IIT Kanpur
http://turbulencehub.org
![Page 2: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/2.jpg)
![Page 3: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/3.jpg)
Hardware
![Page 4: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/4.jpg)
A Growth-Factor of a Billion in Performance in a Career
IBMBG/L
ASCI WhitePacific
EDSAC 1UNIVAC 1
IBM 7090CDC 6600
IBM 360/195CDC 7600
Cray 1
Cray X-MPCray 2
TMC CM-2
TMC CM-5 Cray T3D
ASCI Red
1950 1960 1970 1980 1990 2000 2010
1 KFlop/s
1 MFlop/s
1 GFlop/s
1 TFlop/s
1 PFlop/s
Scalar
Super Scalar
Parallel
Vector
1941 1 (Floating Point operations / second, Flop/s) 1945 100 1949 1,000 (1 KiloFlop/s, KFlop/s) 1951 10,000 1961 100,000 1964 1,000,000 (1 MegaFlop/s, MFlop/s) 1968 10,000,000 1975 100,000,000 1987 1,000,000,000 (1 GigaFlop/s, GFlop/s) 1992 10,000,000,000 1993 100,000,000,000 1997 1,000,000,000,000 (1 TeraFlop/s, TFlop/s) 2000 10,000,000,000,000 2005 131,000,000,000,000 (131 Tflop/s)
Super Scalar/Vector/Parallel
(103)
(106)
(109)
(1012)
(1015)
2X Transistors/Chip Every 1.5 Years
From Karniakadis’s course slides
2018
1018200 PF TITAN
![Page 5: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/5.jpg)
Flop rating for 2 procs: 2*32*24 = 1536 GF
https://www.amd.com/en/products/cpu/amd-epyc-7551
Wants data ~ 8 TB/sec.
Cache, RAM, HD
NODE: 2 proc/node; Focus on a node
![Page 6: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/6.jpg)
Memory BW = 341 GB/s
SSD: transfer rate = 6 Gbit/s
peak IB Switch speed/port = 200 Gb/s
FLOPS free, data transfer expensive (Saday)
Data transfer
![Page 7: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/7.jpg)
software challenges
![Page 8: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/8.jpg)
• Abundance (MPI, OpenMP, CUDA, ML)
• Leads to confusion and non-start..
• Structured programming
• Pressure to do the science..
• Some times CS tools are too complex to be practical.
For beginners
![Page 9: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/9.jpg)
For advanced users• Optimised use of hardware.
• Structured and modular, usable code with documentation.
• Keeping up with upgrades and abundance (MPI3, ML, C++11, Vector processors, GPU, XeonPhi, Rasberry Pi).
• Optimization
• Interactions with users + programers
![Page 10: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/10.jpg)
Now CFD (Computational fluid
dynamics)
![Page 11: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/11.jpg)
Applications
• Weather prediction and climate modelling
• Aeroplane and cars (transport)
• defence / offences
• Turbines, dams, water management
• Astrophysical flows
• Theoretical understanding
![Page 12: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/12.jpg)
Field reversalwith Mani Chandra
![Page 13: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/13.jpg)
Polarity reversals after random time intervals
(tens of millions of years to 50K years).
Last reversal took place around 780,000
years ago.
Glatzmaier & RobertsNature, 1995
Geomagnetism
![Page 14: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/14.jpg)
Nek5000 (Spectral-element) simulation
(1,1)➞(2,2) ➞(1,1)
Chandra & Verma, PRE 2011, PRL 2013
spectral-element code Nek5000
![Page 15: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/15.jpg)
Methods
• Finite difference
• Finite volume
• Finite element
• Spectral
• Spectral element
![Page 16: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/16.jpg)
Spectral method
![Page 17: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/17.jpg)
Example: Fluid solver
![Page 18: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/18.jpg)
Ext. ForcePressurevelocity
field
kinematic viscosity
Incompressibility ULνReynolds no =
∇⋅u = 0
∂tu+ (u ⋅∇)u = −∇p +ν∇2u+ F
![Page 19: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/19.jpg)
Procedure
![Page 20: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/20.jpg)
f (x) = f̂ (kx )kz∑ exp[i(kxx)]
df (x) / dx = [ikx f̂ (kx )kz∑ ]exp[i(kxx)]
![Page 21: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/21.jpg)
Time advance (e.g., Euler’s scheme)
Set of ODEs
ui(k, t + dt) = ui(k, t) + dt � RHS(u(k), t)
Stiff equation for small viscosity ν (use exponential trick)
dui (k)dt
= − jkmum (r)ui (r)! − jki p(k)−νk2ui (k)
ui (k,t + dt) = ui (k)+ dt ×RHSi (k,t)
![Page 22: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/22.jpg)
Nonlinear terms computation:
Fourier transforms take around 80% of total time.
(pseudo-spectral)
![Page 23: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/23.jpg)
Tarang = wave (Sanskrit)
Opensource, download from http://turbulencehub.org
One code to do many turbulence & instabilities problems
Spectral code (Orszag)
Chatterjee et al., JPDC 2018
VERY HIGH RESOLUTION (61443)Cores: 196692 of Shaheen II of KAUST
![Page 24: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/24.jpg)
Fluid MHD, Dynamo
Scalar Rayleigh-Bénard convection
Stratified flows Rayleigh-Taylor flow Liquid metal flows
Rotating flow Rotating convection
Periodic BC Free-slip BC
Instabilities Chaos
Turbulence
No-slip BC Cylinder sphere Toroid
(in progress)
![Page 25: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/25.jpg)
Rich libraries to compute Spectrum
Fluxes Shell-to-shell transfer Structure functions
Tested up to 61443 grids
New things Fourier modes
Real space probes Ring-spectrum
Ring-to-ring transfer
![Page 26: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/26.jpg)
Object-oriented design
![Page 27: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/27.jpg)
We can use these general functions to simulate MHD, convection etc.
Basis-independent universal function (function overloading)
e.g., compute_nlin (u. ∇)u, (b. ∇)u, (b. ∇)b, (u. ∇)T.
Basis functions (FFF, SFF, SSF, SSS, ChFF)
General PDE solver
![Page 28: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/28.jpg)
Generated by Doxygen
![Page 29: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/29.jpg)
Parallelization
![Page 30: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/30.jpg)
Spectral Transform (FFT, SFT, Chebyshev)
Multiplication in real space
Input/Output HDF5 lib
![Page 31: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/31.jpg)
FFT Parallelizationf (x, y, z) = f̂ (kx ,ky ,kz )
kz∑
ky∑
kx∑ exp[i(kxx + kyy + kzz)]
![Page 32: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/32.jpg)
Slab decomposition
Data divided among 4 procs
![Page 33: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/33.jpg)
Inter-process
Communication
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
Nx
Ny
Nx
Ny
Transpose-free FFT
12-15% faster compared to FFTW
p0
p1
p0 p1
MPI vector, conconsecutive data transfer
![Page 34: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/34.jpg)
Pencil decomposition
![Page 35: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/35.jpg)
FFT scalingOn Shaheen 2 at KAUST
with Anando Chatterjee, Abhishek Kumar, Ravi Samtaney, Bilel Hadri, Rooh Khurram
Cray XC40 ranked 9th in top500
Chatterjee et al., JPDC 2018
![Page 36: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/36.jpg)
7683
15363
30723
n0.7
p1
![Page 37: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/37.jpg)
Tarang scalingOn Shaheen at KAUST
![Page 38: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/38.jpg)
• Weak scaling: When we increase the size of the problem, as well as number of procs, then should get the same scaling.
![Page 39: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/39.jpg)
Average flop rating/core (~1.5 %)
Overlap Communication & Computation ??
GPUs ??
Xeon Phi ??
Compare with BlueGene/P (~8 %)
![Page 40: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/40.jpg)
To Petascale & then Exascale
![Page 41: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/41.jpg)
Finite difference code
General code: Easy porting to GPU, MiC
Collaborators: Roshan Samuel
Fahad Anwer (AMU) Ravi Samtaney (KAUST)
![Page 42: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/42.jpg)
Summary★Code development
★Module development
★Optimization
★Porting to large number of processors
★GPU Porting
★ Testing
![Page 43: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/43.jpg)
AcknowledgementsStudents:
Anando Chatterjee
Abhishek Kumar
Roshan Samuel
Sandeep Reddy
Mani Chandra
Sumit Kumar & Vijay
Faculty:
Ravi Samtaney
Fahad Anwer
Ported to:
PARAM, CDAC
Shaheen, KAUST
HPC system IITK
Funding
Dept of Science and Tech., India
Dept of Atomic Energy, India
KAUST (computer time)
![Page 44: Challenges in fluid flow simulations using Exa-scale computing · Sandeep Reddy Mani Chandra Sumit Kumar & Vijay Faculty: Ravi Samtaney Fahad Anwer Ported to: PARAM, CDAC Shaheen,](https://reader033.fdocuments.in/reader033/viewer/2022060801/6085ef1ea28e1e7f06416b41/html5/thumbnails/44.jpg)
Thank you!