Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf ·...
Transcript of Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf ·...
![Page 1: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/1.jpg)
Numerical Modelling in Fortran: day 8Paul Tackley, 2017
![Page 2: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/2.jpg)
Today’s Goals
1. Introduction to parallel computing (applicable to Fortran or C; examples are in Fortran)
2. Finite Prandtl number convection
![Page 3: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/3.jpg)
Motivation: To model the Earth, need a huge number of grid points / cells
/elements!• e.g., to fill mantle volume:
– (8 km)3 cells -> 1.9 billion cells
– (2 km)3 cells -> 123 billion cells
![Page 4: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/4.jpg)
Huge problems => huge computer
www.top500.org
![Page 5: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/5.jpg)
Huge problems => huge computer
www.top500.org
![Page 6: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/6.jpg)
![Page 7: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/7.jpg)
Progress: iPhone > fastest computer in 1976 (cost: $8
million)
(photo taken at NCAR museum)
![Page 8: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/8.jpg)
![Page 9: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/9.jpg)
In Switzerland
Each node: 12-core Intel CPU + GPU
Each node: 2* 18-core Intel CPU
Piz Dora
![Page 10: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/10.jpg)
Shared memory: several cpus (or cores) share the same memory. Parallelisation can often be done by the compiler (sometimes with help, e.g., OpenMP instructions in the code)
Distributed memory: each cpu has its own memory. Parallelisation usually requires message-passing, e.g. using MPI (message-passing interface)
![Page 11: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/11.jpg)
A brief history of supercomputers
1983-54 CPUs,Shared memory
1991: 512 CPUs, distributed memory
2010: 224,162 Cores, distributed + shared memory (12 cores per node)
![Page 12: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/12.jpg)
Another possibility: build you own (“Beowulf” cluster)
Using standard PC cases: or using rack-mounted cases
![Page 13: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/13.jpg)
MPI: message-passing interface• A standard library for communicating
between different tasks (cpus)– Pass messages (e.g., arrays)– Global operations (e.g., sum, maximum)– Tasks could be on different cpus/cores of the
same node, or on different nodes• Works with Fortran and C• Works on everything from a laptop to the
largest supercomputers. 2 versions are:– http://www.mcs.anl.gov/research/projects/mpic
h2/– http://www.open-mpi.org/
![Page 14: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/14.jpg)
How to parallelise a code: worked example
![Page 15: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/15.jpg)
Example: Scalar Poisson eqn.
�
∇2u = fFinite-difference approximation:
�
1h 2 ui+1 jk + ui−1 jk + uij+1k + uij−1k + uijk+1 + uijk+1 − 6ui, j( ) = fij
Use iterative approach=>start with u=0, sweep through grid updatingu values according to:
�
˜ u ijn +1 = ˜ u ij
n + αRijh2
6
Where Rij is the residue (“error”):
�
R = ∇2 ˜ u − f
![Page 16: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/16.jpg)
Code
![Page 17: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/17.jpg)
Parallelisation:domaindecomposition
CPU 0 CPU 1
CPU 3CPU 2
CPU 5CPU 4
CPU 6 CPU 7
Single CPU 8 CPUs
Each CPU will do the same operations but on different parts of the domain
![Page 18: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/18.jpg)
You need to build parallelization into the code using MPI
• Any scalar code will run on multiple CPUs, but will produce the same result on each CPU.
• Code must first setup local grid in relation to global grid, then handle communication
• Only a few MPI calls needed:– Init. (MPI_init,MPI_com_size,MPI_com_rank)– Global combinations (MPI_allreduce)– CPU-CPU communication (MPI_send,MPI_recv…)
![Page 19: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/19.jpg)
Boundaries• Whenupdatingpointsatedgeofsubdomain,needvaluesonneighboringsubdomains
• Holdcopiesoftheselocallyusing“ghostpoints”
• Thisminimizes#ofmessages,becausetheycanbeupdatedallatonceinsteadofindividually
=ghost points
![Page 20: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/20.jpg)
ScalarGrid
Red=boundary points (=0)Yellow=iterated/solved
(1…n-1)
![Page 21: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/21.jpg)
Parallelgrids
Red=ext. boundariesGreen=int. boundariesYellow=iterated/solved
![Page 22: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/22.jpg)
First things the code has to do:
• Call MPI_init(ierr)• Find #CPUs using MPI_com_size• Find which CPU it is, using MPI_com_rank
(returns a number from 0…#CPUs-1)• Calculate which part of the global grid it is
dealing with, and which other CPUs are handling neighboring subdomains.
![Page 23: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/23.jpg)
Example: “Hello world” program
![Page 24: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/24.jpg)
Moving forward
• Update values in subdomain using ‘ghost points’ as boundary condition, i.e.,– Timestep (explicit), or– Iteration (implicit)
• Update ghost points by communicating with other CPUs
• Works well for explicit or iterative approaches
![Page 25: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/25.jpg)
Boundarycommunication
Step 1: x-faces
Step 2: y-faces (including corner values from step 1)
[Step 3: z-faces (including corner values from steps 1 & 2)]
Doing the 3 directions sequentially avoids the need for additional messages to do edges & corners (=>in 3D, 6 messages instead of 26)
![Page 26: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/26.jpg)
![Page 27: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/27.jpg)
![Page 28: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/28.jpg)
![Page 29: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/29.jpg)
Main changes
• Parallelisation hidden in set_up_parallelisation and update_sides
• Many new variables to store parallelisation information
• Loop limits depend on whether global domain boundary or local subdomain
![Page 30: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/30.jpg)
![Page 31: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/31.jpg)
Simplest communication
Not optimal – uses blocking send/receive
![Page 32: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/32.jpg)
Better: using non-blocking (isend/irecv)
![Page 33: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/33.jpg)
Performance: theoretical analysis
![Page 34: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/34.jpg)
How much time is spent communicating?
• Computation time ∝ volume (Nx^3)• Communication time ∝ surf. area (Nx^2)• =>Communication/Computation ∝ 1/Nx
• =>Have as many points/cpu as possible!
![Page 35: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/35.jpg)
Is it better to split 1D, 2D or 3D?• E.g., 256x256x256 points on 64 CPUs• 1D split: 256x256x4 points/cpu
– Area=2x(256x256)=131,072• 2D split: 256x32x32 points/cpu
– Area=4x(256x32)=32,768• 3D split:64x64x64 points/cpu
– Area=6x(64x64)=24,576• =>3D best but more messages needed
![Page 36: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/36.jpg)
Model code performance
Computation : t=aN3
Communication �: t=nL+bN2 /B(L=Latency, B=bandwidth)
TOTAL �: t= aN3 +nL+bN2 /B
(Time per step or iteration)
![Page 37: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/37.jpg)
Example: Scalar Poisson equation
t= aN3 +nL+bN2/B
�
∇2u = f
Assume 15 operations/point/iteration & 1 Gflop performanceÞa=15/1e9=1.5e-8
If 3D decomposition, n=6, b=6*4 (single precision)
Gigabit ethernet: L=40e-6 s, B=100 MB/sQuadrics: L=2e-6 s, B=875 MB/s
![Page 38: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/38.jpg)
Time/iteration vs. #cpusQuadrics, Gonzales-size cluster
![Page 39: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/39.jpg)
Up to 2e5 CPUs (Quadrics communication)
![Page 40: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/40.jpg)
Efficiency
![Page 41: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/41.jpg)
Now multigrid V cycles
4x4x4
8x8x8
16x16x16
32x32x32
Exact solution
corre
ctio
ns
Resid
ues (
=erro
r)
Smooth
Smooth
Smooth
![Page 42: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/42.jpg)
Application to StagYYCartesian or spherical
![Page 43: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/43.jpg)
StagYY iterations:
3D Cartesian
Simple-minded multigrid:Very inefficient coarse levels!Exact coarse solution can takelong time!
Change in scaling from same-node to cross-nodecommunication
![Page 44: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/44.jpg)
New treatment: follow minima• Keep #points/core > minimum (tuned for system)• Different for on-node and cross-node communication
![Page 45: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/45.jpg)
Multigrid – now (& before): yin-yang
1.8 billion
![Page 46: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/46.jpg)
Summary• For very large-scale problems, need to
parallelise code using MPI• For finite-difference codes, the best method is
to assign different parts of the domain to different CPUs (“domain decomposition”)
• The code looks similar to before, but with some added routines to take care of communication
• Multigrid scales fine on 1000s CPUs if:– Treat coarse grids on subsets of CPUs– Large enough total problem size
![Page 47: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/47.jpg)
For more information
• https://computing.llnl.gov/tutorials/parallel_comp/
• http://en.wikipedia.org/wiki/Parallel_computing
• http://www.mcs.anl.gov/~itf/dbpp/• http://en.wikipedia.org/wiki/Message_Pa
ssing_Interface
![Page 48: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/48.jpg)
Programming: Finite Prandtl number convection
(i.e., almost any fluid)
Ludwig Prandtl (1875-1953)
![Page 49: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/49.jpg)
Values of the Prandt number Pr
• Liquid metals: 0.004-0.03• Air: 0.7• Water: 1.7-12• Rock: ~1024 !!! (effectively infinite)
Pr = νκ
Viscous diffusivity
Thermal diffusivity
![Page 50: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/50.jpg)
Finite-Prandtl number convection
• Existing code assumes infinite Prandtl number– also known as Stokes flow– appropriate for highly-viscous fluids like
rock, honey etc.• Fluids like water, air, liquid metal have a
lower Prandtl number so equations must be modified
![Page 51: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/51.jpg)
Applications for finite Pr
• Outer core (geodynamo)• Atmosphere• Ocean• Anything that’s not solid like the mantle
![Page 52: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/52.jpg)
Equations• Conservation of mass (=‘continuity’)• Conservation of momentum (‘Navier-
Stokes’ equation: F=ma for a fluid)• Conservation of energy
Claude Navier(1785-1836)
Sir George Stokes(1819-1903)
![Page 53: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/53.jpg)
Finite Pr Equations
�
∇ ⋅ v = 0
ρ ∂ v
∂ t+ v ⋅∇v⎛
⎝⎜⎞⎠⎟ = −∇P + ρν∇2v + 2ρ
Ω× v + gραTy
�
∂T∂t
+ v ⋅ ∇T = κ∇2T + Q
Valid for constant viscosity only
ρ=density, ν=kinematic viscosity, g=gravity, α=thermal expansivity
Coriolis force
“ma”
Navier-Stokes equation: F=ma for a fluid
continuity and energy equations same as before
![Page 54: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/54.jpg)
Non-dimensionalise the equations
• Reduces the number of parameters• Makes it easier to identify the dynamical
regime• Facilitates comparison of systems with
different scales but similar dynamics (e.g., analogue laboratory experiments compared to core or mantle)
![Page 55: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/55.jpg)
Non-dimensionalise to thermal diffusion scales
• Lengthscale D (depth of domain)• Temperature scale (T drop over domain)• Time to • Velocity to • Stress to
�
D 2 /κ
�
κ /Dρνκ /D2
![Page 56: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/56.jpg)
Nondimensional equations
�
∇⋅ v = 0
�
1Pr
∂ v ∂t
+ v ⋅ ∇ v
⎛ ⎝ ⎜
⎞ ⎠ ⎟ = −∇P + ∇2 v + 1
Ek Ω × v + Ra.Tˆ y
∂T∂ t
+ v ⋅∇T = ∇2T
�
Pr = νκ
�
Ra = gα∇TD 3
νκEk = ν
2ΩD2
Prandtl number Ekman number Rayleigh number
![Page 57: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/57.jpg)
As before, use streamfunction
vx =∂ψ∂ y vy = − ∂ψ
∂ x
Also simplify by assuming 1/Ek=0
![Page 58: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/58.jpg)
Eliminating pressure• Take curl of 2D momentum equation: curl
of grad=0, so pressure disappears• Replace velocity by vorticity: • in 2D only one component of vorticity is
needed (the one perpendicular to the 2D plane),
�
ω = ∇× v
1Pr
∂ω∂ t
+ vx∂ω∂ x
+ vy∂ω∂ y
⎛⎝⎜
⎞⎠⎟= ∇2ω − Ra ∂T
∂ x
∇2ψ =ω z
![Page 59: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/59.jpg)
=> the streamfunction-vorticity formulation
�
∇2ψ = −ω
1Pr
∂ω∂ t
+ vx∂ω∂ x
+ vy∂ω∂ y
⎛⎝⎜
⎞⎠⎟= ∇2ω − Ra ∂T
∂ x
vx ,vy( ) = ∂ψ∂ y,− ∂ψ
∂ x⎛⎝⎜
⎞⎠⎟
�
∂T∂t
+ v ⋅ ∇T = ∇2T + Q
![Page 60: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/60.jpg)
Note: Effect of high Pr
1Pr
∂ω∂ t
+ vx∂ω∂ x
+ vy∂ω∂ y
⎛⎝⎜
⎞⎠⎟= ∇2ω − Ra ∂T
∂ x
If Pr->infinity, left-hand-side=>0 so equation becomes Poissonlike before:
∇2ω = Ra ∂T∂ x
![Page 61: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/61.jpg)
Taking a timestep∇2ψ =ω
vx ,vy( ) = ∂ψ∂ y,− ∂ψ
∂ x⎛⎝⎜
⎞⎠⎟
(ii) Calculate v from ψ
(iii) Time-step ω and T using explicit finite differences:
∂ω∂ t
= −vx∂ω∂ x
− vy∂ω∂ y
+ Pr∇2ω − RaPr ∂T∂ x
∂T∂ t
= −vx∂T∂ x
− vy∂T∂ y
+∇2T
(i) Calculate ψ from ω using:
![Page 62: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/62.jpg)
Tnew −ToldΔt
= −vx∂Told∂ x
− vy∂Told∂ y
+∇2Told
Tnew = Told + Δt ∇2Told − vx∂Told∂ x
− vy∂Told∂ y
⎛⎝⎜
⎞⎠⎟
ω new −ω old
Δt= −vx
∂ω old
∂ x− vy
∂ω old
∂ y+ Pr∇2ω old − RaPr
∂Told∂ x
ω new =ω old + Δt Pr∇2ω old − vx∂ω old
∂ x− vy
∂ω old
∂ y− RaPr ∂Told
∂ x⎛⎝⎜
⎞⎠⎟
T time step is the same as before
w must now be time stepped in a similar way
![Page 63: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/63.jpg)
![Page 64: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/64.jpg)
Stability condition
Diffusion:
Advection:
Combined:
dtdiff = adiffh2
max(Pr,1)
dtadv = aadvminh
maxval(abs(vx)), hmaxval(abs(vy))
⎛⎝⎜
⎞⎠⎟
dt =min(dtdiff ,dtadv )
![Page 65: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/65.jpg)
Modification of previous convection program
• Replace Poisson calculation of w with time-step, done at the same time as T time-step
• Get a compiling code!• Make sure it is stable and convergent for
values of Pr between 0.01 and 1e2• Hand in your code, and your solutions to the
test cases in the following slides• Due date: 18 December (2 weeks from
today)
![Page 66: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/66.jpg)
Test cases• All have nx=257, ny=65, Ra=1e5,
total_time=0.1, and random initial T and w fields, unless otherwise stated
• Due to random start, results will not look exactly as these, but they should look similar (i.e.. width of upwellings & downwellings & boundary layers similar, but number and placement of upwellings/downwellings different).
![Page 67: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/67.jpg)
Pr=10
![Page 68: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/68.jpg)
Pr=1
![Page 69: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/69.jpg)
Pr=0.1
![Page 70: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/70.jpg)
Pr=0.01
![Page 71: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/71.jpg)
Pr=0.01, time=1.0
![Page 72: Numerical Modelling in Fortran: day 8 - macOS Serverjupiter.ethz.ch/~pjt/fortran/class8.pdf · •Works with Fortran and C •Works on everything from a laptop to the ... Exact solution)](https://reader031.fdocuments.in/reader031/viewer/2022021820/5ae4d6eb7f8b9a90138f82c8/html5/thumbnails/72.jpg)
Pr=0.1, Ra=1e7