Model and Data Hierarchies for Simulating and Understanding Climate

Marco A. Giorgetta

Demands on next generation dynamical solvers

(My) wish list for the dynamical solver

• Conserves mass, tracer mass (and energy)• Numerical consistency between continuity

and transport equation• Well behaved dynamics

– Low numerical noise– Low numerical diffusion– Physically reasonable dispersion relationship

• Accurate• Grid refinements (of different kinds)• Fast

Numerical consistency between continuity and transport equation

Example: ECHAM

– A spectral transform dynamical core solving for:• Relative vorticity• Divergence• Temperature• Log(surface pressure)

… using 3 time level “leap frog” time integration scheme– A hybrid “eta” vertical coordinate:

• Pressure at interface between layers: p(k,t) = a x p(t) + b

• Mass of air in a layer x g: dp(k,t) = p(k+1,t) - pi(k,t)

– A flux form transport scheme• q, cloud water, cloud ice, (and tracers for chemistry or aerosols

… using a 2 time level scheme

PROBLEM: tracer mass not conserved

Illustration:

• ICON dynamical core + transport scheme– Triangular grid– Hydrostatic dynamics– Hybrid vertical “eta” coordinate– 2 time level semi-implicit time stepping– Flux form semi-Lagrangian transport scheme

• Jablonowski-Williamson test– Initial state = zonally symmetric, but dynamically instable flow– Initial perturbation Baroclinic wave develops over ~10 days– 4 Tracers, of which Q4(x,y,z,t=0) = 1

Daniel Reinert, DWD

Grid refinements (of different kinds)

Options/Questions

• Grid refinement– static or dynamic/adaptive?

– Re-distribute grid points or create/destroy grid points?

– 2d or 3d? Boundary layer, troposphere, stratosphere, mesosphere

– Single time integration scheme or recursive schemes?

– Conservation properties?

• Dynamical core– Adjust scheme to expected errors ( FE schemes)

• Parameterizations:– Submodels: embedded dynamical models – “super-parameterizations”

• Cost function– How to predict the need for refinement, and what for? Target/goal?

– How to confine computational costs?

Generating the icosahedral triangular grid

Other kinds of grid refinements

(A) Hexagon (B) re-distributing cells

Refining the grid

Grid refinement• Refinement by bi-section of triangle edges: 1

triangle 4 triangles• 1 or more refined regions• 1 or more refinement levels per region

Two-way nesting1. Compute one time step on parent domain dX/dt2. Interpolate the tendencies to the lateral boundary of

the nested domain3. Perform 2 time steps(*) on the nested domain4. Feed back the increments

(*) For more levels Apply recursion

Numerical discontinuities! Leonidas Linardakis, MPI-M

(B) global resolution 35 km

(A) global resolution 140 km (C) global res. 140 km, regional res. 35 km

Mountain induced Rossby waveVorticity at ~ 3 km MSL after 20 days of simulation

• 2000-m circular mountain at 30°N/90°E• Initial condition: Zonal flow with

maximum speed of 20 m/s• Experiments:

– (A) Global with 140 km resolution– (B) Global with 35 km resolution– (C) As (A), but 2-step refined circular

region with resolution of (B)

Günther Zängl, DWD

High Performance Computing

• How to get faster:– Faster CPUs– Faster connections between CPU, memory and disk faster– Parallelization over more CPUs

• CPUs share memory• CPUs have their own memory

– Modify code design to account for architecture of CPUs• Scalar/vector CPUs• Sizes of intermediate, fast access memories (“Caches”)

• The past was dominated by improved CPUs• The future will be dominated by more CPUs

Parallelzation

• Distribute the work to many CPUs– Works well for local computations : cells, columns, (levels), …– Works badly for non-local task: integrals, global organization, …

Serial and parallel sections in a code On 1 CPU:

On 4 CPUs:

Amdahl’s law

The serial fraction of work limits the speedup

P = fraction of the workthat can be parallelized

1-P = remainder, whichcannot be parallelized

S = 1/(1-P) =maximum speedupfor N ∞

(Wikipedia)

The serial fraction of work limits the maximum speedup!

For illustration: Computer at DKRZ

• IBM Power6 CPUs– 250 nodes– 16 CPUs/node total = 4000 CPUs– 2 cores/CPU total = 8000 cores– 2 floating point units/core 64 parallel processes/node

• ECHAM GCM at ~1° resolution– Scales “well” up to 20 nodes = 640 cores (with parallel I/O)– Problem: Spectral transform method used for dynamical core

Requires transformations between spherical harmonics and grid point fields global data exchange, transpositions.

• Future: ~105 cores New model necessary

Strategies

• Select numerical scheme, which is– Sufficiently accurate with respect to your problem– Computationally efficient

• Fast on single CPUs• Minimize global data exchange (Transformations, “fixers”, I/O)• Find optimal way to distribute work

• Practical issues:– Optimization the code for the main computer platform– Account for strengths/weaknesses of available compilers– Avoid “tricks” which will stop the code to work on other platforms– Optimize first the most expensive parts

Other problems in HPC

• Data storage:– Disk capacities grow less than computing power– Bandwidth between computer and storage system– ESMs can produce HUGE amounts of data– Finite lifetime of disks or tapes backups or re-computing?

• Data accessibility:– Bandwidth between disks/tapes and post-processing computer– Post-processing software must be parallelized

• Data description– Documentation of model, experimental setup, formats etc.

• Climate models are no longer a driver for the HPC development

END