P. Jöckel Max Planck Institute for Chemistry, Mainz, Germany · 2009-02-05 · Max Planck...

The Modular Earth Submodel System (MESSy):Atmospheric Chemistry in Earth System Models

http://www.messy-interface.org

P. Jöckel Max Planck Institute for Chemistry, Mainz, Germany

ScicomP13, July 16-20, 2007, Supercomputing Center Garching

The Modular Earth Submodel System

MESSy communityMax Planck Institute for Chemistry,

Air Chemistry Department, Mainz

DLR Institute for Physics of the Atmosphere,Oberpfaffenhofen

Institute for Meteorology and Climate Research,Karlsruhe Research Centre

MPG Supercomputing Center Garching (RZG), Garching

Institute for Meteorology, Free University Berlin

Max Planck Institute for Meteorology,Hamburg (ECHAM5)

Aristotle University of Thessaloniki


MESSy hall of fame

Baumgärtner, Andreas; Brinkop, Sabine; Brühl, Christoph; Buchholz, Joachim;Burrows, Susannah; Butler, Tim; Dameris, Martin; Erhardt, Gabriele;

Ganzeveld, Laurens; Giner, Bernhard; Giorgetta, Marco; Grewe, Volker;Gromov, Sergey; Jöckel, Patrick; Kerkweg, Astrid; Ketelsen, Klaus ;Kirner, Ole;

Kubin, Anne; Kunze, Markus; Kurz, Christian; Lawrence, Mark; Langematz, Ulrike; Lauer, Axel; Lelieveld, Jos; Metzger, Swen; Nissen, Katrin;

Ponater, Michael; Pozzer, Andrea; Rhodin, Andreas; Riede, Hella; Ruhnke, Roland; Sander, Rolf; Sausen, Robert; Schumann, Ulrich;Steil, Benedikt; Steinkamp, Jörg; Stier, Philip; Tanarhte, Meryem;

Taraborrelli, Domenico; Tost, Holger; Tourpali, Kleareti; Traub, Michael; van Aalst, Maarten; van Aardenne, John; ...

Developers, contributors, users, promoters:

... and much more data users ...


I. MotivationII. Overview of the concept of MESSyIII. MESSy for atmospheric chemistryIV. Evaluation of the AC-GCM

ECHAM5/MESSy1 (E5/M1)V. Some notes on the implementation

and performance

Outline


I. Motivation


Atmospheric Chemistry:● major player in the Earth Climate system● provides link between different domains and processes:

● radiative budget (CO2, Ozone, clouds)

● interaction with ocean and biosphere (C-,N-,S-,...-cycles)● human impact (pollution) and impact on humans (air quality)

● self-cleansing capacity of the atmosphere prevents runaway

Atmospheric Chemistry in Earth System Models (ESMs):● a large number of processes involved ...● ... which are “chemically” coupled● highly non-linear processes:

>>> avoid recompilation wherever possible● wide range of time-scales (10-2 s ... 106 years)● large number of species (all relevant ?)

Motivation


Emission

Transformation

Deposition

Transportnatural

emissions

anthropogenicemissions

gas phase clouds

aerosol

advection convection diffusion

wetdeposition

sedimentation

drydeposition

scavenging

OCEAN AND BIOSPHERE

Motivation


MotivationDevelopment aspects:● always state-of-the-art● prepared for the future

Scientific aspects:● increasing complexity requires increasingly transparent

control● variable complexity depending on the scientific

questions/demands -> high flexibility● minimized effort for extension (plug&play, exchangeable code)● efficient future developments (standardised interfaces)● feedback mechanisms between processes (quantification)● process oriented● high degree of internal consistency● reproducibility● ...

continuous further development


... consistency

many projects / ONE code

com

ple

xity

simulation time

GCM

BOX

MESSy

climatestudies

processstudies

Motivation


II. Overview of the conceptof MESSy


process oriented approach of MESSy

MESSy is (a project providing) ...● an interface with infrastructure to couple 'processes' (=submodels) to a GCM (= base model)● a set of processes coded as switchable submodels● a (simple) coding standard

What is MESSy ?Jöckel et al., ACP, 2005

advection

radiation

chemistry

microphysics

convection...

clock/runcontrol(base model)

standard interface

soilvegetation

land use

...

dynamics

biologychemistry

... ... AtmosphereOcean

Land surface

...

MESSy The Systemis a coupled

set ofcommunicating

processes.


The key: Standardisation

Fortran90/95 ISO/IEC 1539:1991languagelevel

user interface:namelists

subroutines as 'operators'in smallest meaningful entity

(box, column, column-vector, ...)

INTENT(IN) INTENT(OUT)process

level

interfacelevel

SWITCH /CALL

infra-structure

level

MEMORY /OUTPUT

TRACER RE-GRIDDING

TOOLS ...

Standardisation on the lowest possible level ...

process specific interface

“sciencecode”


How does MESSy work ?

Base Model Interface Layer: multiple socket outlet

Submodel InterfaceLayer: connector

Submodel Core Layer:the machinery ...

Base Model Layer:power supply


INITIALISATIONPHASE

INTEGRATIONPHASE

FINALISINGPHASE

user friendly flexibility: 'model setup'

PERFORMANCE !!!time loop, decomposition loop, ...

INPUT of boundary conditions

OUTPUT of results

Basic flow chart


III. MESSy for atmosphericchemistry


Atmospheric chemistry

Emission Transformation

Deposition Transport

ConVective TRANSport

LightningNOx

ONLineEMissions

OFFLineEMissions

TNUDGE(pseudo emissions/depositions)

CONVECTion

GCM

advection

radi ationRAD4ALL

DRYDEPosition

AIRSEAexchange

MECCA

incloud chem.,SCAVenging &wet deposition

CLOUD

JVAL/PHOTO

Aerosol:MECCA_AERO,M7, SEDI, PSC,HETCHEM, EQSAM

coupling viaTRACER


TRACER

META-DATA

MEMORY

_FAMILY

_PDEF

● transport families● tagging (linearisation)

● keep positive● track corrections● global mass diagnostics

SET 1

META-DATA

MEMORY

SET 2

META-DATA

MEMORY

SET n

● “generic” submodel (i.e., infrastructure coded as submodel)● provides “chemical coupling” between different processes

(e.g., emission -> chemical reaction -> advection -> deposition)

TRACER


TRACERMETA-DATA: characteristics of tracers (= chemical species)

-> concatenated list of nested Fortran90-structures

TYPE t_trinfo TYPE(t_ident) :: ident ! IDENTIFICATION TYPE(t_proc) :: proc ! PROCESS FLAGS TYPE(t_infra) :: infra ! INFRASTRUCTURE TYPE(t_typ_single) :: typ_single ! TYPE: SINGLE TYPE(t_typ_family) :: typ_family ! TYPE: FAMILY TYPE(t_typ_isotope) :: typ_isotope ! TYPE: ISOTOPE TYPE(t_med_aerosol) :: med_aerosol ! MEDIUM: AEROSOLEND TYPE t_trinfo

TYPE t_trinfo_list TYPE(t_trinfo) :: info TYPE(t_trinfo_list), POINTER :: nextEND TYPE t_trinfo_list

TYPE t_typ_single REAL(DP) :: molarmass = 0.0_DP ! REAL(DP) :: henry = 0.0_DP ! henry coefficient [mol/L/atm] REAL(DP) :: dryreac_sf = 0.0_DP ! dry reaction coeff.END TYPE t_typ_single


chemical equation setsulfurhalogenshydro-

carbonsbasic

tro

po

sph

.st

rato

sph

.aq

ueo

us

kpp solver

radau5

ros3

ros2

auto

solver type timestep

manually

log

9

log

5

...

chosen chemical setup:e.g. basic gas phase

chosen solver:e.g. ros3-auto

F90-code

MBL5) MBL MBL

MBLMBLMBL

MA-ECHAM4/CHEM3)

MATCH1) MISTRA2)

MISTRAMISTRA

1) v. Kuhlmann et al. 20032) v. Glasow et al. 20023) Steil et al. 19984) Meilinger, 20005) Kerkweg, 2005

MSBM4)

MECCA(_AERO)high flexibility >>> automatic code generation

Sander et al., ACP, 2005Kerkweg et al., ACP, 2007


MECCA(_AERO) / KPP


IV. Evaluation of the AC-GCM ECHAM5/MESSy1 (E5/M1)


Evaluation simulation (S1):

● ECHAM5/MESSy● T42 resolution (~2.8°x~2.8°)● 90 vertical levels (up to 0.01 hPa)● nudged in free-troposphere (200-700 hPa) towards

analysis data from ECMWF operational model● 245 reactions including hydrocarbons up to 4 carbon

atoms● feedback between chemistry and dynamics via

radiation● off-line anthropogenic emissions compiled for the

year 2000● 8 years of simulation (1998-2005)

Evaluated Setup


MECCA* MECCA_MBL

AIRSEA(air-sea Δc)

PTRAC*(tracer properties)

DRYDEP(or EMDEP)

CVTRANS(massflux)

CONVECT(convectivecloud/rainproperties)

CLOUD(large scalecloud/rainproperties)

SEDI

M7*(aerosol mass, number, radius, σ, ρ)

SCAV*

LNOX PSC(psc region, khet*)

TROPOP(tropopause height)

TNUDGE

ONLEM(emission fluxes)

(or EMDEP)

HETCHEM(khet*)

JVAL(or PHOTO)

(J_*)

OFFLEM(emission fluxes)

H2O*

RAD4ALLTRACER_FAMILY*

couplingswitchable

* definetracers

coupling todynamics

modifytracers

usetracers

no tracermodific.

not used

Complexity


4 year average (2000-2004, excl. 2002) of ozone (DJF) [μmol/mol]

consistentsimulation of(dynamical and chemical)state of theatmospherebetween surface and 0.01 hPa

OzoneJöckel et al., ACP, 2006


Jan

Jan Jun

Jun

40 hPa

400 hPa

Jan

Jan Jun

Jun

OzoneComparison with O

3 database (monthly averages)

Logan et al., JGR, 1999; Jöckel et al., ACP, 2006


E5/M1NOAA/GMD

Example:tropospheric CO

Carbon monoxide

Novelli et al., JGR, 1998Jöckel et al., ACP, 2006Pozzer et al., ACP, 2007


Example: stratospheric temperature (SON 2003)

E5/M1 (S2) E5/M1 - MIPAS

K K

Temperature

Jöckel et al., ACP, 2006


E5/M1 (S2) TOMSTotal Ozone [DU] 26 Sep 2002

SH vortex split 2002 reproduced

SH vortex split 2002

Jöckel et al., ACP, 2006


Christmas Island (C2H

6)Japan (C

3H

8)

Emmons et al.,JGR, 2000; Pozzer et al., ACP, 2007

Organicspecies

Example: Propane Example: Ethane

Organic species


Philipine Sea (C3H

6)South Atlantic (C

3H

6)

Emmons et al.,JGR, 2000; Pozzer et al., ACP, 2007

Example: Propene

Organic species

...somefail...


Wet deposition

Example: Nitrate [mg(N)/m2/year]

Tost et al., ACP, 2007


Tost et al., ACP, 2007

pH

precipitation pH

cloud pH


V. Some notes on theimplementation andperformance


Evaluation simulation (S1):● ECHAM5/MESSy T42L90MA, 1998-2005● IBM P4 “regatta”; 256 CPUs; (psi @ RZG):

~ 8 hours (wall clock) / simulation month~ 1 Tb output / simulation year

>>> parallelisation of ECHAM5: ● regional decomposition in grid-point space

NCPUS = NPROCA x NPROCB● “vector”-packing on every CPU:

(NGPBLKS-1)xNPROMA + NPROMZ

>>> 70-80% of CPU time consumed by chem. kinetics>>> high I/O load (MPI, dedicated I/O CPU, gather)

-> parallel-netCDF (MPI-2); work in progress

Performance... is highly dependent on the chosen setup ...


NCPUS=16 NPROCB = 2N

PR

OC

A =

8

CPU 0

CPU 0CPU 1

CPU 1

CPU 7CPU 7

CPU 8

CPU 8CPU 9

CPU 9

CPU 15CPU 15

EXAMPLE: T21L19 >>> 64 longitudes x 32 latitudes x 19 levels

CP

U#

Parallelisation

choice ofNPROCANPROCB(NCPUS)

limitedby

advection


CPU 0

CPU 0

CPU 11

CPU 11

NPROMA = 101JP=1JP=1 ...

JP

JP= ...101

JROW=1 ...

JROW= ... 2

JP=1

Vectorisation

optimalchoice ofNPROMA ?

vector-architectures: as large as possible

scalar-architecture: cache ???

on each CPU ...


Example: T42L90MA (128 longitudes x 64 latitudes x 90 levels)WITH complex chemistry setup

NPROMAmax=NLON×NLATNCPUS

(32)(1024)

Vectorisation


Problem (with parallelisation):role of involved processes (with different computational demands)varies with space and time

Examples:- convection / convective transport / scavenging

>>> tropics vs. polar region- chemical transformations

>>> annual and diurnal cycle (photo-chemistry !)>>> involved species~ ~ ~ stiffness of the ODE system describing the chemical

kinetics ...(reaction rates vary over several orders of magnitude)SOLVER:- 'fast'

- 'reliable', accurate- stable (!)

Performance issues

>>> 3rd order Rosenbrock with automatic time stepping


Kerkweg et al., ACP, 2007number of kpp (ros3-auto) substeps

lowest modellayer (~ 70m)with (sea-salt-)aerosolchemistry

>>>

very stiffODE system

22 April 1999, 05:00 UTC

Automatic time-stepping


Kerkweg et al., ACP, 2007number of kpp (ros3-auto) substeps

70 hPa(~ 50 km)only gasphasechemistry

>>>

lower stiffnesof the ODEsystem

22 April 1999, 05:00 UTC

Automatic time-stepping


Performance ???

“FAST CHEMISTRY”

CODE OPTIMISATIONS REDUCED CHEMICAL SETUP

complex setups:- NMHC, > C

5

- Aerosol Chem.- SOA

reduced setups:- simplified chemistry

on long time scales


Vectorisation of KPP code

Δt(1)

OUTER LOOP: grid-boxINNER LOOP: solver (automatic time stepping)

OUTER LOOP: solver (automatic time stepping) (applied to vector of grid-boxes)

+ (de-)compression algorithm

Δt(2)

Δt(3)

Δt(4)

Δt(5)

Δt(6)

Δt(7)

Δt(8)

Δt(...)

Δt(1)

Δt(2)

Δt(3)

Δt(4)

Δt(5)

Δt(6)

Δt(7)

Δt(8)

Δt(...)

“sort-out” every box for which solver finished time-stepping

“scalar”implementation

“vector”implementation


on p4/p5:

scalarimplementationis much fasterthan vectorimplementation

>>>

overhead of(de-compression)exceeds gainof cache usageoptimisation

Vectorisation of KPP code


no de-indexing 64.97s -qnooptde-indexing (1) 35.08 s -qnoopt

de-indexing (2) 13.84 sde-indexing (2) 13.04 s JVS(:) -> JVS(LU_NONZERO)de-indexing (2) 13.11 s JVS(:) -> JVS(LU_NONZERO)

-Q-kppdecompde-indexing (1) 12.85 s JVS(:) -> JVS(LU_NONZERO)

-Q-kppdecomp

scalar sparse LU-decomposition

p4256 CPUsT42L90MAcomplex chemistry

average wall-clock timeper simulation time step

... loop with indirect indexing ... replaced by sequence of operations


ScalingExample: T42L90MA (128 longitudes x 64 latitudes x 90 levels)

WITH complex chemistry setup (1 day integration) ;measured with “-pg” - Option on p4/p5

P5 (8 CPUs) P4 (256 CPUs)(CPU 2) (CPU 40)

TOTAL 6938 s 645 s 10.76CHEM. SOLVER 5382 s 262 s 20.54ADVECTION 381 s 23 s 16.57

256 / (2*8) = 16

!!! preliminary results: choice of arbitrary CPU for comparison isNOT appropriate


www.messy-interface.org

special issue in Journal ofAtmospheric Chemistryand Physicshttp://www.atmos-chem-phys.net/special_issue22.html

Thank you very muchfor your attention !

P. Jöckel Max Planck Institute for Chemistry, Mainz, Germany · 2009-02-05 · Max Planck...

Documents

Transcript of P. Jöckel Max Planck Institute for Chemistry, Mainz, Germany · 2009-02-05 · Max Planck...