© Crown copyright Met Office Atmosphere-Chemistry Model Coupling in the UK Earth System Model...

© Crown copyright Met Office

Atmosphere-Chemistry Model Coupling in the

UK Earth System Model (UKESM)Richard Hill, April 2015

Credits: Marc Stringer, Colin Johnson, Mohit Dalvi and many more including several in this room.


Contents

1. UKESM Overview

2. Background on performance issues

3. 3D coupling and OASIS3-MCT

4. Possible approaches

5. Progress to date

Questions


UKESM – UK Earth System Model• A Joint Met Office/NERC Collaboration

• Pooling expertise and resources from various centres

• The UK community contribution to CMIP6

• Aims to include various component models:

• UM: atmosphere

• JULES: land surface

• NEMO: ocean

• CICE: sea-ice

• BICICLES: Ice shelf

• MEDUSA: ocean biogeochemistry

• UKCA: UK community atmospheric chemistry-aerosol

• Wave model – one day e.g. WAVEWATCH III?

JULESLand surf UM Atmosphere

NEMOOcean

CICESea Ice

OASIS3or

OASIS3-MCT

Current Met Office climate models

JULESLand surf UM Atmosphere

UKCAChemistry

NEMOOcean

Ice Shelf

Wave?

CICESea Ice

MEDUSAOcean Biogeochem

OASIS3-MCT

UKESM potential component models


But: Chemistry is a BIG cost!UM+UKCA profiling: UKCA increases cost by ~400%


Chemistry Costs! UKCA features heavy dynamic load imbalance

• Typically ~25%-30% of cost is dead time as work follows the sun!

– Cray XC40 task positioning might help load balance but won't get us to where we need to be.

• Cost is prohibitive for intended global resolution (N216 = 432x324x85)

• Optimisation work on UKCA code over the years has not delivered more than a few percent improvement

• We need more like ~80%

• Tinkering with the code has no chance of delivering that.

We need another way!We need another way!.

.


How about this?

• Run UKCA at a REDUCED resolution (somehow) and couple to higher resolution atmosphere?

• Maybe.... but...

– UKCA is embedded in UM atmosphere

– UKCA is tied to UM atmosphere resolution

– UKCA is not currently a viable stand-alone code!

– We'd need to exchange (potentially) lots of 3D fields somehow

Can we even do that?Can we even do that?

© Crown copyright Met Office© Crown copyright Met Office

Is 3D coupling viable? OASIS3-MCT offers hope with 3D fields.

• Two possible approaches:

• One allows us to deal with each 3D field as a single entity

• The other allows us to bundle multiple 2D fields together in order to act as a 3D field.

• Both components have the same number of vertical levels (typically 85). This appears not to be an issue as far as the science is concerned.

• i.e. We only need HORIZONTAL regridding.


3D Field Coupling

Option 1): define 3D fields as a true 3D record.

• Requires special 3D remapping weights files which repeat the same weights 85 times in the same record! (assuming no vertical interpolation)

• Requires special program and operations to generate such a file

• Such files are huge.

• Separate files required for fields of differing numbers of vertical levels

• i.e. You can't just re-use weights files in the same way that you can with 2D equivalents.

• Generation, management and debugging of such files is much more difficult than the 2D case.

• But you CAN define and put an entire 3D field in one go!


Level 8

Level 7

Level 6

Level 5

Level 4

Level 3

Level 2

Level 1

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

Coupling using full 3D regridding weights files

Source 3D fildSource 3D fieldTarget 3D field

Full set of 3D weights in 1 file


3D Field Coupling

Option 2): Treat each vertical level as a 2D field

• Use functional features of OASIS3-MCT to allow definition of multiple 2D fields in a single namcouple record – thereby defining a 3D entity.

• OASIS3-MCT effectively bundles multiple fields together. So the “put” operation only actually takes place once the final level is supplied to the “put” call.

• Allows use of a single 2D remapping weights file by each “2D” field at each vertical level.

• 2D weights files are much smaller and easier to create, manage and debug than anything which might constitute a 3D weights file.

• But we have to define and put lots of separate 2D fields.


Coupling using 2D regridding weights files

Source 3D fildSource 3D fieldTarget 3D field

Single level 2D weights file

© Crown copyright Met Office© Crown copyright Met Office

3D coupling is viable!OASIS3-MCT offers hope with 3D fields.

• Tests with toy model systems and iteration with OASIS3-MCT developers:

• We can successfully exchange ~200 3D fields (85 vertical levels) between N216 and N96 toy models –

• i.e. Equivalent to 17,000 2D fields!!! (atmos-ocean uses ~53 2D fields in total !)

• Cost appears acceptable – 5-10% of projected model run times, massively offset by potential reduction in elapsed time and overall resources.


3D Field Coupling

So 3D field coupling looks viable

Option 2) “multi-2D” approach looks preferable

Even if we end up having to define and exchange large numbers of fields... (~85 x option 1)... long term maintenance and usability is overriding factor.

OASIS namcouple file generation and field naming is an issue.

– Potentially 1000's of different field names

– OASIS EXPOUT gets “interesting”...but we can probably cope

So what about the component models?.....

UKCA Component Options Create a stand alone UKCA component?

But remember....

UKCA is embedded in UM atmosphere

UKCA is tied to UM atmosphere resolution

UKCA is not currently a viable stand-alone code!


UKCA Component Options Unraveling UKCA from UM is like....

Unraveling

two of these


UKCA Component Options

WITHOUT

one of these

UKCA Component Options Create a stand alone UKCA component?

Very involved

Requires restart and diagnostic handling system, MPP decomposition, launching mechanism, etc, etc. In fact a whole new independent supporting infrastructure.

ALL BEFORE YOU START WORRYING ABOUT THE ALL BEFORE YOU START WORRYING ABOUT THE COUPLINGCOUPLING

• Also requires coupling of huge numbers of 3D, 2D, 4D (and 1D) fields, mostly chemical tracers, some integers and even some logicals!

• Some good news: Tests with toy models and OASIS3-MCT Some good news: Tests with toy models and OASIS3-MCT suggest volume of coupling data would suggest volume of coupling data would notnot be a limiting factor. be a limiting factor.

JULESLand surf(High res)

UM Atmosphere(High res)

UKCAChemistry(Any res)

OASIS3-MCT

So we can’t do this… (at least not easily)

JULESLand surf(High res)


UKCAChemistry(Low res)

OASIS3-MCT

…or this… (at least not easily)



Alternative:

Simply run an entire “junior” UM+UKCA at the lower res coupled to a “senior” UM at the higher res.

● Pass key coupling fields from higher resolution senior component, to keep junior UM evolution in line.

● Nudging/assimilation or complete overwriting of fields.

● Simpler than creating a brand new stand-alone UKCA component.

● Requires fewer coupling fields than a true stand-alone UKCA - 3D fields only and no awkward integers, etc.

Senior – Junior coupled models proposed set-up


Senior UM

N216

Junior UM

N96

UKCA

NEMOOcean

CICESea-ice

MEDUSA

OBGC

OASIS3-MCT Coupling



Running a “junior” UM-UKCA needs...

● Modifications to user interface to allow TWO UM components to be specified and submitted (along with existing NEMO-CICE and XIOS capability)

● Modify system control to allow two UM components to run concurrently... naming conventions of each component.

● Archiving support.

● Ability for both UM components to start from “the same” initial conditions... the UM reconfiguration and coupling exchanges at timestep zero can facilitate this.



Running a “junior” UM-UKCA features...

●Tracer advection is performed in junior.

●Extra cost in terms of duplicated operations in junior is only about 20% of total junior cost (remember earlier performance chart.)

●Ready-made IO, decomposition, diagnostic, etc system.

●Generation of namcouple files on-the-fly is desirable due to flexible scientific configuration needs.

● Which in turn demands more code and data handling developments to enable this.

Current position

• Routinely running a prototype system coupling UM N216 with UM N96 + Chemistry.

• Currently exchanging ~10 3D fields using “option 2)” mentioned previously in the N216 -> N96 direction.

The main purpose of these is to ensure the “junior” model evolves in the same way as the “senior”.

e.g. U, V, W, temperature, moisture fields

• Currently setting up the N96 -> N216 coupling to pass back critical fields from chemistry which will affect atmos evolution.

– e.g. Aerosols, dust etc. © Crown copyright Met Office

JULESLand surf


UKCAChemistry(Low res)

NEMOOcean

CICESea Ice

MEDUSAOcean Biogeochem

OASIS3-MCT

UKESM current prototype

JULESLand surf(Low res)

UM Atmosphere(Low res)

Current position

• Investigations concentrated on:

– Most effective interpolation methods

– Nudging/assimilation of “junior” with coupled fields versus complete overwriting?

– Likely resource requirements

• Bottom line(s)...

– Technical viability of 3D coupling is not a major concern!

– Performance of 3D coupling is not a major concern!

At least for now!.....At least for now!.....© Crown copyright Met Office

Summary

● UKESM aims to include chemistry in climate model

● Chemistry is very expensive – BUT SCALABLE!

● Aim to run chemistry at reduced resolution cf. atmos

● OASIS3-MCT facilitates this by allowing 3D field coupling

● Toy model tests indicate technical and speed viability

● Actual model developments and tests to date still indicate viability..... still relatively early stages.

● Detailed scientific tests yet to be completed – potential issues with time stepping and position of coupling exchanges.

● Only tested on IBM power7.... Cray XC40 porting soon!© Crown copyright Met Office


Questions and answers

© Crown copyright Met Office Atmosphere-Chemistry Model Coupling in the UK Earth System Model...

Documents

Transcript of © Crown copyright Met Office Atmosphere-Chemistry Model Coupling in the UK Earth System Model...