ESPC Infrastructure Update on HYCOM in CESM (Alex Bozec, Fei Liu, Kathy Saint, Mat Rothstein,...
-
Upload
baldric-flowers -
Category
Documents
-
view
217 -
download
0
Transcript of ESPC Infrastructure Update on HYCOM in CESM (Alex Bozec, Fei Liu, Kathy Saint, Mat Rothstein,...
ESPC Infrastructure• Update on HYCOM in CESM (Alex Bozec, Fei Liu,
Kathy Saint, Mat Rothstein, Mariana Vertenstein, Jim Edwards, …)
• Update on ESMF integration with MOAB (Bob Oehmke)
• Update on accelerator-aware ESMF (Jayesh Krishna, Gerhard Theurich)
• Update on the Earth System Prediction Suite (Cecelia DeLuca)
ESPC Air-Ocean-Land-Ice Global Coupled PredictionProject Meeting, 19-20 Nov 2014
University of Colorado UMC, Boulder
ESPC Infrastructure• Update on HYCOM in CESM (Alex Bozec, Fei Liu,
Kathy Saint, Mat Rothstein, Mariana Vertenstein, Jim Edwards, …)
HYCOM in CESM -3rd step (in progress)
Produce a CESM-HYCOM baseline run in a G-compset : • based on 1º resolution grid from the POP gx1v6 grid (320x384),• v2.2.35 of the HYCOM source (for testing purposes) • Initial Conditions Levitus- PHC2• CORE atmospheric forcing provided by DATM• Ice component: CICE v4.0 (CESM version)
(Ice initial condition: ice cap provided by CESM)
For now coupling frequency:DATM: 1 hrCICE: 1 hrHYCOM: 3 hrs
First Results to be compared with CESM-POP
Fre
sh
wa
ter
Flu
x H
eat
Flu
x
CESM-POP CESM-HYCOM
Comparison of the total heat and freshwater flux (monthly mean Jan/ 1st year) – from Bozec
NUOPC Layer
• HYCOM-CICE coupling in CESM was performed using ESMF/NUOPC Layer software in combination with existing CESM Coupler 7 (CPL7) infrastructure
• The NUOPC Layer is a set of code templates and conventions for ESMF-based codes
• NUOPC components provide:1. A GNU makefile fragement that defines a small set of prescribed variables, which
a NUOPC application uses to compile and link with the component.2. A single point of entry, called SetServices.3. A set of initialize phases with defined behaviors (e.g. advertise what fields can
be produced).4. A set of run phases , where each phase must check the incoming clock of the
driver and the timestamps of incoming fields against its own clock for compatibility.
5. Time stamps on its exported fields consistent with the internal clock of the component.
6. A finalize method that cleans up all allocations and file handles.
CESM/NUOPC Architecture
NUOPC Driver
Model:ATM
Med
iato
rModel:
ICE
Model:OCN
Model:LND
Model:ROF
Model:WAV
Model:GLC
xatmdatm
xicedicecice
xocndocnpop2hycom
xlnddlndslnd
xrofdrof
xwavswav
xglcsglc
NUOPC Layer “generic components” are templates that encode these rules for Drivers, Models, Mediators (custom coupling code) and Connectors (simple data transfers and transformations)
Incremental Implementation
• Initial work refactored CESM CPL7 to better accommodate NUOPC rules (last year)
• Created NUOPC_Driver and NUOPC_Models for dead CESM components• Created NUOPC_Models for data components and added
NUOPC_Mediator and NUOPC_Connectors• Next active components - created NUOPC_Model for POP2 and validated
it running standalone and coupled with CICE• Integrated HYCOM into the CESM build and created a NUOPC_Model for
HYCOM• Modified the data exchange (between coupler/Mediator and
components) to use NUOPC Fields• Connected fields between HYCOM and CICE using CPL7 through NUOPC
interfaces
To be Done – Technical Side
• Remove sequential execution limitation• Currently working on other active components (atmosphere, land, …)
which will be tested in additional component configurations• Introduce standard name brokering so that multiple conventions (CESM,
CF) can be used to name the same Fields, allowing the same component (e.g. HYCOM) version to be used at different sites
• Performance evaluation
ESPC Infrastructure• Update on ESMF integration with MOAB (Bob
Oehmke)
MOAB Integration
• MOAB: A Mesh-Oriented datABase• Library for representing an unstructured finite-element mesh• Department of Energy/University of Wisconsin project
• ESMF’s Internal Mesh representation (here called IntMesh)• Part of ESMF source for representing an unstructured finite-element mesh• Custom ESMF code developed and maintained by ESMF developers
• Goal: To put MOAB alongside IntMesh underneath ESMF functionality to compare them. The goal is to decide if we should use MOAB instead of IntMesh.
• Why use MOAB?• Offload development and maintenance • Take advantage of improvements and optimizations• Added functionality • Share mesh data structure with Model Coupling Toolkit (MCT)
Current Status
• Allowed internal ESMF mesh (IntMesh) to be replaceable
• Added capability to build ESMF_Mesh on top of MOAB mesh• ESMF_Mesh is ESMF’s unstructured mesh class
• Added capability to build ESMF_Fields on top of MOAB mesh• ESMF_Field is ESMF’s distributed data class
• Can do first-order conservative regridding using MOAB: • Can calculate regridding weights• Can apply weights to data contained within an ESMF Field
• Other ESMF capabilities that now work via MOAB:• Retrieving information (dimensions, etc.) from ESMF_Mesh• Writing ESMF_Mesh to a file (useful for debugging)• Filling ESMF_Field with cell areas (useful for checking regrid)
Making Mesh Replaceable
ESMF_Mesh
IntMesh
Old Way
ESMF_Mesh
MeshCap
ORIntMesh MoabWrap
MOAB
New Way
Experiences
• Using MOAB instead of IntMesh has mostly been straightforward• Replacing object names: EntityHandle vs. pointer to MeshObj• New ways of accessing objects: Range vs. iterators to objects
• Some things nicer in IntMesh• Getting global ids• Going from global ids to objects in mesh• Adding new global information
• Some things nicer in MOAB• Data fields support different types• Can pull out data for a set of objects in one chunk• Documentation!
• Still investigating• How to use automatic differentiation code with MOAB
Used for some weight calculation (e.g. patch)
Wrapping Up
• In depth comparison of ESMF capabilities using MOAB vs. IntMesh• Accuracy• Performance
• Decision about whether to replace IntMesh with MOAB
ESPC Infrastructure• Update on accelerator-aware ESMF (Jayesh Krishna,
Gerhard Theurich)
Host – multi-core CPUGPU
.
.
.
O(10k)nodes
RAM
Host – multi-core CPUMIC
RAM
Host – multi-core CPU
RAM
GPU
GPU
GPU
cores
cores
cores MIC
Increasing course grain parallelism:
Decom
position and distribution.
Increasing fine grain parallelism:Threading and SIMD on CPU cores, and device cores.
Modern supercomputer architecture
ESMF/NUOPC accelerator project
GOAL To support migration of optimization strategies from the ESPC Coupling Testbed to infrastructure packages and coupled model applications, and provide support for coupling of optimized components in the ESPC program.
Host – multi-core CPUGPU
.
.
.
Host – multi-core CPUGPU
Host – multi-core CPUGPU
Comp-A Comp-B Comp-C
Example of interleaved components
cores
cores
cores
● Can components that use different programming models (OpenCL, OpenACC, Intel-MIC-Directives, …) run under the same single ESMF executable?
● Do the different programming models provide enough control for a component to decide at run-time whether to use a specific accelerator device or not?
● Is it possible to uniquely identify the available devices? Across programming models? Across the distributed parts of a component? Across components?
● Jayesh Krishna (ANL) developed a set of prototypes to answer questions/explore behavior.
Initial questions and considerations
ESMF Implementation
● Jayesh has forked from the ESMF repository and has implemented the following features:
● Accelerator awareness has been implemented through an extension of the ESMF_VMGet() call to allow each ESMF Persistent Execution Thread (PET) to query the number of accelerator devices it can access.
● Accelerator awareness is supported for NVIDIA GPU and Intel MIC devices.
● Prototype applications were developed that use the new capability of the ESMF_VMGet() call to query the number of accelerator devices, and to allocate (i.e. access) the device from a single PET.
Prototypes with OpenCL, OpenACC,and Intel-MIC-Directives
• An ESMF application with an OpenCL accelerated component, a component accelerated using Intel MIC directives - and a coupler
• An ESMF application with MPI non-accelerated components, an OpenCL accelerated component - and a coupler that uses the newly added support in ESMF to query the number of accelerator devices. The application allocates all devices attached to a node to the lowest ranked MPI process running on the node.
• An ESMF application with MPI non-accelerated components, an OpenACC accelerated component - and a coupler that uses the newly added support in ESMF to query the number of accelerator devices. The application allocates all devices attached to a node to the lowest ranked MPI process running on the node.
• An ESMF application with MPI non-accelerated components, an IntelMIC accelerated component - and a coupler that uses the newly added support in ESMF to query the number of accelerator devices. The application allocates all devices attached to a node to the lowest ranked MPI process running on the node.
Jayesh developed ~12 example applications, e.g.:
Available Features
Feature OpenCL OpenACC Intel-MIC-DirectivesFind number of accelerator devices
YES YES YES
Can ESMF applications use the accelerator framework?
YES YES YES
Turn off offloading to device at runtime
YES YES NO
Check if current thread is executing on the device
YES YES YES
Mix accelerator frameworks in the same application?
YES YES YES
Next Steps
● Merge the development implementation of accelerator awareness into the main development trunk of ESMF. This includes turning the prototype codes into regular ESMF system tests.
● Prototype framework-guided support for interleaved component for the most effective use of available cores and accelerator devices.
● Explore extending the ESMF distributed data classes (Array and Field) to become aware of the memory topology introduced by the accelerator hardware. Prototype the efficient interaction between data that is distributed differently and located in or out of device memory, minimizing expensive data movements.
ESPC Infrastructure• Update on the Earth System Prediction Suite
(Cecelia DeLuca)
Earth System Prediction Suite
The Earth System Prediction Suite (ESPS) is a collection of flagship U.S. weather and climate models and model components that are instrumented to conform to interoperability conventions, documented following metadata standards, and available either under open source terms or to credentialed users.
Inclusion Criteria1. ESPS components and coupled modeling systems are NUOPC-compliant.2. A minimal, prescribed set of model documentation is provided for each
version of the ESPS component or modeling system.3. ESPS codes must have clear terms of use (e.g. public domain statement,
open source license, proprietary status), and must have a way for credentialed ESPC collaborators to request access.
4. Regression tests are provided for each component and modeling system.
5. There is a commitment to continued NUOPC compliance and ESPS participation for new versions of the code.
Status of Modeling Systems usingNUOPC Conventions
1. ESPS components and coupled modeling systems are NUOPC-compliant.
• NOAA Enviromental Modeling System (NEMS) – now producing early results for Global Spectral Model atmosphere coupled to HYCOM and MOM5 oceans – ice coupling results anticipated soon
• CESM – now producing early results for HYCOM-CICE• COAMPS – system in use includes coupled ocean, atmosphere and wave
components• NavGEM-HYCOM – system in use served as a model for other efforts• Model E – atmosphere and ocean components can be driven, but not
yet producing results
Other ESPS Prep
2. A minimal, prescribed set of model documentation is provided for each version of the ESPS component or modeling system.• Metadata will be based on the Common Information Model (CIM) – same
schema used for CMIP5/IPCC• Questionnaires are now
set up for NUOPCcomponents – very littleinformation currentlyincluded
• Tools include Viewerand Comparator
See: https://www.earthsystemcog.org/projects/esps/
New metadata linksfor ESPS code listings
Other ESPS Prep
3. ESPS codes must have clear terms of use (e.g. public domain statement, open source license, proprietary status), and must have a way for credentialed ESPC collaborators to request access.
• Now reviewing and documenting code access paths
Have not focused on these yet inclusion criteria yet …
4. Regression tests are provided for each component and modeling system.
5. There is a commitment to continued NUOPC compliance and ESPS participation for new versions of the code.
Next Steps
• ESPS BAMS paper proposal accepted in July 2014 – have 6 months to submit
• Finalizing paper for submission – had waited on:– More coupled codes producing results– At least one open source coupled code available for examination– Initial metadata setup– Greater precision in discussion, esp. around behavioral guarantees– Reformatting and images suitable for BAMS
• Last version here:https://www.earthsystemcog.org/projects/esps/resources/
Keep working toward fulfilling ESPS criteria