Optimizing Performance Management Gina Fisk, LANL Senior Cyber Security Manager gina@lanl
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Layer 2 nd...
-
Upload
charlene-bruce -
Category
Documents
-
view
217 -
download
0
Transcript of NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Layer 2 nd...
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Infrastructure Layer
2nd ESMF Community Meeting, GFDL, May 15th 2003
http://www.esmf.ucar.edu, [email protected]
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Talk Outline
• What is the infrastructure layer– What is it for– What it contains
• ESMF infrastructure in this release – ESMF version 1.0 function call† examples (code!!!)
(†aka API or methods)
• Next steps….• Infrastructure Utility Example
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Infrastructure Layer
1. A standard software platform for enabling interoperability (developing couplers, ensuring performance portability).
2. Set of reusable software for Earth science applications. Streamlined development for researchers.
NCARAtmosphere
GFDLOcean NSIPP
Land
NCARAtmosphere
GFDLOcean NSIPP
Land
MySea Ice
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Infrastructure Layer Scope
SupportSupport for• Physical Grids• Regridding• Decomposition/composition• Communication• Calendar and Time• I/O• Logging and Profiling
ESMF Infrastructure
User Code
ESMF Superstructure
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
2. ESMF provides a toolkit that components use toi. ensure interoperabilityii. abstract common services
Location Within ESMFLocation Within ESMF
Component: run(), checkpoint()
Field: halo(), import(), export() + I/O
Grid: regrid(), transpose() + Metrics
Layout, PEList, Machine Model
Application ComponentGridded
ComponentsCoupler
Components
1. ESMF provides an environment for assembling components.
INFRASTRUCTURE
INFRASTRUCTURE
LAYERLAYER
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Talk Outline
• What is the infrastructure layer– What is it for– What it contains
• ESMF infrastructure in this release – ESMF version 1.0 function call† examples (code!!!)
(†aka API or methods)
• Next steps….• Infrastructure utility example
Infrastructure Internal Organization
MachineModel
DELayout
DistGridPhysGrid
Bundle
Array
Time, Alarm, Calendar, LogErr, I/O, Attributes
F90
C++
Field
Grid
Regrid
Comm
Data Communications
Route
Two tiers
Class hierarchy for data and communications
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Infrastructure Fields and Grids
Field Contains array, grid, array to grid Contains array, grid, array to grid mapping and metadatamapping and metadata
Array Holds actual data e.g. temperature, wind Holds actual data e.g. temperature, wind speed etc…speed etc…
Grid Contains physical and distributed gridContains physical and distributed grid
Physical Grid Grid metrics, e.g. lat-lon coords, Grid metrics, e.g. lat-lon coords, spacings, areas, volumes etc..spacings, areas, volumes etc..
Distributed Grid Parallel decomposition informationParallel decomposition information
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Creating an ESMF Field
1. Fortran Array
2. Array Attach
3. Field Attach
Array
real, dimension(100,100) :: arr
esArr = ESMF_ArrayCreate(arr,…)
ESMF_Array
info.
ESMF_Array
info.
ESMF_FieldAttachArray(esFld,esArr,…)
ESMF_Fieldmetadata
ESMF_Grid
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Creating an ESMF Grid
ESMF_Grid
PhysGrid
DistGrid
call ESMF_PhysGridCreate…(…) call ESMF_DistGridCreate…(…)
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Field and Grid Together :grid = ESMF_GridCreate(…,layout,…) :field_u = ESMF_FieldCreate(grid, array) :
1. Create field distributed over a set of decomposition elements (DE’s).
2. Domain decomposition determined by DELayout, layout.3. Each object (grid and field_u) has internal representation.4. Other parts of the infrastructure layer use the internal
representation e.g.• Regrid() – interpolation/extrapolation + redistribute over DE’s• Redistribution() - general data rearrangement over DE’s• Halo() – specialized redistribution
ESMF_Array
info.
ESMF_Fieldmetadata
ESMF_Grid
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Regrid• Function mapping field’s array to a different physical and distributed grid.
• RegridCreate() – creates a Regrid structure to be used/re-used
regrid = ESMF_RegridCreate(src_field, dst_field, method, [name], [rc])
Source, Dest field can be empty of field data (RegridCreate() uses grid metrics)– PhysGrid, DistGrid info used for setting up regrid– Resulting regrid can be used for other fields sharing same Grid
• Method specifies interpolation algorithm. For example, bilinear, b-spline, etc…
},,{},,{ DPADPA
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Regrid Interface (cont)• RegridRun() – performs actual regridding call ESMF_RegridRun(src_field, dst_field, regrid,[rc])
– Communication and interpolation handled transparently.
• RegridDestroy() – frees up memory call ESMF_RegridDestroy(regrid,[rc])
src_field
dst_field
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Redistribution• No interpolation or extrapolation• Maps between distributed grids, field’s array and physical grid are the
same i.e.
• Example: layout() created two distributed grids one decomposed in X and one decomposed in Y.
– Redistribution() function maps array data between the distributions (a transpose/corner turn)
– Communication handled transparently
},,{},,{ DPADPA
src_field dst_field
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Halo • Fields have distributed index space of
– Exclusive {E}, compute {C} and local {L} region where
• Halo() fills points not in {E} or {C} from remote {E} e.g
{E}
{C}{L}
{C}{L}
{E}
ECL
call ESMF_FieldHalo(field_foo, status)
DE3 DE4
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICHMore functions in reference manual e.g
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Bundle and Location Stream
• ESMF_Bundle– collection of fields on the same grid
• ESMF_LocationStream– Like a field but…..– unstructured index space with an associated
physical grid space– useful for observations e.g. radiosonde, floats
• Functions for create(), regrid(), redistribute(), halo() etc…
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Infrastructure Utilities• Clock• Alarm• Calendar• I/O• Logging• Profiling• Attribute• Machine modeland comms
Ensures consistent time between components
Provides field level I/O in standard forms – netCDF, binary, HDF, GRIB, Bufr
Consistent monitoring and messaging
Consistent parameter handling
Hardware and system software hiding. Platform customizable
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time• Standard type for any component
• Calendar (support for range of calendars)
! initialize stop time to 13May2003, 2:00 pmcall ESMF_TimeInit(inject_stop_time, & YR=int(2003,kind=ESMF_IKIND_I8), & MM=off_month, DD=off_day, H=off_hour, M=off_min, & S=int(0,kind=ESMF_IKIND_I8), & cal=gregorianCalendar, rc=rc)
do while (currTime .le. inject_stop_time ) : call ESMF_ClockAdvance(localclock, rc=rc) call ESMF_ClockGetCurrTime(localclock, currtime, rc)end
call ESMF_CalendarInit(gregorianCalendar, ESMF_CAL_GREGORIAN, rc)
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
I/O• Field level binary, netCDF, HDF, GRIB, bufr
– Currently I/O piped through 1 PE call ESMF_FieldAllGather(field_u, outarray, status) if (de_id .eq. 0) then write(filename, 20) "U_velocity", file_no call ESMF_ArrayWrite(outarray, filename=filename,
rc=status) endif call ESMF_ArrayDestroy(outarray, status)
– Current system fixed textual.
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Attributes• Flexible parameter specs, configuration e.g. file
of parameters
Code
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Internal Classes
• Machine model– Captures system attributes, CPU, mem,
connectivity graph– Useful for defining decomposition, load-balance,
performance predictions.
• Comms– Communication driver, allows bindings to MPI,
shared memory, vendor system libraries
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICHComms Performance Test
Right mix (green) on Compaq gives x2 realized bandwidth (in large message limit)
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Talk Outline• What is the infrastructure layer
– What is it for– What it contains
• ESMF infrastructure in this release – ESMF version 1.0 function call† examples (code!!!)
(†aka API or methods)
• Next steps….• Infrastructure utility example
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Regrid Next Steps• Support for all ESMF Grids• Support for regridding methods:BilinearBicubic1st-order Conservative2nd-order ConservativeRasterized ConservativeNearest-neighbor distance-weighted averageSpectral transforms1-d interpolations (splines)Index-space (shifts, stencils)Adjoints of many above
Halo Next Steps• Support for periodicity• More general haloing
– in tandem with distributed grid evolution
• Adjoint forms
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Distributed Grid • Regular 2d already supported• Next steps
– Generalized 1d decomposition– Extend support for 2d and quasi regular
decomposition – Spectral grid decompositions
• Larger set of metrics, grids• High level routines for rapid definition of common grids.
Physical Grid Next Steps
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
I/O Next Steps• Broaden format set, binary, netCDF, HDF, GRIB,
bufr• Improve parallelization
• Full support for alarms• Broader functionality
Time/Logging/Profiling Next Steps
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Summary
• ESMF Infrastructure Layer– Comprehensive class structure available in
version 1.0– Over the coming year significant extension of
functionality will take place.– Feedback and comments on version 1.0 welcome
http://www.esmf.ucar.edu
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Infrastructure Utility detailed example
Earl Schwab, ESMF Core Team, NCAR
Time Manager
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
What is Time Manager?
• Clock for time simulation• Time representation• Time calculator• Time comparisons• Time queries• F90 API, C++ implementation
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Clock for time simulation
type(ESMF_Clock) :: clock
call ESMF_ClockInit(clock, timeStep, startTime, stopTime, rc=rc)
do while (.not.ESMF_ClockIsStopTime(clock, rc))
! Do application work
.
.
.
call ESMF_ClockAdvance(clock, rc=rc)
end do
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Clock (cont.)
• Clock queries/commands
call ESMF_ClockGetCurrTime(clock, currTime, rc)call ESMF_ClockSetCurrTime(clock, currTime, rc)call ESMF_ClockGetTimeStep(clock, timeStep, rc)call ESMF_ClockSetTimeStep(clock, timeStep, rc)call ESMF_ClockGetAdvanceCount(clock, advanceCount, rc)call ESMF_ClockGetStartTime(clock, startTime, rc)call ESMF_ClockGetStopTime(clock, stopTime, rc)call ESMF_ClockGetPrevTime(clock, prevTime, rc)call ESMF_ClockSyncToWallClock(clock, rc)
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time Representation
type(ESMF_Calendar) :: calendar1
call ESMF_CalendarInit(calendar1, ESMF_CAL_GREGORIAN, rc)
- ESMF_CAL_GREGORIAN (3/1/-4800 to 10/29/292,277,019,914)
- ESMF_CAL_JULIAN (+/- 106,751,991,167,300 days)
- ESMF_CAL_NOLEAP
- ESMF_CAL_360DAY
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time Representation (cont.)
type(ESMF_Time) :: time1call ESMF_TimeInit(time1, YR=int( 2003 ,kind=ESMF_IKIND_I8), &
MM= 5 , DD= 15 , H= 15 , cal=calendar1, rc=rc)
- YR, MM, DD, H, M, S F90 optional- D, H, M, S arguments
type(ESMF_TimeInterval) :: timeInterval1call ESMF_TimeIntervalInit(timeInterval1,
D=int( 90 ,kind=ESMF_IKIND_I8), rc=rc)
- D, H, M, S F90 optional arguments
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time Calculator
• Time differencing• Time increment/decrement by a time interval• Time interval arithmetic (+, -, *, /)
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time Calculator (cont.)
call ESMF_TimeInit(time1, YR=int( 2003 ,kind=ESMF_IKIND_I8), & MM= 5 , DD= 15 , cal=gregorianCalendar, rc=rc)
call ESMF_TimeInit(time2, YR=int( 2003 ,kind=ESMF_IKIND_I8), & MM= 3 , DD= 26 , cal=gregorianCalendar, rc=rc)
call ESMF_TimeIntervalInit(timeInterval1, D=int( 90 ,kind=ESMF_IKIND_I8, rc=rc)
timeInterval2 = time2 - time1time1 = time1 + timeInterval1 ! Uses F90 overloaded timeInterval3 = timeInterval1 * 2 ! operators
double precision :: ratioratio = timeInterval1 / timeInterval2
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time Comparisons
>, <, >=, <=, ==, <> F90 overloaded operators between any 2 times or time intervals
if (time1 < time2) then…
end if
if (timeInterval1 .ge. timeInterval2) then…
end if
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Time Queries
call ESMF_TimeGet(time1, YR=yr, MM=mm, DD=dd, H=h, M=m, S=s, rc=rc)call ESMF_TimeIntervalGet(timeInterval1, D=d, H=h, M=m, S=s)
call ESMF_TimeGetDayOfYear(time1, dayOfYear, rc) ! double or integercall ESMF_TimeGetDayOfMonth(time1, dayOfMonth, rc)call ESMF_TimeGetDayOfWeek(time1, dayOfWeek, rc)call ESMF_TimeGetMidMonth(time1, midMonth, rc)
call ESMF_TimeGetString(time1, string, rc) ! 2003-05-14T12:20:19 (ISO 8601)call ESMF_TimeIntervalGetString(timeInterval1, string, rc) ! P1DT12H0M0S (ISO)
call ESMF_TimeGetRealTime(time1, rc)
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
More information
• ESMF User’s Guide• ESMF Reference Manual• ESMF Requirements Document• Time Manager F90 API & examples source code
esmf_1_0_0_r/src/Infrastructure/TimeMgr/interface
esmf_1_0_0_r/src/Infrastructure/TimeMgr/examples