ESMF Performance Evaluation and Optimization Peggy Li(1), Samson Cheung(2), Gerhard Theurich(2),...
1
ESMF Performance Evaluation and Optimization Peggy Li(1), Samson Cheung(2), Gerhard Theurich(2), Cecelia Deluca(3) (1)Jet Propulsion Laboratory, California Institute of Technology, USA (2) Silicon Graphics Inc., USA (3) National Center for Atmospheric Research (NCAR), USA Objective: We report the results of two performance studies conducted on ESMF applications. The first one is a grid redistribution overhead benchmark based on two different-resolution grids used in the CCSM (Community Climate System Model) and the second one is a scalibility evaluation of the ESMF superstructure functions on large processors. 1. CCSM Grid Redistribution Benchmark Background: CCSM is a fully-coupled, global climate model that provides state-of- the-art computer simulations of the Earth’s past, present, and future climate states. The CCSM 3.0 consists of four dynamical geophysical models, namely, the Community Atmosphere Model (CAM), the Community Land Model (CLM), the Parallel Ocean Program (POP) and the Community Sea-Ice Model (CSIM), linked by a central coupler. CCSM Coupler controls the execution and time evolution of the coupled CCSM system by synchronizing and controlling the flow of data between the various components. Current CCSM Coupler is built on top of MCT (The Model Coupling Toolkit). In this study, we benchmark the performance of one major CCSM coupler function: the grid redistribution from the atmosphere model to the land model . The CCSM3 atmosphere model (CAM) and land model (CLM) share a common horizontal grid. The two resolutions been benchmarked are T85 - a Gaussian grid with 256 longitude points and 128 latitude points and T42 - a Gaussian grid with 128 longitude points and 64 latitude points. Figure 1.a CAM T42 Grid (128x64) Decomposition on 8 processors Figure 1.b CLM T42 Grid (128x64) Decomposition on 8 processors Benchmark Program Our benchmark program contains four components: an Atmosphere Grid Component (ATM), a Land Grid Component (LND), an Atmosphere-to-Land Coupler Component (A2L) and a Land-to-Atmosphere Coupler Component (L2A). The ATM component creates a 2D arbitrarily distributed global rectangular grid and a bundle of 19 floating-point fields associated with the grid. The decomposition of a T42 resolution ATM grid on 8 processors is depicted in Figure 1.a. The LND component contains a bundle of 13 floating-point fields on the land portion of the same 2D global rectangular grid. The LND grid is arbitrarily distributed on 8 processors as shown in Figure 1.b where the dark blue represents no data. The A2L and L2A components perform grid redistribution from ATM grid to the LND grid and vise versa. ESMF handles data redistribution in two stages: the initialization stage that precomputes the communication pattern required for performing the data distribution and the actual data redistribution stage. Our benchmark program measures the performance of the bundle level Redist functions ESMF_BundleRedistStore() and ESMF_BundleRedistRun() between an arbitrarily distributed ATM grid and another arbitrarily distributed LND grid. Contact: [email protected] Full Reports: www.esmf.ucar.edu/main_site/performance.htm Acknowledgment: This task is sponsored by the Modeling, Analysis and Prediction (MAP) Program, National Aeronautics and Space Administration (NASA). Results We ran the benchmark program on the IBM SP Cluster at NCAR and the Cray X1E at Cray Inc using 8 to 128 processors. We measured ESMF_BundleRedistStore() and ESMF_BundleRedistRun() in both A2L and L2A components and compared the timing results on the two platforms. In summary, the Cray X1E performs worse than the IBM SP in both functions. The performance of the data redistribution using ESMF is comparable to CCSM’s current MCT- based approach on both IBM SP and Cray X1E. A. T42 Grid B. T85 Grid CCSM T42 Grid: 128x64 (Time in Milliseconds) # Nodes Init (X1E) Init (IBM) run (X1E) run (IBM) 8 357.0178 40.5002 16.5927 1.2776 16 218.2901 34.1019 14.8972 1.5684 32 389.4656 31.3586 34.928 1.9814 64 425.2421 29.9956 59.4735 2.9228 ESMF_BundleRedistStore ESMF_BundleRedistRun CCSM T85 Grid: 256x128 (Time in milliseconds) # Nodes Init (X1E) Init (IBM) run (X1E) run (IBM) 16 924.6599 150.6831 30.0566 4.0421 32 1087.6294 140.4841 40.592 3.3827 64 1149.7676 124.6631 64.6535 4.6124 128 1728.3291 128.1008 129.1839 7.5746 ESMF_BundleRedistStore ESMF_BundleRedistRun ESMF Grid Redistribution Run Time (256x128 grid) 0 20 40 60 80 100 120 140 0 50 100 150 Number of Processors run (X1E) run (IBM) Optimization: 1. We optimized ESMF_BundleRedistStore() by redesigning a ESMF Route function ESMF_RoutePrecomputeRedistV() that calculates the send and receive route tables in each PET. The new algorithm sorts the local and the global grid points in the order of its grid index to reduce the time to calculate the intersection of the source and the destination grid. 2. We identified two functions that perform poorly on X1E, namely, MPI_Broadcast() and memcpy(). We replaced a loop of MPI_Broadcast() by a single MPI_AllGatherV() in ESMF_BundleRedistStore(). We also replaced memcpy() by assignment statements that was used to copy user data into message buffer in ESMF_BundleRedistRun(). These two modification improves the X1E performance significantly. 2. ESMF Superstructure Scalability Benchmark This benchmark program evaluates the performance of the ESMF Superstructure Functions on large number of processors, i.e., over 1000 processors. The ESMF superstructure functions include the ESMF initialization and termination (ESMF_Initialize(), ESMF_Finalize()), and the component creation, initialization and execution and termination (ESMF_GridCompCreate(), ESMF_GridCompInit(), ESMF_GridCompRun() and ESMF_GridCompFinalize()). We conducted the performance evaluation on the Cray XT3, jaguar, at Oak Ridge National Laboratory and the SGI Altix superclusters, columbia, at NASA Ames. We ran the benchmark from 4 procesors up to 2048 processors. Timing Results on XT3 The performance of ESMF_Initialize()and ESMF_Finalize() is dominated by the parallel I/O performance on the target machine because, by default, each processor opens an Error Log file at ESMF initialization (defaultLogType = ESMF_LOG_MULTI). By setting defaultLogType to ESMF_LOG_NONE, ESMF_Initialize()and ESMF_Finalize() run 200 times faster for 128 processors and above. The timings for these two functions with and without an error log file are shown below. ESMF component functions overheads are very small. ESMF_GridCompRun() time is below 20 microseconds for processors up to 2048. However, except for ESMF_GridCompFinalize(), the other three functions have complexity of O(n) where n is the number of processors. The following table and figures depict the timings of these four component functions on XT3. ESM F Com p o n e n t R o u tin e s Tim in g 1 10 100 1000 10000 100000 1 10 100 1000 10000 N um b er o f P ro cesso rs T im e (m icro seco n d s) GridC om pC reate GridC om pInit G ridC om pRun GridC om pFinalize # Processor s GridComp Create GridComp Init GridComp Run GridComp Finalize 4 59.10 37.20 1.98682 7.90 8 76.10 47.90 1.98682 10.00 16 95.80 64.10 1.98682 12.10 32 122.10 74.10 2.30471 14.80 64 161.80 91.00 2.62260 16.90 128 266.10 105.10 3.01996 19.80 256 604.90 108.90 4.60942 23.80 512 1871.80 115.10 6.99361 27.90 1024 6957.10 430.01 11.28514 35.00 2048 28701.00 998.97 20.98083 47.20 IBM SP2 (bluevista) Cray X1E (earth) Cray XT3/XT4 (jaguar) SGI Altix (columbia) CPU Type and Speed IBM POWER5/ 7.6 GFLOPS/sec MSP (Multi- streaming processor)/ 18 GFLOPS/sec Dual-core AMD Opteron/ 2.6GHz Intel Itanium/ 6GFLOPS/sec Memory per Processor 2GB/processor 16GB shared/node 1TB total 16 GB/compute module, 512GB total 4 GB/processor, 46TB total 2GB/ processor 1TB shared/node Total number of processors 624 (78 8- processor nodes) 128 MSPs (32 4-MSP compute modules) 11,508 processors 10,240 (16 512- processor nodes and one 2048-E system) Aggregated performance 4.74 TFLOPS 2.3 TFLOPS 119 TFLOPS 51.9 TFLOPS Network IBM High Performance Switch (HPS) 5 microsecond DSM Architecture, 2D Torus, 34GB/s memory Cray Seastar Router, 3D Torus SGI NumaLink fabric, 1 microsecond latency XT3 and Altix Comparison We compared the timing results for the six ESMF superstructure functions on Cray XT3 and SGI Altix. The timing charts are shown below. ESM F_ In itialize Tim in g 1 10 100 1000 10000 100000 1 10 100 1000 10000 N um b e r o f P ro ce sso rs T im e (m secs) M ultiple Log Files N o Log Files ESM F_ Fin alize Tim ing 0.01 0.10 1.00 10.00 100.00 1000.00 10000.00 1 10 100 1000 10000 N um b er o f P rocessors Tim e (m secs) M ultiple Log Files N o Log File ESM F_G ridC o m pC re ate T im e (X T3 vs A ltix ) 10 100 1000 10000 100000 4 8 16 32 64 128 256 512 1024 N um b e r o f P ro ce sso rs Tim e (m icro se co n d s Colum bia C ray XT3 ESM F_G ridC om p In it Tim e (X T3 vs A ltix ) 1 10 100 1000 10000 N um b e r o f P ro ce sso rs T im e (m icro se co n d s Colum bia C ray XT3 ESM F_G rid C om pR u n T im e (X T3 vs A ltix) 1 10 100 4 8 16 32 64 128 256 512 1024 N um b e r o f P ro ce sso rs T im e (m icro se co n d s Colum bia C ray X T 3 ESM F_G ridC o m p Fin alize T im e (X T3 vs A ltix ) 1 10 100 N um b e r o f P ro ce sso rs T im e (m icro se co n d s Colum bia C ray XT3 ESMF Component functions overhead on XT3 (numbers are in microseconds) Comparison of the Four Benchmark Machines ESMF Grid Redistribution Initialization Time (128x64 grid) 0 50 100 150 200 250 300 350 400 450 0 10 20 30 40 50 60 70 Number of Processors Time (milliseconds) Init (X1E) Init (IBM) ESMF Grid Redistribution Run Time (128x64 grid) 0 10 20 30 40 50 60 70 0 20 40 60 80 Number of Processors Time (milliseconds) run (X1E) run (IBM) ESMF Grid Redistribution Init Time (256x128 grid) 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 50 100 150 Number of Processors Time (milliseconds) Init (X1E) Init (IBM) ESMF_Initialize() and ESMF_Finalize() time shown in (A) and (B) was made with defaultLogType set to ESMF_LOG_NONE. Altix performs poorer than XT3 in both functions due to synchronization problem and MPI implementation. For ESMF_Initialize(), the time difference between the two machines was due to a global synchronization in the first MPI global operation called, MPI_Comm_Create() in the function. On Altix, MPI_Finalize() takes about 1 seconds regardless the number of processors used, which dominates the time for ESMF_Finalize(). The component functions on both machines have similar performance. ((C) to (F)). The timing for ESMF_GridCompRun() (E) are very close on two machines where XT3 is slightly better for all the configurations. On 1024 processors, it takes 11.28 microseconds on XT3 and 13.84 microseconds on Altix. (E ) (B ) (C ) (D ) (A ) (F ) ESMF_Initialize Time (XT3 vs Altix) 1 10 100 1000 10000 4 8 16 32 64 128 256 512 1024 Number of Processors Time (msecs) Columbia Cray XT3 ESMF_Finalize (XT3 vs Altix) 0.01 0.10 1.00 10.00 100.00 1000.00 10000.00 4 8 16 32 64 128 256 512 1024 Number of Processors Time (msecs) Columbia Cray XT3
-
Upload
august-wilkerson -
Category
Documents
-
view
215 -
download
2
Transcript of ESMF Performance Evaluation and Optimization Peggy Li(1), Samson Cheung(2), Gerhard Theurich(2),...
ESMF Performance Evaluation and OptimizationPeggy Li(1), Samson Cheung(2), Gerhard Theurich(2), Cecelia Deluca(3)
(1)Jet Propulsion Laboratory, California Institute of Technology, USA(2) Silicon Graphics Inc., USA (3) National Center for Atmospheric Research (NCAR), USA
Objective:We report the results of two performance studies conducted on ESMF applications. The first one is a grid redistribution overhead benchmark based on two different-resolution grids used in the CCSM (Community Climate System Model) and the second one is a scalibility evaluation of the ESMF superstructure functions on large processors.
1. CCSM Grid Redistribution Benchmark
Background:CCSM is a fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth’s past, present, and future climate states. The CCSM 3.0 consists of four dynamical geophysical models, namely, the Community Atmosphere Model (CAM), the Community Land Model (CLM), the Parallel Ocean Program (POP) and the Community Sea-Ice Model (CSIM), linked by a central coupler.
CCSM Coupler controls the execution and time evolution of the coupled CCSM system by synchronizing and controlling the flow of data between the various components. Current CCSM Coupler is built on top of MCT (The Model Coupling Toolkit).
In this study, we benchmark the performance of one major CCSM coupler function: the grid redistribution from the atmosphere model to the land model . The CCSM3 atmosphere model (CAM) and land model (CLM) share a common horizontal grid. The two resolutions been benchmarked are T85 - a Gaussian grid with 256 longitude points and 128 latitude points and T42 - a Gaussian grid with 128 longitude points and 64 latitude points.
Figure 1.a CAM T42 Grid (128x64) Decomposition on 8 processors
Figure 1.b CLM T42 Grid (128x64) Decomposition on 8 processors
Benchmark ProgramOur benchmark program contains four components: an Atmosphere Grid Component (ATM), a Land Grid Component (LND), an Atmosphere-to-Land Coupler Component (A2L) and a Land-to-Atmosphere Coupler Component (L2A). The ATM component creates a 2D arbitrarily distributed global rectangular grid and a bundle of 19 floating-point fields associated with the grid. The decomposition of a T42 resolution ATM grid on 8 processors is depicted in Figure 1.a. The LND component contains a bundle of 13 floating-point fields on the land portion of the same 2D global rectangular grid. The LND grid is arbitrarily distributed on 8 processors as shown in Figure 1.b where the dark blue represents no data. The A2L and L2A components perform grid redistribution from ATM grid to the LND grid and vise versa.
ESMF handles data redistribution in two stages: the initialization stage that precomputes the communication pattern required for performing the data distribution and the actual data redistribution stage. Our benchmark program measures the performance of the bundle level Redist functions ESMF_BundleRedistStore() and ESMF_BundleRedistRun() between an arbitrarily distributed ATM grid and another arbitrarily distributed LND grid.
Contact: [email protected]
Full Reports: www.esmf.ucar.edu/main_site/performance.htm
Acknowledgment: This task is sponsored by the Modeling, Analysis and Prediction (MAP) Program, National Aeronautics and Space Administration (NASA).
Contact: [email protected]
Full Reports: www.esmf.ucar.edu/main_site/performance.htm
Acknowledgment: This task is sponsored by the Modeling, Analysis and Prediction (MAP) Program, National Aeronautics and Space Administration (NASA).
ResultsWe ran the benchmark program on the IBM SP Cluster at NCAR and the Cray X1E at Cray Inc using 8 to 128 processors. We measured ESMF_BundleRedistStore() and ESMF_BundleRedistRun() in both A2L and L2A components and compared the timing results on the two platforms. In summary, the Cray X1E performs worse than the IBM SP in both functions. The performance of the data redistribution using ESMF is comparable to CCSM’s current MCT-based approach on both IBM SP and Cray X1E.
A. T42 Grid
B. T85 Grid
CCSM T42 Grid: 128x64 (Time in Milliseconds)
# Nodes Init (X1E) Init (IBM) run (X1E) run (IBM)
8 357.0178 40.5002 16.5927 1.2776
16 218.2901 34.1019 14.8972 1.5684
32 389.4656 31.3586 34.928 1.9814
64 425.2421 29.9956 59.4735 2.9228
ESMF_BundleRedistStore ESMF_BundleRedistRun
CCSM T85 Grid: 256x128 (Time in milliseconds)
# Nodes Init (X1E) Init (IBM) run (X1E) run (IBM)
16 924.6599 150.6831 30.0566 4.0421
32 1087.6294 140.4841 40.592 3.3827
64 1149.7676 124.6631 64.6535 4.6124
128 1728.3291 128.1008 129.1839 7.5746
ESMF_BundleRedistStore ESMF_BundleRedistRun
ESMF Grid Redistribution Run Time (256x128 grid)
0
20
40
60
80
100
120
140
0 50 100 150Number of Processors
Tim
e (
mil
lisecon
ds)
run (X1E)
run (IBM)
Optimization:1. We optimized ESMF_BundleRedistStore() by redesigning a ESMF Route function
ESMF_RoutePrecomputeRedistV() that calculates the send and receive route tables in each PET. The new algorithm sorts the local and the global grid points in the order of its grid index to reduce the time to calculate the intersection of the source and the destination grid.
2. We identified two functions that perform poorly on X1E, namely, MPI_Broadcast() and memcpy(). We replaced a loop of MPI_Broadcast() by a single MPI_AllGatherV() in ESMF_BundleRedistStore(). We also replaced memcpy() by assignment statements that was used to copy user data into message buffer in ESMF_BundleRedistRun(). These two modification improves the X1E performance significantly.
2. ESMF Superstructure Scalability Benchmark
This benchmark program evaluates the performance of the ESMF Superstructure Functions on large number of processors, i.e., over 1000 processors. The ESMF superstructure functions include the ESMF initialization and termination (ESMF_Initialize(), ESMF_Finalize()), and the component creation, initialization and execution and termination (ESMF_GridCompCreate(), ESMF_GridCompInit(), ESMF_GridCompRun() and ESMF_GridCompFinalize()). We conducted the performance evaluation on the Cray XT3, jaguar, at Oak Ridge National Laboratory and the SGI Altix superclusters, columbia, at NASA Ames. We ran the benchmark from 4 procesors up to 2048 processors.
Timing Results on XT3
The performance of ESMF_Initialize()and ESMF_Finalize() is dominated by the parallel I/O performance on the target machine because, by default, each processor opens an Error Log file at ESMF initialization (defaultLogType = ESMF_LOG_MULTI). By setting defaultLogType to ESMF_LOG_NONE, ESMF_Initialize()and ESMF_Finalize() run 200 times faster for 128 processors and above. The timings for these two functions with and without an error log file are shown below.
ESMF component functions overheads are very small. ESMF_GridCompRun() time is below 20 microseconds for processors up to 2048. However, except for ESMF_GridCompFinalize(), the other three functions have complexity of O(n) where n is the number of processors. The following table and figures depict the timings of these four component functions on XT3.
ESMF Component Routines Timing
1
10
100
1000
10000
100000
1 10 100 1000 10000
Number of Processors
Tim
e (
mic
rose
con
ds)
GridCompCreate
GridCompI nit
GridCompRun
GridCompFinalize
# Processors
GridCompCreate
GridCompInit
GridCompRun
GridCompFinalize
4 59.10 37.20 1.98682 7.90
8 76.10 47.90 1.98682 10.00
16 95.80 64.10 1.98682 12.10
32 122.10 74.10 2.30471 14.80
64 161.80 91.00 2.62260 16.90
128 266.10 105.10 3.01996 19.80
256 604.90 108.90 4.60942 23.80
512 1871.80 115.10 6.99361 27.90
1024 6957.10 430.01 11.28514 35.00
2048 28701.00 998.97 20.98083 47.20
IBM SP2
(bluevista)
Cray X1E
(earth)
Cray XT3/XT4
(jaguar)
SGI Altix
(columbia)
CPU Type and Speed
IBM POWER5/
7.6 GFLOPS/sec
MSP (Multi-streaming processor)/
18 GFLOPS/sec
Dual-core AMD Opteron/ 2.6GHz
Intel Itanium/
6GFLOPS/sec
Memory per Processor
2GB/processor
16GB shared/node
1TB total
16 GB/compute module, 512GB total
4 GB/processor,
46TB total
2GB/processor
1TB shared/node
Total number of processors
624 (78 8-processor nodes)
128 MSPs (32 4-MSP compute modules)
11,508 processors
10,240 (16 512-processor nodes and one 2048-E system)
Aggregated performance
4.74 TFLOPS 2.3 TFLOPS 119 TFLOPS 51.9 TFLOPS
Network
IBM High Performance Switch (HPS) 5 microsecond latency
DSM Architecture, 2D Torus, 34GB/s memory bandwidth
Cray Seastar Router, 3D Torus
SGI NumaLink fabric, 1 microsecond latency
XT3 and Altix ComparisonWe compared the timing results for the six ESMF superstructure functions on Cray XT3 and SGI Altix. The timing charts are shown below.
ESMF_ Initialize Timing
1
10
100
1000
10000
100000
1 10 100 1000 10000Number of Processors
Tim
e (
msecs)
Multiple Log FilesNo Log Files
ESMF_ Finalize Timing
0.01
0.10
1.00
10.00
100.00
1000.00
10000.00
1 10 100 1000 10000
Number of Processors
Tim
e (
msecs)
Multiple Log FilesNo Log File
ESMF_ GridCompCreate Time (XT3 vs Altix)
10
100
1000
10000
100000
4 8 16 32 64 128 256 512 1024
Number of Processors
Tim
e (
mic
rosecon
ds)
ColumbiaCray XT3
ESMF_ GridCompInit Time (XT3 vs Altix)
1
10
100
1000
10000
Number of Processors
Tim
e (
mic
roseconds)
ColumbiaCray XT3
ESMF_ GridCompRun Time (XT3 vs Altix)
1
10
100
4 8 16 32 64 128 256 512 1024
Number of Processors
Tim
e (
mic
roseconds)
Columbia
Cray XT3
ESMF_ GridCompFinalize Time (XT3 vs Altix)
1
10
100
Number of Processors
Tim
e (
mic
roseconds)
ColumbiaCray XT3
ESMF Component functions overhead on XT3 (numbers are in microseconds)
Comparison of the Four Benchmark Machines
ESMF Grid Redistribution Initialization Time(128x64 grid)
0
50
100
150
200
250
300
350
400
450
0 10 20 30 40 50 60 70
Number of Processors
Time (milliseconds)
Init (X1E)
Init (IBM)
ESMF Grid Redistribution Run Time(128x64 grid)
0
10
20
30
40
50
60
70
0 20 40 60 80
Number of Processors
Time (milliseconds)
run (X1E)
run (IBM)
ESMF Grid Redistribution Init Time (256x128 grid)
0200400600800
10001200
1400160018002000
0 50 100 150
Number of Processors
Time (milliseconds) Init (X1E)
Init (IBM)
ESMF_Initialize() and ESMF_Finalize() time shown in (A) and (B) was made with defaultLogType set to ESMF_LOG_NONE. Altix performs poorer than XT3 in both functions due to synchronization problem and MPI implementation. For ESMF_Initialize(), the time difference between the two machines was due to a global synchronization in the first MPI global operation called, MPI_Comm_Create() in the function. On Altix, MPI_Finalize() takes about 1 seconds regardless the number of processors used, which dominates the time for ESMF_Finalize().
The component functions on both machines have similar performance. ((C) to (F)). The timing for ESMF_GridCompRun() (E) are very close on two machines where XT3 is slightly better for all the configurations. On 1024 processors, it takes 11.28 microseconds on XT3 and 13.84 microseconds on Altix.
ESMF_Initialize() and ESMF_Finalize() time shown in (A) and (B) was made with defaultLogType set to ESMF_LOG_NONE. Altix performs poorer than XT3 in both functions due to synchronization problem and MPI implementation. For ESMF_Initialize(), the time difference between the two machines was due to a global synchronization in the first MPI global operation called, MPI_Comm_Create() in the function. On Altix, MPI_Finalize() takes about 1 seconds regardless the number of processors used, which dominates the time for ESMF_Finalize().
The component functions on both machines have similar performance. ((C) to (F)). The timing for ESMF_GridCompRun() (E) are very close on two machines where XT3 is slightly better for all the configurations. On 1024 processors, it takes 11.28 microseconds on XT3 and 13.84 microseconds on Altix.
(E)
(B)
(C)(D)
(A)
(F)
ESMF_Initialize Time (XT3 vs Altix)
1
10
100
1000
10000
4 8 16 32 64 128 256 512 1024
Number of Processors
Time (msecs)
Columbia
Cray XT3
ESMF_Finalize (XT3 vs Altix)
0.01
0.10
1.00
10.00
100.00
1000.00
10000.00
4 8 16 32 64 128 256 512 1024
Number of Processors
Time (msecs)
Columbia
Cray XT3
