Showcasing the hierarchical testing ... - verif.rap.ucar.edu*[email protected] % Currently affiliated...
1
Showcasing the hierarchical testing framework established by the Global Model Test Bed (GMTB) Michelle Harrold* 1,4 , Grant Firl 1,4 , Man Zhang 2,3,4 , Jamie Wolff 1,4 , Judy Henderson 2,4 , Ligia Bernardet 2,3,4 , Joshua Hacker % 1 National Center for Atmospheric Research/Research Applications Laboratory ● 2 NOAA/Earth System Research Laboratory/Global Systems Division 3 Cooperative Institute for Research in the Environmental Sciences ● 4 Developmental Testbed Center The Global Model Test Bed (GMTB) team was established within the Developmental Testbed Center (DTC) to facilitate community involvement in the development of the Next Generation Global Prediction System (NGGPS) by supporting a hierarchical testing framework. The GMTB is actively developing a uniform ‘test harness’ to enable in-depth investigation of various physical parameterizations and advanced physics suites. The goals of the hierarchical testing framework are to provide: • A common infrastructure for testing physics developments that works across all temporal and spatial scales and facilitates an efficient R2O pipeline • Simple-to-complex testing • A framework for evidenced-based decision making • Streamlined testing to accelerate transfer of worthy improvements into operations What is the GMTB? Hierarchical Testing • GMTB successfully developed an SCM that has been a valuable tool for physics developers within the hierarchical testing framework • GMTB successfully implemented a framework for performing coarse resolution global forecasts – including pre-processing, forecasts, post-processing, graphics, and verification • Using the SCM and global workflow, the GMTB undertook a testing and evaluation effort to compare 2 schemes: scale-aware simplified Arakawa Schubert and the untuned Grell-Freitas • GMTB will continue to expand aspects of the hierarchical testing framework • Input and collaboration from the community is essential to the success of the testbed! Acknowledgments: Collaboration with Georg Grell was instrumental to the success of implementing the GF code in NEMS. Vijay Tallapragada and the EMC global team shared their workflow capabilities and knowledge of their verification procedures that were heavily leveraged in this effort. The DTC is funded by the National Oceanic and Atmospheric Administration, the Air Force, and the National Center for Atmospheric Research (NCAR). NCAR is sponsored by the National Science Foundation (NSF). Parameterization Simulator Limited Area Domain LR/MR Global Reforecast/Forecast Coupled/Tuning/Climate Modes Single Column Model Physics Testing Hierarchy Physics Dev. DTC Operational Centers Responsibility Legend LR/MR Global DA Mode Operational Pre-Implementation • The hierarchical testing capability within the GMTB was used to support a testing and evaluation effort to compare the GFS’s operational convective parameterization [Scale-Aware Simplified Arakawa Schubert (SASAS)] against an experimental configuration using a more advanced, scale-aware parameterization, the Grell–Freitas (GF) scheme • Test plan was created jointly with EMC, NGGPS, Program Office, and the physics developer (G. Grell) • Invokes concept of hierarchical testbed § SCM § Global retrospective runs ü 20160601 – 20160615; 00 UTC initializations ü Global diagnostics ü Global verification • All experiments were performed at a resolution of T574; cycled DA was employed for global runs Initialization Data Graphics Verification Pre-processing Forecast Post-processing • Total precipitation partitions: • GF has more explicit and less convective precip than SASAS • GF produces more precip over ocean than SASAS; both produce similar amounts over land Single Column Model GMTB Workflow ü Highly flexible and configurable ü Python for graphics ü DTC’s Model Evaluation Tools for verification Workflow from EMC ü GMTB keeping pace with EMC ü GMTB & EMC collaboration Hierarchical Testing Results Conclusion & Next Steps *[email protected] % Currently affiliated w/ Jupiter. GF SASAS GF-SAS Global Runs – Diagnostics Global Runs – Verification Total precip rate Verification performed by GMTB: • Grid-to-grid ü Precipitation (6-h and daily accum.) § Frequency bias § Equitable threat score ü 500 hPa height § Anomaly correlation • Grid-to-point ü Upper-air (T, RH, wind, height) § Bias, RMSE ü Near-surface (T, RH, wind, PRMSL) § Bias, RMSE • Performed over: ü Global sub-domain (Grid 3, 1 ∘ ×1 ∘ ) ü CONUS sub-domain (Grid 218, 12km) Global Workflow Diagnostics • For full DTC testing of GF scheme: ü No tuning was performed ü Deep and shallow convection turned on; mid- level scheme was not used ü Diurnal cycle adjustment was employed ü For closure option, average of all possible closures was used • Testing was first performed with non-cycled runs à progression to next tier of advanced testing included cycled DA • The DTC Visitor Program supports visitors to work with the DTC to test new forecasting and verification techniques, models and model components: http://www.dtcenter.org/visitors/ • Currently accepting applications! • Potential areas related to GMTB: ü Additional cases in SCM ü Additional model diagnostics ü Physics developers modifying schemes for inclusion into Common Community Physics Package (CCPP) DTC Visitor Program Diagnostics provided by GMTB: • For a single case (20160610), several diagnostics were investigated to better understand the behavior of the two configurations: ü Precipitation § Precipitation rate § Convective/explicit partition ü Components of water budget ü Cloud and radiation variables § Low, mid, and high cloud fraction § LW and SW and surface and TOA • Regional investigation over the Tropics: • Tropical region: 20° S – 20° N • Uses gridded CMORPH analyses (satellite microwave observations) as truth • Compared to CMORPH, both SASAS and GF overestimate precipitation, with GF overestimating at a higher level • Both SASAS and GF display an increasing trend in precipitation as the forecast progresses in time Convective precip rate Explicit precip rate Precip rate over Ocean Precip rate over Land • Given the results in both the SCM and case study, the average convective precipitation over the retrospective period was calculated to visualize the differences between SASAS and GF • Both GF and SASAS are appropriately capturing the ITCZ, including the SPCZ • Large differences over northern part of South America, where GF has minimal convective precipitation • With exception to areas off the west coasts of India and Indonesia, SASAS typically has more organized areas of stronger convective precipitation GF SASDA GF-SASDA 6-h Accum. Convective Precip. – f240 NH SH TROP NH SH TROP • In all regions, overall transition from high bias at lower thresholds to low bias at higher thresholds • NH and SH: Minimal statistically significant (SS) differences • Tropics: GF often SS better 6-h Accumulated Precipitation Frequency Bias – F120 6-h Accumulated Precipitation ETS – F120 • In NH and Tropics, skill decreases as threshold increases • In NH and Tropics, SS differences show SASAS as having higher skill • In the SH, performance is similar – minimal differences • “Scorecards” are an excellent tool to identify patterns in the difference of performance between two configurations, including level of significance, for metrics, variables, levels, regions, and times • Scorecard of near-surface temperature, relative humidity (RH), and wind: • Clear diurnal signal in statistical results, with differences often at times valid at 18 and 00 UTC • SASAS typically better for RH and wind RMSE • GF frequently outperforms SASAS when considering temperature and RH bias • Plot on right shows 2-m temperature bias over the CONUS; corresponds to the first line of scorecard • Note the diurnal signal and relationship to performance 2-m Temperature Bias over CONUS SCM developed by GMTB: • Uses Interoperable Physics Driver (IPD) to connect to GFS operational physics • Portable code has minimal dependencies • Two cases currently available: ü Maritime deep convective GCSS case based on TWP-ICE field campaign ü Continental deep convective GCSS case based on ARM SGP Summer 1997 IOP • Available to friendly users via NOAA’s Vlab • Features complete documentation and User’s Guide: http://www.dtcenter.org/GMTB/gmtb_scm_doc/ Case Setup • GCSS case from TWP-ICE field campaign • Deep and suppressed convection near Darwin, Australia during 1/20-2/12/2006 • Forced by fixed SSTs, prescribed horizontal advective tendencies of T, q, prescribed vertical velocity, and nudged u,v • 100-member forcing ensemble that varies forcing based on uncertainty in precip measurement -20 -15 -10 -5 0 5 10 15 T tendencies (K day -1 ) 0 20000 40000 60000 80000 p (Pa) SASAS GF force PBL Conv MP LW SW -8 -6 -4 -2 0 2 4 6 8 q tendencies (g kg -1 day -1 ) 0 20000 40000 60000 80000 p (Pa) SASAS GF force PBL Conv MP Jan 20 2006 Jan 21 2006 Jan 22 2006 Jan 23 2006 Jan 24 2006 Jan 25 2006 date 0 1 2 3 4 5 6 (mm/hr) SASAS GF total convective 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 total surface precipitation rate (mm h -1 ) 0.5 0.6 0.7 0.8 0.9 1.0 convective precipitation ratio 0.0 0.5 1.0 1.5 2.0 2.5 3.0 total surface precipitation rate (mm h -1 ) 0.0 0.2 0.4 0.6 0.8 1.0 convective precipitation ratio Precip Partition: Deep Convective Period Physics Scheme Control / GF Surface Noah (ocean surface) Radiation RRTMG PBL Hybrid EDMF Microphyics Zhao-Carr Deep & Shallow Con. SASAS / GF • Key SCM results: • GF produces weaker tendencies, leaving the microphysics to do more “work" to balance the forcing • The GF suite, for both the deep and shallow convective periods, produced much lower convective precipitation ratio • SASAS produces more convective precipitation q tendencies during active phase T tendencies during active phase Precip Partition: Shallow Convective Period Active Phase Precipitation SASAS GF Total Convective ---- SASAS GF ---- SASAS GF
Transcript of Showcasing the hierarchical testing ... - verif.rap.ucar.edu*[email protected] % Currently affiliated...
Showcasing the hierarchical testing framework established by the Global Model Test Bed (GMTB)
Michelle Harrold*1,4, Grant Firl1,4, Man Zhang2,3,4, Jamie Wolff1,4, Judy Henderson2,4, Ligia Bernardet2,3,4, Joshua Hacker%
1 National Center for Atmospheric Research/Research Applications Laboratory ● 2 NOAA/Earth System Research Laboratory/Global Systems Division3 Cooperative Institute for Research in the Environmental Sciences ● 4 Developmental Testbed Center
The Global Model Test Bed (GMTB) team was established within the Developmental Testbed Center (DTC) to facilitate community involvement in the development of the Next Generation Global Prediction System (NGGPS) by supporting a hierarchical testing framework.The GMTB is actively developing a uniform ‘test harness’ to enable in-depth investigation of various physical parameterizations and advanced physics suites. The goals of the hierarchical testing framework are to provide:
• A common infrastructure for testing physics developments that works across all temporal and spatial scales and facilitates an efficient R2O pipeline
• Simple-to-complex testing• A framework for evidenced-based decision making• Streamlined testing to accelerate transfer of worthy
improvements into operations
What is the GMTB? Hierarchical Testing
• GMTB successfully developed an SCM that has been a valuable tool for physics developers within the hierarchical testing framework
• GMTB successfully implemented a framework for performing coarse resolution global forecasts – including pre-processing, forecasts, post-processing, graphics, and verification
• Using the SCM and global workflow, the GMTB undertook a testing and evaluation effort to compare 2 schemes: scale-aware simplified Arakawa Schubert and the untuned Grell-Freitas
• GMTB will continue to expand aspects of the hierarchical testing framework• Input and collaboration from the community is essential to the success of the testbed!
Acknowledgments: Collaboration with Georg Grell was instrumental to the success of implementing the GF code in NEMS. Vijay Tallapragada and the EMC global team shared their workflow capabilities and knowledge of their verification procedures that were heavily leveraged in this effort. The DTC is funded by the National Oceanic and Atmospheric Administration, the Air Force, and the National Center for Atmospheric Research (NCAR). NCAR is sponsored by the National Science Foundation (NSF).
Parameterization Simulator
Limited Area Domain
LR/MR Global Reforecast/Forecast
Coupled/Tuning/Climate Modes
Single Column Model
Physics Testing Hierarchy
Physics Dev.
DTC
Operational Centers
Responsibility Legend
LR/MR Global DA Mode
Operational Pre-Implementation
• The hierarchical testing capability within the GMTB was used to support a testing and evaluation effort to compare the GFS’s operational convective parameterization [Scale-Aware Simplified Arakawa Schubert (SASAS)] against an experimental configuration using a more advanced, scale-aware parameterization, the Grell–Freitas (GF) scheme
• Test plan was created jointly with EMC, NGGPS, Program Office, and the physics developer (G. Grell)
• Invokes concept of hierarchical testbed§ SCM§ Global retrospective runs
ü20160601 – 20160615; 00 UTC initializationsüGlobal diagnosticsüGlobal verification
• All experiments were performed at a resolution of T574; cycled DA was employed for global runs
Initialization Data
Graphics
Verification
Pre-processing
Forecast
Post-processing
• Total precipitation partitions:• GF has more explicit and less
convective precip than SASAS• GF produces more precip over
ocean than SASAS; both produce similar amounts over land
Single Column Model
GMTB WorkflowüHighly flexible and
configurableüPython for graphicsüDTC’s Model
Evaluation Tools for verification
Workflow from EMCüGMTB keeping
pace with EMCüGMTB & EMC
collaboration
Hierarchical Testing Results
Conclusion & Next Steps
% Currently affiliated w/ Jupiter.
GF SASAS GF-SAS
Global Runs – Diagnostics
Global Runs – Verification
Total precip rate
Verification performed by GMTB:•Grid-to-gridü Precipitation (6-h and daily accum.)
§ Frequency bias§ Equitable threat score
ü 500 hPa height§ Anomaly correlation
•Grid-to-pointü Upper-air (T, RH, wind, height)
§ Bias, RMSEü Near-surface (T, RH, wind, PRMSL)
§ Bias, RMSE•Performed over:ü Global sub-domain (Grid 3, 1∘×1∘)ü CONUS sub-domain (Grid 218, 12km)
Global Workflow
Diagnostics
• For full DTC testing of GF scheme:ü No tuning was performedü Deep and shallow convection turned on; mid-
level scheme was not usedü Diurnal cycle adjustment was employedü For closure option, average of all possible
closures was used• Testing was first performed with non-cycled
runs à progression to next tier of advanced testing included cycled DA
• The DTC Visitor Program supports visitors to work with the DTC to test new forecasting and verification techniques, models and model components: http://www.dtcenter.org/visitors/
• Currently accepting applications!• Potential areas related to GMTB:
ü Additional cases in SCMü Additional model diagnosticsü Physics developers modifying schemes for inclusion into Common
Community Physics Package (CCPP)
DTC Visitor Program
Diagnostics provided by GMTB:• For a single case (20160610),
several diagnostics were investigated to better understand the behavior of the two configurations:üPrecipitation§ Precipitation rate§ Convective/explicit partition
üComponents of water budgetüCloud and radiation variables§ Low, mid, and high cloud fraction§ LW and SW and surface and TOA
• Regional investigation over the Tropics:• Tropical region: 20° S – 20° N• Uses gridded CMORPH analyses
(satellite microwave observations) as truth• Compared to CMORPH, both SASAS and
GF overestimate precipitation, with GF overestimating at a higher level
• Both SASAS and GF display an increasing trend in precipitation as the forecast progresses in time
Convective precip rate
Explicit precip rate
Precip rate over Ocean
Precip rate over Land
• Given the results in both the SCM and case study, the average convective precipitation over the retrospective period was calculated to visualize the differences between SASAS and GF
• Both GF and SASAS are appropriately capturing the ITCZ, including the SPCZ
• Large differences over northern part of South America, where GF has minimal convective precipitation
• With exception to areas off the west coasts of India and Indonesia, SASAS typically has more organized areas of stronger convective precipitation
GF
SASDA
GF-SASDA
6-h Accum. Convective Precip. – f240
NH SH
TROP
NH SH
TROP • In all regions, overall transition from high bias at lower thresholds to low bias at higher thresholds
• NH and SH: Minimal statistically significant (SS) differences
• Tropics: GF often SS better
6-h Accumulated Precipitation Frequency Bias – F120 6-h Accumulated Precipitation ETS – F120
• In NH and Tropics, skill decreases as threshold increases
• In NH and Tropics, SS differences show SASAS as having higher skill
• In the SH, performance is similar – minimal differences
• “Scorecards” are an excellent tool to identify patterns in the difference of performance between two configurations, including level of significance, for metrics, variables, levels, regions, and times
• Scorecard of near-surface temperature, relative humidity (RH), and wind: • Clear diurnal signal in statistical results, with
differences often at times valid at 18 and 00 UTC• SASAS typically better for RH and wind RMSE• GF frequently outperforms SASAS when
considering temperature and RH bias• Plot on right shows 2-m temperature bias over the CONUS; corresponds to the first line of scorecard• Note the diurnal signal and relationship to
performance
2-m Temperature Bias over CONUS
SCM developed by GMTB:• Uses Interoperable Physics Driver (IPD)
to connect to GFS operational physics• Portable code has minimal dependencies• Two cases currently available:
ü Maritime deep convective GCSS case based on TWP-ICE field campaign
ü Continental deep convective GCSS case based on ARM SGP Summer 1997 IOP
• Available to friendly users via NOAA’s Vlab
• Features complete documentation and User’s Guide: http://www.dtcenter.org/GMTB/gmtb_scm_doc/
Case Setup• GCSS case from TWP-ICE field campaign• Deep and suppressed convection near Darwin, Australia
during 1/20-2/12/2006• Forced by fixed SSTs,
prescribed horizontal advective tendencies of T, q, prescribed vertical velocity, and nudged u,v
• 100-member forcing ensemble that varies forcing based on uncertainty in precip measurement
�20 �15 �10 �5 0 5 10 15
T tendencies (K day�1)
0
20000
40000
60000
80000
p(P
a)
SASASGF
forcePBLConvMPLWSW
�8 �6 �4 �2 0 2 4 6 8
q tendencies (g kg�1 day�1)
0
20000
40000
60000
80000
p(P
a)
SASASGF
forcePBLConvMP
Jan 20 2006 Jan 21 2006 Jan 22 2006 Jan 23 2006 Jan 24 2006 Jan 25 2006date
0
1
2
3
4
5
6
(mm
/hr)
SASASGF
totalconvective
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
total surface precipitation rate (mm h�1)
0.5
0.6
0.7
0.8
0.9
1.0
conv
ectiv
epr
ecip
itatio
nra
tio
0.0 0.5 1.0 1.5 2.0 2.5 3.0
total surface precipitation rate (mm h�1)
0.0
0.2
0.4
0.6
0.8
1.0
conv
ectiv
epr
ecip
itatio
nra
tio
Precip Partition:Deep Convective Period
Physics Scheme Control / GFSurface Noah (ocean surface)
Radiation RRTMG
PBL Hybrid EDMF
Microphyics Zhao-Carr
Deep & Shallow Con. SASAS / GF
• Key SCM results:• GF produces weaker tendencies,
leaving the microphysics to do more “work" to balance the forcing
• The GF suite, for both the deep and shallow convective periods, produced much lower convective precipitation ratio
• SASAS produces more convective precipitation
q tendencies during active phase T tendencies during active phase
Precip Partition: Shallow Convective Period
Active Phase Precipitation
SASASGF
TotalConvective ----
SASASGF ----
SASASGF
