Verification of Global Ensemble Forecasts

Fanglin Yang Yuejian Zhu, Glenn White, John Derber

Environmental Modeling Center National Centers for Environmental Prediction

College Park, Maryland, USA

GCWMB Bi-weekly Briefing, September 19, 2013

1

http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/ensm/

A new Web Page: Verification of Ensemble Mean Forecasts

• The site displays verification statistics for global ensemble mean forecasts. It does not contain any verification of ensemble probabilistic forecast.

• For a complete list of ensemble verification products, please visit Ensemble Team Web and its Global Ensemble Verification Products.

• The NWP models evaluated are: • GFS: NCEP Global Deterministic Forecast;       • GEFSM: Mean of NCEP Global Ensemble Forecasts;    • CMCEM: Mean of Canadian Global Ensemble Forecasts;    • FENSM: Mean of US Navy Global Ensemble Forecasts;    • NAEFSM: Biased Corrected Mean of GEFS and CMCE;   

ECMWF ensemble is yet to be included in this verification page.2

Grid-to-Grid Verification Metrics: verified against each model’s own high-res deterministic analyses, including NH, SH, Tropics, Globe and PNA regions, up to 16 days of forecasts

• Anomaly correlation• Bias• RMS• Murphy’s MSE Skill Scores• Ratio of Standard Deviation• Regional means of surface variables (T2m, RH2m, clm water, SLP etc)

Grid-to-Obs Verification Metrics: verified against surface and Rawinsonde observations, including bias and RMS, up to 7 days of forecasts

• Surface T2m, RH2m, 10 Wind, SLP against station obs, over the CONUS

• Upper air T, Q, RH, and Wind against RAOBS, over NH, SH, Tropics, the Globe and PNA.

Weather forecast maps

3

500-hPa Height Anomaly Correlation

NH NH

SHSH

• Ensembles GEFS, CMCE, and NAEFS have higher AC scores than GFS• GEFS is better than CMCE and FENS 4

SLP Anomaly Correlation

NHSH

• Biased corrected NAEFS is the best• It seems the initial conditions of CMC ensembles are very

different from its deterministic high-res analysis (why)5

WIND RMSE vs Variance

• All ensembles have smaller RMSE than the deterministic GFS• However, GFS maintains the best its variance. All ensembles lose variances with

forecast leading time. Fields become much smoothers probably due to lower resolution and multi-member averaging.

• Note that for ensemble forecast, ensemble spread is a more meaningful metric.

NH, 200hPa

6

WIND RMSE vs Variance

• Note that FENS performed quite well in the tropics• GEFS tends to lose its variance quicker than others in the tropics

Tropics, 850hPa

7

Comparison of A few near-surface forecast variables

Tropics 10m U

FENS has the strongest wind

CMCE is the weakest

NH T2m

FENS is much warmer than others for daily high

CMCE has the lowest daily low8

Comparison of A few near-surface forecast variables

Does CMCE has SSTs in the forecast hours set to its initial conditions? It seems there is no SST relaxation to its climatology !

SH T2m

9

Verification of T2m to Surface Observations over the CONUS West

10

• GFS is too warm in the afternoon, while ECMWF (deterministic, 12-hourly) is too warm in the morning

• GEFS performed the best• FENS is too warm in both day and night, while CMCE is too cold

July28 – Sept 14, 2013

Verification of T2m to Surface Observations over the CONUS East

11

• Again, FENS is too warm in both dayt and night• Both GEFS and GFS are too cold in the afternoon• While GFS is slightly too warm for daily high, GEFS is slightly too cold.• Both CMCE and ECMWF are slightly too warm

July28 – Sept 14, 2013

Verification of 10-m Wind to Surface Observations over the CONUS West

12

• All models have weaker 10-m wind than the observed in the CONUS west, although they all captured the observed diurnal variation.

• CMCE has the slowest wind speed.

July28 – Sept 14, 2013

Verification of Forecasts Against Rawinsonde Observations, d-6 Wind

13

• Winds from all ensembles are too weak compared to the RAOBS• Deterministic GFS and EMWF are also slightly weaker than the observed in

the troposphere.

Tropics NH

Forecast of Hurricane Humberto, IC 2013091000

14

Colorado Flooding, 6-d forecasts from high-res deterministic models

15

24-h rainfall, valid at 2013091212 24-h rainfall, valid at 2013091312

GFS too far north

Colorado Flooding, 3-d forecasts from high-res deterministic models

16

24-h rainfall, valid at 2013091212 24-h rainfall, valid at 2013091312

good forecast

Colorado Flooding, 3-d ENS FCST, IC 2013091000

17

Colorado Flooding, 7-d ENS FCST, IC 2013090600

18