Post on 17-Jan-2016
description
© Crown copyright Page 1
Sarah Beare (nee John)
Neill Bowler, Marie Dando, Anette Van der Wal, Rob Darvell
Performance of the MOGREPS Regional Ensemble
© Crown copyright Page 2
Outline
Overview of MOGREPS Configuration
Overview of probabilistic verification
Results
Summary and future work
© Crown copyright Page 3
MOGREPS-Regional
Regional Ensemble
Regional ensemble over N. Atlantic and Europe (NAE)
23 perturbed members + control
IC Perturbations from regional ETKF
LBC from global ensemble
Stochastic physics
- random parameters
24km resolution, 38 levels
Run at 6Z & 18Z to T+54
© Crown copyright Page 4
MOGREPS Operational System diagram
New global analysis
Perturbations mixed and scaled by ETKF
Global ensemble forecast using stochastic physics
New NAE analysis
0Z 12Z 18Z06Z
New NAE analysis
LBCs
MOGREPS Verification
© Crown copyright Page 6
Verification
Aim to verify products issued from MOGREPS system
Two systems available
Focus on surface parameters and higher thresholds
Further results available in verification report recently published on met office website.
http://www.metoffice.gov.uk/research/nwp/publications/papers/technical_reports/index.html
© Crown copyright Page 7
Site-specific Verification system
- 79 sites across the UK and Europe.
- Probabilistic forecast data binned into 10% categories
- Ability to compare with ECMWF forecasts
- Ability to compare RAW model forecasts and post-processed data
- Can calculate and plot :
Reliability diagrams, Empirical ROC Curves, Brier Score and decompositions Brier Skill Score
- CRPS in the process of being added
© Crown copyright Page 8
Area Based Verification System
- Can verify against observations or analyses - Forecast probabilities not binned
- Can calculate and plot :
Reliability and attributes diagrams, Empirical and parametric ROC, Economic value, Brier Score and decompositions Brier Skill Score Spread and Skill based measures
- Would like to include the RPS, CRPS
- Can compare Global v Regional ensemble.
- THORPEX group introducing ECMWF EPS data for comparison purposes (no facility to account for different ensemble sizes).
© Crown copyright Page 9
Performance of ensemble and high-res (12km)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 12 24 36 48
Lead time (h)
RM
SE
(1
0m
win
d s
pe
ed
)
High-res
Control
Ensemble mean
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
0 12 24 36 48
Lead time (h)
RM
SE
(s
tati
on
te
mp
era
ture
)
High-res
Control
Ensemble mean
Ensemble mean has lower RMSE than high-res for day 2 forecasts, indicating presence of substantial uncertainties
Verification for UK stations for Dec 2006 to Feb 2007. T+48 data only available after 6 Feb
10m w/s
2m Temp
© Crown copyright Page 10
Spread-skill relationship – wind speed T+30
Spread-skill for wind-speed binned into equal population bins by spread
Skill corrected for observation error
Blue – MOGREPS Pink – No SkillGreen – Perfect
Grad = 0.93
Grad = 0.05
Grad = 0.59
0
1
2
3
4
5
0 1 2 3 4 5
Average Standard Deviation (ms-1)
RM
SE
(m
s-1)
MOGREPS
No Skill
Perfect Spread
Grad = 0.60
Grad = -0.05
Grad = 0.98
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
Average Standard Deviation (ms-1)
RM
SE
(m
s-1
)MOGREPS
No Skill
Perfect Spread
DJF 06/07
JJA 06
© Crown copyright Page 11
Spread-skill relationship – temperature
Spread-skill for temperature binned into equal population bins by spread Skill corrected for
observation error
Blue – MOGREPS Pink – No SkillGreen – Perfect
DJF 06/07
JJA 06
Grad = 0.91
Grad = 0.63
Grad = 0.03
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
Average Standard Deviation (K)
RM
SE
(K
)
MOGREPS
No Skill
Perfect Spread
Grad = 0.96
Grad = 0.01
Grad = 0.08
0
1
2
3
4
5
0 1 2 3 4 5
Average Standard Deviation (K)
RM
SE
(K)
MOGREPS
No Skill
Perfect Spread
Temperature
© Crown copyright Page 13
Temp DJF06/07 (79 sites UK & Europe)
Temp < 0oC T+36 Temp < -2oC T+36
© Crown copyright Page 14
Brier Skill Score components - Temperature > 10oC 6 Nov 2006 - 28 Feb 2007
Temp>10C
79 sites UK & Europe 6 Nov 2006 – 28 Feb 2007
Brier Skill score
Reliability
Resolution
© Crown copyright Page 15
Reliability diagram for surface temperature
Reliability diagram for Temp>10C
79 sites UK & Europe
6 Nov 2006 – 28 Feb 2007
© Crown copyright Page 16
Temperature > 10oC KFMOS
Kalman-filter bias-corrected forecasts.
Reliability
Resolution
Brier Skill score
Precipitation
© Crown copyright Page 18
Brier Skill Score components for Precipitation
12-hour precip>0.5mm
79 sites UK & Europe 1 July 2006 – 31 March 2007
12-hour precip>5mm
© Crown copyright Page 19
Reliability diagram for precipitation
Reliability diagram for 12h precip>5mm
79 sites UK & Europe
1 July 2006 – 31 March 2007
© Crown copyright Page 20
6hr precip > 0.3mm against gridded analysis
•Reliability and sharpness diagram for T+36 forecast.
•6h precip > 0.3mm
•Verification against Nimrod Analysis over the UK at 1.0 degree resolution.
•1st Jan 06 – 28th Feb 07
© Crown copyright Page 21
•Reliability and sharpness diagram for T+36 forecast.
•6h precip > 5mm
•Verification against Nimrod Analysis over the UK at 1.0 degree resolution.
•1st Jan 06 – 28th Feb 07
6hr precip 5mm against gridded analysis
© Crown copyright Page 22
•Reliability and sharpness diagram for T+36 forecast.
•6h precip > 25mm
•Verification against Nimrod Analysis over the UK at 1.0 degree resolution.
•1st Jan 06 – 28th Feb 07
6hr precip 25mm against gridded analysis
Wind speed
© Crown copyright Page 24
Reliability diagram for wind speed
Reliability diagram for wind speed >F5 at T+30
79 sites UK & Europe
6 Nov 2006 – 31 March 2007
© Crown copyright Page 25
Wind speed at least gale force 8
•Reliability and sharpness diagram for T+36 forecast.
•10m Wind > F8
•Verification against surface obs over UK and Europe.•1 Jan 06 – 28 Feb 07
© Crown copyright Page 26
Wind speed at least severe gale force 9
•Reliability and sharpness diagram for T+36 forecast.
•10m Wind > F9
•Verification against surface obs over UK and Europe.•1 Jan 06 – 28 Feb 07
© Crown copyright Page 27
Wind speed at least storm force 10
•Reliability and sharpness diagram for T+36 forecast.
•10m Wind > F10
•Verification against surface obs over UK and Europe.•1 Jan 06 – 28 Feb 07
© Crown copyright Page 28
Summary of verification
Ensemble mean beats control and hi-res from ~18h
Very good reliability for non-extreme events
Useful skill for some more severe events
T<-2C
Windspeed > F9
6h Precip > 5mm/12h
KFMOS bias correction effective in reducing many biases
Better forecasts from NAE than ECMWF, even after KFMOS for some
parameters
Combination of resolution and perturbation strategy
Useful spread-skill in many cases
Exception is temperature in summer
© Crown copyright Page 29
Future Plans
Fully operational early 2008
SKEB2 scheme including convection for improved stochastic physics
Vertical localization of ETKF to improve near-surface perturbation scaling
Perturbations to lower boundary (eg soil moisture, SST)
Enhanced resolution: Global 60km from 2009
Regional 16km from 2009 and 12 from 2011
Small convection-resolving ensemble from 2011 ~1.5km over UK
~6 members
Nested on members selected by clustering
© Crown copyright Page 30
Any Questions?
© Crown copyright Page 31
More Information….
Technical Reports
453 Comparison of error breeding, singular vectors,random perturbations and ensemble Kalman filter perturbation strategies on a simple model. N Bowler (2004)
452 Results of an initial stochastic physics scheme for the Met Office Unified Model. A. Arribas (2004)
503 The MOGREPS short-range ensemble prediction system: Verification report - Trial Performance of MOGREPS January 2006 - March 2007 N Bowler, M Dando, S Beare and K Mylne (2007)
497 The MOGREPS short-range ensemble prediction system Part 1: System Description
N Bowler, A Arribas, K Mylne and K Robertson, (2007)
© Crown copyright Page 32
NEW! Regional ETKF
Until end of May MOGREPS Perturbations for the NAE were interpolated from the global at T+6
Large scale perturbations Spread too large (especially at upper levels)
Since beginning of June perturbations to regional ensemble generated by a regional ETKF.
Smaller, more realistic perturbations Better representation of important short length scales But surface perturbations may be too small
© Crown copyright Page 33
Subjective Evaluation of Severe Weather
Forecaster comments/feedback Severe Gale Event – 02/09/06
…. the NAE MOGREPS output clearly captured the strength and gave a good forecast for potential severe gale gusts…. (Paul Davies, Chief Forecaster)
Intense rainfall - 20/07/07
… The risk of very high totals is confirmed by NAE MOGREPS members and also other models… (Steve Willington, Chief Forecaster)