Post on 05-Jan-2016
© Crown copyright Page 1
Scale selective verification of precipitation forecasts
Nigel Roberts and Humphrey Lean
© Crown copyright Page 2
Background - What do we want to know?
1km model – should improve precipitation forecasts
…. But, smaller scales are less predictable (Zhang et al 2003) – yet higher expectation
Can a 1-km model provide more accurate and useful forecasts of rainfall events on the scales of river catchments?
On what scales should the output be presented?
What does data assimilation do to different scales?
© Crown copyright Page 3
Verification approach
Verify over different spatial scales using a conceptually simple approach. Fractions/probabilities from nearest neighbouring points. (also Ebert)
Verify against radar – good spatial coverage. Stable network over UK.
Verify accumulations - smooth out temporal noise.
Use accumulation exceedance thresholds e.g. > 4 mm, > 8 mm ….
© Crown copyright Page 4
Radar 12 km forecast 1 km forecast
0.125 0.5 1 2 4 8 16 32 mm
The problem we face
0 100 km
Six hour accumulations 10 to 16 UTC 13th May 2003
© Crown copyright Page 5
Schematic example - different scales
© Crown copyright Page 6
1-km forecast Radar
0.125 0.5 1 2 4 8 16 32 mm
Six hour accumulations 10 to 16 UTC 13th May 2003
© Crown copyright Page 7
4 mm threshold, Fractions at grid scale (1 or 0)
Model Radar> 4 mm > 4 mm
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Fraction
© Crown copyright Page 8
Model Radar> 4 mm > 4 mm
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Fraction
4 mm threshold, Fractions within 35x35 km squares
© Crown copyright Page 9
Model Radar
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Fraction
4 mm threshold, Fractions within 75x75 km squares
© Crown copyright Page 10
Model Radar
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Fraction
4 mm threshold, Fractions within 105x105 km squares
© Crown copyright Page 11
Brier score for comparing fractions
Skill score for fractions/probabilities - Fractions Skill Score (FSS)
A score for comparing fractions with fractions
© Crown copyright Page 12
Graphical behaviour of the Fractions Skill Score
© Crown copyright Page 13
Evaluation of precipitation forecasts from a 1km NWP forecast model
11 convective cases (Summer 2003/2004)44 forecasts out to T+7
12km with data assimilation (3-hour cycle)
Concentrating on 1km spun up from 12km fields at T+1 (no additional data assimilation at 1km)
© Crown copyright Page 14
Results for 6-hour accumulations
© Crown copyright Page 15
Results for 6-hour accumulations
© Crown copyright Page 16
Results for 1-hour accumulations
© Crown copyright Page 17
Scale selective evaluation of operational 12km modelprecipitation forecasts during 2003
hourly accumulations
© Crown copyright Page 18
Conclusions about the approach
Verification approach is useful the concept is intuitively easy to understand gives information about spatial accuracy links output products to verification agreement with subjective assessment
Issues …. choice and interpretation of scores, dealing with a bias radar is regarded as ‘truth’! variability from case to case should be compared with other approaches – e.g. Barbara
Casati
© Crown copyright Page 19
Conclusions about the results
Forecasts of the spatial distribution of rainfall are more accurate at 1km than 12km at scales > ~20-30km.
The scale at which satisfactory skill is achieved is improved at 1km. e.g. from 70km (12km model) to 55km (1km model) for a 10% threshold
The more isolated the events being predicted the poorer the forecast skill, however the 1km model gave the greatest improvement for the most isolated events.
The impact of resolution is greater for hourly than for 6-hourly accumulations. Skill at 1km is maintained after the spin-up period.
Operational 12km model - Skill drops off most quickly after the analysis time at the smallest scales (~ <100 km).
© Crown copyright Page 20
Caution - bias
© Crown copyright Page 21
Scale selective evaluation of operational 12km modelprecipitation forecasts during 2003
widespreadlocalised
© Crown copyright Page 22
Variation between forecasts
© Crown copyright Page 23
Scale selective evaluation of operational 12km modelprecipitation forecasts during 2003