May 30, 2003 Tony Eckel, Eric Grimit, and Cliff Mass UW Atmospheric Sciences This research was...
-
date post
19-Dec-2015 -
Category
Documents
-
view
220 -
download
5
Transcript of May 30, 2003 Tony Eckel, Eric Grimit, and Cliff Mass UW Atmospheric Sciences This research was...
May 30, 2003
Tony Eckel, Eric Grimit, and Cliff Mass
UW Atmospheric Sciences
This research was supported by the DoD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745.
Overview
Review Ensemble Forecasting Theory and Introduce UW’s SREFs
Discuss Results of Model Deficiencies on SREF- Need for Bias Correction- Impact on ensemble spread- Impact on probabilistic forecasts skill
Conclusions
T
The true state of the atmosphere exists as a single point in phase space that we never know exactly.
A point in phase space completely describes an instantaneous state of the atmosphere.For a model, a point is the vector of values for all parameters (pres, temp, etc.) at all grid points at one time.
An analysis produced to run a model like the eta is in the neighborhood of truth. The complete error vector is unknown, but we have some idea of its structure and magnitude.
e
Chaos drives apart the forecast and true trajectories…predictability error growth.
EF can predicted the error magnitude and give a “probabilistic cloud” of forecasts.
12hforecast
36hforecast
24hforecast
48hforecast
T
48hverification
phasespace
e
a
uc
j
tg
n
2 1 0 1 2 34
2
0
2
4
6
7
-2.96564
Core ,i 2
Cent ,1 2
32.5 ,Core ,i 1 Cent ,1 1
M
T
T
Analysis Region
48h forecast Region
12hforecast
36hforecast
24hforecast
Diagram for: PME ACMEcore
or ACMEcore+
phasespace
Plug each IC into the MM5 to create an ensemble of mesoscale forecasts (cloud of future states encompassing truth).
1) Reveal uncertainty in forecast 2) Reduce error by averaging M 3) Yield probabilistic information
e
a
uc
j
tg
n
c
2 1 0 1 2 34
2
0
2
4
6
7
-2.96564
Core ,i 2
Cent ,1 2
32.5 ,Core ,i 1 Cent ,1 1
T
M
T
Analysis Region
48h forecast Regionphasespace
ACME’s Centroid
e
n
ac
u
tg
T
j
M
T
Analysis Region
48h Forecast Region
e
a
uc
j
tg
n
c
2 1 0 1 2 34
2
0
2
4
6
7
-2.96564
Core ,i 2
Cent ,1 2
32.5 ,Core ,i 1 Cent ,1 1
phasespace
ACME’s Mirrored Members
FP = 93%
Parameter Threshold(EX: precip > 0.5”)
FP = ORF = 72%
Fre
qu
en
cy
Initial State
Forecast Probability from an Ensemble
• EF provides an estimate (histogram) of truth’s Probability Density Function (red curve).
• In a large, well-tuned EF, Forecast Probability (FP) = Observed Relative Frequency (ORF)
24hr Forecast State 48hr Forecast State
Fre
qu
en
cy
• In practice, things get wacky from• Under-sampling of the PDF (too few ensemble members)• Poor representation of initial uncertainty• Model deficiencies
-- Model bias causes a shift in the estimated mean-- Sharing of model errors between EF members leads to reduced variance
• EF’s estimated PDF does not match truth’s PDF, and Fcst Prob Obs Rel Freq
UW’s Ensemble of Ensembles
# of EF Initial Forecast Forecast Name Members Type Conditions Model(s) Cycle Domain
ACME 17 SMMA 8 Ind. Analyses, “Standard” 00Z 36km, 12km1 Centroid, MM58 Mirrors
ACMEcore 8 SMMA Independent “Standard” 00Z 36km, 12km Analyses MM5
ACMEcore+ 8 PMMA “ “ 8 MM5 00Z 36km, 12km variations
PME 8 MMMA “ “ 8 “native” 00Z, 12Z 36km large-scale
Hom
egro
wn
Impo
rted
ACME: Analysis-Centroid Mirroring Ensemble
PME: Poor Man’s Ensemble MM5: PSU/NCAR Mesoscale Modeling System Version 5
SMMA: Single Model Multi-Analysis
PMMA: Perturbed-model Multi-Analysis
MMMA: Multi-model Multi-Analysis
Resolution (~ @ 45 N ) ObjectiveAbbreviation/Model/Source Type Computational Distributed Analysis
gfs, Global Forecast System, Spectral T254 / L64 1.0 / L14 SSINational Centers for Environmental Prediction ~55km ~80km 3D Var cmcg, Global Environmental Multi-scale (GEM), Spectral T199 / L28 1.25 / L11 3D VarCanadian Meteorological Centre ~70km ~100km eta, Eta limited-area mesoscale model, Finite 12km / L60 90km / L37 SSINational Centers for Environmental Prediction Diff. 3D Var gasp, Global AnalysiS and Prediction model, Spectral T239 / L29 1.0 / L11 3D VarAustralian Bureau of Meteorology ~60km ~80km
jma, Global Spectral Model (GSM), Spectral T106 / L21 1.25 / L13 OIJapan Meteorological Agency ~135km ~100km ngps, Navy Operational Global Atmos. Pred. System, Spectral T239 / L30 1.0 / L14 OIFleet Numerical Meteorological & Oceanographic Cntr. ~60km ~80km
tcwb, Global Forecast System, Spectral T79 / L18 1.0 / L11 OITaiwan Central Weather Bureau ~180km ~80km ukmo, Unified Model, Finite 5/65/9/L30 same / L12 3D VarUnited Kingdom Meteorological Office Diff. ~60km
“Native” Models/Analyses of the PME
Design of ACMEcore+
8 5 3 2 5 3 2 2 8 8 = 921,600
Total possible combinations:
vertical 36km 12km shallow SSTIC ID# Soil diffusion Cloud Microphysics Domain Domain cumulus Radiation Perturbation Land Use Table
MRF 5-Layer Y Simple Ice Kain-Fritsch Kain-Fritsch N cloud standard standard
avn plus01 MRF LSM Y Simple Ice Kain-Fritsch Kain-Fritsch Y RRTM SST_pert01 LANDUSE.TBL.plus1
cmcg plus02 MRF 5-Layer Y Reisner II (grpl), Skip4 Grell Grell N cloud SST_pert02 LANDUSE.TBL.plus2
eta plus03 Eta 5-Layer N Goddard Betts-Miller Grell Y RRTM SST_pert03 LANDUSE.TBL.plus3
gasp plus04 MRF LSM Y Shultz Betts-Miller Kain-Fritsch N RRTM SST_pert04 LANDUSE.TBL.plus4
jma plus05 Eta LSM N Reisner II (grpl), Skip4 Kain-Fritsch Kain-Fritsch Y cloud SST_pert05 LANDUSE.TBL.plus5
ngps plus06 Blackadar 5-Layer Y Shultz Grell Grell N RRTM SST_pert06 LANDUSE.TBL.plus6
tcwb plus07 Blackadar 5-Layer Y Goddard Betts-Miller Grell Y cloud SST_pert07 LANDUSE.TBL.plus7
ukmo plus08 Eta LSM N Reisner I (mx-phs) Kain-Fritsch Kain-Fritsch N cloud SST_pert08 LANDUSE.TBL.plus8
Perturbations to:
1) Moisture Availability
2) Albedo
3) Roughness Length
ACMEcore+
CumulusPBL
ACME
Total of 129, 48-h forecasts (Oct 31, 2002 – Mar 28, 2003) all initialized at 00z- Missing forecast case days are shaded
Parameters:- 36 km Domain: Mean Sea Level Pressure (MSLP), 500mb Geopotential Height (Z500)- 12 km Domain: Wind Speed @ 10m (WS10), Temperature at 2m (T2)
Research Dataset
36 km Domain (151127)
12 km Domain(101103)
Verification:
- 36 km Domain: centroid analysis (mean of 8 independent analyses, available at 12h increments)
- 12 km Domain: ruc20 analysis (NCEP 20 km mesoscale analysis, available at 3h increments)
November December January
February March
3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 11 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4
1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 25 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
cmcg*
The ACME Process
STEP 1: Calculate best guess for truth (the centroid) by averaging all analyses.
STEP 2: Find error vector in model phase space between one analysis and the centroid by differencing all state variables over all grid points.
STEP 3: Make a new IC by mirroring that error about the centroid.
cmcgC cmcg*
Sea
Lev
el P
ress
ure
(mb)
~1000 km
1006
1004
1002
1000
998
996
994
cent
170°W 165°W 160°W 155°W 150°W 145°W 140°W 135°W
eta
ngps
tcwbgasp
avn
ukmo
cmcg
MSLP analysis south of the Aleutians at 00Z on Jan 16, 2003
tcwb centroid centroid + (centroid tcwb)
bias correction…
Overview
The two flavors of model deficiencies play a big role in SREF:
1) Systematic: Model bias is a significant fraction of forecast error and must be removed.
2) Stochastic: Random model errors significantly increase uncertainty and must be accounted for.
Bias Correction: A simple method gives good results
Model Error*: Impact on ensemble spread
Final Results: Impact of both on probabilistic forecasts skill
* bias-corrected
980
990
1000
1010
1020
1030
1040
1050
980 990 1000 1010 1020 1030 1040 1050
MSLP Forecast (mb)
MS
LP
An
aly
sis
(m
b)
Often difficult to completely remove bias within a model’s code Systematic but complex; involving numerics, parameterizations, resolution, etc.
Depend upon weather regime (time of day, surface characteristics, stability, moisture, etc.)
Cheaper and easier to remove bias through post-processing Sophisticated routines such as MOS require long training periods (years) The bulk of bias can be removed with the short term mean error
Need for Bias Removal
NGPS Forecast vs Analysis
Data Info
Single model grid point in eastern WA
Verification: centroid analysis
70 forecasts (Nov 25, 2002 – Feb 7, 2003)
Lead time = 24h
980
990
1000
1010
1020
1030
1040
1050
980 990 1000 1010 1020 1030 1040 1050
GASP Forecast vs Analysis
980
990
1000
1010
1020
1030
1040
1050
980 990 1000 1010 1020 1030 1040 1050
GFS Forecast vs Analysis
980
990
1000
1010
1020
1030
1040
1050
980 990 1000 1010 1020 1030 1040 1050
GFS-MM5 Forecast vs Analysis
TrainingPeriod
Bias-correctedForecast Period
TrainingPeriod
Bias-correctedForecast Period
TrainingPeriod
Bias-correctedForecast Period
Gridded Bias Removal
N
n nji
tjitji o
f
Nb
1 ,
,,,,
1 N number of forecast cases (14) fi,j,t forecast at grid point (i, j ) and lead time (t)oi,j verifying observation
For the current forecast cycle:
1) Calculate bias at every grid point and lead time using previous 2 weeks’ forecasts
2) Post-process current forecast to correct for bias:
tji
tjitji b
ff
,,
,,*,, fi,j,t bias-corrected forecast at grid point (i, j ) and lead time (t)*
November December January
February March
3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 11 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4
1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 25 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
Spatial and Temporal Dependence of Bias
GFS-MM5 MSLP Bias at f24
Common Bias Forecast Error
> 1 too low < 1 too high
Spatial and Temporal Dependence of Bias
GFS-MM5 MSLP Bias at f36
Common Bias Forecast Error
> 1 too low < 1 too high
Bias Vs Corrected at f36
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
avn cmcg eta gasp jma ngps tcw b ukmo mean
MS
LP
RM
SE
(m
b2)
0%
5%
10%
15%
20%
25%
30%
35%
0 12 24 36 48Lead Time (hours)
% Im
pro
ve
me
nt avn
cmcg
eta
gasp
jma
ngps
tcwb
ukmo
mean
Bias Correction Results
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
avn cmcg eta gasp jma ngps tcw b ukmo mean
bia
sed
bia
s-co
rrec
ted
PME
Bias Vs Corrected at f36
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
avn cmcg eta gasp jma ngps tcw b ukmo mean
MS
LP
RM
SE
(m
b2)
0%
5%
10%
15%
20%
25%
30%
35%
0 12 24 36 48Lead Time (hours)
% Im
pro
ve
me
nt avn
cmcg
eta
gasp
jma
ngps
tcwb
ukmo
mean
ACMEcore
Bias Correction Results
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
avn cmcg eta gasp jma ngps tcw b ukmo mean
bia
sed
bia
s-co
rrec
ted
ACMEcore+
Bias Correction Results
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
avn cmcg eta gasp jma ngps tcw b ukmo mean
bia
sed
bia
s-co
rrec
ted
Bias Vs Corrected at f36
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
avn cmcg eta gasp jma ngps tcw b ukmo mean
MS
LP
RM
SE
(m
b2)
0%
5%
10%
15%
20%
25%
30%
35%
0 12 24 36 48Lead Time (hours)
% Im
pro
ve
me
nt avn
cmcg
eta
gasp
jma
ngps
tcwb
ukmo
mean
Lead Time (hours)
Verification Rank
Pro
babi
lity
Verification Rank HistogramRecord of where verification fell (i.e., its rank) among the ordered ensemble members:
Flat Well calibrated EF (truth’s PDF matches EF PDF)
U’d Under-dispersive EF (truth “gets away” quite often)
Humped Over-dispersive EF
12
34
56
78
9
12
24
36
48
0.00
0.05
0.10
0.15
0.20
0.25
0.30
ACMEcore
Lead Time (hours)
Verification Rank
Pro
babi
lity
Verification Rank HistogramRecord of where verification fell (i.e., its rank) among the ordered ensemble members:
Flat Well calibrated EF (truth’s PDF matches EF PDF)
U’d Under-dispersive EF (truth “gets away” quite often)
Humped Over-dispersive EF
12
34
56
78
9
12
24
36
48
0.00
0.05
0.10
0.15
0.20
0.25
0.30
*ACMEcore
Model Error Impact on ensemble spread …
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
0 12 24 36 48
Lead Time
Var
ian
ce /
MS
E
*PME
MSE of *PME Mean
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
0 12 24 36 48
Lead Time
Var
ian
ce /
MS
E
*ACMEcore
*ACMEcore+
MSE of *ACMEcore Mean
Ensemble Dispersion
(MSLP)
Analysis Error
Error Growth due to Analysis ErrorE
nse
mb
le V
aria
nce
(m
b2 )
Error Growth due to Model Error
EF Mean’s MSE adjusted by n / n+1 to account for small sample size
MS
E o
f E
F M
EA
N
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
0 12 24 36 48
Var
ian
ce /
MS
E
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
0 12 24 36 48
Lead Time
Var
ian
ce /
MS
E
*PME
MSE of *PME Mean
*ACMEcore
*ACMEcore+
MSE of *ACMEcore Mean
12
34
56
78
9
12
24
36
48
0.00
0.05
0.10
0.15
0.20
*ACMEcore
Lead Time (hours)
Verification Rank
Pro
babi
lity
Verification Rank HistogramRecord of where verification fell (i.e., its rank) among the ordered ensemble members:
Flat Well calibrated EF (truth’s PDF matches EF PDF)
U’d Under-dispersive EF (truth “gets away” quite often)
Humped Over-dispersive EF
12
34
56
78
9
12
24
36
48
0.00
0.05
0.10
0.15
0.20
*ACMEcore+
Lead Time (hours)
Verification Rank
Pro
babi
lity
Verification Rank HistogramRecord of where verification fell (i.e., its rank) among the ordered ensemble members:
Flat Well calibrated EF (truth’s PDF matches EF PDF)
U’d Under-dispersive EF (truth “gets away” quite often)
Humped Over-dispersive EF
Lead Time (hours)
Verification Rank
Pro
babi
lity
Verification Rank HistogramRecord of where verification fell (i.e., its rank) among the ordered ensemble members:
Flat Well calibrated EF (truth’s PDF matches EF PDF)
U’d Under-dispersive EF (truth “gets away” quite often)
Humped Over-dispersive EF
12
34
56
78
9
12
24
36
48
0.00
0.05
0.10
0.15
0.20
PME
Impact of both on probabilistic forecasts skill
Explain probabilistic forecast verification
P(MSLP < 1001mb) by uniform ranks method, 36h lead time, Sub-domain AReliability Diagram Comparison
PME ACMEcore
Sample Climatology
Resolution Reliability Uncertainty Skill Score
Biased 0.1860 0.0010 0.2111 0.8714
Corrected 0.1859 0.00005 0.2111 0.8752
Resolution Reliability Uncertainty Skill Score
Biased 0.1860 0.0010 0.2111 0.8714
Corrected 0.1859 0.00005 0.2111 0.8752
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Forecast Probability
Ob
serv
ed R
elat
ive
Fre
qu
ency
Resolution Reliability Uncertainty Skill Score
0.1443 0.0013 0.1756 0.8138
0.1465 0.0002 0.1756 0.8330
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Forecast Probability
Ob
serv
ed R
elat
ive
Fre
qu
ency
Resolution Reliability Uncertainty Skill Score
0.1526 0.0008 0.1756 0.8641
0.1522 0.0002 0.1756 0.8655
36km Verification Sub-domain A
0.70
0.75
0.80
0.85
0.90
0.95
1.00
0 12 24 36 48
Lead Time (hours)
Sk
ill S
co
re
(re
lativ
e to
sa
mp
le c
lima
tolo
gy)
*PME
PME
*ACMEcore
ACMEcore
*ACMEcore+
ACMEcore+
~6hr improvement by bias correction
~11hr improvement by multi-model diversity and “global” error growth
Skill vs. Lead Time (Sub-domain A)
36km Verification Sub-domain B
Skill vs. Lead Time(all bias –corrected)
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
0 12 24 36 48
Lead Time (hours)
Sk
ill S
co
re
(re
lativ
e to
sa
mp
le c
lima
tolo
gy)
*PME
*ACMEcore
*ACMEcore+
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
0 12 24 36 48
Lead Time (hours)
36km Sub-domain A (2/3 ocean)P(MSLP < 1001mb)
Sample Climatology 23%
36km Sub-domain B (mostly land)P(MSLP < 1011mb)
Sample Climatology 20%
~11hr improvement by PME ~22hr improvement by PME
~3hr improvement by ACMEcore+
Conclusions
Caveats:- Consider non-optimal ICs and small EF size? (still fair comparison between PME and ACMEcore)- What about higher skill of PME members? (not so dramatic after bias correction)- Does higher resolution of MM5 make comparison unfair? (fitting to lower res. would decrease 2)
Why bother with ACMEcore?
PME is certainly more skilled at the synoptic level, but has little to no mesoscale info.
Should these conclusions hold true for mesoscale?
YES! Model deficiencies for surface variables (precip, winds, temperature) can be even stronger, so the effect on SREF may be even greater.Demonstrating that is now the focus of my research…
P(precip > 0.25” in 6hr)
An ensemble’s skill is dramatically improved by:
1) Correcting model bias
2) Accounting for model uncertainty
UW’s Ensemble of Ensembles
# of EF Initial Forecast Forecast Name Members Type Conditions Model(s) Cycle Domain
ACME 17 SMMA 8 Ind. Analyses, “Standard” 00Z 36km, 12km1 Centroid, MM58 Mirrors
ACMEcore 8 SMMA Independent “Standard” 00Z 36km, 12km Analyses MM5
ACMEcore+ 8 PMMA “ “ 8 MM5 00Z 36km, 12km variations
PME 8 MMMA “ “ 8 “native” 00Z, 12Z 36km large-scale
ACNE 9 hybrid? 8 Ind. Analyses, 9 MM5 00Z, 12Z 36km, 12kmMMMA 1 Centroid variationsPMMA
ACNE: Analysis-Centroid Nudged Ensemble
SMMA: Single Model Multi-Analysis
PMMA: Perturbed-model Multi-Analysis
MMMA: Multi-model Multi-Analysis
Pro
pose
d
?
Skill Score (SS ) Details
climclimperfect
clim 1BS
BS
BSBS
BSBSBSS
2
1
1
n
iii OBSFP
nBS SCSCSCORFN
nORFFPN
nBS i
M
iiii
M
ii
111 2
1
2
1
***
(reliability) (resolution) (uncertainty)
Brier Score
Brier Skill Score
yuncertaint
reliabiltyresolution
uncertainy00
uncertainyresolutionyreliabilit1
SS
n: number of data pairsFPi: forecast probability {0.0…1.0}ORFi: observation {0.0 = yes, 1.0 = no}
M : number of probability bins (normally 11)N : number of data pairs in the binFP*
i : binned forecast probability {0.0, 0.1,…1.0}ORF*
i : observation for the bin {0.0 = yes, 1.0 = no}SC : sample climatology (total occurrences / total forecasts)
Decomposed Brier Score (uses binned FP as in rel. diag.)
Skill Score
FP = 77.1%
For a certain threshold, say Ws 20kt, the FP is then simply the area under the PDF to the right (1 p value)
Ws = {16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8}
Wind Speed (kt)
Fre
quen
cy
Ideal Calculation of Forecast Probability (FP)Given a very large ensemble, a PDF could be found a grid point for any parameter (e.g., wind speed, Ws).
Unfortunately, we work with very small ensembles so we can’t make a good estimate of the PDF.Plus, we often do not even know what PDF shape to fit.
So we are forced to estimate FP by other means, for a set of Ws forecasts at a point such as:
Note: These are random draws from the PDF above
0 10 20 30 40 50 60 700
0.01
0.02
0.03
0.04
0.05
FP = 7/8 = 87.5%
FP = 7/9 + [ (21.1 – 20.0) / (21.1 – 16.5) ] * 1/9 = 80.4%
16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8
8/8 7/8 6/8 5/8 4/8 3/8 2/8 1/8 0/8
Democratic Voting FP
Uniform Ranks FP
9/9 8/9 7/9 6/9 5/9 4/9 3/9 2/9 1/9 0/9
“pushes” FP towards the extreme values, so high FP is normally over-forecast and low FP is normally under-forecast.
a continuous, more appropriate approximation.
20.0
FP = [ (1 – GCDF(50.0)) / (1 – GCDF(47.8)) ] * 1/9 = 8.5%
0 10 20 30 40 50 60 70 800
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 a b
fraction = a / b
16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8
Uniform Ranks FP
9/9 8/9 7/9 6/9 5/9 4/9 3/9 2/9 1/9 0/9
FP When Threshold Falls in an Extreme Rank
0.0,-
50.0
Use the tail of a Gumbel PDF to approximate the fraction for the last rank.
0 10 20 30 40 50 60 70 800
0.01
0.02
0.03
0.04
0.05
x
xCDF expexp)(G
6ˆ s
ˆˆ x
FP = [ (1 – CDF(50.0)) / (1 – CDF(47.8)) ] * 0.17 = 13.0%
16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8
Weighted Ranks FP
1.0 0.83 0.72 0.62 0.54 0.45 0.36 0.27 0.17 0.0
Calibration by Weighted Ranks
0.0,-
50.0
Use the verification rank histogram from past cases to define non-uniform, “weighted ranks”.
The ranks to sum up and fraction of the rank where the threshold falls are found the same way as with uniform ranks, but now the probability within each rank is the chance that truth will occur there.
Sample Climatology
Skill Zone
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Forecast Probability
Ob
se
rve
d R
ela
tiv
e F
req
ue
nc
y
Uniform Ranks vs. Democratic Voting
Data Info
P(MSLP < 1002mb)
Verification: centroid analysis
70 forecasts (Nov 25, 2002 – Feb 7, 2003)
Applied 2-week, running bias correction
36km, Outer Domain
Lead time = 48h
UR
DV
References
Hamill, T. M. and S. J. Colucci, 1997: Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev., 125, 1312–1327
Eckel, F. A., 1998: Calibrated probabilistic quantitative precipitation forecasts based on the MRF Ensemble. Masters Thesis, 133 pp
-----, and M. K. Walters, 1998: Calibrated probabilistic quantitative precipitation forecasts based
on the MRF Ensemble. Wea. and Fcst., 13, 1132–1147