Continued Development of Tropical Cyclone Wind Probability Products
John A. Knaff – PresentingCIRA/Colorado State University
and Mark DeMariaNOAA/NESDIS
Presentation at 60th Interdepartmental Hurricane Conference
Mobile, AL22 March 2006
60th IHC 22 March 2006 - Mobile, AL 2
Outline
• Training Activities• Verification Activities• Results/insights from examination of
hurricane warning break points
60th IHC 22 March 2006 - Mobile, AL 3
Training Activities
• Mark DeMaria coordinated with Rick Knabb to provide feedback on a TPC/NHC training session.
• Several cases rerun for 2004 and 2005– For Pablo Santos, Miami WFO for an experimental
algorithm that uses the probabilities.– Web page with examples and a product description
http://rammb.cira.colostate.edu/projects/tc_wind_prob
60th IHC 22 March 2006 - Mobile, AL 4
60th IHC 22 March 2006 - Mobile, AL 5
Verification: Current Status
• Developed:– Input data handling (GRIB, ATCF….)– Statistical Methods
• Scalar measures of skill, accuracy, confidence• Conditional measures
– Methods to assess deterministic forecasts• Remaining
– Treatment of the OFCL forecast & wind radii through 5 days.
– Integrating the pieces.– How to Interpret the statistics and optimize use.
60th IHC 22 March 2006 - Mobile, AL 6
Statistical methods: Probability Bias
• Mean Forecast Probabilities (Fi) minus the Mean Observed Frequencies (Ei) = 1 or 0
Determines if the probabilities over/under forecast the outcome.
N
kk
N
kk
E
FBias
1
1
60th IHC 22 March 2006 - Mobile, AL 7
Statistical Methods: Brier Score
• Mean of square of the Forecast Probabilities (Fi) minus the Observed Frequencies (Ei) = 1 or 0
Measures the Mean Square Errors (Accuracy) associated with a probabilistic forecast
n
kkk EF
NBS
1
21
60th IHC 22 March 2006 - Mobile, AL 8
Statistical Methods: Brier Skill Score
• A scalar skill score comparing a given Brier Score with the Brier Score of a reference forecasts (OFCL, CLIPER etc.).
Assess relative accuracy (skill) of a probabilistic forecast with respect to a reference forecast.
refBSBSBSS 1
60th IHC 22 March 2006 - Mobile, AL 9
Statistical methods: Discrimination Distance
• Distance between the mean forecast probability (F) for all event (E) and all non-events (E’).
Measures the ability of a forecast scheme to discriminate events.
'EFEFd
60th IHC 22 March 2006 - Mobile, AL 10
Statistical Methods: Conditional Distributions
Obs
erve
d fr
eque
ncy
of e
vent
s
Bins of Forecast Probabilities
60th IHC 22 March 2006 - Mobile, AL 11
Statistical Methods: Relative Operating Characteristics
Forecasts (contingent on the forecast probability)Observation Warning (W) No Warning (W’) TotalEvent (E) h m E
Nonevent (E’) f c E’
Total w w’ N
A series of 2x2 contingency tables, which are conditional on a range of forecast probabilities are constructed.
For instance a warning would be issued if the probability exceeded 1,2,3,4,…100 %
The results of the contingency tables can be quantified in terms of hit rate (hr) = h/(h+m) and false-alarm rate (far)=f/(f+c)
A plot of far vs. hr can be created and a skill score created from the area under the curve.
60th IHC 22 March 2006 - Mobile, AL 12
Statistical Methods: ROC diagram & Skill Score
12 ASSROC,where A is the area under the curve
Mason and Graham (1999)
skill
No skill
ROC Skill Score
60th IHC 22 March 2006 - Mobile, AL 13
Verification Procedure: Test Dataset
• Dataset– 5-day cumulative 64-kt
wind probabilities were generated for 342 coastal break points (195 official breakpoints + 147 additional points)
– These were analyzed when warnings were issued for the 14 storms to the right
– N=128250 points
Storm Name Year
Alex 2004
Charley 2004
Frances 2004
Gaston 2004
Ivan 2004
Jeanne 2004
Arlene 2005
Cindy 2005
Dennis 2005
Emily 2005
Katrina 2005
Ophelia 2005
Rita 2005
Wilma 2005
60th IHC 22 March 2006 - Mobile, AL 14
60th IHC 22 March 2006 - Mobile, AL 15
Verification Procedure: Summary Skill Measures
Bias = 0.893 (under forecasts)BS = 0.0248 BSOFCL= 0.0346BSzero = 0.0392
ROC
Relative Operating Characteristics5-d Cummulative Probabilities for Landfalling Atlantic TC 2004-2005
BSSOFCL = 28.30%
BSSzero = 36.75%
60th IHC 22 March 2006 - Mobile, AL 16
Verification Procedure: Conditional Distribution of Break Point Probabilities
Slight under forecast of probabilities for this dataset – due to Wilma
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0 to
55
to 1
010
t o 1
515
t o 2
020
t o 2
525
to 3
030
t o 3
535
t o 4
040
t o 4
545
to 5
050
to 5
555
to 6
060
t o 6
565
to 7
070
t o 7
575
t o 8
080
to 8
585
to 9
090
to 9
595
t o 1
00Forecast Probability Bins
Obs
erve
d Fr
eque
ncy
of E
vent
s
Forecast Probabilities
Observed Frequencies
Linear (ObservedFrequencies)
60th IHC 22 March 2006 - Mobile, AL 17
Verification Procedure: Summary
• The probabilities are skillful– Brier Skill Score 28% more accurate than the
OFCL deterministic forecast• Note 50% of the OFCL forecasts verified
– ROC Skill Score 88%– The discrimination distance d=26% is large– Probabilities slightly under forecast and are
well calibrated for this limited dataset
60th IHC 22 March 2006 - Mobile, AL 18
Can the wind speed probabilities be used do decrease the area warned
or increase lead time?
60th IHC 22 March 2006 - Mobile, AL 19
Probability Model for NHC Hurricane Warnings
0
200
400
600
800
1000
1200
1400
1600
1/1/
1963
1/1/
1967
1/1/
1971
1/1/
1975
1/1/
1979
1/1/
1983
1/1/
1987
1/1/
1991
1/1/
1995
1/1/
1999
1/1/
2003
Date
War
ning
Len
gth
(nm
i)
0
12
24
36
48
60
72
84
1/1/
1963
1/1/
1966
1/1/
1969
1/1/
1972
1/1/
1975
1/1/
1978
1/1/
1981
1/1/
1984
1/1/
1987
1/1/
1990
1/1/
1993
1/1/
1996
1/1/
1999
1/1/
2002
1/1/
2005
Date
War
ning
Lea
d Ti
me
(hr)
NHC storm total hurricanewarning lengths 1963-2005
NHC storm average hurricanewarning lead times 1963-2005
Since 2000 warning areas have decreased and lead times increased.
60th IHC 22 March 2006 - Mobile, AL 20
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0
0 to
5
5 to
10
10 to
15
15 to
20
20 to
25
25 to
30
30 to
35
35 to
40
40 to
45
45 to
50
50 to
55
55 to
60
60 to
65
65 to
70
70 to
75
75 to
80
80 to
85
85 to
90
90 to
95
95 to
100
Bins (%)
Perc
ent O
ccur
ence
Avg. Probabilities at Warned Break Points (Warnings in Place)
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0
0 to
5
5 to
10
10 to
15
15 to
20
20 to
25
25 to
30
30 to
35
35 to
40
40 to
45
45 to
50
50 to
55
55 to
60
60 to
65
65 to
70
70 to
75
75 to
80
80 to
85
85 to
90
90 to
95
95 to
100
Bins (%)
Perc
ent O
ccur
ence
Avg. Probabilities at Unwarned Break Points (Warnings in Place)
Warned Break Points(Warnings in Place)
Unwarned Break Points(Warnings in Place)
%2.29%,8.27 EFEF %4.3%,4.1 '' EFEF
%4.26d
Discrimination Distance
N=5025 N=123225
60th IHC 22 March 2006 - Mobile, AL 21
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0
0 to
5
5 to
10
10 to
15
15 to
20
20 to
25
25 to
30
30 to
35
35 to
40
40 to
45
45 to
50
50 to
55
55 to
60
60 to
65
65 to
70
70 to
75
75 to
80
80 to
85
85 to
90
90 to
95
95 to
100
Bins (%)
Perc
ent O
ccur
ence
Avg. Probabilities at Warned Break Points (Warnings in Place)
%2.29%,8.27 EFEF
?
Why are there so many low probabilities at the break points?
60th IHC 22 March 2006 - Mobile, AL 22
Distribution of Probabilities at the Ending Break Points
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
0
0 to
5
5 to
10
10 to
15
15 to
20
20 to
25
25 to
30
30 to
35
35 to
40
40 to
45
45 to
50
50 to
55
55 to
60
60 to
65
65 to
70
70 to
75
75 to
80
80 to
85
85 to
90
90 to
95
95 to
100
Bins (%)
Perc
ent O
ccur
ence
Probabilities at the ending break points
9.14
8.8
EF
EF
end
end
?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
MX01
TX/MEX B
ORDER
BROWNSVILL
E
PORT ISABEL
STATE PARK P
ADRE ISLA
NDTX01
PORT MANSFIE
LDTX02
TX03TX04
BAFFIN B
AYTX05
TX06
CORPUS CHRIS
TI
PORT ARANSAS
TX07
ROCKPORTTX08
TX09
PORT OCONNOR
TX10
PALACIO
S
MATAGORDATX11
SARGENTTX12
FREEPORT
SAN LUIS
PASS
TX13
GALVESTON
BOLIVAR P
ENINSULA
TX14
HIGH IS
LANDTX15
PORT ARTHUR
SABINE P
ASS
SABINE R
IVERLA
01
CAMERON
LAKE C
HARLESLA
02LA
03LA
04LA
05LA
06
INTRACOASTAL C
ITY
VERMILION B
AYLA
07LA
08
MORGAN CITY
LA19LA
09LA
10LA
11LA
12
GRAND ISLELA
20LA
13
MISSIS
SIPPI R
IVERLA
21LA
14LA
15LA
16LA
17
NEW O
RLEANS
LA22LA
18
PEARL RIV
ER
BAY SAIN
T LOUIS
GULFPORT
Break Points
Prob
abili
ty/W
arni
ngs
(1 o
r 0)
-43.5 hr Prob-43.5 hr Warn-31.5 hr Prob-31.5 hr Warn-19.5 hr Prob- 19.5 hr Warn-7.5 hr Prob -7.5 hr WarnVerification
Example: Hurricane Rita
Warnings are brought down too slowly in this case.
00UTC 24 Sep.
12 UTC 23 Sep. 00 UTC
23 Sep.
12 UTC 22 Sep
t=0 at landfall
60th IHC 22 March 2006 - Mobile, AL 24
Summary of Warning Break Points
• Probabilities are useful in the watch/warning process.– Objectively assign of the warnings at fixed
lead time?– Average at warnings = 28%– Average at end points of the Warnings = 9%
• It appears that warnings can be dropped sooner, thus decreasing the area warned area.
60th IHC 22 March 2006 - Mobile, AL 25
Future Plans
• Seasonal Verification Code• See what we can learn from the
verification.• Report to this audience
Questions?
Top Related