Pablo Santos WFO Miami, FL Mark DeMaria NOAA/NESDIS David Sharp WFO Melbourne, FL
description
Transcript of Pablo Santos WFO Miami, FL Mark DeMaria NOAA/NESDIS David Sharp WFO Melbourne, FL
Pablo SantosWFO Miami, FL
Mark DeMariaNOAA/NESDIS
David Sharp
WFO Melbourne, FL
2010 AMS Annual Meeting, Atlanta, GA PS/MM/DS
Determining Optimal Thresholds of Tropical Cyclone Incremental Wind Speed
Probabilities to Support Expressions of Uncertainty within Text Forecasts
2
Probability Definitions • Cumulative – The probability of sustained winds reaching or exceeding
the specified wind speed threshold between the 00 hour forecast and a specified forecast hour. • Available in Grid (NDFD)/Graphic Forms as well as within PWSAT# product.
• Individual (Interval) – The probability of sustained winds reaching or exceeding the specified wind speed threshold beginning during a specified forecast period ending at the specified forecast hour (e.g., period of onset).• Computed from Cumulative and explicitly available in PWSAT# text product.
Incremental –The probability of sustained winds reaching or exceeding the specified threshold during a specified forecast period ending at the specified forecast hour.• Available in Grid (NDFD) but not within the PWSAT#. • These are the probabilities referenced throughout this presentation.
(A Quick Review)
3Hypothetical Test Case
Wind Impacts Are A Concern Well Outside Of The Error
Cone
3
Based on 1938 Long Island Express
Hurricane - a large fast-moving storm
Tropical Storm Force wind probabilities are
almost certain in a large area from the Carolinas to New
England well outside the cone!
4
• During potential high impact weather events such as tropical cyclones (TC), users not only demand a WFO’s ‘best forecast’, but also require a corresponding expression of uncertainty for decision-making purposes.
• Therefore, a methodology was developed that uses TC Incremental Wind Speed Probabilities to elevate the utility of official text forecast products:
– By including situational expressions of uncertainty within the Zone Forecast Product (ZFP) and the Coastal Waters Forecast (CWF) which responsibly ‘scopes’ NHC’s forecast (temporally and spatially) for use within local forecasts.
• For WFOs, this effort crosses a major threshold for incorporating probabilistic data (other than PoP) within their text forecast products.
Proposed WFO Use (Item 42-05 from NOAA Hurricane Conference, 2005)
5
(The Nine Expressions of Uncertainty) • HURRICANE CONDITIONS *• HURRICANE CONDITIONS EXPECTED• HURRICANE CONDITIONS POSSIBLE• TROPICAL STORM CONDITIONS *• TROPICAL STORM CONDITIONS * WITH HURRICANE CONDITIONS POSSIBLE• TROPICAL STORM CONDITIONS EXPECTED• TROPICAL STORM CONDITIONS EXPECTED WITH HURRICANE CONDITIONS POSSIBLE• TROPICAL STORM CONDITIONS POSSIBLE• TROPICAL STORM CONDITIONS POSSIBLE WITH HURRICANE CONDITIONS ALSO POSSIBLE
• The * phrases are equivalent to imminent or ongoing conditions as reflected in the 00-12 hour period simultaneously by the hazard, wind, and probability grids. • The word EXPECTED is used in situations mainly during the Warning period, but is also selectively used in the Watch period (e.g., Day 1 & 2, or out to period 4).• The word POSSIBLE is used during the extended period (e.g., Day 3 - 5), but is
also used in specific situations within Watch/Warning period. • Also, certain situations require compound phrases (e.g., when there is a Tropical Storm Warning with a Hurricane Watch).
Baseline Phraseology
6
Determining Optimal Probability Thresholds
• NHC verification program evaluates gridded products.• In this case as it pertains to WFO Forecast applications.
– Sample: • 19 storms 2004-2008 cases with U.S. hurricane warnings along
East and Gulf of Mexico Coasts from Brownsville, TX to Portland, Maine.
• All probability runs starting 3 days before 1st warning.– 2008 MC model re-runs for 400 forecast cases.
• Evaluated probabilities at 343 coastal “breakpoints” as well as 286 inland points (approx. 50 km inland).
– Use Threat Score or ROC diagrams to select optimal probability thresholds.
• Emphasis placed on 12 h incremental probabilities.
• Results for 34 & 64 knot probabilities were based on– 6 hourly NHC best track wind radii interpolated every hour to
produce the model verification grid (used to determine points with observed hurricane/tropical storm force winds).
7
2010 Wind Speed Probability Model
• Track and Intensity error statistics updated with previous 5 year sample– 2005-2009
• Track error distributions stratified by Goerss Predicted Consensus Error (GPCE)– GPCE estimates track forecast uncertainty from model spread
and other forecast parameters • GPCE version implemented at NHC in 2010
– Includes option to run without GPCE input • 2010 model (GPCE and non-GPCE versions) run for all
2004-2008 WFO test cases– Coastal and inland points
8
Storm Cases For 400 MC Model Runs
• Coastal Points• Inland Points
9
Conversion of Probabilities into Yes-No Forecast
• Pick probability threshold (Pt)• If P ≥ Pt, forecast YES• If P < Pt, forecast NO• Best choice of Pt depends on particular
application • Use appropriate validation metric to select Pt
objectively • Metrics determined from contingency table
10
Contingency Table for Validation of Yes-No Forecasts
ObservedYes No
Yes
Forecast
No
a b
c d
Threat Score (TS) = a/(a + b + c)Hit Rate (HR) or Probability of Detection (POD) = a/(a + c)False Alarm Rate (FR) = b/(b + d)
11
Regional Stratification
• Estimate optimal probabilities using Threat Score– ROC score probabilities too low
• Region 1– Brownsville, TX to Mobile, AL
• Region 2– Mobile, AL to GA/SC Border
• Includes all Florida breakpoints• Region 3
– GA/SC Border to Eastport, ME
12
Optimal 34 knot Probability Thresholds(Coastal vs. Inland)
Based on Threat Score…collectively, and by each region.
13
Optimal 64 knot Probability Thresholds(Coastal vs. Inland)
Based on Threat Score…collectively, and by each region.
14
Optimal 34 & 64 knotProbability Thresholds
(Combined)
Based on Threat Score…by region, for all points.
15
Associated Reliability Diagrams(Coastal vs. Inland)
16
Associated Reliability Diagrams(Combined)
17
2010 Evaluation of GPCE input:Reduction in Brier Score
In
land
34
Cum
Inla
nd 3
4 In
c
Inla
nd 6
4 Cu
m
Inla
nd 6
4 In
c
Coas
tal 3
4 Cu
m
Coas
tal 3
4 In
c
Coas
tal 6
4 Cu
m
Coas
tal 6
4 In
c
0
1
2
3
4
5
Perc
ent I
npro
vem
ent
18
Summary• Optimal Probability Threshold curves decrease almost
exponentially with time (for both 34 & 64 knot).• Optimal thresholds are correspondingly larger for inland
points, particularly in the shorter ranges (e.g., watch/warning periods).
• Optimal thresholds are a function of geographical location:– More so for 34 knot probabilities, in the short range, and for coastal
points.• Overall, probabilities are deemed to be “reliable” although:
– They tend to under predict 64 knot or greater events particular for coastal points. For all points combined, this bias is smaller.
• Results indicate that the empirically-derived thresholds (original; based on histograms) need an upward adjustment in the short range to better support operational use.
19
Summary
• That there is inherent operational sensitivity regarding event-specific reliability and continuity (forecast-to-forecast, region-to-region, coastal-to-inland, etc.),
• That the current ZFP/CWF text formatters will only support one set (each) of 34 and 64 knot thresholds,
• And that newer Monte Carlo based results by region are still forthcoming which will also include more cases,
• …the following (interim) values to the right are the ones currently in effect for operational use.
Period PWS64 PWS3400-12 hr 30% 55%13-24 hr 25% 45%25-36 hr 20% 40%37-48 hr 15% 35%49-60 hr 10% 30%61-72 hr 7% 25%73-84 hr 6% 20%85-96 hr 5% 15%97-108 hr 4% 12.5%
109-120 hr 3% 10%
So Given this and…
Proposed Single Set Values
20
Future Work
• Iteratively transition optimal thresholds to ZFP and CWF text formatters, as well as point-n-click web forecast code.– single set– multi-set
• coastal vs. inland• by region
• Complete verification using 2010 version of the Monte Carlo model by regions.
21
END
22
Discussion Items 10-49 to 10-51Official Implementation of EoU
• Problems Encountered During 2010 Season: – ER problem with phrases dropping Wx in presence of EoU.
Investigating.– Offices editing probability grids and questioning EoU with winds
of 10 to 15 mph. Training Issue.– However, technical team suggests we can consider dropping the
phrase if winds less than 15 kts and there is no warning.
• Iteratively transition optimal thresholds to ZFP and CWF text formatters, as well as point-n-click web forecast code.– single set, multi-set, coastal vs. inland, by region
23
Discussion Items 10-49 to 10-51Official Implementation of EoU
• Back Up sites will need training to be aware of this capability. Configuration wise it should not be a problem.
• • Public and Marine Directives will need to be updated to include set of
rules that prompts EoU.
• Work on instructions for field offices to enable Tropical SAF Formatter. MIA and MLB already have it. Baseline code is the same as the ZFP. It is a matter of configuration.
• RECOMMENDATION: Official or Not? Reconvene Wind Team to oversee all issues above are taken care of whether it is by 2011 or 2012.
24
BACK UP SLIDES
25
REFERENCES
• DeMaria, M., J. A. Knaff, R. Knabb, C. Lauer, C. R. Sampson, and R. T. DeMaria, 2009: A new method for estimating tropical cyclone wind speed probabilities. Wea. Forecasting, 24, 1573-1591.
• Santos, P., D. Sharp, M. DeMaria, and S. Kiser, 2009: The Determination of Optimal Thresholds of Tropical Cyclone Incremental Wind Speed Probabilities to Support Expressions of Uncertainty in Text Forecasts. Preprints, Symposium on Urban High Impact Weather/89th AMS Annual Meeting, P1.5, Phoenix, AZ. [Available online http://ams.confex.com/ams/pdfpapers/147019.pdf]
• Santos, P., D. W. Sharp, M. Volkmer, and G. Rader, 2008: Employing Hurricane Wind Probabilities to Convey Forecast Uncertainty and Potential Impact Through NWS Field Office Forecast Products. Preprints, Tropical Meteorology Special Symposium/19th Conf. on Prob. And Statistics, JP1.3, New Orleans, LA. [Available online http://ams.confex.com/ams/pdfpapers/131748.pdf]
• Sharp, D. W., P. Santos, M. Volkmer, G. Rader, and M. Sardi, 2006: Employing Hurricane Wind Probabilities to Enhance Local Forecasts and Improve Guidance for Decision-Makers. Preprints, 27th Conf. on Hurricanes and Tropical Meteorology, 9A.3, Monterey, CA. [Available online http://ams.confex.com/ams/pdfpapers/108409.pdf]
26
Wind Impacts Are A Concern Well Outside Of The Error
Cone
26
Based on 1938 Long Island Express
Hurricane - a large fast-moving storm
Hurricane-force winds extended outside the cone over much of
NJ, srn NY, LI, and CT
Hypothetical Test Case
27
Storm Cases For 400 MC Model Runs
28
Optimal 34 and 64 kt Probability Thresholds based on Threat Score and
ROC Diagram for Coastal Points
0
10
20
30
40
50
60
0-12 hr
12-24 hr
24-36 hr
36-48 hr
48-60 hr
60-72 hr
72-84 hr
84-96 hr
96-108 hr
108-120 hr
Time Interval (hr)
Opt
imal
Pro
babi
lity
Thre
shol
d (%
) 34 kt Threat Score-based64 kt Threat Score-based34 kt ROC Diagram-based64 kt ROC Diagram-based34 kt Hist-Based64 kt Hist-Based
2929
Note:• Go to weather.gov• Click on Graphical Forecasts• Scroll to the Bottom• Click on Tropical• Notice both Cumulative and
Incremental Probabilities are available here
• Notice the Resolution• What About Interval
or Individual Probabilities?
Tropical Cyclone Wind Speed Probabilities
Through NDFD
30
How Can I get the grids from the NDFD
• Download DeGrib (NDFD GRIB2 Decoder from):• http://www.weather.gov/mdl/NDFD_GRIB2Decoder/register.php
• Then go to this site and read on how turn the NDFD gridded data into GIS Files (Creating GIS files from the downloaded NDFD grids):• http://www.weather.gov/ndfd/gis/ndfd_GIS_tutorial.html
31
Pt Optimization Procedures
• Threat Score– Threat Score useful for low probability events since it
does not include No-No cases– TS ranges from 0 to 1 (1 is best)– Pick Pt that maximizes TS
• Hit Rate/False Alarm Rate– Plot HR vs. FR in 2-D plane for range of Pt
• Relative Operating Characteristic (ROC) diagram– Pick Pt for point closest to upper left corner in ROC
diagram • Upper left corner has Hit Rate=1, False Alarm Rate=0
32
Hurricane Charley - Update
33
Hurricane Charley (ZFP)
CHARLEY (ZFP)* Punta Gorda
Advisory Time: 20040813_1500 (0-12 Hours – First Period)A Second Algorithm Example
34
Fay – WFOs MFL and MLB