Eidgenössisches Departement des Innern EDIBundesamt für Meteorologie und Klimatologie MeteoSchweiz
The challenge to verify operational weather warnings
Tanja Weusthoff and Marco Arpagaus
EMS, 14.09.2011
MeteoSwiss official warnings (24h acc. precip.) for 4th September 2011
2 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Introduction 152 warning regions
5 level
8 different hazards
current project: development of an application for an automatic verification of weather warnings
3 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Introduction
specifications: • develop an automatic verification for operational warnings,
which allows a certain tolerance (space and time)
• use synergies (e.g. with GIN, a common platform for all natural hazards in Switzerland: http://www.gin-info.ch/index.html)
motivation:• replace current manual (subjective) verification in order to
release resources
• prove the quality of official warnings
• single official voice (SOV, since January 2011) distribution via media for level 4 and 5
4 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Challenges
• how to evaluate the usefulness of warnings without knowing the needs of individual users (and their cost/loss)
• how to interprete „tolerant results“ and what should be communicated to the users
Precautions causes Costs
Having no protection results in Losses
5 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Challenges
• representativity of observations
• account for feedback of authorities
• small statistics (rare events)
> 2000 m
< 800 m
e.g. station Magadino / Cadenazzo (203 m asl) is only wind station for two warn regions with complex terrain (308 and 309)
6 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Basic Concept
• event-oriented verification, warnings as binary events
• tolerant in space, time and threshold
• distinguish two types of warnings• SHORT: short-term events (e.g. thunderstorms)• AKKU: accumulated events (e.g. 24h precipitation accumulation)
• event definition and evaluation is (in principle) the same within each group; differentiate between basic verification (strict) and detailed verification
• verification per warning region, summary for specific regions or whole Switzerland
7 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Example: AKKU
Rain level 3-5Snowfall (lowlands and mountains) level 3-5Snowmelt level 3-5Heat wave level 3
• snowmelt cannot be verified due to a lack of observations; • for snow and rain use of radar data (spatial information) and
psychrometer temperature
8 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Event-Definition
• each warning is an event; duration of warning at least as long as accumulation period duration
• observation: consider hourly 24h, 48h and 72h sum; important is first threshold exceedance (t2)
Hit An event is observed during a valid warning and the first threshold exceedance (t2) occurs at least accumulationPeriod/3 hours (i.e. 8h,16h,24h) after the beginning of the warn event (ts) and not later then the end of the warn event (te).Miss A threshold exceedance is observed without an active warning or the first threshold exceedance occurs bevor accumulationPeriod/3 hours after the beginning of the warning (ts). False Alarm A warning has been issued, but no threshold exceedance has been observed.
AKKU
t2 - ti > tv?
tiissue time
WarnEvent (ts to te)
ts t1 t2 te
ObsEvent (t1 to t2)
acc/3
9 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Evaluation A: Basic-verification
Present results as:
• POD
• FAR
• FBI
• (TS)
derived from contingency table
AKKU
Event observed No event
Warning issued A (Hit) B (False Alarm)
No warning C (Miss) D (Correct Rejection)
Evaluation B: detailed verification• in principle like Evaluation A (Hit, Miss, False Alarm)
• introduce additional category
• combination of Hit, Miss und False Alarm see definitions on next slide
Missing-D-problem: what is a „non-event“?
10 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Evaluation B: detailed verification
(1.)
Specifications
Miss + Hit + False Alarm the first threshold exceedance (t2) occurs maximal tshift hours after the end of the warn event (te) or less than accumulatioPeriod/3 (8,16,24h) but more than accumulationPeriod/4 (6,12,18h) after the start of the warning (ts).
tiissue time
WarnEvent (ts to te)
ts t2 te
ObsEvent (t1 to t2)
(2.)
AKKU
tiissue time
WarnEvent (ts to te)
ts t1 t2te
ObsEvent (t1 to t2)
t2 - te < tshift
acc/4 acc/3
11 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Evaluation B: detailed verification
Presentation of results:
„perfect“ hit
„useful“ combined categories including a hit
„bad“ false alarm, miss
AKKU
„good“
... adapted from DWD
12 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Tolerance
• Allow tolerances in:• threshold: LowHit = 90% of threshold• time: variation of time components (e.g. tshift) • space: if possible, take into account neighbouring warn
regions
• apply tolerances to basic and detailed verification
• SHORT: possibility to request a minimum leadtime tv for the warning (i.e. a hit is only possible if the warning has been issued at least tv (= t1 - ti) hours before the observed event, otherwise it is classified as a miss)
SHORT + AKKU
13 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Application Flow Diagram
obs DataBase hourly data per warn region
warn DataBase warn events
store warn and obs events on a monthly basis
store evaluation results for each event on a monthly basis
present results• per region• single event• …
Java Application AWV• derive obs events• perform event-based
verification for each setting and each region
• aggregate results and calculate scores for specific period
14 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Observations (surface stations): 24 h acc.MeteoSwiss official warnings (24h acc. precip.)
for 4th September 2011
A qualitative example…Level 3 warning:
WarnIssue 04.09.2011 08:56
WarnStart 04.09.2011 12:00
WarnEnd 05.09.2011 12:00
24h acc. precip.
35 mm/24h
50 mm/24h
Bern
Luzern
precipitation analysis for 4th September 2011 (prel.)
15 Verification of Weather Warnings | EMS 2011Tanja Weusthoff
Thanks for listening …
Top Related