ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
description
Transcript of ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
ASSESSING THE IMPACT OF COLLABORATIVE
RESEARCH PROJECTS ON NWS PERFORMANCE
Jeff Waldstreicher
Scientific Services Division – Eastern Region
Northeast Regional Operational Workshop (NROW)
November 4-5, 2003
Albany, NY
ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS
• IMPACT ON VERIFICATION SCORES– Performance Metrics
• CASE STUDY ANALYSIS– Event Verification– Subjective Evaluation of Impact on Forecast
Process• AFDs• Event Reviews and Impact Reports
COMPLICATIONS
• Many Factors Influence Performance Metrics– Factors Not Independent– Difficult to tie performance changes to a specific factor
• Cannot Analyze “Null Case” – If forecasters have “knowledge” or “data,” cannot
directly answer “What if they did not have the knowledge?”
FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES
• Infusion of New Technologies – Hardware (New systems or Processors) – Software (New algorithms or Models)
• Applied Research and Development
– National (Research Laboratory)– Local/Regional– Independent– Collaborative
• Changes to Operational Procedures– Implementation of Best Practices
FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES
• Climate Variability– Variations in frequency and type of events
• External Outreach and Education– Including development of spotter networks
• Personnel/Staffing Issues– Including forecaster experience levels
• Training– All other factors are also tied to training issues
HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?
• Can be global (national), regional, or local– Technology usually national
– Climate Variability typically regional
– Staffing usually local • Impact can be long or short term
– Technology infusion typically has long term impact
– Climate Variability is usually short term
• Factors are often inter-related– New technology or staffing changes require training
– Applied R&D often yields/suggests changes to operations
COMET PROJECTS IN EASTERN REGION
• Different types of projects– Cooperative (2-3 year ~$35K/yr)
• 19 ER Cooperative Projects funded since 1991
– Partners (1 year ~$9K/yr)• 39 ER Partners Projects funded since 1991
• 20 ER WFOs and 3 RFC have participated– 90 Offices Nationwide
• 21 Universities have participated in ER projects– 70+ Universities Nationwide
EVALUATING COMET PROJECTS
• Examined COMET Cooperative and Partners Projects in Eastern Region– Projects completed between 1995 and mid-2001– Projects specifically addressing warning programs
• Tornado• Severe Thunderstorms• Flash Flooding• Winter Storms
• Study was designed to minimize as much as possible the impact of the factors previously discussed.
METHODOLOGY• 3-year running verification scores used
– Minimize impact of short-term factors such as variability of events
• Compared 3 years before project to 3 years following project– 1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores
and a full 3 years of “after” scores
• 3-year “expected” improvements calculated based on long term trend of ER-wide scores– Compared rate of improvement for WFO involved in collaborative project to ER-
wide improvement– Use of ER scores as a baseline minimizes the impact of national/region-wide
factors such as AWIPS and radar system improvements
• Very difficult to evaluate the impact of project results beyond the primary WFO
INDIVIDUAL PROJECT COMPARISON
• Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period
PROBABILITY OF DETECTION (POD)
# of COMET Projects With
WFO Greater Improvement
# of COMET Projects With ER Greater
Improvement
# of COMET Projects With No
Difference in Improvement
Tornado Warnings 3 2 1
Severe Tstm Warnings 6 1 0
Flash Flood Warnings 5 1 0
Winter Storm Warnings 4 3 0
FALSE ALARM RATIO (FAR)
# of COMET Projects With
WFO Greater
Improvement
# of COMET Projects With ER Greater
Improvement
# of COMET Projects With No
Difference in Improvement
Tornado Warnings 2 4 0
Severe Tstm Warnings 3 4 0
Flash Flood Warnings 3 3 0
Winter Storm Warnings 3 3 1
LEAD TIME
# of COMET Projects With
WFO Greater Improvement
# of COMET Projects With ER Greater
Improvement
# of COMET Projects With No
Difference in Improvement
Tornado Warnings 5 1 0
Severe Tstm Warnings 5 1 1
Flash Flood Warnings 4 1 1
Winter Storm Warnings 4 3 0
IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES
• WFO RAH and NCSU– Continuous collaborative projects since January 1991– COMET Projects 1991-2000
• 3 Cooperative Projects• 3 Partners Projects• 1 Graduate Fellowship
– CSTAR Projects• 2000-2003• 2003-2006
• Unique “laboratory” to examine the impact of long-term collaborative activities
IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES
• WFO ALY and U-Albany– Continuous since February 1995– Numerous COMET Projects
• 3 Cooperative Projects
• 7 Partners Projects
• Several additional projects between UA and NCEP
– CSTAR PROJECT 2001-2004
SUMMARY
• Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance
• There are indications that certain
performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.