Marine Verification at WFO Key WestMarine Verification at WFO Key West

-Where to go and how to get there…

Andy Devanas, SOO and

Matt Parke, Verification Focal Point

There are three kinds of lies: There are three kinds of lies: lies, damned lies, and statistics.lies, damned lies, and statistics.

Benjamin Disraeli (1804 - 1881)

WFO KEY WEST COUNTY WARNING AREA (CWA)HIGHLIGHTED

Figure 1

X

XX

XX

X

X – No Surface Data Available

NO WAVE DATA AT ALL

USF BuoyUSF Buoy

Not Not OperationalOperational

Marine Area approx. Marine Area approx.

22000 square miles 22000 square miles

Massachusetts – Massachusetts –

10555 square miles10555 square miles

Key West Verification System Needs…Key West Verification System Needs…

Need to integrate verification into the forecast Need to integrate verification into the forecast processprocess

Need to provide meaningful, frequent and timely Need to provide meaningful, frequent and timely verification data to forecasters ( e.g. Percent verification data to forecasters ( e.g. Percent Correct vs. ESS)Correct vs. ESS)

Need to identify forecast and guidance biasesNeed to identify forecast and guidance biases

Enhanced discrimination in critical categoriesEnhanced discrimination in critical categories

How To Get There…How To Get There…

*Use MVerify for rapid feedback to forecasters*Use MVerify for rapid feedback to forecasters-Local AWIPS application-Local AWIPS application-Provides most flexibility in stratification of desired -Provides most flexibility in stratification of desired

parametersparameters Minimum forecast period adjustableMinimum forecast period adjustable Can segregate by forecasterCan segregate by forecaster

*Investigate various statistical measures/distributions which *Investigate various statistical measures/distributions which best track forecast performancebest track forecast performance

*Set reasonable goals of forecast performance.*Set reasonable goals of forecast performance.

*Monitor forecast performance.*Monitor forecast performance.-Accountability-Accountability

*Use verification statistics in post-mortem evaluations as *Use verification statistics in post-mortem evaluations as feedback to forecasters.feedback to forecasters.

APPLICATION OF THE NATIONAL MARINE APPLICATION OF THE NATIONAL MARINE VERIFICATION PROGRAM (NMVP) TO LOCAL VERIFICATION PROGRAM (NMVP) TO LOCAL MARINE VERIFICATIONMARINE VERIFICATION

DRAWBACKSDRAWBACKS*NMVP is not timely. Must wait until after end of month to access NMVP is not timely. Must wait until after end of month to access statistics.statistics.*NVMP does not allow defining of periods.*NVMP does not allow defining of periods.

Minimum period is one month.Minimum period is one month.*Can’t define inter-month periods.*Can’t define inter-month periods.

*NMVP does not provide access to individual forecast-observation pairs.*NMVP does not provide access to individual forecast-observation pairs.-Cannot do post-mortem analysis.-Cannot do post-mortem analysis.-Do not know which matched pairs are producing statistics.-Do not know which matched pairs are producing statistics.-No Quality control of forecasts and observations-No Quality control of forecasts and observations..

*Some measures (ESS) are heavily weighted towards rare climatological *Some measures (ESS) are heavily weighted towards rare climatological events.events.

-ESS is volatile.-ESS is volatile.-Critical missing matched pairs can have large -Critical missing matched pairs can have large

impact on ESS.impact on ESS.

BENEFITSBENEFITS*Provides relatively easy and quick statistics spanning > 3 year dataset*Provides relatively easy and quick statistics spanning > 3 year dataset*Can stratify data with regards to forecast period, buoy (cman), and *Can stratify data with regards to forecast period, buoy (cman), and seasonseason

Equitable Skill Score reward/penalty Equitable Skill Score reward/penalty Characteristics (from NMVP)- Characteristics (from NMVP)-

•A relatively small reward for correctly forecasting common A relatively small reward for correctly forecasting common eventsevents

•A large reward for correctly forecasting rare events A large reward for correctly forecasting rare events

•Less penalty is assigned to an incorrect forecast of a rare Less penalty is assigned to an incorrect forecast of a rare event than a similar size error of a common event. "Near event than a similar size error of a common event. "Near hits" of rare event receive a modest reward. hits" of rare event receive a modest reward.

•Uses a graduated reward/penalty system, whereby a large Uses a graduated reward/penalty system, whereby a large forecast error is penalized more than a small forecast error forecast error is penalized more than a small forecast error for a given category of events. for a given category of events.

What’s wrong with this What’s wrong with this picture??picture??

FORECAST

OBS

  < 8 8 - 12 13 - 17 18 - 22 23 - 27 28 - 32 > 32TOTA

L   

< 8 10 24 8 0 0 0 0 42    

8 - 12 2 61 66 16 0 0 0 145    

13 - 17 0 15 114 91 7 0 0 227    

18 - 22 0 1 32 51 28 0 0 112    

23 - 27 0 0 1 6 1 0 0 8    

28 - 32 0 0 0 0 0 0 0 0    

> 32 0 0 0 0 0 0 0 0    

TOTAL 12 101 221 164 36 0 0 534

SS

CC

EE

CC

SS

CC

AA

MOST CRAFT

REMAIN

IN PORT

ESS Does Not Account For Value of ESS Does Not Account For Value of Forecast to Our Customers…Forecast to Our Customers…

FCST< 8 8 - 12 13 - 17 18 - 22 23 - 27 28 - 32 > 32 TOTAL

< 8 11 17 0 0 0 0 0 288 - 12 2 31 4 1 0 0 0 38

13 - 17 0 7 13 6 0 0 0 26OBS 18 - 22 0 0 4 15 1 0 0 20

23 - 27 0 0 0 1 5 0 0 628 - 32 0 0 0 0 0 0 0 0

> 32 0 0 0 0 0 0 0 0TOTAL 13 55 21 23 6 0 0 118

Bias 0.46 1.45 0.81 1.15 1 -- --

FCST< 8 8 - 12 13 - 17 18 - 22 23 - 27 28 - 32 > 32 TOTAL

< 8 19 9 0 0 0 0 0 288 - 12 2 24 12 0 0 0 0 38

13 - 17 0 4 15 7 0 0 0 26OBS 18 - 22 0 0 4 14 2 0 0 20

23 - 27 0 0 0 1 4 3 0 828 - 32 0 0 0 0 0 0 0 0

> 32 0 0 0 0 0 0 0 0TOTAL 21 37 31 22 6 3 0 120

Bias 0.75 0.97 1.19 1.1 0.75 -- --

1 2 3 4 5 - 71 1.12 0.048 -0.406 -0.732 -1

OBS 2 0.048 0.375 -0.08 -0.406 -0.674CATEGORY 3 -0.406 -0.08 0.476 0.149 -0.118

WEIGHTS 4 -0.732 -0.406 0.149 1.221 0.9535 - 7 -1 -0.674 -0.118 0.953 4.703

FCST CATEGORY WEIGHTS

Illustration indicating sensitivity of ESS to missing observations at climatologically improbable bins.

WFO KEY WEST FORECAST VS OBS SEP 2002

AVN FORECAST VS OBS SEP 2002

PURPLE REPRESENTS BINS COLLAPSED AND MULTIPLIED BY WEIGHTS

OVERFORECAST

UNDERFORECAST

THEREFORE AVN ESS IS IMPROVED GREATLY BY WEIGHTING OF CLIMATOLOGICALLY IMPROBABLE EVENTS WHICH ARE NOT INCLUDED IN THE EYW ESS

TO MAKE MEANINGFUL COMPARISONS BETWEEN DIFFERENT FORECAST SYSTEMS, MUST HAVE MATCHED DATASETS - QUALITY CONTROL

WFO KEY WEST ESS = 0.69PERCENT CORRECT = 63.56 %PERCENT >1 CATEGORY = 0.85 %

AVN ESS = 0.76PERCENT CORRECT = 63.33 %PERCENT >1 CATEGORY = 0.00 %

Before Matched Sets at NMVP Site

CONTINGENCY TABLECONTINGENCY TABLE

FORECASTFORECAST

OBSOBS

   < 8< 8 8 - 128 - 12 13 - 13 - 1717

18 - 18 - 2222

23 - 23 - 2727

28 - 28 - 3232 > 32> 32 TOTATOTA

LL     

< 8< 8 2222 4646 77 44 00 00 00 7979      

8 - 128 - 12 2727 116116 5959 1414 00 00 00 216216      

13 - 1713 - 17 00 3939 134134 4747 22 11 00 223223      

18 - 2218 - 22 00 00 1313 4545 77 00 00 6565      

23 - 2723 - 27 00 00 00 88 00 00 00 88      

28 - 3228 - 32 00 00 00 00 00 00 00 00      

> 32> 32 00 00 00 00 00 00 00 00      

TOTALTOTAL 4949 201201 213213 118118 99 11 00 591591      

BiasBias 0.620.62 0.930.93 0.960.96 1.821.82 1.121.12 ---- ----         

This Forecast Was Never MadeThis Forecast Was Never Made

Where Did This Number Come From?Where Did This Number Come From?

This Will Severely This Will Severely Skew ESS Score. As Skew ESS Score. As Well As Other Well As Other Statistical Measures.Statistical Measures.

EYW Forecasts: January 2004EYW Forecasts: January 2004

2x2 2x2 Contingency Contingency

TableTable

Event ObservedEvent Observed

YESYES NONO

Event Event ForecastForecast

YESYES AA BB

NONO CC DD

Percent Correct Definition From Percent Correct Definition From NOAA Verification Glossary NOAA Verification Glossary

PC= (A+D)/(A+B+C+D). PC= (A+D)/(A+B+C+D).

Percent CorrectPercent Correct Definition: Percentage of the correct forecasts (hits) from Definition: Percentage of the correct forecasts (hits) from the total number of events. the total number of events.

Formula:Formula:                   

Percent Correct Percent Correct Definition From Definition From

Marine Verification Program GlossaryMarine Verification Program Glossary

In Other Words: A/(A+B+C+D)In Other Words: A/(A+B+C+D)

FCSTa. < 8 8 - 12 13 - 17 18 - 22 23 - 27 28 - 32 > 32 N p(x)

< 8 43.8 50.2 5.6 0.3 0.1 0 0 5599 26.98 - 12 11.1 56.8 30.1 1.9 0.1 0 0 7481 35.913 - 17 1.5 20.9 56.5 19.8 1.2 0 0 5439 26.1

OBS 18 - 22 0.3 3.7 23.1 59.3 13.2 0.3 0.1 1837 8.823 - 27 0.2 0.5 4.6 40.1 51.1 3.2 0.2 409 228 - 32 0 0 5.3 23.7 52.6 13.1 5.2 38 0.2 > 32 0 0 0 14.3 42.9 14.3 28.6 7 0

20810FCST

b. < 8 8 - 12 13 - 17 18 - 22 23 - 27 28 - 32 > 32< 8 72.9 34.0 5.1 0.6 0.2 0 0

8 - 12 24.5 51.4 37.0 5.8 1.1 0 013 - 17 2.4 13.7 50.5 43.1 12.2 0 0

OBS 18 - 22 0.1 0.8 6.7 43.5 44.3 24.0 28.623 - 27 0.1 0.1 0.3 6.6 38.1 52.0 14.328 - 32 0 0 0.3 0.4 3.6 20.0 28.6 > 32 0 0 0 0 0.5 4 28.6

N 3375 8267 6086 2501 549 25 7 20810p(f) 16.2 39.7 29.2 12.1 2.6 0.1 0.03

Table 1. Factorization of the joint distribution of wind speed forecasts and observations with a.) representing the conditional distribution of the forecast bins based on a particular observation bin as well as the marginal distribution of the observations (also known as the likelihood-base rate factorization) and b.) representing the conditional distribution of observation bins based on a particular forecast bin as well as the marginal distribution of forecasts (also known as the calibration-refinement factorization).

Difficulty discriminating between bins.

Overforecast area

WFO KEY WEST JAN 1999 - OCT 2003 WIND SPEED FORECAST - OBSERVATION PAIRS (5 C-MAN LOCATIONS)

Overforecast this bin 42.1% of the time.

Yellow diagonal line represents the perfect forecast.

Underforecastarea

“gaming”

•The majority, almost 89%, of our observations The majority, almost 89%, of our observations (marginal distribution p(x)) were ≤ 17 kts.(marginal distribution p(x)) were ≤ 17 kts.

•Our forecast system had a hard time discriminating Our forecast system had a hard time discriminating between light winds (< 8kts) and winds of 8 – 12 kts.between light winds (< 8kts) and winds of 8 – 12 kts.•Our forecast system is slightly better discriminating in Our forecast system is slightly better discriminating in the 8 – 12 kt range, but still tends to overforecast more the 8 – 12 kt range, but still tends to overforecast more often than underforecast.often than underforecast.•At 13 – 17 kts, our system is the least biased.At 13 – 17 kts, our system is the least biased.

•We underforecast (bias > 1) moderate to high wind We underforecast (bias > 1) moderate to high wind events.events.

•Forecasters tend to “game” light wind forecasts which Forecasters tend to “game” light wind forecasts which do not have much value to customers.do not have much value to customers.

Some EYW verification results:Some EYW verification results:

Gridded Verification at Gridded Verification at EYW….EYW….

Otherwise known as…Otherwise known as…

What are you What are you KiddingKidding me??me??

From Murphy 1993…From Murphy 1993…

““Since Forecasters’ judgments Since Forecasters’ judgments necessarily contain an element of necessarily contain an element of uncertainty, their forecasts must uncertainty, their forecasts must reflect this uncertainty accurately in reflect this uncertainty accurately in order to satisfy the basic maxim of order to satisfy the basic maxim of forecasting. In general, then, forecastsforecasting. In general, then, forecasts must be expressed in probabilistic must be expressed in probabilistic terms.”terms.”

Melbourne’s ADAS

Closer to grid verification

*Supported by relatively dense surface data network over land

*Buoy in Marine Zones

LAPS analysis highly LAPS analysis highly influenced by single data influenced by single data point.point.

Winds in this region most Winds in this region most likely NE 10-15 knots.likely NE 10-15 knots.

Again – note influence Again – note influence of single data point of single data point over huge region…over huge region…

We do not have ANY guidance to forecast wave height inside the reef at this resolution and detail, and little skill – therefore we do not attempt to forecast this region in our grids and zero out this region – but do include in our CWF.

Wavewatch III is poor at best outside of the reef – and does not account for gulf stream interactions.

AND NOT ONE WAVE HEIGHT DATA POINT!!

So How Do You Verify Wave Height??

Answer: You don’t.

Marine ForecastMarine Forecast730 PM EST SUN MAR 14 2004730 PM EST SUN MAR 14 2004

SMALL CRAFT ADVISORYSMALL CRAFT ADVISORY

TONIGHTTONIGHT: EAST WINDS AROUND 20 KNOTS. SEAS OUTSIDE THE : EAST WINDS AROUND 20 KNOTS. SEAS OUTSIDE THE REEF 5 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. REEF 5 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. SEAS SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY TO INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY TO ROUGH. SCATTERED SHOWERS.ROUGH. SCATTERED SHOWERS.

MONDAYMONDAY: EAST WINDS 15 T0 20 KNOTS...DECREASING TO 10 TO 15 : EAST WINDS 15 T0 20 KNOTS...DECREASING TO 10 TO 15 KNOTS BY NOON. SEAS OUTSIDE THE REEF 4 TO 7 FEET...EXCEPT KNOTS BY NOON. SEAS OUTSIDE THE REEF 4 TO 7 FEET...EXCEPT HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3 FEET. HIGHER IN THE GULF STREAM. SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS CHOPPY. SCATTERED SHOWERS. NEARSHORE WATERS CHOPPY. SCATTERED SHOWERS. MONDAY NIGHT: SOUTHEAST WINDS NEAR 15 KNOTS. SEAS MONDAY NIGHT: SOUTHEAST WINDS NEAR 15 KNOTS. SEAS OUTSIDE THE REEF 3 TO 5 FEET. OUTSIDE THE REEF 3 TO 5 FEET. SEAS INSIDE THE REEF 2 TO 3 SEAS INSIDE THE REEF 2 TO 3 FEET. NEARSHORE WATERS A MODERATE CHOP. ISOLATED FEET. NEARSHORE WATERS A MODERATE CHOP. ISOLATED SHOWERS. SHOWERS.

GRID VERIFICATION GRID VERIFICATION “THOUGHTS”…“THOUGHTS”…

•You must have relatively dense You must have relatively dense surface data network to support surface data network to support gridded verification field.gridded verification field.

•Perception that numerical models Perception that numerical models are the better performer are are the better performer are incorrect and not supported.incorrect and not supported.

•Need high resolution shallow Need high resolution shallow water wave guidance.water wave guidance.

Given the current state of model guidance and Given the current state of model guidance and surface data available for the EYW marine area –surface data available for the EYW marine area –

- It is entirely possible that our forecasters will have - It is entirely possible that our forecasters will have to DEGRADE our forecasts to match verification to DEGRADE our forecasts to match verification fields…fields…