Aviation Verification and Aviation Verification and ConvectionConvection

Chris LeonardiChris LeonardiWFO RLXWFO RLX

August 31, 2005August 31, 2005

GPRA GoalsGPRA Goals

Most GPRA goals at the WFO level are Most GPRA goals at the WFO level are associated with significant/severe weather.associated with significant/severe weather.

As far as routine WFO products are As far as routine WFO products are concerned, there is a goal only for IFR concerned, there is a goal only for IFR aviation forecasts, and for day 1 aviation forecasts, and for day 1 precipitation threat scores.precipitation threat scores.– NONE for temperature forecasts, you WRKVER NONE for temperature forecasts, you WRKVER

worshippers!worshippers!

Aviation GPRA goalAviation GPRA goal

The FY2005 national goal for operational The FY2005 national goal for operational impact forecasts (more on this later) for IFR impact forecasts (more on this later) for IFR conditions is:conditions is:– 0.46 POD0.46 POD– 0.68 FAR0.68 FAR

Our stats for FY 2005 through 8/27:Our stats for FY 2005 through 8/27:– 0.39 POD0.39 POD– 0.68 FAR 0.68 FAR …not horrible, but……not horrible, but…

Stats breakdownStats breakdown

IFR stats from 6/1/05 to 8/27/05:IFR stats from 6/1/05 to 8/27/05:– 0.29 POD0.29 POD– 0.76 FAR0.76 FAR

Why are we doing worse?Why are we doing worse?– Fog?Fog?– Convection?Convection?– Confusion over who gets the TAFs (H/J shift)?Confusion over who gets the TAFs (H/J shift)?– Natural stat drop when changing regimes (ie. starting up the Natural stat drop when changing regimes (ie. starting up the

enhanced short-term program)?enhanced short-term program)?– The high ERA of the Red Sox bullpen?The high ERA of the Red Sox bullpen?

Most of these reasons are probably at least partially Most of these reasons are probably at least partially responsible...but one problem appears to be easily fixable.responsible...but one problem appears to be easily fixable.

Stats breakdownStats breakdown

For which scheduled TAF (00Z, 06Z, 12Z, 18Z) did For which scheduled TAF (00Z, 06Z, 12Z, 18Z) did we have the worst IFR scores over the past few we have the worst IFR scores over the past few months (through 8/20)?months (through 8/20)?

CycleCycle PODPOD FARFAR CSICSI

00Z00Z .18.18 .78.78 .11.11

06Z06Z .34.34 .74.74 .17.17

12Z12Z .20.20 .69.69 .14.14

18Z18Z .30.30 .92.92 .07.07

Stats breakdownStats breakdown

What kind of conclusions can we draw?What kind of conclusions can we draw?– Doing the best on 06Z TAFs – when we expect Doing the best on 06Z TAFs – when we expect

the most fogthe most fog– Doing the worst on 18Z TAFs (with an alarming Doing the worst on 18Z TAFs (with an alarming

FAR!) – when we expect the most convectionFAR!) – when we expect the most convection– 12Z TAFs are second best (lingering fog) and 12Z TAFs are second best (lingering fog) and

00Z TAFs are second worst (lingering 00Z TAFs are second worst (lingering convective chances plus incipient fog)convective chances plus incipient fog)

Convection has been the bigger problem!Convection has been the bigger problem!

StatsStats

So how are these verification stats determined So how are these verification stats determined anyway?anyway?– Only scheduled TAFs (00Z, 06Z, 12Z, and 18Z) are Only scheduled TAFs (00Z, 06Z, 12Z, and 18Z) are

tracked. This is to discourage ob-chasing.tracked. This is to discourage ob-chasing.– The “operational impact” category forecast is evaluated The “operational impact” category forecast is evaluated

against the current ob every 5 minutes. A contingency against the current ob every 5 minutes. A contingency table is created from these evaluations and IFR table is created from these evaluations and IFR POD/FAR are calculated from them…along with a TON POD/FAR are calculated from them…along with a TON of other stats.of other stats.

……so what is “operational so what is “operational impact???”impact???”

““Operational Impact”Operational Impact”• When no TEMPO or PROB forecasts are in effect, When no TEMPO or PROB forecasts are in effect, the operational impact (OI) forecast is the same as the operational impact (OI) forecast is the same as the prevailing forecast.the prevailing forecast.

•When a TEMPO or PROB forecast is in effect, the When a TEMPO or PROB forecast is in effect, the OI forecast is defined as the forecast in effect that OI forecast is defined as the forecast in effect that is most likely to impact operations. is most likely to impact operations.

OI and TEMPOsOI and TEMPOs If a TEMPO forecast is in effect, a 2-step process If a TEMPO forecast is in effect, a 2-step process

is used to find the OI forecast: is used to find the OI forecast: – Step 1 - Variability test. The program looks for at least Step 1 - Variability test. The program looks for at least

two changes in the observations +/- 90 minutes of the two changes in the observations +/- 90 minutes of the end of the 5-minute interval being evaluated. end of the 5-minute interval being evaluated.

– Step 2 - If the variability test passed (two or more Step 2 - If the variability test passed (two or more changes +/- 90 minutes), then the OI forecast is the changes +/- 90 minutes), then the OI forecast is the forecast (prevailing or TEMPO) (a) closest categorically forecast (prevailing or TEMPO) (a) closest categorically to the observation for ceiling, visibility, and flight to the observation for ceiling, visibility, and flight category, or (b) closest in knots for wind speed and wind category, or (b) closest in knots for wind speed and wind gusts.gusts.

OI and TEMPOs cont.OI and TEMPOs cont.

If the variability test failed, the OI forecast is If the variability test failed, the OI forecast is defined as the most pessimistic of the two defined as the most pessimistic of the two forecasts (e.g., lowest ceiling category, forecasts (e.g., lowest ceiling category, highest wind speed)highest wind speed)

This is what hurts our stats the most.

OI exampleOI example

Let’s look at a couple of scenarios from a recent day Let’s look at a couple of scenarios from a recent day at RLX.at RLX.- Both CRW/PKB TAFs have prevailing VFR with - Both CRW/PKB TAFs have prevailing VFR with TEMPO IFR conditions in TSRA for the first 6 TEMPO IFR conditions in TSRA for the first 6 hours.hours.- PKB receives a TSRA with IFR conditions from - PKB receives a TSRA with IFR conditions from 1814Z to 1830Z…and is VFR during the remaining 1814Z to 1830Z…and is VFR during the remaining portion of the 6-hour period.portion of the 6-hour period.- CRW gets no TSRA and remains VFR through - CRW gets no TSRA and remains VFR through the period (with plenty of CB/TCU around!)the period (with plenty of CB/TCU around!)How would these TAFs be verified at 1900Z?How would these TAFs be verified at 1900Z?

OI example for PKBOI example for PKB

PKB receives a TSRA with IFR conditions PKB receives a TSRA with IFR conditions from 1814Z to 1830Z…and is VFR during from 1814Z to 1830Z…and is VFR during the remaining portion of the 6-hour periodthe remaining portion of the 6-hour period

At 1930Z…visibility is 10SM (VFR). Since there has At 1930Z…visibility is 10SM (VFR). Since there has been variability (2 category changes) in last 90 been variability (2 category changes) in last 90 minutes, the forecast with the least amount of minutes, the forecast with the least amount of error is used.error is used.

P6SM. P6SM. A VFR hit!A VFR hit! Note, though, that this does Note, though, that this does NOT help our IFR stats.NOT help our IFR stats.

OI example for PKBOI example for PKB

So when does the IFR forecast hit? At So when does the IFR forecast hit? At 1815Z, 1820Z, and 1825Z…three 5-minute 1815Z, 1820Z, and 1825Z…three 5-minute intervals.intervals.

From 1945Z on, there is no longer any From 1945Z on, there is no longer any variability. Therefore, the most pessimistic variability. Therefore, the most pessimistic forecast is used (2SM). EVERY 5-minute forecast is used (2SM). EVERY 5-minute interval from 1945Z to 2400Z is an IFR interval from 1945Z to 2400Z is an IFR miss… miss… 51 of them.51 of them.

OI example for CRWOI example for CRW

CRW gets no TSRA and remains VFR CRW gets no TSRA and remains VFR through the period.through the period.

At 1930Z…vis is P6SM. The variability test fails, At 1930Z…vis is P6SM. The variability test fails, since there is no change +/- 90 minutes from this since there is no change +/- 90 minutes from this time. Therefore, the most pessimistic condition is time. Therefore, the most pessimistic condition is used.used.

2SM. 2SM. An IFR miss! In fact, during this period there An IFR miss! In fact, during this period there were *72* IFR misses.were *72* IFR misses.

OI example statsOI example stats

So, how do the combined IFR stats look for So, how do the combined IFR stats look for these two TAFs?these two TAFs?– POD = 1.0POD = 1.0 (since all 3 IFR occurrences were (since all 3 IFR occurrences were

covered by an IFR forecast)covered by an IFR forecast)– FAR = 0.98FAR = 0.98 (123 IFR misses, 3 IFR hits) (123 IFR misses, 3 IFR hits)– Also recorded 18 VFR hitsAlso recorded 18 VFR hits– CSI = 0.02CSI = 0.02

What to do??What to do??

So, it would seem statistically that it would So, it would seem statistically that it would behoove us to forecast IFR rarely…behoove us to forecast IFR rarely…if ever…if ever…for convectionfor convection

BUT…we want to provide good service to BUT…we want to provide good service to our customers and present the realm of our customers and present the realm of possibilities.possibilities.

We need to find a way to balance service and We need to find a way to balance service and verification as much as possible.verification as much as possible.

Some suggestions…Some suggestions…

For the first 3 hours of a TAF:For the first 3 hours of a TAF:– If you feel/observe that IFR for convection is If you feel/observe that IFR for convection is

justified, pick the *most likely* hour and use a justified, pick the *most likely* hour and use a TEMPO during that hour. Otherwise…use CB or TEMPO during that hour. Otherwise…use CB or MVFR conditions.MVFR conditions. This strategy will inform pilots of the risk, yet This strategy will inform pilots of the risk, yet

somewhat minimize the impact to our stats if there is somewhat minimize the impact to our stats if there is a miss. Possible drawback…implied precision.a miss. Possible drawback…implied precision.

Yes, you may pick the wrong hour…but that is what Yes, you may pick the wrong hour…but that is what amendments are for, AND at least the presence of amendments are for, AND at least the presence of such convection in the TAF will inform the users.such convection in the TAF will inform the users.

Some suggestions…Some suggestions…

Between 3-6 hours of a TAF…Between 3-6 hours of a TAF…– Use of IFR TEMPO groups for convection is Use of IFR TEMPO groups for convection is

STRONGLY discouraged…unless you have a STRONGLY discouraged…unless you have a clear justification (e.g. squall line). Even then, clear justification (e.g. squall line). Even then, be *brief*. Monitor trends, and amend if needed.be *brief*. Monitor trends, and amend if needed.

– Preferred method…use CB cloud groups, or a Preferred method…use CB cloud groups, or a TEMPO MVFR thunder group (for a situation TEMPO MVFR thunder group (for a situation such as likely pops for popup convection)such as likely pops for popup convection)

Some suggestions…Some suggestions…

Beyond 6 hours into a TAF…Beyond 6 hours into a TAF…– Use of IFR TEMPO groups for convection is Use of IFR TEMPO groups for convection is

PROHIBITED (even though doing so would not PROHIBITED (even though doing so would not affect our stats).affect our stats).

– Use either a CB cloud group, or Use either a CB cloud group, or perhapsperhaps an an MVFR TEMPO thunder group in a rare instance MVFR TEMPO thunder group in a rare instance (well-timed strong cold front, for example).(well-timed strong cold front, for example).

Bottom line…Bottom line…

Be deterministic!Be deterministic!– Focus on what WILL happen, not what COULD Focus on what WILL happen, not what COULD

happen.happen.– Use TEMPOs sparingly…keep them short and Use TEMPOs sparingly…keep them short and

specific.specific.– Be ESPECIALLY careful with convection. CB is Be ESPECIALLY careful with convection. CB is

your friend!your friend! Check the stats occasionally, both yours Check the stats occasionally, both yours

and the office’s (instructions to come)and the office’s (instructions to come)

Aviation gridsAviation grids

GridsGrids– Remember to do them!Remember to do them!– Tool usage (more than just Tool usage (more than just

PredHgtFmModelRH)PredHgtFmModelRH)– Starting up visibility grids next week (prevailing Starting up visibility grids next week (prevailing

vis, same time range)vis, same time range) Goal: become comfortable enough with Goal: become comfortable enough with

grids/AVNFPS formatting to begin TAF grids/AVNFPS formatting to begin TAF generation from grids by early Octobergeneration from grids by early October