1

NEXTOR

Monitoring and Modeling NAS Monitoring and Modeling NAS Performance at the Daily Performance at the Daily

LevelLevelMark Hansen

Performance Metrics TIM

May 2002

2

NEXTORNormalizationNormalization

Translate before/after performance comparisons to with/without comparisons

Enable effects “Beyond FAA Control” to be removed while examining metrics of direct economic relevance

Enable comparison of observed and predicted (by models) impacts of technology deployments

3

NEXTORNEXTOR NormalizationNEXTOR Normalization

Effects of FFP1 on Terminal Area and En Route Performance

Focus on Delays and Time-in-System Metrics

Presented HereEffect of TTMA at LAXEffect of URET

4

NEXTORModeling CycleModeling Cycle

Models

R&D and Deployment Decisions Introduction

of New Systems

Benefits and Impacts of New Systems

Normalization

5

NEXTORConceptual FrameworkConceptual Framework

Demand

Weather

Conditionsat otherAirports

Time at Origin

AirborneTime

Taxi-In Time

Total Flight Time

System Performance

•En Route

•Terminal Area

•ATM

•Aircraft

6

NEXTORDaily Flight Time Index (DFTI)Daily Flight Time Index (DFTI)

Daily weighted average of flight times to a given airport from a set of origins

Flight time=Actual Arrival Time - Scheduled Departure TimeScheduled Flight Time+Departure Delay+Flight Time DelayTime-at-Origin+Airborne Time+Taxi-In Time

Origins have at least one completed flight in each day of sample

Weights reflect origin share of flights to study airport over study period

7

NEXTOR

Figure 2.1 Daily DFTI Values for LAX

100

120

140

160

180

200

220

240

1/1/

00

2/1/

00

3/1/

00

4/1/

00

5/1/

00

6/1/

00

7/1/

00

8/1/

00

9/1/

00

10/1

/00

11/1

/00

12/1

/00

1/1/

01

2/1/

01

3/1/

01

4/1/

01

5/1/

01

6/1/

01

Date

DFT

I (in

min

ute

s)

8

NEXTORFigure 2.2 Daily DFTI Moving Average, Three Components

0

20

40

60

80

100

120

140

160

180

200

Jan

-00

Fe

b-0

0

Ma

r-0

0

Ap

r-0

0

Ma

y-0

0

Jun

-00

Jul-

00

Au

g-0

0

Se

p-0

0

Oct

-00

No

v-0

0

De

c-0

0

Jan

-01

Fe

b-0

1

Ma

r-0

1

Ap

r-0

1

Ma

y-0

1

Jun

-01

Date

Av

era

ge

Tim

e P

er

Flig

ht

(Min

ute

s)

Taxi_inAir_BorneT_Origin

9

NEXTORWeather NormalizationWeather Normalization

Based on CODAS hourly weather observations for LAX

Factor analysis of weather dataCreate small number of factors that capture

variation in large number of variablesFactors are linear combinations of original

variablesFactors correspond to principal axes of N-

dimensional data elipse

10

NEXTORFactor Analysis with Two Factor Analysis with Two

VariablesVariables

X1

X2

First Factor

11

NEXTOR

Table 5. LAX Weather Factor Interpretations, Rotated Factors

Factor Interpretation 1 High temperature throughout the day

2 High cloud ceiling and VFR conditions in the afternoon and evening

3 High cloud ceiling and VFR conditions in the morning

4 High visibility throughout the day

5 High wind speed in the afternoon and evening

6 Medium cloud ceiling in the evening

7 Medium cloud ceiling in the morning hours

12

NEXTORDemand NormalizationDemand Normalization

Deterministic Queuing Analysis Arrival Curve from ASPM OAG Departure Curve Based on Either

Called AARs (Demand and Weather Normalization)

Hypothetical Rates

13

NEXTORDeterministic Queuing Deterministic Queuing

DiagramDiagram

14

NEXTOR

Figure 2.6 Hypothetical Queuing Diagram at for LAX Arrivals on June 8, 2001

0

200

400

600

800

1000

1200

0:00 2:24 4:48 7:12 9:36 12:00 14:24 16:48 19:12 21:36 0:00

Time of Day

Cu

mu

lati

ve

Nu

mb

er

Completed, Capacity=60

Completed, Capacity=80

Completed, Capacity=40

Scheduled

15

NEXTORTrends in HDD Parameters for the Study Period (7-Day Moving Average)

0

0.5

1

1.5

2

2.5

3

3.5

4

1/7/00 4/16/00 7/25/00 11/2/00 2/10/01 5/21/01

Date

HD

D R

ela

tive

to

Fir

st W

ee

k o

f 20

00

HDD50

HDD70

HDD80

OAG_Arrivals

16

NEXTORNormalization for Conditions Normalization for Conditions

at other Airportsat other Airports Consider airports included in DFTI

average For each compute daily average

departure delay for flights not bound to LAX region

Average airport departure delays using DFTI weights

17

NEXTOR

Figure 2.4 Origin Airport Congestion at LAX

0

5

10

15

20

25

30

35

40

45

1/1/

00

2/1/

00

3/1/

00

4/1/

00

5/1/

00

6/1/

00

7/1/

00

8/1/

00

9/1/

00

10/1

/00

11/1

/00

12/1

/00

1/1/

01

2/1/

01

3/1/

01

4/1/

01

5/1/

01

6/1/

01

Date

Ori

gin

Air

port

Del

ay (

min

)

18

NEXTORPerformance ModelsPerformance Models

ttttt ODELDMDWXfY ),,(

Where:Yt is DFTI or DFTI component for day t;WXt is vector of weather factors for day t;DMDt is vector of demand factors for day t;ODELt is average origin departure delay for day t;t is stochastic error term.

19

NEXTORFunctional Forms ConsideredFunctional Forms Considered

ParametricLinear (with 3, 6, 9, and 12 weather

factors)Quadratic response surfaceNon-linear

Non-parametric9 clusters based on 3 weather factors12 clusters based on 9 weather factors

20

NEXTORLinear Model Estimation Results Linear Model Estimation Results

(Pre-TTMA)(Pre-TTMA)

Variable Description Estimate T - statistic P - value

INTERCEPT Intercept 138.055 567.065 0.0001 OAC Origin airport congestion 1.128 44.351 0.0001 WX1 Warm daily temperatures -1.357 -12.101 0.0001 WX2 VFR ops, no low cloud ceiling in the morning -0.988 -7.116 0.0001 WX3 VFR ops, no low cloud ceiling in the afternoon -1.123 -7.583 0.0001 WX4 High visibility throughout day -0.449 -3.575 0.0004 WX5 Medium cloud ceiling throughout day 1.440 10.555 0.0001 WX6 High winds throughout the day 0.512 4.531 0.0001 WX7 High cloud ceiling throughout day 0.911 4.172 0.0001 WX8 Precipitation in late morning and afternoon 1.871 8.324 0.0001 WX9 Precipitation in early morning -0.379 -2.614 0.0091 DMD1 Peak demand 0.075 0.725 0.4685 DMD2 Base demand 0.440 4.574 0.0001

ADJUSTED R2 0.743

21

NEXTORPredicted vs Actual ValuesPredicted vs Actual Values

130.00

140.00

150.00

160.00

170.00

180.00

190.00

200.00

210.00

130.00 140.00 150.00 160.00 170.00 180.00 190.00 200.00

Observed Values (min)

Pre

dic

ted

Val

ues

(min

)

22

NEXTOROutliersOutliers

Used TMU logs to investigate days for which predictions have large errors

Reasons for higher than predicted DFTIEast flowRadar outagesAir Force OneOver-stringent ground delay program

No clear explanations for lower-than-average DFTI: “No news is good news”

23

NEXTOR Variable Parameter Estimates DFTI At Origin Airborne Taxi-in intercept 139.29 15.23 115.9 8.11 pFAST -1.99 -1.71 -0.19 -0.07 OAC 1.39 1.29 0.03 0.06 Peak Demand -0.35 -0.1 -0.24 -0.01 Base Demand 0.97 0.74 0.04 0.19 Weather Factor1 -3.37 -0.89 -2.63 0.14 Weather Factor2 -2.66 -1.8 -0.75 -0.12

Weather Factor3 -1.88 -1.36 -0.48 -0.04 Weather Factor4 0.19 -0.24 0.49 -0.07 Weather Factor5 1.48 0.73 0.79 -0.03 Weather Factor6 0.46 0.12 0.31 0.02 Weather Factor7 0.64 0.27 0.29 0.81 Adjusted R-Square 0.79 0.83 0.55 0.39 Significant at 5% level Significant at 10% level

DFTI Model ResultsDFTI Model Results

24

NEXTOREffect of TTMA at LAXEffect of TTMA at LAX

Used same methodology but included TTMA (PFAST) dummy variable for Post 2/9/01

Used AARs in computing deterministic delay variable

Considered both DFTI and Average Delay as Dependent Variables

25

NEXTOR

Model Model1 2

Variable Estimate EstimateIntercept 4.03 5.49pfast -1.74 -3.76oac 0.58 0.54ahdd 0.59 0.22pct EDCT 0.18wx_fact1 -0.79 -1.50wx_fact2 -1.30 -1.15wx_fact3 -0.98 -0.27wx_fact4 0.01 0.15wx_fact5 0.87 0.72wx_fact6 0.25 -0.02wx_fact7 0.24 -0.10

Adj R2 0.77 0.87

Delay Model Results Delay Model Results

26

NEXTORURET ImpactURET Impact

Performed Flight Level Analysis Observed Airborne Time for ASQP Flights for

Corresponding Months Before and After URET (about 800,000 per analysis)

Effects EstimatedBefore vs After URETUse URET Sectors vs Don’t Use Use URET Sectors After URET

Control for Distance, Direction, and Originating Airport and Destination Airport

27

NEXTORURET Model ResultsURET Model Results

Variable Estimate

Intercept 10.14distance 0-200 0.16distance 200-500 0.14distance 500-1000 0.14distance 1000+ Miles 0.13change in latitude 5.00change in longitude 39.77after URET 0.42Use URET sector 0.45Use URET sector after URET -0.24

Adj R2 0.98

28

NEXTORConclusionsConclusions

A Priori Modeling without Ex Post Analysis is One Hand Clapping

Normalization Required to Compare Observed with Predicted Outcomes and Account for Factors Beyond FAA Control

We can Do This