FAA EDR Standards Project

RTCA Plenary Sub-Group 4 & 6

September 16, 2014

Presented by: Sal Catapano - Exelis

Outline

• Background / FAA EDR Standards Project Report • Current In Situ EDR Algorithms / Aircraft Equipped• FAA EDR Standards Project Overview• 1-Minute Mean In Situ EDR Report Performance and

Recommendations• 1-Minute Peak In Situ EDR Report Performance and

Recommendations• Variability Analysis Findings• Follow-on Recommendations• Open Question / Discussion Period

Background &

Current EDR Algorithms / Aircraft Equipped

Background

• In 2001, ICAO made EDR the turbulence metric standard• In 2011, ADS-B ARC recommended the FAA “establish

performance standards for EDR”• In 2012, RTCA SC-206, developed an Operational Services

and Environmental Definition (OSED) identifying the necessity for:

An international effort to develop performance standards for aircraft (in situ) EDR values, independent of computation approach

• In response, the FAA sponsored an EDR Standards Project from July, 2012 to September, 2014 that:

Provides the analysis, inputs, and recommendations required to adopt

in situ EDR performance standards

FAA EDR Standards Final Report

• Delivered to FAA August 29, 2014• Documents research, analysis,

and findings of project• Includes performance tolerance

recommendations for 1 minute mean and peak in situ EDR reports

• Report and appendices submitted to RTCA on September 10, 2014

6

Panasonic Longitudinal Wind-Based

ATR Accelerometer-Based Input: TAS, Altitude, Vertical Acceleration, Weight, Frequency Response

Users: American Airlines, others

Windowing: 5 sec running window

Average Calc: N/A

Peak Calc: Largest EDR in 30 seconds

NCAR Vertical Acceleration-Based Input: TAS, Altitude, Vertical Acceleration, Weight, Frequency Response, Mach, Flap Angle, Autopilot Status, QC Parameters

Users: United Airlines

Windowing: 10 sec window every 5 sec

Average Calc: Arithmetic mean over 1 min

Peak Calc: 95th percentile over 1 minute

Input: TAS, Roll Angle for QC, TAMDAR Icing for QC (if using TAMDAR Sensor)

Users: TAMDAR - Regional Airlines

Windowing: 9 sec window

Average Calc: 1, 3, 7min; 300, 1500ft

Peak Calc: Largest EDR in 1, 3, 7min; 300, 1500ft

NCAR Vertical Wind-BasedInput: TAS, Altitude, Inertial Vertical Velocity, Body Axis AoA, Pitch Rate, Pitch, Roll Angle, QC, Filter Parameters

Users: Delta and Southwest Airlines

Windowing: 10 sec running

Average Calc: Median over 1 min

Peak Calc: Largest EDR in 1 Minute

Current Operational In Situ EDR Algorithms

Aircraft Equipped

Airline Type Method Count

American and

others

B737-800, B757-200 B767-

300, A320, A321, A330-300 Vertical Acceleration 500+

Delta B737NG, B767 Vertical Wind 167

Southwest B737-700, B737NG Vertical Wind 156

UnitedB757 (EDR equipped B737

no longer in fleet)Vertical Acceleration

54

(reducing to 15 by Dec

31, 2015)

Regional Airlines

via TAMDAR

SAAB 340, ERJ-145, ERJ-

190, ERJ-195, Beech

1900C, Dash 8 (Q-100, Q-

300, Q-400)

Longitudinal Wind

(via TAS)256

Total: 1133

7

FAA EDR Standards Project Overview

Project Team and Key Stakeholders

Standards Research Process

Commercial Output

Simulator Output

Research Quality Data

Raw Vertical Winds

Input Winds Development

• Homogenous – exercise mean EDR

Maintains single EDR on average throughout wind dataset (e.g. 0.5 EDR)

• Non-Homogenous – exercise peak EDR Simulate “burst” of turbulence embedded in

background field of ambient turbulence

X =

Minute 1 Minute2 Minute 3

.5 EDR

Input Winds Datasets

X =

Case Title Case DescriptionInput Wind Datasets Specifications

Turbulence Intensity or

Modulation ShapeLength Scales

Random Phenomena(Homogenous)

Turbulence is a random process, therefore datasets were created to

replicate

5 von Karman

0.1, 0.2, 0.3, 0.5, 0.7 EDR 500 Meters

Varying Length Scale Nonconformance w/

Algorithm Assumptions(Homogenous)

Operational In situ EDR algorithms all assume a 500 meter length scale,

however in real-world conditions the length scale varies, therefore datasets were created with varying length scales

20 von Karman (including the 5 for random)

0.1 0.2, 0.3, 0.5, 0.7 EDR 200 Meters

0.1, 0.2, 0.3, 0.5, 0.7 EDR 500 Meters

0.1, 0.2, 0.3, 0.5, 0.7 EDR 750 Meters

0.1, 0.2, 0.3, 0.5, 0.7 EDR 1000 Meters

NoiseFloor

(Homogenous)

Aircraft sensors and avionics introduce noise to EDR calculations, so datasets

at low EDR severity levels were generated to determine impacts to EDR

calculations

15 von Karman

0.01, 0.02, 0.03, 0.04, 0.06 EDR 200 Meters

0.01, 0.02, 0.03, 0.04, 0.06 EDR 500 Meters

0.01, 0.02, 0.03, 0.04, 0.06 EDR 750 Meters

Nonconformance w/ Algorithm Assumptions(Non-Homogeneous)

In real-world conditions not all turbulence is homogenous, so

modulation was applied to von Karman datasets to simulate convective and

mountain wave turbulence

von Karman at 500 meters and 0.06 EDR background

Narrow/Tall Pulse Shape (SPIKE)A1 = 20

L_Sigma = 500M

Mid/Mid Pulse Shape (MID)A1 = 15

L_Sigma =1000M

Wide/Short Pulse Shape (FLAT)A1 = 5

L_Sigma =1500M

‘Expected’ EDR Value

• Homogeneous Wind Datasets – Mean EDR Single ‘expected’ EDR value corresponding to each

dataset (i.e., .5 EDR dataset has ‘expected’ EDR value of .5 EDR)

• Non-homogeneous Wind Dataset – Peak EDR Spectral Scaling Method developed by the project

used to calculated EDR ‘expected’ values based on individual implementation window length and modulation characteristics

.5 EDR

Expected Mean vs. Sample Mean

Statistical Sets

Sample Mean

Sample Mean

Sample Mean

1-Minute Mean EDR

• Implementations performed well compared with EDR ‘expected’ value

Consistency across turbulence severity levels and length scales, as well as across implementations

• Noise floor for the project’s simulation determined to be at or below 0.02 EDR

Below noise floor bias is high for all implementations Above noise floor bias quickly improves, as signal-to-noise ratio

increases

In Situ EDR Mean Report Findings

Real-world operational noise floors are dependant on avionics and sensor characteristics

.01 .02 .03 .04 .06 0.1 0.2 0.3 0.5 0.70%

5%

10%

15%

20%

25%

30%

35%32.2%

19.4%18.3% 17.6% 17.9% 18.7% 18.9% 19.5%

21.1%22.8%

14.3%

7.5% 7.2% 7.3% 7.3% 7.5% 7.4% 7.5% 8.3% 9.0%

19.1%

4.7%3.2% 2.4% 1.7% 2.1% 2.2% 2.3% 2.3% 2.3%

1-Minute Mean EDR Performance

99%-Band70%-BandBias

Expected EDR Value

P

erfo

rman

ce (

% o

f E

xpec

ted

ED

R

Val

ue)

Performance Category EDR Range Bias 70%-band 99%-band

Supplemental 0.01 - 0.02 +5% +10% +20%

Minimum >0.02 – 0.20 +5% +10 +20%

Minimum >0.20 – 0.70 +5% +10 +25%

Note: Statistics normalized to expected value

1-Minute Mean EDR Performance & Recommendations

.01 .02 .03 .04 .06 0.1 0.2 0.3 0.5 0.70%

5%

10%

15%

20%

25%

30%

35%32.2%

19.4%18.3% 17.6% 17.9% 18.7% 18.9% 19.5%

21.1%22.8%

14.3%

7.5% 7.2% 7.3% 7.3% 7.5% 7.4% 7.5% 8.3% 9.0%

19.1%

4.7%3.2% 2.4% 1.7% 2.1% 2.2% 2.3% 2.3% 2.3%

1-Minute Mean EDR Performance

99%-Band70%-BandBias

Expected EDR Value

P

erfo

rman

ce (

% o

f E

xpec

ted

ED

R

Val

ue)

1-Minute Mean EDR Performance Tolerance Threshold Example

Example: 0.1 Mean EDR Performance Tolerance Thresholds

Bias +5%Expected Value

0.1 EDRBias must be between

0.095 – 0.105 EDR

70%-band +10% Sample Mean At least 70% of results must be

Sample Mean + 0.01 EDR

99%-band +20% Sample Mean At least 99% of results must be

Sample Mean + 0.02 EDR

Note: Statistics normalized to expected value

.01 .02 .03 .04 .06 0.1 0.2 0.3 0.5 0.70%

10%

20%

30%

40%

50%

60%

70%

80%

32.2%

19.4% 18.3% 17.6% 17.9% 18.7% 18.9% 19.5% 21.1% 22.8%

75.0%

60.0%

45.0%

30.0%

20.0%

99%-Band

Wake 99%-Band

Expected EDR Value

P

erfo

rman

ce (

% o

f E

xpec

ted

E

DR

Val

ue)

Wake Vortex Decay Modeling Performance Objectives (99%-Band)

Note: Wake decay modeling community has performance objectives for null to light turbulence

1-Minute Peak EDR

5 Sec Window

10 Sec Window

Window Length (Meters)

ED

R E

xpec

ted

Val

ue

Spike

Mid

Flat

Standard Deviation Increases

.7

.5.6

Window Length Flat Mid Spike

5 Second Window

0.32 0.51 0.70

8 Second Window

0.28 0.42 0.56

10 Second Window

0.26 0.38 0.50

Bias IncreasesExpected Value

Decreases

1-Minute Peak EDR Expected Value

‘Representative’ Expected Value

Performance Recommendations for1-Minute Peak In Situ EDR Reports

Statistic Set Performance (FLT) Performance (MID) Performance (SPIKE)

1Bias +15.6% +18.3% +23.2%

270%-band +23.8% +26.9% +35.7%

299%-band +65.6% +69.2% +82.9%

Statistic SetRecommended Peak

Standard (FLT)Recommended Peak

Standard (MID)

Recommended Peak Standard

(SPIKE)

1Bias +20% +20% +25%

270%-band +25% +30% +40%

299%-band +70% +70% +85%

1Bias is normalized to the “representative” expected value2 70% and 99% bands are normalized to the “window length specific” expected value

.51Bias Increases

Bias Increases

1-Minute Peak EDR Example

Window Length (Meters)

ED

R E

xpec

ted

Val

ue

Spike

Mid

Flat

5 Sec Window

10 Sec Window

.38

.445.51

Example: MID Dataset Peak EDR Performance Tolerance Thresholds

Bias +20%‘Representative’ Expected Value

0.445 EDR

Bias must be “Representative’ Expected

Value + 0.089 EDR

70%-band +30%

9 Second Window Specific Expected Value

0.40 EDR

At least 70% of results must be Window Specific

Expected Value + 0.12 EDR

99%-band +70%

9 SecondWindow Specific Expected Value

0.40 EDR

At least 99% of results must be Window Specific

Expected Value + 0.28 EDR

‘Representative’ Expected Value

Today • Current user applications employ peak EDR data only for

strategic forecasting and planning Interested in receiving numerous peak EDR reports User objectives for peak EDR are subjective and require further

sensitivity analysis

Future• EDR used in tactical decision making (e.g., cross-linking

between aircraft) Improved inter-implementation consistency may be required across

EDR implementations Today’s peak EDR performance may not meet future use

performance requirements

Peak EDR Performance Objectives

In Situ EDR Peak Report Findings

• All implementations performed well compared with EDR ‘expected’ value

However, all existing in situ EDR implementations employ different window lengths, parameter settings, and calculation methodologies

Some of these component differences lead to inconsistent peak EDR values across implementations in non-homogeneous turbulence (i.e., convective, mountain wave)

• Project elected to perform variability analysis on a set of EDR algorithm components to discover sources of variability

Variability Analysis

Scatter Plot of Results Consistency Performance Curve

Algorithm Parameters included in Variability Analysis

Input Type Window Length Window Function

Window Overlap Lower Cutoff Wavenumber Upper Cutoff Wavenumber

Consistency Improvement Potential

Follow-on Recommendations

• Performance standard adoption Validate in situ recommendations Determine how compliance will be enforced

• Define operational requirements Pursue broad ConOps for EDR Perform application specific sensitivity analyses

• Continue variability analyses Research additional algorithm components Define parameter values for all components

• Pursue additional research into the science of EDR Analyze impact of distorting assumptions Define an approach to develop vertical EDR profiles

• Consider non-in situ EDR performance standards

Leverage momentum of Project

Team’s Success

Follow-on activities MUST have operational significance and benefit

Questions / Discussions

Name Phone Email

Michael Emanuel 609-485-4873 michael.emanuel@faa.gov

Joe Sherry 571-224-8926 joseph.sherry@exelisinc.com

Sal Catapano 301-867-2879 salvatore.catapano@exelisinc.com

Back-up Slides

Inertial Sub-Range

Work Element Relationship

EDR Standards Process

Algorithm Input Data Development

Modulation Pulse

Notional Depiction of Relationships Associated with Differing Window

Length

Window Length (Seconds)

1 2 3 4 5 6 7 8 9 10

SPIKE Dataset Expected

Values1.19 1.05 0.90 0.79 0.70 0.64 0.60 0.56 0.53 0.50

MID Dataset

Expected Values0.65 0.63 0.59 0.55 0.51 0.48 0.45 0.42 0.40 0.38

FLT Dataset

Expected Values- - - - 0.32 - - 0.28 - 0.26

Spike, Mid, and Flat Expected Values

ISE Comparison of Project Generated Non-homogenous Wind Data Sets

99%-band

+49.5%-49.5%

Example: 0.1 Mean EDRBias

Sample Mean

70%-band

-35% +35%

Sample Mean

Expected Value = 0.1 EDRResults Center = 0.1021 EDRPerformance Stnd = 0.105 EDR

Sample Mean = 0.1 EDRRight Tolerance Band = 0.1075 EDRLeft Tolerance Band = 0.0925 EDRPerformance Stnd = At least 70% of results must be between 0.09 EDR and 0.11 EDR

Sample Mean = 0.1 EDRRight Tolerance Band = 0.1187 EDRLeft Tolerance Band = 0.0813 EDRPerformance Stnd = At least 99% of results must be between 0.08 EDR and 0.12 EDR