CARE/ASAS Validation Framework Guidelines & Case Studies Mark Watson NATS.

CARE/ASAS Validation Framework Guidelines & Case

Studies

Mark Watson

NATS

Contents

WP4 & MAEVA VGH The Validation Framework The Case Studies

Work Package 4

Align previous work packages to MAEVA VGH

Write guidelines and include Activity 3 case studies

Guideline Report

Update EMERALD RTD Plan

(presented later)

Master ATM European Validation Plan (MAEVA)

European Commission funded 5th framework project

Promote a common framework for validation of 5th FP ATM projects

Proposes top-down approach rather than enabler-targeted bottom-up approach

Describes lifecycle of ATM steps from concept to operational implementation

Wider intended adoption throughout Europe

WP1: Initial Validation Framework

WP2: System Performance Metrics

WP3: Human Performance Metrics

Compare To MAEVA VGH

Include Activity 3 Case Studies

CARE/ASAS

Validation

Framework

Guidelines

CARE/ASAS Validation Framework

… five steps to enlightenment!Step 1: Identification Of Validation Aims, Objectives And

Hypotheses

Step 2: Validation Design - Plan & Prepare The Validation Exercise

Step 3: Conduct Of Validation Exercise Runs

Step 4: Analysis of the Results

Step 5: Develop and Report Conclusions & Recommendations

but with 16 actions...

Action 1. Understanding the ATM problem

Action 2. Selection of the ASAS application

Action 3. Identification of stakeholders

Action 4. Identification of validation aims

Action 5. Definition of the high level objectives (HLO)

Action 6. Definition of the low level objectives (LLO)

Action 7. Establishing validation platform requirements and selection of the validation technique

Action 8. Selection of system performance and human performance metrics and hypotheses

Action 9. Definition of the high level experimental design

Action 10. Operational and statistical significance

Step 1: IDENTIFICATION OF VALIDATION AIMS, OBJECTIVES AND HYPOTHESES

Step 2: Validation Design - Plan & Prepare The Validation ExerciseAction 11: Selection Of The Validation Platform/Tool

Action 12: Scenario Definition

Action 13: Production Of Detailed Experiment Design


Action 14: Execution


Action 15: Results Analysis

Step 5: Develop and Report Conclusions Recommendations

Action 16: Conclusion & Recommendations

MAEVA VGH vs. CARE/ASAS VF

Case Study Examples Time based sequencing in approach

Airborne Separation category sequencing & merging operations from Top of Descent

until Final Approach Fix time is the separation criteria limited separation responsibility delegated to pilot Airborne separation minima may be lower than ATC

separation minima Mixed levels of ADS-B equipage Example airspace Madrid

Case Study Examples Airborne Self Separation in Segregated En-

route Airspace Airborne self separation category Free flight segregated airspace Aircraft fly preferred route between entry and exit Flight crews responsible for self-separation from all

aircraft example airspace Mediterranean

Order Of The VF Presentations

Scenario Template & Database - Juan Alberto Herreria, ISDEFE

System Performance Metrics - Mike Sharples, QinetiQ

Case Study (Time based sequencing) Actions part 1 - Mark Watson, NATS

Discussion Forum

Lunch

Human Performance Metrics & Experimental Design - Brian Hilburn, NLR

Case Study Actions part 2 - Mark Watson, NATS

Coffee Break

Case Study of the Validation Framework

Time Based Sequencing In Approach

The Validation Framework

Step 1: Identification Of Validation Aims, Objectives And Hypotheses (10 actions)

Step 2: Validation Design - Plan & Prepare The Validation Exercise (2 actions)



Step 5: Develop and Report Conclusions & Recommendations





Action 5. Definition of the high level objectives (HLO)

Action 6. Definition of the low level objectives (LLO)

Action 7. Establishing validation platform requirements and selection of

the validation technique

Action 8. Selection of system performance and human performance

metrics and hypotheses


Action 11: Selection Of The Validation Platform/Tool


Action 13: Production Of Detailed Experiment Design

Step 2: VALIDATION DESIGN - PLAN & PREPARE THE VALIDATION EXERCISE

Action 9. Definition of the high level experimental design

Action 10. Operational and statistical significance

Constrained capacity on approach within TMA airspace.




Increase capacity on approach by aircraft flying the minimum aircraft separation.

Selected ASAS application is Time Based Sequencing In Approach.

Separation responsibility should be delegated to the pilot to decrease controller

workload.

Maintain present level of safety.




Airline operator

Pilot

ATSP

Airport Operator

ATCO

Assess the application for its effect on capacity in TMA on approach.

Assess the impact on controller and pilot workload and TMA capacity.


Action 5: Definition of the High-Level Objectives (HLO)

Action 6: Identification of Low-Level Objectives (LLO)

Safety

Capacity

Economics

Airspace throughput

Controller & Pilot Workload

Voice Communications

Conflicts

Traffic densities


Action 7: Establishing Validation Platform Requirements and selection of validation technique

Scope of ATM system

Fidelity/Resolution

Geography

Time-based Requirements

Action 8: Identification of System Performance and Human Performance Metrics & Hypotheses


SYSTEM:

Planned versus Actual Flight Profiles

Sector Entry/Exit

Conflicts

Workload per controller

Number of Time Based Clearances


Action 8: Identification of System Performance and Human Performance Metrics & Hypotheses (4 of 4) - IF IT WAS A REAL TIME SIMULATION!!

Pilot metric to assess peak workload

Various performance based and physiological based objective measures are

available

Physiological measures of EEG potentials (brainwaves) dismissed as too

intrusive. Pupil diameter is therefore chosen

Action 9: Definition of High Level Design


Action 10: Operational and Statistical Significance

Initial 2005 baseline sample with no ASAS application to prove representativeness.

Three measured runs - 2005, 2010, 2015.

Three levels of separation delegation for each run.

2005 measured run with minimum separation delegation

decreased controller and communications workload by 5%

All other measured runs must improve on this.

95% statistical significance required

Action 11: Selection Of Platform/Tool

Step 2: Plan and Prepare the Validation Exercise


MAEVA VGH describes available European platforms and their capability. Suitability of TAAM for addressing HLO of Economics and Capacity through fast time

simulations.

Adaptable to the airspace of this validation exercise.

Safety can be addressed through analysis of results.

Use scenario template as aide-memoir

Helps develop a detailed scenario definition document

Action 13: Production Of Detailed Experimental Design

Step 2: Plan and Prepare the Validation Exercise

Detailed planning of the exercise runs. Preparation of the Measurement and Analysis Specification.

and finally… Conclusions CARE/ASAS VALIDATION FRAMEWORK closely

aligns with MAEVA VGH (some interim steps differ in order or are tailored)

Step by Step route map for the creation of validation exercises for any ASAS application

An iterative process of design

sufficient detail for organisations with limited ASAS or validation experience

Will encourage uniformity of ASAS validations

Forum Discussion

CARE/ASAS Validation Framework Guidelines & Case Studies Mark Watson NATS.

Documents

Transcript of CARE/ASAS Validation Framework Guidelines & Case Studies Mark Watson NATS.