Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr....

Elizabeth Scottie-Beth Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr. Ute Fischer Sponsored by the FAA, Tom McCloy as Technical Monitor DEVELOPING A TRAINING PROGRAM FOR THE TRAFFIC ALERT AND COLLISION AVOIDANCE SYSTEM IN CONTEXT

Overview +Introduction to TCAS and Background Information +Design of Training Program +Evaluation of the Training Program +Results +Conclusions 2 Develop a training program intended to improve pilots understanding of TCAS use for collision avoidance in a range of traffic situations

Traffic alert and Collision Avoidance System (TCAS) 3 +Pilot always shown the Traffic Situation Display (TSD) +TCAS delivers a two stage advisory and vertical avoidance maneuver Traffic Advisory (TA) - Traffic Traffic Resolution Advisory (RA) - Climb Climb +Pilot is to follow an RA, even if it conflicts with ATC instructions, unless the pilot believes that safe flight would be jeopardized Federal Aviation Administration (2000). Introduction to TCAS II Version 7. Washington, D.C.

Collision Avoidance in a Broader Context Pilots dont only interact with TCAS, they also receive information from ATC and the environment 4 Pilots Response Advised Maneuvers for Collision Avoidance Notification and Awareness Awareness of other aircraft via TSD TCAS Traffic Advisory TCAS Resolution Advisory ATC Traffic Callout Visual acquisition of other aircraft ATC Maneuver Compliance to TCAS Compliance to ATC Personal Assessment and Maneuver Decision Party-line Information

Current TCAS Training Requirements: Ground Training +Classroom or computer based +>60 training requirements +Measures performance through quizzes and/or activities 5 To comply to the RA, you should Pull the stick back Push the stick forward Federal Aviation Administration (2001). Advisory Circular No. 120-55B: Air Carrier Operational Approval and Use of TCAS II. Washington, D.C.

Current TCAS Training Requirements: Flight Training 6

Observed TCAS Use and Operation o The TCAS Operational Performance Assessment (TOPA) monitored the terminal area of 8 major airports and examined pilot compliance to climb and descend advisories 7 41% to Climb RAs 59% to Descend RAs TOPA observed compliance rates of Misunderstanding/ Confusion Aggressiveness Non-Compliance Olson, W. and J. Olszta (2010). TCAS Operational Performance Assessment in the U.S. National Airspace. IEEE/AIAA Digital Avionics Systems Conference. Olszta, J., & Olson, W. (2011). Characterization and Analysis of Traffic Alert and Collision Avoidance Resolution Advisories Resulting for 500' and 1,000' Vertical Separation. Paper presented at the Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011), Berlin, Germany.

Observed TCAS Use and Operation 8 [My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a person should fly 'away' from the red VSI indication, if instructed via RA. ACN:785761, 2008 [My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a person should fly 'away' from the red VSI indication, if instructed via RA. ACN:785761, 2008 Pilot reports flying into the red region on the VSI Non-Compliance Aggressiveness Misunderstanding/ Confusion NASA. (2009). Aviation Safety Reporting System. Retrieved August 1, 2010: http://asrs.arc.nasa.gov/ Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the initial guidance was so clearly wrong ACN: 854982, 2009 Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the initial guidance was so clearly wrong ACN: 854982, 2009 Pilot disagrees with advised descend RA

Observed TCAS Use and Operation 9 Non-Compliance Misunderstanding/ Confusion Aggressiveness Far Eastern Air B757 Response to Descend RA (TCAS advised a descent rate of 1500 FPM) Image copied from Lacagnina (2008). Easy Does It. Aero Safety World : http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1

Research Statement and Objectives Develop a training program intended to improve pilots understanding of TCAS use for collision avoidance in a range of traffic situations 10 (1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment (2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories (1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment (2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories

11 Demonstration Based Training Event Based Training Approach to Training Design Complement of Two Methods

Demonstration Based Training (DBT) +Computer-Based Training ~25 minutes Provides conceptual understanding of TCAS Outlines rules for compliance +6 segments Introduction to TCAS Traffic Situation Display TCAS Advising Logic Traffic Advisories Resolution Advisories Example Timeline of RA Evolution +Mid-Training Quizzes 12

DBT: Demonstrations of RAs 13

Event Based Training (EBT) +Presents traffic events that create the requirement to act +Builds context and complexity into each scenario as the flight progresses +Uses a more realistic training environment Fowlkes, J., Dwyer, D., Oser, R., & Salas, E. (1998). Event-Based Approach to Training (EBAT). The International Jounal of Aviation Psychology, 8(3), 209-221. 14

Air Traffic Transcripts Charts & Checklists Audio Communications (Aviation Intercom) TCAS Alerts Simulation Architecture Eyetracker PartyLine First Officer Captain Experimenter/Instructor Coded Log of Flights B747-400 Simulator (RFS) TSD PFD ND PFD ND Touch screen SideStick ATC->TSD VGA ATC Air Traffic Simulator (TGF) TCAS Logic 15 Simulator Study in Integrated Flightdeck ATC Environment Pritchett, A., Fleming, E., Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012). Pilot Interaction with TCAS and Air Traffic Control. Paper presented at the 2nd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS), London.

Structure of Events: Descend RA Example 16 Training Objective Context of EventPerformance Measures Feedback Session Accurate interpretation of and response to TCAS Descend RA. Instrument Meteorological Conditions (daytime, clouds, no winds) ATC provides no traffic information Conflict caused by IFR traffic enroute (most likely on departure) RA maneuvering should not violate ATC instructions No conflicting ATC or party-line information Pilot disengages autopilot and flight directors If the pilot did not meet any particular performance measure, review the correct response in regards to that measure Pilot responds to advisory with appropriate vertical speed Pilot ensures vertical speed is not excessive o Pilot notifies ATC of response to TCAS advisory as the maneuver is performed Pilot reengages autopilot and flight directors o Pilot notifies ATC of TCAS advisory and response after clear of conflict Pilot returns to original clearance (if needed)

Evaluating the training programs impact

+Baseline Study +Training Study +6 scenarios with 2 traffic events per scenario +Traffic events defined by RA type ATC information Traffic density Overview of Study 18 Comparison to identify the impact of training

Overview of Study Pre-Training Data Collection (50 minutes) Pre-Experiment Questionnaire Pre-Experiment Quiz TCAS Training Program (50 minutes) Introduction to TCAS Demonstration Based Training Short Section Quizzes Event Based Training Evaluating TCAS Training Program (120 minutes) Flight Scenarios Post Scenario Questionnaires Debrief (15 minutes) Post- Experiment Questionnaire 19

Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior 20 Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?

21 Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS? Decrease time to achieve compliance Decrease time to disconnect autopilot Reduce aggressive response features Increase percentage of RA duration in compliance Increase appropriate response post-Clear of Conflict Increase understanding of TCAS Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Assessing the Pilots Response: Skill-Based Behaviors 22 Vertical Speed time 5 seconds RA Climb 2 sec TCAS weakens required vertical rate TCAS assumed g pull-up Clear of Conflict TCAS assumed constant vertical RA rate Autopilot Disconnect Time TCAS RA Maneuver Pilots Response Time Pilot First Achieves Compliance Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior X

Mean Time to Comply: During Training 23 Skill-Based Behavior p MM = Significance for the mixed model p 2 = Significance of the variance p = Significance of the means Rule-Based Behavior Knowledge- Based Behavior

+During training, no significant differences +Post training Autopilot disconnect time decreased Pilot response was more consistent for one event Autopilot Disconnect: Post Training 24 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Impact of Training Program 25 Decrease time to achieve compliance Decrease time to disconnect autopilot Reduce aggressive response features Increase percentage of RA duration in compliance Increase appropriate response post-Clear of Conflict Increase understanding of TCAS Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior Skill-Based Behavior Rule-Based Behavior

Assessing the Pilots Response: Aggressiveness 26 Vertical Speed time 5 seconds RA Climb 2 sec TCAS weakens required vertical rate TCAS assumed g pull-up Clear of Conflict TCAS assumed constant vertical RA rate Maximum vertical rate Maximum vertical rate difference Vertical rate difference TCAS RA Maneuver Pilots Response Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior *Altitude Deviation

Assessing the Pilots Response: Compliance 27 Vertical Speed Not in Compliance In Compliance TCAS RA Maneuver Pilots Response time 5 seconds RA Climb 2 sec TCAS weakens required vertical rate TCAS assumed g pull-up Clear of Conflict TCAS assumed constant vertical RA rate Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Assessing the Pilots Response: Return to Clearance 28 TCAS RA Maneuver Pilots Response TCAS assumed constant vertical RA rate TCAS weakens required vertical rate Clear of Conflict Altitude time 5 seconds RA Climb TCAS assumed g pull-up Cleared altitude at time of RA Pilot holds new altitude Contacts ATC and request new clearance or ask for further instructions Pilot begins descent back to originally cleared altitude May inform ATC of response to RA Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Before Training +89% of the pilots did not know that an RA should cause less than 500 feet of altitude deviation +56% of the pilots responded that they would hold current altitude achieved after responding to an RA, as opposed to returning to their clearance +When asked about airline procedures for following TCAS, all 18 pilots noted the need for complying with an RA BUT 28% of the pilots commented compliance wasnt necessary is there was a TCAS malfunction or if the RA would cause an unsafe situation 29 [Pilots] must always comply with a TCAS RA unless [aircraft] performance is hindered (i.e. operating single engine) or [there is] an obvious TCAS malfunction (ie you can see traffic and it is not a threat) Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

+During training, aggressive response features decreased +Post training, same decreasing trend observed Aggressiveness 30 Altitude Deviation Average Vertical Rate Difference Maximum Vertical Rate Difference All decreased Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

+During training, percentage compliance decreased in training event with Climb RA (93.1% compared to 99.6%) +Post training No significant differences in means observed But, trained pilots had a more consistent response Percentage Compliance 31 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Return to Clearance 32 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior Pilot returns to original clearance No Yes

Impact of Training Program 33 Decrease time to achieve compliance Decrease time to disconnect autopilot Reduce aggressive response features Increase percentage of RA duration in compliance Increase appropriate response post-Clear of Conflict Increase understanding of TCAS Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Before Training +94% pilots agreed with statement I understand TCAS maneuvers when they are issued, but Only 50% of the pilots correctly identified the assumptions made by TCAS advisory logic When asked to interpret TSD symbols, only 50% of the pilots got all parts of the associated questions correct 34 Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

My understanding of TCAS has increased +50% pilots reported an increase in understanding TCAS logic +27% pilots claimed to have learned about different types of TCAS RAs (notably, Crossing RAs) Post Training Understanding of TCAS 35 Types of RA's were not previously taught. We were taught simply to comply Strongly Disagreed 0 0 Neutral 6% Agreed 61% Strongly Agreed 33% Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Post Training Trust in TCAS I am more likely to trust TCAS after completing todays training 36 My trust in TCAS was already at a maximum so I wouldnt be more likely to trust it Strongly Disagreed 11% Disagreed 0 Neutral 39% Agreed 39% Strongly Agreed 11% Skill-Based Behavior Rule-Based Behavior Knowledge- Based Behavior

Post-Training 37 Decrease time to achieve compliance Decrease time to disconnect autopilot Reduce aggressive response features Increase percentage of RA duration in compliance Increase appropriate response post-Clear of Conflict Increase understanding of TCAS Increase trust in TCAS

Conclusions and Future Work Conclusions +Pilots may not need more training, but instead need better training Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive EBT structure allows for the design of purposeful training events +Current FAA mandated TCAS training objectives may not fully reflect all training areas Language needed to address the reduction of excessive responses to RAs Future Work +What facilities and technologies would be required to implement this type of training program? +What implications arise when considering training design versus system design? Would incorporating human factors considerations in initial design stages decrease the amount of required training? 38

Conclusions and Future Work Conclusions +Pilots may not need more training, but instead need better training Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive EBT structure allows for the design of purposeful training events +Current FAA mandated TCAS training objectives may not fully reflect all training areas Language needed to address the reduction of excessive responses to RAs Future Work +What facilities and technologies would be required to implement this type of training program? +What implications arise when considering training design versus system design? Would incorporating human factors considerations in initial design stages decrease the amount of required training? 40

Acknowledgements +Work sponsored by the FAA, Tom McCloy as Technical Monitor +34 pilot participants +Dr. Amy Pritchett +Dr. Karen Feigh +Dr. Ute Fischer +Dr. Wesley Olson, MIT Lincoln Labs +Wayne Gallo, FAA +Roger Sultan, FAA +Kylie Garey +TCAS Team: William Cleveland, Vlad Popescu, Justin Mullins, Anil Bozan, Henry Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, Dhruv Thaakar, Jonathan Zoetrum, Jelle Wissink +CEC Lab Members 41

Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr....

Documents

Transcript of Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr....