Converging Measures: Federal Aviation Administration ...
Transcript of Converging Measures: Federal Aviation Administration ...
Federal AviationAdministration
Converging Measures:
Integrating Diverse Data
Sources in ATM Human-
in-the-Loop Simulations,
FAA
Ben Willems, FAA Kenneth Allendoerfer, Ph.D.
Federal Aviation Administration
Human Factors Branch, ANG-E25
Ben Willems
Federal Aviation Administration
ATO Program Management Office, AJM-1310
FAA-EUROCONTROL
Technical Interchange Meeting –
Modeling & Simulation
October 16-17, 2019
2
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
•Why
Why conduct human-in-the
loop simulations at all?
3
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
What do we want to learn from ATM HITL
simulations?
• Effects of proposed…
– Procedure or operational concept
– Airspace or airport design
– Technology or equipment
– Training program
• On controllers’…
– Physical and physiological states
– Cognition
– Decisions
– Actions
• And on…
– overall human-system performance.
4
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Why conduct HITL simulations at all?
• Human behavior is hard to predict especially in complex and novel situations
– Fast-time models typically do not capture this unpredictability very well.
• Human actions are driven by physiological and cognitive processes and states
– What’s going on is a different question than what does the human know about what’s going on.
• Project may have explicit human-performance goals
– “… improve pilot awareness of hazardous weather that may affect the flight…”
• Humans are going to be asked to work in this new environment and to live with the consequences
– Rhetorical impact & credibility
5
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
SAFETYEFFICIENCY
& CAPACITY
WORKLOAD
& EFFORTACCEPTANCE
& USAGE
High-Level Constructs
6
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Analytic Weight
• Analysis-HEAVY
– Incident reviews, replays, watch and code videos
– Qualitative and quantitative data
– Requires human expertise to design and conduct the analysis
• Analysis-LITE
– Data collected automatically by system and processed by algorithms
– Quantitative data
– Requires human expertise to design the analysis, but the analysis itself can be conducted automatically
– Lend themselves to incorporation into fast-time models
• HITLs lend themselves to Analysis-HEAVY metrics
– Effects on humans (especially cognitive) are hard to measure automatically
7
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Efficiency & Capacity
• Fast-time modelers will be familiar with these
• Typically calculated from track data or output
of ATM automation systems
– Number of aircraft handled per hour
– Number of arrivals
– Average spacing over the arrival fix
– Median time between departures
– Total Distance flown
– Trajectory conformance
– Etc.
8
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Safety
• Operational errors
• Runway incursions
• Near misses
– How to define?
– Quantification of risk
• Unlikely to occur organically during a HITL sim
– Controllers see problems ahead of time and fix them
– Script 2x the conflicts that you want
9
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Safety
• Precursors – Increased Risk
– Workload (very high / very low)
– Fatigue
– Small errors
– Lack of experience / recent training
– “Sharpness”
• Awareness – Attention
– Mental model of situation vs. situation itself
– Situation Awareness (SA)
– Distraction / inattention
– “Tricky” situations – things look normal but aren’t –inattentional blindness, confirmation bias
10
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Eye Tracking to Measure
Awareness / Attention
• Point of gaze
• Fixations, saccades
– Time since last fixation
– Sequences and patterns of fixations
– Visual scan pattern
• Reaction time to alerts, surprises
• Pupil diameter (cognitive activity)
11
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Karsten, G., Goldberg, B., Rood, R., & Sulzer, R. (1975). Oculometer measurement of air traffic
controller visual attention (FAA-NA-74-61). Atlantic City, NJ: Federal Aviation Administration
National Aviation Facilities Experimental Center.
12
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Eye Tracking (Old School)
13
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Eye Tracking (Today)
14
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Eye Tracking
15
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Eye Tracking – Point of Gaze
16
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Analysis-HEAVY
• Because safety events are rare in sims, we tend to go “deep”
• Replays of incidents with SMEs
– Communications, screen-capture videos
– Over-the-shoulder videos
– Identify where controller’s mental model disconnected from the true state of the world
• Review of point-of-gaze data for a chosen aircraft pair
• We don’t have strong algorithms to identify a good scan pattern or a risky scan pattern
17
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
FALCON
18
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Workload & Effort
• Subjective Ratings
– Air Traffic Workload
Input Technique
(ATWIT)
– NASA-TLX
• Ratings by SME
observers
• Post-Scenario
Ratings / Interviews
Workload Assessment Keypad (WAK)
19
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Task Metrics
• Control actions
– Altitude, speed, heading, route changes
• Communications
– Number, duration, errors
• UI interactions
– Number, duration, errors
20
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Physiological Metrics
• Neuroergonomics
– Electroencephalography (EEG)
– Functional Near Infrared Spectroscopy (fNIRS)
• Pupil diameter
• Heart rate variability
• Stress
– Galvanic skin response
– Hormone levels
– Heart rate
– Speech characteristics
21
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
EEG
22
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
fNIRS
23
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Sample of fNIRS Data
Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B. (2012). Optical brain monitoring for operator training and mental
workload assessment, NeuroImage, 59, 36–47.
24
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Acceptance & Usage
• Do the humans choose to use the new thing?
– Did controllers take the advice given by the decision-
support tool? Under what circumstances?
• Trust / Distrust
• Reliance / Over-reliance
• Does it feel intuitive? Do they enjoy using it?
• We are a long way from having models that can
predict these effects (but they are important).
25
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Complex Event Processing
• Must be able to analyze a stream of data
across different sources
– Identify patterns within a source but also across
sources
– Abstract the patterns into complex events
– Count or measure complex events
• How long did it take?
• How many steps did it take?
• What was the effect of the new thing on the complex event?
– Example: Controller issues a reroute.
26
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Complex Event Processing Example
TIME
Cognitive Activity (EEG)
↓ Cognitive activity starts to climb ↓ Cognitive activity reaches maximum
Attention (eye tracker)
↓ Controller visually attends to AAL1234 datablock ↓ Controller visually attends to route menu
Situation Awareness (SAGAT)
↓ Probe reveals high awareness
of AAL1234 position and route
System Events (UI states)
↓ TFM notifies that reroute is needed ↓ New route appears in route field
Taskload (UI interactions)
↓ C activates route menu ↓ C starts editing route field ↓ C finishes editing route field
Taskload (voice comm)
↓ C initiates A-G call ↓ C completes A-G call
Operational Events (aircraft states)
↓ AAL1234 enters sector ↓ AAL1234 begins to follow new route
27
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Considerations for the future
• Cheap biometric sensors
– Wearable technology
• Big data analytics and visualization
techniques
– Neuroergonomics tools provide a mountain of data
– Pattern derivation within and across metrics
– Potential application of machine learning & AI
28
William J. Hughes Technical Center
Human Factors Branch, ANG-E25
October 2019
Federal AviationAdministration
Considerations for the future
• Use of better operational performance metrics from fast-time sims into HITLs
• Fast-time models with more sophisticated human models
– Stochastic models of human decisions and actions
– Output metrics in human terms
– Use of HITLs to develop algorithms and parameters for fast-time models
• Use of fast-time sims to validate and expand results from HITLs
• Use of fast-time sims to identify areas that need additional investments in HITLs
Federal AviationAdministration
Kenneth Allendoerfer, Ph.D.
(609) 485-4864
Ben Willems
(609) 485-4191
Human Factors Branch, ANG-E25
Federal Aviation Administration
William J. Hughes Technical Center, Building 28
Atlantic City International Airport, NJ 08405
http://hf.tc.faa.gov
Questions &
Contact Information