2. NCAT _Marcia Nealy (1)

Ecological Approach to Model Human Judgment Performance under Uncertain Environment with Autonomous Aids TECHLAV Annual Meeting, July 24-25, 2016 Marcia Nealy (MS student) | Dr. Younho Seong (NCAT) | Dr. Joseph Stephens (NCAT) Hybrid Lens Model of Operator Judgment with Autonomous Aids Based on the Lens model with its extension, a Hybrid Lens model was developed to precisely identify the effect of automated decision aids on operators’ judgment performance, depending on the quality of decision aids. The model identifies : • Human operators’ sole judgment performance • Decision aids’ quality of producing estimates – Decision aids’ competence • Effect of decision aids’ on judgment performance • Correspondence between the decision aid and human operator Introduction • Simulated Aircraft Identification Task with Automated Decision Aid – Participants received raw information (radar, speed, altitude) and output from ADA • Varied Properties of the Aid – Aid Validity (how well the aid predicts the aircraft type) – Aid Reliability (how consistently the aid makes decisions) – “Meta-information” – whether the aid reports on its own performance • Participants Completed Multiple Scenarios, Over 2 Days – Day 1 – No Aid | Day 2 – Aid, under different conditions • High(+) vs. Low (-) Validity (V), Reliability (R), Meta Information/ Understandability (U) Technical Experimentation Methods Results • Brunswik (1955), Hammond, Stewart, Brehmer, & Steinmann, (1975) • Achievement – The Correspondence Between Judgments, and the Environment to be Judged, Based on Available information (the cues) – Example: Judge if an aircraft is hostile or friendly based on known information about the aircraft – achievement measures how well your judgments correspond to the true nature of the aircraft • Multiple Regression Based Model Formulation Provides Measures of: – Achievement (r a ) – Degree to which judgment policies of the person match a normative model of the environment (G) – Degree to which the environment is well modeled with a linear model (R e ) – Degree to which people make consistent judgments (R s ) • Apply This Modeling Framework to: – Human Operator (HO) making judgments unaided, based on a set of cues – Automated Decision Aid (ADA) making judgments, based on a set of cues – Human Operator making judgments in conjunction with the decision aid (HO-ADA), where there is an additional cue – the output of the ADA – Achievement of these three systems, as well as the correspondence (or reliance/use) of the HO and the ADA, and the difference between the unaided HO, and aided HO, can then be determined Research Thrust Area: 3-7 Acknowledgement This research is supported by Air Force Research Laboratory and OSD under agreement number FA8750-15-2-0116, and also Army Research Laboratory through the second author. Achievement r a Y S Judgments Y E Criterion Knowledge G Un-modeled Agreement C Cognitive Control, R S Environmental Predictability, R E Cue1 Cue2 Cue3 Cue4 Δ Residuals β 1E β 2E β 3E β 4E Ŷ E Linear Model Output Δ Residuals β 1S β 2S β 3S β 4S Ŷ S Linear Model Output YENV X1 YADA X2 X3 X4 Environmental State Automated Decision Aid CUES Validity Understand -ability Reliability

Upload
marcia-nealy
Category

Documents
view
23
download
0

Embed Size (px):

Transcript of 2. NCAT _Marcia Nealy (1)

Ecological Approach to Model Human Judgment Performance under Uncertain Environment with

Autonomous Aids

TECHLAV Annual Meeting, July 24-25, 2016

Marcia Nealy (MS student) | Dr. Younho Seong (NCAT) | Dr. Joseph Stephens (NCAT)

Hybrid Lens Model of Operator Judgment with

Autonomous Aids Based on the Lens model with its extension, a Hybrid Lens

model was developed to precisely identify the effect of automated decision aids on operators’ judgment performance, depending on the quality of decision aids. The model identifies :

• Human operators’ sole judgment performance • Decision aids’ quality of producing estimates – Decision

aids’ competence • Effect of decision aids’ on judgment performance • Correspondence between the decision aid and human

operator

Introduction

•  Simulated Aircraft Identification Task with Automated Decision Aid – Participants received raw information (radar, speed, altitude)

and output from ADA

• Varied Properties of the Aid – Aid Validity (how well the aid predicts the aircraft type) – Aid Reliability (how consistently the aid makes decisions) – “Meta-information” – whether the aid reports on its own

performance

• Participants Completed Multiple Scenarios, Over 2 Days – Day 1 – No Aid | Day 2 – Aid, under different conditions

•  High(+) vs. Low (-) Validity (V), Reliability (R), Meta Information/ Understandability (U)

Technical Experimentation

Methods Results • Brunswik (1955), Hammond, Stewart,

Brehmer, & Steinmann, (1975) • Achievement – The Correspondence

Between Judgments, and the Environment to be Judged, Based on Available information (the cues) – Example: Judge if an aircraft is hostile or friendly based on

known information about the aircraft – achievement measures how well your judgments correspond to the true nature of the aircraft

• Multiple Regression Based Model Formulation Provides Measures of: – Achievement (ra) – Degree to which judgment policies of the person match a

normative model of the environment (G) – Degree to which the environment is well modeled with a

linear model (Re) – Degree to which people make consistent judgments (Rs)

• Apply This Modeling Framework to: – Human Operator (HO) making judgments unaided, based on a

set of cues – Automated Decision Aid (ADA) making judgments, based on a

set of cues – Human Operator making judgments in conjunction with the

decision aid (HO-ADA), where there is an additional cue – the output of the ADA

– Achievement of these three systems, as well as the correspondence (or reliance/use) of the HO and the ADA, and the difference between the unaided HO, and aided HO, can then be determined

Research Thrust Area: 3-7

Acknowledgement This research is supported by Air Force Research Laboratory and OSD under agreement number FA8750-15-2-0116, and also Army Research Laboratory through the second author.

Achievement ra

YS Judgments

YE Criterion

Knowledge G

Un-modeled Agreement C

Cognitive Control,

Environmental Predictability,

Cue1

Cue2

Cue3

Cue4

Δ Residuals