Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall...
-
Upload
brady-dorrance -
Category
Documents
-
view
215 -
download
0
Transcript of Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall...
![Page 1: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/1.jpg)
Automation Usage Decisions: Controlling Intent and Appraisal Erros
in a Target Detection Task
Hall P. Beck
Appalachian State University
Mary T. Dzindolet
Cameron University
Linda G. Pierce
Army Research Laboratory
![Page 2: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/2.jpg)
Automation Usage Decisions (AUDs)
AUDs: Operator must choose between automated and/or a manual control or less technologically sophisticated means of control.
Misuse: The overutilization of automation
Disuse: The underutilzation of automation
Purpose: To determine if feedback and scenario training mitigate misuse and/or disuse.
![Page 3: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/3.jpg)
Two Causes of Suboptimal AUDs
Appraisal Errors: Operators are unable to determine if automation or a non-automated alternative maximizes the likelihood of task success.
Intent Errors: Operators know the relative utilities of the options but disregard this information when deciding whether to use or not use automation.
![Page 4: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/4.jpg)
Design and Procedure
A 2 (Feedback, No Feedback) x 2 (Scenario Training, No Scenario Training) x 2 (Superior, Inferior Machine) between-subjects in which an AUD is the dependent variable
![Page 5: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/5.jpg)
Design and Procedure
1. Receive scenario training or control information
2. Perform 280 target detection trials
3. Feedback group is told how many errors that they and the machine made. No feedback group does not receive this information.
4. Operator either makes twice (superior machine condition) or half (inferior machine condition) as the machine.
![Page 6: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/6.jpg)
High Noon
Ten trials will be replayed from the preceding 280. Extra credit depends on the number of these trials which are correct.
Operator bases credit on their own or the machine‘s performance.
![Page 7: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/7.jpg)
Underlying Logic
Suboptimal AUDs in the no feedback conditions could be due to appraisal and/or intent errors.
Suboptimal AUDs in the feedback condition should only be due to intent errors.
Feedback should reduce or eliminate appraisal errors.
Scenario Training should decrease intent errors.
![Page 8: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/8.jpg)
Scenario Training x Feedback
0
20
40
60
80
100
No Scen 84 55
Scenario 87 29
No Feedback Feedback
![Page 9: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/9.jpg)
Conclusions
Results suggest that operators‘ AUDs were determined by multiple contingencies.
Operator training programs should include procedures should as scenario training to reduce intent errors.
Advances in the reliability of decision aids and other automated devices will be a hollow achievement unless our knowlege of hardware and software is matched by an equally sophisticiated comprehension of the causes of misuse and disuse.
![Page 10: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/10.jpg)
To Shoot Or Not To Shoot
![Page 11: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/11.jpg)
To Shoot Or Not To Shoot
Since 1900, 10% to 25% of US war fatalities in resulted from fratricide
![Page 12: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/12.jpg)
Targeting Decisions: Possible Outcomes
1) Soldier and CID detect a friend.
2) Soldier and CID fail to detect a friend.
3) Soldier detects a friend and CID fails to detect a friend.
4) Soldier fails to detect a friend and CID detects a friend.
![Page 13: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/13.jpg)
Automation Usage Decisions (AUDs)
AUDs- Choices in which a human operator has the option of relying upon manual control or one or more levels of automation (LOAs) to perform a task.
Optimal AUD-Soldier relies upon the form of control that is most likely to result in a correct decision.
![Page 14: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/14.jpg)
Types of Suboptimal AUDs
Misuse is over reliance, soldier employs automation when manual control or a relatively low LOA has a greater likelihood of success
Disuse is the under utilization of automation, soldier manually performs a task that could best be done by a machine or a higher LOA.
![Page 15: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/15.jpg)
Beck, Dzindolet, & Pierce (2002)
Appraisal Errors-Soldier misjudges the relative utilities of the automated (CID) and non-automated (e.g., view through gun site) options.
Intent Errors-Soldier disregards the utilities of the alternatives when making AUDs.
![Page 16: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/16.jpg)
Intent Errors: Two Images of an Operator
An operator is a single-minded individual whose sole object is to maximize task performance
An operator‘s decision to rely on automation is based on a number of contingencies only one of which is to achieve a successful performance.
![Page 17: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/17.jpg)
John Henry Effect
John Henry Effect: Operators respond to automation as a challenger, competitor, or threat
Increasing the operator’s personal involvement with the non-automated alternative augments the likelihood of a John Henry Effect.
![Page 18: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/18.jpg)
John Henry Effect
Variables that increase the strength of a John Henry Effect augment operators‘ preference for the non-automated over the automated alternative
Heightened preference for the non-automated option should: 1) increase disuse and 2) decrease misuse
![Page 19: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/19.jpg)
Design
2 (Operator: Self-reliant, Other-reliant) x 2 (Machine Performance: Inferior, Superior) x 14 (Trial Blocks) design
Dependent Variable: Suboptimal AUDs (Superior Machine: Basing credit point on the operator’s performance; Inferior Machine: Basing credit on the machine’s performance)
![Page 20: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/20.jpg)
Credit Choice Screen
![Page 21: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/21.jpg)
Sample Helicopter Photograph
![Page 22: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/22.jpg)
Sample Helicopter Photograph
![Page 23: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/23.jpg)
Operator Response Screen
![Page 24: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/24.jpg)
CID Response Screen
![Page 25: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/25.jpg)
Results Screen
![Page 26: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/26.jpg)
Hypotheses
• Self-reliant operators will be less likely to base credit points on the CID than other-reliant operators
• Therefore– Disuse will be greater in the self-superior
than in the other-superior condition – Misuse will be higher among other-inferior
than self-inferior persons
![Page 27: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/27.jpg)
Disuse
• Figure 1. Mean suboptimal automation usage decisions (AUDs) as a function of operator and trial block for persons working with the superior machine.
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Trial Blocks (20 Trials Per Trial Block)
Mean S
ubop
tim
al
AU
Ds
Self
Other
![Page 28: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/28.jpg)
Misuse
• Figure 2. Mean suboptimal automation usage decisions (AUDs) as a function of operator and trial block for persons working with the inferior machine.
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Trial Blocks (20 Trials Per Trial Block)
Mean
Su
bo
pti
mal
AU
Ds
Other
Self
![Page 29: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/29.jpg)
Conclusions
1) Self-reliant and other-reliant operators were yoked. Each had the same information. It seems reasonable to conclude that the difficulty in determining the optimal AUD was approximately equal in both conditions. Thus, the large differences in suboptimal AUDs were probably due to intent rather than appraisal errors.
2)Results support the hypotheses that factors which augment the degree of personal involvement or challenge from automated devices will increase the probability of disuse and decrease the likelihood of misuse
![Page 30: Automation Usage Decisions: Controlling Intent and Appraisal Erros in a Target Detection Task Hall P. Beck Appalachian State University Mary T. Dzindolet.](https://reader036.fdocuments.in/reader036/viewer/2022081515/56649ca65503460f94968fc2/html5/thumbnails/30.jpg)
A Few Implications
1) Operator training programs should attempt to attenuate intent as well as appraisal errors.
2) At least on this task, intent errors were a significant source of suboptimal AUDs
3) Both appraisal and intent errors are sufficient to produce suboptimal AUDs although neither is necessary
4) It will be a hollow achievement if advances in our knowledge of hardware and software is matched by an equally sophisticated comprehension of the causes and control of misuse and disuse.