Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy...
-
Upload
cordelia-ryan -
Category
Documents
-
view
213 -
download
0
Transcript of Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy...
![Page 1: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/1.jpg)
Collaborative Time Sensitive Targeting
aka
Technologies for Team Supervision
March 31, 2008
Missy CummingsHumans & Automation Lab
MIT Aeronautics and Astronauticshttp://halab.mit.edu
![Page 2: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/2.jpg)
Review of Original ProposalReview of Original Proposal
Project Goal Develop technological support for futuristic unmanned aerial vehicle (UAV) teams
engaged in TST operations (but really any time critical resource allocation task)
Technology to Support the Team Supervisor:
Activity Awareness Displays
Interruption Recovery Tools
Technology to Support UAV Operators:
Activity Awareness Displays
Objective Performance Measures
![Page 3: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/3.jpg)
Teamwork in UAV OperationsTeamwork in UAV Operations
Intelligence Consumers (e.g., Ground Troops)
UAV Operators
Operations CenterGround Crew
Vehicle(s)
![Page 4: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/4.jpg)
Generalizable to Other DomainsGeneralizable to Other Domains
3 “Confederates”as UAV Operators
![Page 5: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/5.jpg)
Team Interface Issues in Time-Critical Team Interface Issues in Time-Critical OperationsOperations
• Increasing reliance on collaboration tools- e.g., email, instant messaging
(chat)- Increased communication
overhead
• Need interfaces that intelligently share activity information to improve planning & coordination in networked teams
- Activity awareness
• Research for technology aids to assist supervisors of teams virtually non-existent
• Measurement of technology interventions very difficult- Objective team performance
metrics are elusive
![Page 6: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/6.jpg)
Boeing Team Test FacilityBoeing Team Test Facility
3 “Confederates”as UAV Operators
![Page 7: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/7.jpg)
Simulated Task EnvironmentSimulated Task Environment
Situation Map Display Mission Status Display Remote Assistance Display
Surveillance support for a ground convoy through a hostile region
• UAV team consists of:
- 1 Mission Commander
- 3 UAV operators, each controlling multiple UAVs
• UAVs have camera sensors only
- Team must coordinate with external strike team to destroy identified threats
![Page 8: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/8.jpg)
Team Supervision Decision AidingTeam Supervision Decision Aiding
UAV Operations Team Interruption recovery
tools for the team supervisor
![Page 9: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/9.jpg)
Interruption Recovery Assistance (IRA) toolInterruption Recovery Assistance (IRA) tool
• Interactive event timeline, containing event “bookmarks”
• Increased recovery time but positive impact on decision accuracy, especially in complex task situations.
• IRA tool tended to provide greater benefits to participants without military experience
• Some experimental/interface issues so additional study needed
• Summer 2008 w/ Waterloo
![Page 10: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/10.jpg)
Current Work: Predictive Tool for Team Current Work: Predictive Tool for Team SupervisionSupervision
UAV Team SupervisorTechnologies for
assessing:
- How well is the team doing?
- When & who should received attention?
Team
![Page 11: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/11.jpg)
• The need for more objective team performance metrics
- Our focus is supervisory control in team settings with embedded layers of automation
• Predicting team states versus actual performance
- Artificial intelligence, behavioral pattern detection and performance correlations
• Alert team supervising agent of sub-optimal team behavior/cognitive strategies
- Relying on human knowledge-based reasoning to determine if an anomalous team state exists
• Two stage approach
- Predict an individual operator’s state transition as a proof of concept
- Adapt model to UAV team setting
Current Work: Team Performance PredictionCurrent Work: Team Performance Prediction
![Page 12: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/12.jpg)
• Evaluation and predictions of team behaviors:- Automatic, continuous, and in real-time
• Only relies on easily observable data- User interaction with machine
- User communication
- Eye tracking?
• Use Bayesian pattern prediction methods applied to individual/team behaviors- Hidden Markov Model
• Not predicting performance per se- Anomalous conditions
- Value of human judgment
Team Prediction Model CharacteristicsTeam Prediction Model Characteristics
![Page 13: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/13.jpg)
• Probabilistic state transitions
• Observables vs. Hidden States- Hidden states are not directly visible, but variables influenced by them are.
- Cognitive states vs. behaviors
• HMM uses the observables to infer:- Most likely hidden state sequence- Based on future sequences, the most likely observables- Machine learns the clustering of behaviors (need lots of data!)
Hidden Markov ModelsHidden Markov Models
Modify Flight Parameters
CommsCamera Control
Waypoint Update
H&S MonitoringTarget Monitoring
Mission Replan
![Page 14: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/14.jpg)
Predicting Individual Behavior Predicting Individual Behavior
![Page 15: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/15.jpg)
Human State Estimation - TRACS Human State Estimation - TRACS
![Page 16: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/16.jpg)
Building the Model Building the Model
Group ofCriteria
IndividualCriterion
Group ofMatches
IndividualMatch
DataCluster
DataItem
Evaluate
BacktrackBrowse
SelectFilte
r
Automatch
Create
1 2 3 4 5 6
8 9 10 11 12 13
15 16 17 18 19 20
22 23 24 25 26 27
28 29 30 31 32 33 34
35 36 37 38 39 40 41
0
7
14
21
GrammarPattern
Recognizer/Predictor
Low Level Input
Future TeamBehavior
ILDs
![Page 17: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/17.jpg)
4 State HMM for Mission Planning Task 4 State HMM for Mission Planning Task
• Unsupervised vs. supervised model training for state recognition
Group ofCriteria
IndividualCriterion
Group ofMatches
IndividualMatch
DataCluster
DataItem
Evaluate
BacktrackBrowse
SelectFilte
r
Automatch
Create
1 2 3 4 5 6
8 9 10 11 12 13
15 16 17 18 19 20
22 23 24 25 26 27
28 29 30 31 32 33 34
35 36 37 38 39 40 41
0
7
14
21
Leveraging Automation
ManualPlanning
Exploration
![Page 18: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/18.jpg)
Predicting an Operator’s Strategies Predicting an Operator’s Strategies
![Page 19: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/19.jpg)
Live Demo! Live Demo!
![Page 20: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/20.jpg)
Validating our HMM Approach Validating our HMM Approach
• Currently only behaviors that entail mouse clicks are observed
- A limit, particularly for monitoring interfaces
• Eye tracking arguably gives us additional data about the information operators access
- Noisy
- Cost-benefit analysis?
• Study will be conducted this summer comparing HMM results with and without eye tracking input
![Page 21: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/21.jpg)
Extending our Approach to UV Teams Extending our Approach to UV Teams
Visualization task
Evaluation/
Choice of UV + engage
Choice of UV
Sector A
Sector B
Sector C
a) Replan Path
b) Replan Target
Extending to task to teams: UV hand-off between Sectors
![Page 22: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/22.jpg)
The Plan The Plan
• Develop HMM using experimental data from individual experiments
- Investigate supervised vs. unsupervised learning
• Develop a team version of the interface
• Update HMM with team experiments to be conducted this summer
![Page 23: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/23.jpg)
Additional Future Work: Designing for Team Additional Future Work: Designing for Team Activity Awareness Activity Awareness
UAV Operations Team
![Page 24: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/24.jpg)
Preliminary UAV Operator Display DesignsPreliminary UAV Operator Display Designs
Operator 1
UAV 4
Reroute/ReassignmentTarget ID Task
3UAV1
UAV 3 UAV 5
Health Status
Fuel level: 40%
Connected
- Searching - Searching Health Status
Fuel level: 70%
Health Status
Fuel level: 70%
Connected Connected 0:00:300:00:050:03:59 UAV3
5
Help
Request sent to:
UAV3
4 - Searching
Assistance Requests Overview
Req 3 (Complete)
Request from: Op 1 Elapsed time: 0m35sDetected by: UAV 3 Target type: 5 (Com Eq)
Request from: Op 3 Elapsed time: 0m30sDetected by: UAV 8 Target type: 5 (Com Eq)
Req 5 Request from: Op 1 Elapsed time: 0m05sDetected by: UAV 2 Target type: 3 (IED-V)
Req 4
ATR Information:
Possible Match: Target Type 5: Comm Equipment
Match Confidence: Medium
Found by UAV2 – Location: 43:00E/24:00N
ATR Decision:
SubmitClassification
Confirm ATR Suggested Classification:
Target Type 5 with Medium Confidence
Provide Alternate Target Classification:
Select Target Type…
Select Confidence Level…
Select Target Priority…
Veto Classification:
Image does not contain known target
Operator 3
Cancel
All
Others
Op 1
mCDR
Op 2
Classification:
Target Type 5 High Confidence
(no classification available)
Elapsed time:
0:00:20
Image Received at:
12:56:40
Request new image
Threat distance from the convoy: 15 miles
Elapsed Time:
Current Time:
1: 25: 30 PM
0: 22: 45
Mission Info:
0: 03: 28
UAV Reassignment
Elapsed Time:
Remaining Time:
0: 02: 37
Routes:
View Filter:
Hazards
Connection Blackouts
Targets Range
All
UAV 4
UAV 5
UAV 6
Sendprivate
All
Op 1
Op 3
mCDR
Strike team
others
Type here…
Op 2 – communication box
ALERTS & Clarification
SendType here…
Share Files
mCDR OthersStrike T Op 1 Op 3
Comm Status
0:03:57
On line
Handoff Available[0:00:30] Conference call
[0:01:20] mCDR to OP1:
[0:01:22] Op1 to Op3:
[0:01:25] Op3 to strike team:
[0:01:29] Op3 to Op2:
4 - Searching
Repair
Repair
RepairRepair
Health Status
Health Status
Health Status
Fuel level: 40%
Fuel level: 70%
Fuel level: 70%
- Target ID5
6
Connected
Connected
DisconnectedSafety route in 0:29:59
Reconnecting in 0:00:49
0:00:18
0:00:04
0:03:57
(0:01:50) UAV 7 (area3) has been shot down
(0:03:00) UAV 4 Fuel Level at 40%
(0:03:40) UAV 6 enter in a difficult area to connect
mCDR
Strike team
OP1
Others
OP3
0:03:57
0:03:57
0:01:53 - Changing shift
0:00:33 - Target task
Comm
Comm
Comm
Comm
0:03:57
Repair
Repair
Repair
Repair
Repair
Comm
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
MapDisplay
CommunicationsDisplay
Tasking Display:TargetID & rerouting, reassigning UAVs
![Page 25: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/25.jpg)
DeliverablesDeliverables
• 6 Conference papers- ASNE, HSIS, ICCRTS, HFES
• 4+ Technical reports• 3 Undergraduate theses
- 2 pending• 1 Masters thesis• 1 PhD dissertation (pending)• 3 Journal articles in various stages• 1 workshop (CSCW 2006)• IP disclosure pending HMM validation• The Holy Grail of team research???
![Page 26: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/26.jpg)
• Near Term
- Eye tracking study
- HMM-Multi UV experiment
- Extension for supervisor IRA tool
• Mid Term
- Activity awareness displays for operators
- Decision support display for HMM
- Multi-UV connection to predictive model system architecture research (ONR)
Development of CE metric
• Project slated to end December 2008- Requested no-cost extension to JUN 09
- Additional funding
Future PlansFuture Plans
![Page 27: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/27.jpg)
Backup Backup
![Page 28: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/28.jpg)
Simulated Task Environment Display DetailSimulated Task Environment Display Detail
Map Display
Mission Status Display
![Page 29: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/29.jpg)
Simulated Task Software ArchitectureSimulated Task Software Architecture
Large-Screen Wall
Displays
Situation Map Display Mission Status Display
TabletPC Display
Mission Commander Display
Simulation & Collaboration Server
(Grouplab SharedDictionary)
![Page 30: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/30.jpg)
Situation Map Display
Mission Status Display
Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy safety
• Current & expected operator task performance, relative to convoy safety
4
(nominal)
(down)
8
1
(reviewing ATR imagery)
![Page 31: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/31.jpg)
Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness
Potential threat
envelopeTarget strike indicators
Known threat
envelope
Situation Map Display
Mission Status Display
• UAV status & tasking
• Current & expected convoy safety
• Current & expected operator task performance, relative to convoy safety
![Page 32: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/32.jpg)
Situation Map Display
Mission Status Display
Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy safety
• Current & expected operator task performance, relative to convoy safety
![Page 33: Collaborative Time Sensitive Targeting aka Technologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and.](https://reader035.fdocuments.in/reader035/viewer/2022070408/56649e575503460f94b4f273/html5/thumbnails/33.jpg)
Developing Methodology for Deriving Developing Methodology for Deriving Collaborative System Requirements Collaborative System Requirements
Project Goal:
• To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking