Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Télécom Bretagne

A graduate Engineering Schooland a Research Centre in the field of

Science and Information Technologies

Institut Mines-Télécom

4

Télécom Bretagne

The Engineering School of the (French) Far West

Télécom Bretagne in few numbers350 Researchers (faculty, masters, PhD)9 Research DepartmentsAffiliations to French National Research Agencies (CNRS)International prices (Marconi, SPIE, IEEE) and National representativeness (French Academy of Sciences)

1200 students

160 Academics

45% International students from 50+ countries

Telecom Bretagne ranked in Top 15 among 200+ Graduate Engineering Schools in France

International exchanges

• Out: Our student have to spend between 4 and 6 months abroad

• In: Master of Sciences• One semester with English spoken teaching + intensive classes

of French language

• In: Internship• Research / engineering project in one of our labs

• In/Out: Co-tutelle PhD

Lab-STICCCNRS UMR 6285

From Sensors to Knowledge:Communicate and Decide

Télécom Bretagne

UBO

UBS

ENIB

ENSTA-Bretagne

The Research Unit of the (French) Far West

Lab-STICC teams

Short bio

Airborne systems engineering (Thales - 1986/2000)PhD in Computer Sciences - 1999Professor at Telecom Bretagne - 2000 / presentHead of Department (Cognitive Sciences / Usages)Visiting professor MIT - Aero-Astro Department 2006/2007French representative NATO group HF - UAV 2008/2012Deputy director Lab-STICC Research Unit

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Human supervisory control of complex systems

Human supervisory control

Increasing complexity and autonomy of systems and subsystems have led to • a separation in time and space between command /control and

execution of the task• a mediation through computers between the operator and the

actuators and sensors

Human supervisory control

Operator Situation

Operator’s space

Command

Display

System

Space of task

Informationmanagement

TaskSharing

Coordination

Informationcollection

Command system

• Task sharing and allocation• Human - System InteractionDimension 1

Referring to OODA

• Functional decomposition• Information processing in systemsDimension 2 Functional decomposition

Supervisory control and OODA

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

Operator

Or

Operator’Operator’Operator

D

• Usual functional sharing• ~ Fitts tableDimension 1 + Dimension 2

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Human factors and UVs control: a brief state-of-the-art

RTO-TR-HFM-170 Supervisory Control of Multiple Uninhabited Systems -

Methodologies and Enabling Human-Robot Interface Technologies

www.cso.nato.int/pubs/rdp.asp?RDP=RTO-TR-HFM-170

Annual Workshops on Human Factors of Unmanned Aerial

VehiclesCERI

http://www.cerici.org/ceri_workshops.htm

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

Supervisory control and OODA

Focusing on Observation

Informationcollection

Remote perception (Observe)

Relative orientation in space

Stability of images

ref. Cooper & al., CERI-UAV 2007

by courtesy of TNO / L. van Breda

North-up Platform up UAV-up

MovingOperatorCentered

MovingUAV

Centered

Remote perception (Observe)

Telepresence UGV (TNO Netherlands / US Army): acuity, contrast sensibility, stereo vision and audio

by courtesy of TNO / L. van Breda

EyeRobot

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator’OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

Supervisory control and OODA

Focusing on Action

Remote action

ref. Maj. Martin, CERI-UAV 2007

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

Operator

Ors spaceOperator’Operator’Operator s space

D

Supervisory Control and OODA

Focusing on Orientation

Support to situation assessment

Cognitive task analysis• Identification of data structure that are meaningful (or salient) for the

operator

Situation awareness models• Mica Endsley 3 stages process (perception, understanding,

projection)• Gary Klein RPD (Recognition Primed Decision)• and many others ...

Hunn, CERI 2006

Graphical interfaces, multimodality

MUSCIT (US Army / USAF) : monitoring ground scenes from multiple UAVs

• Graphical interfaces, attention allocation, multimodal presentation

Left Control Station displaytactical situation display, payload control

Right Control Station displayaircraft video, sensor management

by courtesy of AFRL / M Patzek

Supervisory Control and OODAGoing ahead: teaming!!!

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

collection

D

collection

OrOb A

Cognitive assistant unitsCognitive Assistant Units (Germany / UBM Münich)From shared knowledge representation, mimicking human operators and pilots to handle selected tasks

by courtesy of Schulte / IFS

Assistant systems

by courtesy of Donath & al. / IFS

Assistant systems

by courtesy of Donath & al. / IFS

Playbook (Miller)

Delegation: one way humans manage supervisory control with heterogeneous, intelligent assets

Require a shared knowledge of domain goals, tasks and actions

Plays reference a defined range of plan/behavior alternatives (that can be further constrained or simulated)

Supervisor calls plays; agents have autonomy within the play’s scope

by courtesy of Shively / AMRDEC

Playbook

Overwatch: sustained surveillance of a fixed target or area

Track target: sustained surveillance of a moving target

Area recon one-pass reconnaissance of an area

Route recon: one pass reconnaissance of a route

Encircle patrol: circle a target

Protect: surveillance of an area while moving

“Plays”

Miller, Goldman, Funk, Wu & Pate 2004

Constraints on Plays

Example of constraints (for overwatch function)• earliest / latest acceptable “time on target”• earliest / latest acceptable “time off target”• stealth• priority• sensors• etc...

ref: Miller, Goldman, Funk, Wu & Pate 2004

AI constraint basedcontrol algorithms

Human supervisory control

• Next step is autonomy handled by pro-active automata (also called agents)

• From “Human-in-the-loop” to “Human--the-loop” to “Human-on-the-loop”• From supervisory control to authority sharing (a.k.a.

mixed initiative)

Supervisory Control and OODAHow to cooperate?

? ? ? ?

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

How to cooperate?

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

collection

D

collection

OrOb A

Interaction?

Most of the approaches focus on task sharing, in terms of delegation or responsibilityBut very few address the communication and interaction protocol that supports this delegation

Following: a detailed example from the domain of swarm control

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Playing with swarms

Collective Intelligence & Swarms

• Collective Intelligence paradigm : an intelligent system may be composed of a large number of simple entities interacting together (agents) • Natural collectives systems: social insects, multi-cellular organisms,

communities...• Physical systems: mobile robots, sensors networks, distributed

systems

• Computational systems: multi-agents systems, grid computing, cellular automata

• Properties• Robustness, reactivity, self-organisation, adaptativity

by courtesy of LORIA France

Artificial pheromone

Mobile multi-agent system based on social insect mimickingAgents communicate through the environment where they virtually d r o p a m o u n t s o f a r t i f i c i a l pheromoneA pheromone can have

a repulsive effect so that agents avoid to patrol on the same areaan attractive effect so that agents follow the track opened by others

environment where they virtually a r t i f i c i a l

Principles of swarm

by courtesy of MAIA team LORIA - France

Principles of swarms

by courtesy of MAIA team LORIA - France

SMAARTSwarms for surveillance and security

Context: intrusion surveillance

Interaction management for swarms

• A swarm is an autonomous complex system• Main characteristics of UAVs system:

• Many vehicles• Intrinsic collective automation, local decision• Complex interactions

• Several concurrent tasks• Manage threads of interaction• Priorities, turn-taking, interruption, change of

context, etc.

HMI: a global view

HMI details

Pheromones

2 pheromones in order to ensure patrol and tracking functions• Repulsive pheromone for self organization and covering the

area• Attractive pheromone to concentrate UAVs on tracking (and

possible future identification functions)

HMI details (2)

Patrolling with swarms

Results

Between +6,5% and -13% on the first 20 minutes of pure

surveillance

Fail:

1. to control system

2. to self evaluate

Interacting with a swarm?

Objects out of usual representations• Interactions through the environment, fields of pheromone, diffusion,

evaporation, random moves• What happens? Why? What is to happen?How to control?How to set systems parameters?

SUSIESupervising Intelligent Systems based

on Swarms

Interaction management

• Pheromon grids, large amount of subsystems, multiple localized events,”fuzzy” areas ...

• Need to increase the power of expression for the interpretation and for the generation of interaction• New modalities to interact on these

objects• Reflection on an adequate model of

interaction

Control the self-organized systems

Keep human in/on the loopAdapt the algorithm to the new modes of interaction• Speak the language of the

operator• A : Deal with topological +

operational concepts• Control more accurately but

not too much• B : Mix different modes of

control

Changing the point of viewA : Dealing with topological / technical aspects

Mobile Point Open lineClosed

line Area

Watch see followbear regul. a sensor on a given point

bear regul. a sensor on all points / line

bear regul. a sensor on all points /cl. line

bear regul. a sensor on all points / area

Avoid e.g. coll. avoidance

see closed line reduced to a point

do not cross a line

do not cross a given

closed line

get out area if inside

Find find targets systematic search of a

point

find at least one point of

a line

see open line with inside/

outside

see closed line

Follow track N/Amaneuver in

order to align on a line

see open line N.A

Intercept cross mobiletrajectory

N/Amaneuver so as to meet an

open linesee open line N/A

Changing the point of viewB : Mixing modes of control

• Keep pheromone as a basic principle for self organization and surveillance

• Add use of way-points for specific tasks• reach directly the area to patrol• come back to area when shifting because of speed• watch a point (see previous table)

• Add other use of pheromones to extend the expressiveness of commands

• complete exclusion zone (avoid an area - see previous table)• dynamic map of pheromone (work in progress)

Presentation of SUSIE

see video at:

recherche.telecom-bretagne.eu/susieor

deev-interaction.com/project-susie/

Human factors experiment (show)

Human factors experiment (no show)

Experiment configuration

• 47 people

• Non expert but trained

• 15 minutes scenario

• No show (24 people) / Show (23 people)

Same age mean

Experiments conclusions

Show No show

Better time of identification Better results in identification

Less manual transfer of UAVs Less failure in identification

Better usability perceived Lower perceived workload

Lower perceived cognitive load

Lower perceived time pressure

???

!!!

And a NATO price!

http://www.dailymotion.com/video/xjfmut_recherche-exploratoire-susie-supervision-de-systemes-d-intelligence-en-essaim_news

Demonstration French Air Show Le Bourget 2011

Demonstration French Air Show Le Bourget 2011

http://recherche.telecom-bretagne.eu/susie

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

New trends and challenges in dialogue management for human supervisory

control

Did we fix everything?We proposed:

! A user-centered level of language! A way to make algorithms (and machines) compatible! A new intuitive mode of interaction (tabletop)

! but

! what about the cooperation?! what about more complex systems ... needing more complex commands?

Supervisory control is an InteractionSupervisory control is an interaction…

! Between a supervisory and one or more subordinates

! Nature of supervisor! Nature of subordinates! Nature of tasks! Nature/range of subordinate behaviors! Nature of world (operating conditions)

! Reliability! Trust! Criticality/Importance! Social/Emotional issues! Autonomy

from Chris Miller / SIFT 2012

Human-Human Supervisory Control Examples

A Parking ValetA Postal clerk/Fast Food ChefShepherd to sheepdogA young child (~3 yrs)A teenager (~16 yrs)A new secretary/asstA conciergeQuarterback (Am.) to team Project Manager to teamCEO to corporationPresidential AideMovie Director’s GopherRadar O’Reilly

from Chris Miller / SIFT 2012

Describing the Interaction (Simply)

• How frequently do I have to interact?• How explicitly/expressively?• What kinds of things can I ask for?• Two dimensions proposed:

• Intervention Demand = how much time/attention do I have to spend commanding and monitoring for useful work?• How much time must I spend managing the automation vs.

how much time does it perform independently• Scope = what range of functions can the automation provide?

from Chris Miller / SIFT 2012

Scoring Supervisory Control Examples

A Parking ValetA Postal clerk/Fast Food ChefShepherd to sheepdogA young child (~3 yrs)A teenager (~16 yrs)A new secretary/asstA conciergeQuarterback (Am.) to team Project Manager to teamCEO to corporationPresidential AideMovie Director’s GopherRadar O’Reilly

Frequency11610873243331

Scope12326644689910

Useful?+++----++++

++++++

Ratio1.525

1.31.1.75.5.67.38.33.33.1

from Chris Miller / SIFT 2012

A (Partial) Utility Threshold?

Radar O’ReilyValet

Postal Clerk

Sheepdog

Child

Teenager

New Secretary

Concierge

QB to Team

PM to Team

CEO to Corp

Presidential Aide

Gopher

1

10

110Scope

Frequency from Chris Miller / SIFT 2012

New kind of Operator Support System

Vehicles Command & ControlSituation AwarenessDSS & Automation

New kind of Operator Support System (II)

Vehicle Command & ControlSituation Awareness

Automation

Interaction ManagementSemantic Bridge

Automation

Interaction Management+

Situation AwarenessTwo tasks:

1. mission

2. interaction

Need for Semantic Bridge

same principles upstream

interface manipulation

Need for Interaction Management

• New characteristics of UVSs and their OSSs, e.g.• Several vehicles• More automation, decision support• More complex interactions

• Several concurrent tasks• Manage threads of interaction• Priorities, turn-taking, interruption, change of

context, etc.

interactive

trajectory

planner

inter-UV CoordinationMore automation, decision support

multiple failures or events

Priorities, turn-taking, interruption, change of

multiple payloads + automation

Interaction as a Collaborative Activity

Traditional view [Shannon]: unidirectional, one-m e s s a g e , a d d r e s s e e i s p a s s i v e , n o n -understandings are errors

Collaborative view [Clark]: shared goal, shared effort, interactive refinement, feedbacks

Interaction is a Subordinate Activity

Execute the mission Interact with the OSS

The operator does two things:

>more important

Perfect understanding is not requiredSufficient for the current purpose: mission & context

Shared effort, but unequal contributions

Adjusting the Interaction Load

Non-Understanding

Give up / postpone

“Hold on. Intruder detected at XY”

Ask for recast “Please rephrase”

Request refinement

“Which building?”

Propose refinement

“Do you mean the building North of the airport?”

Understanding

understanding

= positive

feedback

Mor

e co

oper

ativ

e st

rate

gies

Adjusting the Interaction Load

Non-Understanding

Generation & Interpretation

Give up / postpone Keywords

Ask for recast Selfish

Request refinement Cooperative

Propose refinement Mutual awareness

Understanding

understanding

= positive

feedback

only one’s

beliefs

only the addressee’s

beliefs

Mutual awarenesscommon beliefs

Mor

e co

oper

ativ

e st

rate

gies

Why Not Always Be Cooperative?

Maintain the operator’s SAImprove the feeling to be working with the OSSConstruct and improve a common language & representations (e.g. “selfish” at first enables “cooperative” later)Develop more accurate models of each otherUnusual operator’s demand, critical task, etc.

with the OSS

similar to “adverse effects of

automation”

*loss of SA

*complacency

*skill degradation

[Parasuraman et al.]

Summary

• Consider interaction as a task, not only as a an overhead

• Principles similar to human-human interaction/dialogue

• The cooperative attitude of the system can be adjusted

• Interaction load adjustment could be useful to mitigate mission workload

Next step: Trust management

Trust

• A major indirect effect of dialog management• Within a team:

• mutual modeling (task driven)• interaction (speech turn, style, etc. impact trust)

• Constructive / based upon observations

Lee’s Model

Trust and interaction

Trust and interaction

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Conclusions

Main asserts

• Interaction is a key concept in the control of autonomous systems

• Interaction must be processed as a task by itself• Interaction has to remain intuitive and close to

operators high-level representations as the autonomous system has to interpret these representations in its own OODA space

(Many) Remaining challenges

Operator multi-tasking / cognitive load• Attention and focus

• Quantitative models

Trust in automation• Extrinsic factors (role of alarms, task environment)

• Intrinsic factors (self-confidence, sharing mental models)

• Situation awareness• Task and mode switching

• Error diagnosis and recovery

(Many) Remaining challenges

• Understanding the operator(s)

• Physiological and cognitive assessment of the operator’s state

• Assessing team cohesion and performance

• Systems of systems

• UAVs within the complete system

• Insertion of UAVs in civil air traffic / regulation

• Psycho-sociological aspects: UAV operators

• Lack of link between UAV operators and field forces

• Ruptures of contexts

Any questions?

gilles.coppin@telecom-bretagne.eu