1 Ruben Strenzke 2011 Assuming the Human‘s Cognitive State as Basis for Assistant System...

1Ruben Strenzke 2011

Assuming the Human‘s Cognitive Stateas Basis for Assistant System Initiative

© UniBwM / LRT-13 / 2011

Ruben Strenzke

Universität der Bundeswehr MünchenFaculty of Aerospace SciencesInstitute of Flight Systems (IFS)

Dagstuhl Seminar on Plan Recognition03.-08. April 2011


• Projects‘ Background– Manned-Unmanned Teaming Application– The Cooperative Automation Approach

for Assistant Systems

• Assuming the Human‘s Cognitive State– Project 1: Mixed-Initiative Mission Planning– Project 2: Operator Workload Estimation

Agenda


Unmanned Aerial Vehicle (UAV) Guidance TrendOperator-to-vehicle ratio

n : 1

1 : 1

1 : n

Helicopter commander

onboardGroundControlStation

1 : n1 : n

Today

Future


Manned-Unmanned Teaming (MUM-T) MissionHB Homebase

C Corrdidor

PZ Pickupzone

DZ1 Dropzone

DZ2 Alt Dropzone

L Leg

N

HB

FLOTFLET

C3

DZ1

C2

L1

L2

L3

PZ

C1

DZ2

L4


MUM-T Human-Machine Concept [Strenzke&Schulte 2011]

MUM-T Work System

Supervisory Control(Task-based Guidance)

Cooperative

Control (M

ixed-Initiative)

proactive,rational,

multi-aircraft-oriented agent

rather reactive,greedy and

single-aircraft-oriented agents


MUM-T Research Simulator [Uhrmann et al. 2009, 2010]


Multi-UAV Guidance: Human-Machine Interface

Task-basedinteraction

RecconaissanceTasks

UAV selection

Operator willbe overtaxed!(multiple UAVs,

mission replanning)


Requirement 1:The assistant system has to guide the attention of the human operator to the most urgent task.

Requirement 2 : If the human operator cannot carry out the most urgent task because of overtaxing, then the assistant system has to take initiative to transfer this situation into one which can be handled normally by operator.

Requirement 3:If there are tasks, the human operator is not capable to accomplish, or which are of too risky or costly, these tasks have to be allocated to the automation.

Basic Requirements for Assistant Systems

Cognitive Automation Approach [Onken&Schulte 2010]

Assistant system shall provide: Situation awareness improvement, work load reduction, error avoidance, error elimination…

but be silent and invisible most of the time – cannot be accessed by human initiative

knows and follows the same work object as the human does


Assistance System: Human-Machine Interface

• Speech synthesis monolog and display dialog

• 3 escalation steps: advice proposal generation task re-allocation


Example for Mixed-Initiative Dialog

• Assistant System takes initiative: “UAV1 needs follow-up task” • Operator presses proposal-button• Assistant System proposes: “add task transit A B for UAV1” • Operator presses accept-button • Assistant System affirms: “added task transit A B for UAV1”• asf.

Assistant Sytem needs to solve these questions:- What is the operator planning?- What is he/she doing?- Is his/her plan good?- Does he/she follow the plan or make errors?- Is he/she overtaxed?


• Projects‘ Background– Manned-Unmanned Teaming Application– The Cooperative Automation Approach

for Assistant Systems

• Assuming the Human‘s Cognitive State– Project 1: Mixed-Initiative Mission Planning– Project 2: Operator Workload Estimation

Agenda


Regarding the Human Operator from Outside

• Application-specific operator model:(disregarding intermediate goals in hierarchical plans, and top-level goals are known)

goals

plan

action

+ ep

+ ea

has

has knows

measuresestimates

chooses


Regarding Human Error and its Source

workload

goals

plan

action

behavior + eb(e.g. + sweating)

effects

+ ep

+ ea

has

has knows

measuresassumes

estimates

Cognitive state

shows

mea

sure

sgenerates

suffe

rs fr

om

chooses

First project:Mixed-Initiative Planning


Online Plan Recognition/Reasoning by Planning

I generate a plan and enter the UAV tasks

into the system.

I cannot know what the human is planning because the entering of the plan is never explicitly finished, but I

treat each planned UAV task as a constraint and I try to solve the

mission problem while regarding these additional constraints.

If there is no solution, the human must have made a mistake (or

problem is unsolvable. If there is a solution, I can check its costs

against my own solution and also check if the human forgot to enter

any tasks. online

possible plan set generation


Mixed-Initiative Mission Planner (MMP) Concept

4. Assistant system takes initiative on basis of assumed human plan

(action scheduling, missing plan elements)

or reference plan(suboptimal assumed human plan,

infeasible system plan)

5. Assistant system urges humanto make system plan convergeto one of its plans if necessary

1. System plan is enteredincrementally

partial or complete human plan

2. Assistant System hasdirect access to system plan

3. Assumed human plan is apossible completion of the

system plan, i.e. it shall converge to human planwith every detail added

to the system plan


Slave MMP

Free MMP

Assistant Core

Diverse informationsources

Mission Control

There is a new objective!

UAV Operator

I want UAV1 to... and helo to…

Ground vehicle spotted!

Mission + userconstraints

Mission

constraints

Assumed human plan Reference plan

Current ta

ctical si

tuation Mission orderAircraft

tasks

The MMP is based on the PDDL 2.2 compatible planner

„LPG-td“ [Gerevini et al. 2004]

user goals,user plan!

non-probabilistic,non-hierarchical, CWA


Given and User Goals Example

Free

Slave


Operator Workload Estimation

workload

goals

plan

action

behavior + eb(e.g. + sweating)

effects

+ ep

+ ea

has

has knows

measuresassumes

suffe

rs fr

om estimates

Cognitive state

shows

generates

chooses

mea

sure

s

Second project:Workload Estimation


Workload Estimation by HMMs [Donath et al. 2010, 2011]

search for objects identify object further tasks...

normal (excl. ea/eb) HMM1 HMM3 ...

too high (incl. ea/eb) HMM2 HMM4 ...

taskworkload

Video!

- Workload can be estimated by regarding operator behavior

- behavior differs between tasks and workload levels

- Measuring of visual and manual interaction with the system

- has been done offline

- for one person:


Challenges: -Online Generative Plan Recognition-Having Data-Fragments from Different Process Steps

workload

goals

plan

action

behavior + eb

effects

+ ep

+ ea

has

has

Cognitive state

shows

If we have assumptions abouthis plan, it‘s easier to

assume what he is doing!

If we have assumptions abouthis goal(s), it‘s easier to

assume what he is planning!

generates

suffe

rs fr

om

chooses

knows

measuresassumes

estimates

mea

sure

s


The Obligatory Last Slide

Thank you very much for your attention!

Comments and questions welcome

[email protected]

1 Ruben Strenzke 2011 Assuming the Human‘s Cognitive State as Basis for Assistant System...

Documents

Transcript of 1 Ruben Strenzke 2011 Assuming the Human‘s Cognitive State as Basis for Assistant System...