1

Beyond Human Factors: An Approach to Human/Automation TeamsHaomiao Huang Jerry Ding Wei Zhang Claire J. TomlinHybrid Systems LabAction Webs Meeting11/17/2010

2[nasa.gov, businessweek.com, tgdaily.com, techeasy.co.za, deere.com, aurore-sciences.org]

Advances in complex multi-agent systems require smart integration of human elements.

3

[foxnews.com] [wikipedia]

[media.weirdworm.com]

[knowyourmeme.com]

[adriandayton.com]

This requires new approaches to analyze humans as part of the system!

Let’s think about humans as part of the solution, not the problem.

4

Two related problems

2) Control - generating useful directives and controls for human agents

1) Modeling- Properly representing humans as components of the overall system

5

OutlineMotivation

Scenario for Research on Human/Automation Teams

Adversarial Game Problem

Reachability Based Approach

Results

Conclusions & Future Work

6

Choosing a Research ScenarioGames are representative of hard, real-world problems, yet provide relatively benign “sandbox” environments for development

Robocup

Chess

What is a good game to capture the aspects of human-automation teams that we want to explore?

Starcraft

Roboflag

7

Time tested and fun

Capture-the-FlagCapture-the-flag embodies the basic research challenges we are trying to address

http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag

Limited InformationMultiple AgentsCompeting Objectives

Human playersAdversarial

8

Automation-Assisted Human Capture-the-FlagUsing mobile phones, computers, and UAVs, we have turned capture-the-flag into a testbed for advanced automation concepts involving human team members

Game software on

Android phones

STARMAC Quadrotor

UAVs

Server-side Management Software

9

Time tested and fun

Narrowing the problem

http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag

Limited InformationMultiple AgentsCompeting Objectives

Human playersAdversarial

10

OutlineMotivation

Scenario for Research on Human/Automation Teams

Adversarial Game Problem Problem statement Related Work Solution Insights

Reachability Based Approach

Results

Conclusions & Future Work

11

Our Problem

Capture Region

Defender

Attacker

Flag

Flag Region

Return Region

Game Domain

Characterize and solve a 1-sided capture-the-flag game with a single attacker and defender

12

Related Work on Adversarial Games Multi-agent games on discrete state spaces

Greedy search

Hespanha, Kim, and Sastry 1999

Approximate DP/Reinforcement Learning

Lagoudakis and Parr 2002

Discrete Play Matching

Browning, Bruce, and Veloso 2005

Pursuit-evasion games with continuous statesReceding-Horizon Control

Mcgrew, How, Bush, Williams and Roy 2008

Sprinkle, Eklund, Kim, and Sastry 2004

Optimal Trajectory PlanningEarl and D’Andrea 2001

Chasparis and Shamma 2005

Analytical game theory approaches

Basar 1989, Lewin 1994,

Stipanovic, Melikyan, Hovakimyan 2010

Hamilton-Jacobi Reachability

Mitchell, Bayen, and Tomlin, 2005

Ding, Sprinkle, and Tomlin 2008

Assumed, learned, or randomized opponent model

13

ReachabilityApproach, derived from pursuit-evasion games: CTF game can be posed as a reachability problem.

Assume system dynamics

Where is the input for Player I and is the input for Player II

Define as the reach-avoid set where a player can arrive in a goal region in at most time while avoiding region , no matter what the other player does

14

Capture-the-Flag as ReachabilityVictory conditions for each player can be encoded as reach-avoid sets in the joint state-space

Defender

Attacker

Joint Capture Set

Joint Return Set

Flag Return Set (For Attacker)

Game Domain

15

1-D Game

16

Geometric insightsGeometric analysis allows some insight into the 2-D capture-the-flag problem

17

Geometric insightsGeometric analysis allows some insight into the 2-D capture-the-flag problem

18

Utility of Reachability AnalysisReachability analysis gives complete characterization of game, and are a natural display tool for guiding human decision-making and allowing least-restrictive control

Teo and Tomlin, 2003

Geometric analysis is not terribly general, though…

19

OutlineMotivation

Scenario for Research on Human/Automation Teams

Adversarial Game Problem

Reachability Based Approach Hamilton-Jacobi Reachability Computation

Results

Conclusions & Future Work

20

Hamilton-Jacobi ReachabilityReachability in continuous state-spaces can analyzed as a terminal cost-only optimization problem, solved backward in time

Reachability Cost Function

Classic Optimal Control Cost Function

Tomlin 2009

21

Level-Set RepresentationSets can be represented using sub-level sets of signed distance functions as terminal cost functions

Set operations using point-wise minimum and maximums can be used to create arbitrary sets

Tomlin 2009, Mitchell 2003

22

Solution Based on HJBI Equation

The cost-to-go function is the unique viscosity solution to the Hamilton-Jacobi-Bellman-Isaacs equation

Classic Optimal Control Cost Function

Hamilton-Jacobi-Bellman-Isaacs Equation

Optimal Hamiltonian

23

Reachability Via Modified HJBI Equation

The backward reachable set is the zero sub-level set of the viscosity solution to a modified HJBI equation

Modified HJBI Equation

Optimal Hamiltonian

Reachability Cost Function

Mitchell, Bayen, Tomlin 2005

24

Numerical Solution to the Modified HJBI Equation

The viscosity solution to the modified HJBI Equation can be computed on a grid using the Level Set Toolbox from UBC

http://www.cs.ubc.ca/~mitchell/ToolboxLS/index.html

26

OutlineMotivation

Scenario for Research on Human/Automation Teams

Adversarial Game Problem

Reachability Based Approach

Results HJBI Reachability applied to capture-the-flag Simulation results Experimental setup

Conclusions & Future Work

27

Problem Formulation for 1v1 Capture-the-Flag

HJBI reachability analysis allows us to fully characterize the game

Dynamics

Optimal Hamiltonian

Optimal Inputs

28

Flag Return & Flag Capture

Winning regions for each portion of the game can be calculated directly from reach-avoid conditions

29

Sequenced Capture and ReturnWinning regions for the full sequence (flag capture and subsequent return) can be computed by using the intersection of the flag return set and flag zone as the initial condition for flag capture

30

Simulation ResultsSimulation results demonstrate the use of the reachability solutions

31

Field Experiments in Progress

Reachability-based control and input directives are being implemented on Droid Incredible phones

Game software on Android phones

Server-side Management Software

Player Positions and State

Reachable sets & optimal control inputs

32

OutlineMotivation

Scenario for Research on Human/Automation Teams

Adversarial Game Problem

Reachability Based Approach

Results

Conclusions & Future Work

33

ConclusionsCapture-the-flag is great platform for developing human-

automation systems research.

A differential game formulation using HJBI reachability solves perfect information, 1v1 CTF

34

Future WorkWe have the “correct” answer to the adversarial problem… now what?

http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag

Limited InformationMultiple AgentsCompeting Objectives

Human playersAdversarial

35

Thank you!Questions?