Astronauts and IoT: Toward True Human-Autonomy Teaming

Dr. Alonso VeraChief, Human Systems Integration Division

Nordic Digital Business Summit 2016Helsinki, Finland

22 September 2016

05/01/2023 2

OverviewThree Themes- Not designing autonomous systems to interact

with humans increases costs- The need ongoing need for human engagement

with increasingly intelligent systems (IISs)- Both cognitive and physical

- IoT allows IISs to reach into the physical world in an interactive way that has not been possible until now

Machine IntelligenceWe appear to be at an exciting time with respect intelligent machines (again).  • Four Related Areas of Development

1. Big Data - volume, velocity and variety2. Deep Learning3. Networked operations and cyber-physical systems4. Moore’s Law (exponential growth, doubling of components on an integrated circuit

every two years): faster, bigger computers driving change with increasing velocity

• Stephen Hawking, Bill Gates and Elon Musk have all recently warned about the potential dangers of AI.

• Also interesting time in terms of self-driving cars and companies with robotic operations/factories like Amazon, Tesla and Toyota

• Big Blue, Watson, Pokerbot• Google DeepMind AI Division beats human at GO (Jan 2016)• First AI investment software hits Wall St. (Feb 2016)

Manpower Reduction: Start with the Human (Not the Technology)The Autonomy Paradox (Blackhurst, Gresham & Stone, 2011)

• Autonomy doesn’t get rid of humans, it changes their roles

• DoD has shifted from Levels-of-Automation to Cognitive Echelons

As machine intelligence advances, the need for better human interfaces increases

The Littoral Combat ShipBuilt to be operated by 45 sailors

Dr. Larry Shattuck, NPS (pg. 13-15) http://human-factors.arc.nasa.gov/workshop/autonomy/download/presentations/Shaddock%20.pdf

Recent DevelopmentsThe Littoral Combat Ship• New, highly autonomous

vehicle

In the end, the ship required 60 sailors, all E5 or above

… and they are still encountering major issues

05/01/2023 6

Apple Research on Teaming of IISs with Humans

http://www.cultofmac.com/444551/why-apple-is-the-new-nasa/?&tc=em

• Characterizing calorie burn during swimming and using learning algorithms to tune the functionality to individual differences

• Developed novel experimental hardware and tested on 700 swimmers• To develop a feature on one app for the new iWatch

Path to Collaborative, Human-in-the-Loop Planning Systems

NEEMO

NASA Extreme Environments Mission Operations (NEEMO): Underwater laboratory off of the coast of Key Largo, Florida.

International Space Station CAST – Crew Autonomous Scheduling

Test: • Playbook Check-Out by Scott Kelly on

Station in August 2015. • Astronaut Peggy Whitson trained on

Playbook in July 2016.• Astronaut self-scheduling study on ISS on

Mission Increment 50/51 Nov 2016.

ISS Mission Control• Three integrated planning

systems: Power, Attitude Control and Crew Activity

• Crew activity includes ESA JAXA and Payloads

05/01/2023 10

What are the Challenges of Working with IISs?• Lack of transparency about intent, state awareness,

risk/confidence posture, etc.

• Part of the challenge is just the reverse though. Given that the IIS does not know what the human is trying to do, it is difficult for the system to know to engage in ways that are useful.

Toward Human-Autonomy Teaming?

Caged Robots

Domesticated Robots

Designed for Teaming

VIDEO REMOVED TO

DECREASE FILE SIZE

05/01/2023 13

IoT and IISs• IoT as a way for increasingly intelligent systems to

have awareness of state in the physical world• Addressing the AI grounding problem• Beyond machine vision

• Applied to manual human tasks• Assembly, maintenance, repair• Reduced training, errors

• Task execution support • For complex tasks• For infrequently performed tasks

05/01/2023 14

Assembly, Maintenance and Repair on International Space Station The Rodent Habitat

05/01/2023 15

How Can IoT Help?Challenge: loss of tools and loss of training• Tools and parts with embedded sensors • Data from sensors transmitted to the network by object• Objects know what they and can also capture and

communicate other information about themselves• Location• Calibration status• History of failures, adjustments, repairs• Motion, torque, force, orientation, relative proximity

May allow intelligent/adaptive procedure execution systems to guide humans though complex tasks with little or no training, as well as detect errors and reduce performance time

Summer 2016 Student Intern Project

Internet of ThingsMicrosoft Hololens

Combine Augmented Reality and Internet-of-Things

Mission

AR-IOT

Existing ProcedureMission

AR-IOT

AR-IOT

Dividing procedure execution research into two parts

Mission

AR-IOT

Task Procedure Path Visualization

AR-IOT

IoT

IoT Pickup Detection

AR-IOT

IOT Technology (server) IoT

AR-IOT

Prototype

Storyboards

AR-IOT

Voice command opens the

selection menu.

Voice or gesture selects the tools option

User selects all wanted tools. Feedback occurs for each selection. Once ready, user selects “Find” to begin path.

1 2 3

Storyboards

AR-IOT

Menu begins to guide user.

Simple HUD cues user to the destination.

Spaced arrows lead user to location. HUD allows user access to a menu to cancel or change a path.

Location has hologram of item over the storage compartment.

4 5 6

Storyboards

AR-IOT

Multiple of similar tools in a

compartment with a hologram over the correct

one. HUD changes to icons

of the tools.

Tool is secured in the compartment and the user picks it up.

Haptic feedback and change of HUD when correct tool is located. Progress indicator fills in the current tool.

7 8 9

Storyboards

AR-IOT

When tool is located the path changes to begin

guide to next location. HUD displays new location goal.

Destination HUD changes to icon HUD.

Line leads directly to the desired tool. A holographic overlay is also placed over the specific tool.

10 11 12

Storyboards

AR-IOT

Haptic and another change of HUD signal found object.

Path changes to lead user to final destination. HUD also indicates location.

Arrow or highlight of doorhandles indicates user to open the door (or module)

13 14 15

Storyboards

AR-IOT

16

Once the user returns to the procedure location there will be indicators of places where they will need to install the tools based off of the IOT sensors and holograms.

Once tools have been placed you

can begin the procedure.

The first step is to place the tools you've found in a secure bag. The user will see a holographic animation that shows you how to do this.

17 18

Step 2 is to attach your bag to the habitat. These animations are similar to the items own movement.

19

Storyboards

AR-IOT

20

When the item is placed

successfully there is a

holographic blinking change

as well as an update in your

heads up display.

Step 3. Open top access hatch. The animation will show the hatch opening and closing.

21

22

Storyboards

AR-IOT

23

Content such as videos or Skype

calls can be displayed on the wall to provide support for the

astronauts while they conduct procedures.

Microsofts “Xray” feature can show objects containing other objects so users view isn't obstructed.

User Test 1B Recording

VIDEO REMOVED TO

DECREASE FILE SIZE

Insights & Opportunities

AR-IOT

Gestures (Select Tools)

Many users said they preferred using voice over hand gesture but during user tests participants tended to use both.“Voice is more comfortable because I wasn’t sure about the depth of my gestures.”

“Instead of navigation through menus there should be a way to simply say, let’s find the flashlight.”

Insights & Opportunities

AR-IOT

Guidance (Path)

Some users felt that they didn’t need such an excessive guidance system which at times could feel unnecessary.“It would be condescending to an astronaut to tell them exactly where to go”

Insights & Opportunities

AR-IOT

Arrow Spacing (Path)

Using calculated arrow spacing along side a linear path gave users an easy guidance system to locate their desired tool.“The arrows were positioned just right so I knew exactly where to turn”

“Following the line was simple and intuitive”

Insights & Opportunities

AR-IOT

Destination (Locate Tool)

Placing a hovering hologram over tools precise location proved to be a successful device to mark the destination.“I noticed the hologram and my immediate reaction was to go there.”

“I saw a star, so it was pretty clear to me that something was there.”

Insights & Opportunities

AR-IOT

Precise Location (Locate Tool)

A single line on the floor leading to a tools exact location was a successful method of marking the destination because the user could follow the line of tape with their eye to the exact destination. “You could clearly see that the highlighted drawer was where the item is”

Insights & Opportunities

AR-IOT

Visual Feedback (Locate Tool)

Visual feedback after task completion was needed after the user found each tool to ensure they were on the right path. “I didn’t have any doubts that I have found the right flashlight but maybe if there was a dozen it would have been more helpful.”

“Visual feedback would definitely be more helpful if there were different types of flashlights”

Insights & Opportunities

AR-IOT

Using Memory (Return)

Many users were able to return to the procedure room while only using their memory and not relying on our guiding system.“To me it was pretty clear because I knew where this room was so I mostly tuned the path out”

Journey Map

AR-IOT

AR-IOT

Procedure Execution

Animations & Procedure

VIDEO REMOVED TO

DECREASE FILE SIZE

AR-IOT

Procedure Execution

Animations & Procedure

Holographic animation

Procedure Execution

AR-IOT

HUDWe wanted an unintrusive heads up display that placed most of the information within the line itself.

Storyboards

Final Test

VIDEO REMOVED TO

DECREASE FILE SIZE

From Cummings, M.L., "Man vs. Machine or Man + Machine?" IEEE Intelligent Systems, (2014) 29(5), p. 62-69.

2015

Architecture based on autonomy performing all skill and rule-based roles, as well as most knowledge-based roles. Manpower reduced by two orders of magnitude with remaining expert humans teaming with machine intelligence to solve complex problem solving under uncertainty. Machine intelligence for airspace management evolves from the outset to support teaming with small set of expert humans to support cooperative problem-solving.

Adaptive

2035

Computers Humans

e.g, current HITL for ATM Next-Gen research

Affordances from the Environment

Using Affordances

• Application of Gibsons’ Ecological Psychology• Alternatives to using

human central attention resource

• A car more like a horse

Teaming of Human and Machine Intelligence• Even as computers get very “intelligent”, it is very likely that the nature

of the their intelligence will be different than that of humans (unless they become omniscient or we program them to function just like humans)

• Humans are particularly good at adaptive problem-solving and discovery, areas where there has been little machine intelligence progress

• Successful efforts going forward will be those that wrap new machine intelligence capabilities around human competencies in order to get the most out of each

Goal: Design the human into the process. Focus on how the system will communicate it’s state to the human so that the human can help in un-anticipated situations, and vice versa. What data and how it is presented to each agent such that each can bring its unique capabilities to bear on it.

The Economics of Human-Centered Automation

• For lower costs, higher efficiencies and overall improved system performance:• Characterize nature of human roles

(skills, rules, knowledge, expertise) and tasks (e.g., proportion of hard and soft constraints)

• Wrap autonomy around remaining human roles from the beginning

Dr. Jon Bornstein, DoD Autonomy Roadmap Autonomy Community of Interest http://www.defenseinnovationmarketplace.mil/resources/AutonomyCOI_NDIA_Briefing20150319.pdf

Critical to shape the autonomy industry• e.g., Apple v. Littoral Combat Ship

05/01/2023 51

Final Thoughts• Humans will remain important components of complex

systems• Use human adaptive expertise as much as possible• Use human cognitive & perceptual system as much as

possible in interactions • Robotics progressing faster than AI• Be aware of areas where you don’t have big data

• Not all problems are associative in nature

• Don’t assume search will solve all problems

Thank you