Post on 10-Jan-2017
Astronauts and IoT: Toward True Human-Autonomy Teaming
Dr. Alonso VeraChief, Human Systems Integration Division
Nordic Digital Business Summit 2016Helsinki, Finland
22 September 2016
05/01/2023 2
OverviewThree Themes- Not designing autonomous systems to interact
with humans increases costs- The need ongoing need for human engagement
with increasingly intelligent systems (IISs)- Both cognitive and physical
- IoT allows IISs to reach into the physical world in an interactive way that has not been possible until now
Machine IntelligenceWe appear to be at an exciting time with respect intelligent machines (again). • Four Related Areas of Development
1. Big Data - volume, velocity and variety2. Deep Learning3. Networked operations and cyber-physical systems4. Moore’s Law (exponential growth, doubling of components on an integrated circuit
every two years): faster, bigger computers driving change with increasing velocity
• Stephen Hawking, Bill Gates and Elon Musk have all recently warned about the potential dangers of AI.
• Also interesting time in terms of self-driving cars and companies with robotic operations/factories like Amazon, Tesla and Toyota
• Big Blue, Watson, Pokerbot• Google DeepMind AI Division beats human at GO (Jan 2016)• First AI investment software hits Wall St. (Feb 2016)
Manpower Reduction: Start with the Human (Not the Technology)The Autonomy Paradox (Blackhurst, Gresham & Stone, 2011)
• Autonomy doesn’t get rid of humans, it changes their roles
• DoD has shifted from Levels-of-Automation to Cognitive Echelons
As machine intelligence advances, the need for better human interfaces increases
The Littoral Combat ShipBuilt to be operated by 45 sailors
Dr. Larry Shattuck, NPS (pg. 13-15) http://human-factors.arc.nasa.gov/workshop/autonomy/download/presentations/Shaddock%20.pdf
Recent DevelopmentsThe Littoral Combat Ship• New, highly autonomous
vehicle
In the end, the ship required 60 sailors, all E5 or above
… and they are still encountering major issues
05/01/2023 6
Apple Research on Teaming of IISs with Humans
http://www.cultofmac.com/444551/why-apple-is-the-new-nasa/?&tc=em
• Characterizing calorie burn during swimming and using learning algorithms to tune the functionality to individual differences
• Developed novel experimental hardware and tested on 700 swimmers• To develop a feature on one app for the new iWatch
Path to Collaborative, Human-in-the-Loop Planning Systems
NEEMO
NASA Extreme Environments Mission Operations (NEEMO): Underwater laboratory off of the coast of Key Largo, Florida.
International Space Station CAST – Crew Autonomous Scheduling
Test: • Playbook Check-Out by Scott Kelly on
Station in August 2015. • Astronaut Peggy Whitson trained on
Playbook in July 2016.• Astronaut self-scheduling study on ISS on
Mission Increment 50/51 Nov 2016.
ISS Mission Control• Three integrated planning
systems: Power, Attitude Control and Crew Activity
• Crew activity includes ESA JAXA and Payloads
05/01/2023 10
What are the Challenges of Working with IISs?• Lack of transparency about intent, state awareness,
risk/confidence posture, etc.
• Part of the challenge is just the reverse though. Given that the IIS does not know what the human is trying to do, it is difficult for the system to know to engage in ways that are useful.
Toward Human-Autonomy Teaming?
Caged Robots
Domesticated Robots
Designed for Teaming
VIDEO REMOVED TO
DECREASE FILE SIZE
05/01/2023 13
IoT and IISs• IoT as a way for increasingly intelligent systems to
have awareness of state in the physical world• Addressing the AI grounding problem• Beyond machine vision
• Applied to manual human tasks• Assembly, maintenance, repair• Reduced training, errors
• Task execution support • For complex tasks• For infrequently performed tasks
05/01/2023 14
Assembly, Maintenance and Repair on International Space Station The Rodent Habitat
05/01/2023 15
How Can IoT Help?Challenge: loss of tools and loss of training• Tools and parts with embedded sensors • Data from sensors transmitted to the network by object• Objects know what they and can also capture and
communicate other information about themselves• Location• Calibration status• History of failures, adjustments, repairs• Motion, torque, force, orientation, relative proximity
May allow intelligent/adaptive procedure execution systems to guide humans though complex tasks with little or no training, as well as detect errors and reduce performance time
Summer 2016 Student Intern Project
Internet of ThingsMicrosoft Hololens
Combine Augmented Reality and Internet-of-Things
Mission
AR-IOT
Existing ProcedureMission
AR-IOT
AR-IOT
Dividing procedure execution research into two parts
Mission
AR-IOT
Task Procedure Path Visualization
AR-IOT
IoT
IoT Pickup Detection
AR-IOT
IOT Technology (server) IoT
AR-IOT
Prototype
Storyboards
AR-IOT
Voice command opens the
selection menu.
Voice or gesture selects the tools option
User selects all wanted tools. Feedback occurs for each selection. Once ready, user selects “Find” to begin path.
1 2 3
Storyboards
AR-IOT
Menu begins to guide user.
Simple HUD cues user to the destination.
Spaced arrows lead user to location. HUD allows user access to a menu to cancel or change a path.
Location has hologram of item over the storage compartment.
4 5 6
Storyboards
AR-IOT
Multiple of similar tools in a
compartment with a hologram over the correct
one. HUD changes to icons
of the tools.
Tool is secured in the compartment and the user picks it up.
Haptic feedback and change of HUD when correct tool is located. Progress indicator fills in the current tool.
7 8 9
Storyboards
AR-IOT
When tool is located the path changes to begin
guide to next location. HUD displays new location goal.
Destination HUD changes to icon HUD.
Line leads directly to the desired tool. A holographic overlay is also placed over the specific tool.
10 11 12
Storyboards
AR-IOT
Haptic and another change of HUD signal found object.
Path changes to lead user to final destination. HUD also indicates location.
Arrow or highlight of doorhandles indicates user to open the door (or module)
13 14 15
Storyboards
AR-IOT
16
Once the user returns to the procedure location there will be indicators of places where they will need to install the tools based off of the IOT sensors and holograms.
Once tools have been placed you
can begin the procedure.
The first step is to place the tools you've found in a secure bag. The user will see a holographic animation that shows you how to do this.
17 18
Step 2 is to attach your bag to the habitat. These animations are similar to the items own movement.
19
Storyboards
AR-IOT
20
When the item is placed
successfully there is a
holographic blinking change
as well as an update in your
heads up display.
Step 3. Open top access hatch. The animation will show the hatch opening and closing.
21
22
Storyboards
AR-IOT
23
Content such as videos or Skype
calls can be displayed on the wall to provide support for the
astronauts while they conduct procedures.
Microsofts “Xray” feature can show objects containing other objects so users view isn't obstructed.
User Test 1B Recording
VIDEO REMOVED TO
DECREASE FILE SIZE
Insights & Opportunities
AR-IOT
Gestures (Select Tools)
Many users said they preferred using voice over hand gesture but during user tests participants tended to use both.“Voice is more comfortable because I wasn’t sure about the depth of my gestures.”
“Instead of navigation through menus there should be a way to simply say, let’s find the flashlight.”
Insights & Opportunities
AR-IOT
Guidance (Path)
Some users felt that they didn’t need such an excessive guidance system which at times could feel unnecessary.“It would be condescending to an astronaut to tell them exactly where to go”
Insights & Opportunities
AR-IOT
Arrow Spacing (Path)
Using calculated arrow spacing along side a linear path gave users an easy guidance system to locate their desired tool.“The arrows were positioned just right so I knew exactly where to turn”
“Following the line was simple and intuitive”
Insights & Opportunities
AR-IOT
Destination (Locate Tool)
Placing a hovering hologram over tools precise location proved to be a successful device to mark the destination.“I noticed the hologram and my immediate reaction was to go there.”
“I saw a star, so it was pretty clear to me that something was there.”
Insights & Opportunities
AR-IOT
Precise Location (Locate Tool)
A single line on the floor leading to a tools exact location was a successful method of marking the destination because the user could follow the line of tape with their eye to the exact destination. “You could clearly see that the highlighted drawer was where the item is”
Insights & Opportunities
AR-IOT
Visual Feedback (Locate Tool)
Visual feedback after task completion was needed after the user found each tool to ensure they were on the right path. “I didn’t have any doubts that I have found the right flashlight but maybe if there was a dozen it would have been more helpful.”
“Visual feedback would definitely be more helpful if there were different types of flashlights”
Insights & Opportunities
AR-IOT
Using Memory (Return)
Many users were able to return to the procedure room while only using their memory and not relying on our guiding system.“To me it was pretty clear because I knew where this room was so I mostly tuned the path out”
Journey Map
AR-IOT
AR-IOT
Procedure Execution
Animations & Procedure
VIDEO REMOVED TO
DECREASE FILE SIZE
AR-IOT
Procedure Execution
Animations & Procedure
Holographic animation
Procedure Execution
AR-IOT
HUDWe wanted an unintrusive heads up display that placed most of the information within the line itself.
Storyboards
Final Test
VIDEO REMOVED TO
DECREASE FILE SIZE
From Cummings, M.L., "Man vs. Machine or Man + Machine?" IEEE Intelligent Systems, (2014) 29(5), p. 62-69.
2015
Architecture based on autonomy performing all skill and rule-based roles, as well as most knowledge-based roles. Manpower reduced by two orders of magnitude with remaining expert humans teaming with machine intelligence to solve complex problem solving under uncertainty. Machine intelligence for airspace management evolves from the outset to support teaming with small set of expert humans to support cooperative problem-solving.
Adaptive
2035
Computers Humans
e.g, current HITL for ATM Next-Gen research
Affordances from the Environment
Using Affordances
• Application of Gibsons’ Ecological Psychology• Alternatives to using
human central attention resource
• A car more like a horse
Teaming of Human and Machine Intelligence• Even as computers get very “intelligent”, it is very likely that the nature
of the their intelligence will be different than that of humans (unless they become omniscient or we program them to function just like humans)
• Humans are particularly good at adaptive problem-solving and discovery, areas where there has been little machine intelligence progress
• Successful efforts going forward will be those that wrap new machine intelligence capabilities around human competencies in order to get the most out of each
Goal: Design the human into the process. Focus on how the system will communicate it’s state to the human so that the human can help in un-anticipated situations, and vice versa. What data and how it is presented to each agent such that each can bring its unique capabilities to bear on it.
The Economics of Human-Centered Automation
• For lower costs, higher efficiencies and overall improved system performance:• Characterize nature of human roles
(skills, rules, knowledge, expertise) and tasks (e.g., proportion of hard and soft constraints)
• Wrap autonomy around remaining human roles from the beginning
Dr. Jon Bornstein, DoD Autonomy Roadmap Autonomy Community of Interest http://www.defenseinnovationmarketplace.mil/resources/AutonomyCOI_NDIA_Briefing20150319.pdf
Critical to shape the autonomy industry• e.g., Apple v. Littoral Combat Ship
05/01/2023 51
Final Thoughts• Humans will remain important components of complex
systems• Use human adaptive expertise as much as possible• Use human cognitive & perceptual system as much as
possible in interactions • Robotics progressing faster than AI• Be aware of areas where you don’t have big data
• Not all problems are associative in nature
• Don’t assume search will solve all problems
Thank you