Safe Testing of Autonomy in Complex, Interactive Environments

(TACE)

Flight Software Workshop

December 16 – 18 2014

David Scheidt david.scheidt@jhuapl.edu Robert Lutz robert.lutz@jhuapl.edu William D’Amico bill.damico@jhuapl.eduSubodh Harmalkar subodh.harmalkar@jhuapl.edu

2

Outline

Test & Evaluation Need and Challenges

Background and Definitions

Current Effort under TRMC’s T&E/S&T program

Path to a Testing Capability

Summary

3

From Tele to Auto to Autonomous Operations

Unmanned systems usually require the operator to be a “user-in-the-loop”

APL has demonstrated the autonomous operation of multiple and cooperative UXVs where the operator is a “user-on-the-loop”

There have been many demonstrations of APL’s Mission Level Autonomy (MLA) under JHU/APL internal research and at Camp Roberts (CA)

MLA has been shown to be “platform” and “vendor” agnostic riding above proprietary control/guidance algorithms

MLA research led to insights on what would be needed for rigorous test and evaluation

UXV

Sensor Control

Autonomous Operations

Operationalobjectives

Measure effect

Measureeffect

Operatingenvironmentaldecisions

A measure of effectiveness of an autonomous system is directly related to

“operational objectives” thus making the test methodology and test results applicable to

both developmental and operational test environments – true joint DT/OT is needed

4

Systems with Autonomous Behaviors Are Here

ACTUV: ANTI-SUBMARINE WARFARE CONTINUOUS TRAIL

UNMANNED VESSEL

AMAS: Autonomous Mobility Appliqué System

AACUS: Autonomous

Aerial Cargo/Utility

System

LDUUV: Large Diameter UUV

5

Autonomy Test & Evaluation Challenge

Autonomous systems respond to unpredictable change by devising a course of action. Before we deploy such systems how can we be sure that autonomous decisions that will be produced will always: (1) achieve objectives set by human supervisors and (2) not produce unacceptable unintended consequences. Testing autonomous systems is particularly challenging since we cannot possibly test all interactions between the autonomous system and the natural world.

6

1970 1990 2000

Air-Exjam(later EXDRONE and BMQ-147 Dragon)

Scheer “Beast”

Autonomy S&T Vision Program

Predator/SSN Interoperability Demo

NEAR burn failureDARPA UUV Program

History of Autonomous Unmanned Systems at APL

1965 20102005

4 UGVs @ APG

2 UAVs & 4 UGVs

@ APG

1 UAV 1 UGS @ APG

Tactical Sensing 1 UAV & Multiple UGS

Unmanned SurfaceVehicles

3 UAVs, 2 UGS,3 Mobile Users

@ Dugway & DOE Range

TNT Experiments6 UAVs, 3 UGS

@ Camp Roberts

1 UUV

2015

OPISR – 4 UAVs, 1 UGV, 2 USV, 1 UUV, 3 UGS

@ Webster

Small Ocean-going USVs

In Atlantic, Pacific & Gulf of Mexico

JHU/APL has been developing autonomous unmanned vehicles since 1962.

Since 2002 JHU/APL has conducted dozens of autonomous vehicle flight programs and flown/launched hundreds of sorties. Currently JHU/APL has 24 active autonomy programs.

JHU/APL Proprietary

New Horizons

TACE

DADFS

Agile UxVAutonomy

7

Combined Autonomous Air/Ground Missions2004 Aberdeen Test Center

OSD NII Swarming Unmanned Vehicle Experiment – Cooperative Search, Patrol & Track (4 UGVs and 2 UAV)

Combinations of autonomous air and ground vehicles (an AACUS/AMAS collaboration) are clearly possible and probably required for many military missions

8

Challenges for Autonomous System Test

• Test & Evaluation (T&E) Need- Build an infrastructure for SAFE/LIVE testing of autonomous unmanned vehicles

(AUVs), especially for tactical UAVs that have significant airspace and platform-related range safety issues

- Demonstrate live, virtual, and constructive (LVC) methods for SAFE/LIVE tests

- LVC methods should be Test and Training Network Architecture (TENA) compliant

- To our knowledge there are no AUV established T&E capabilities with these features

• T&E Challenge– Provide an infrastructure that supports safe testing when testing autonomous systems

operating over the horizon or in denied environments.– Provide an infrastructure that supports safe testing when autonomous systems perform

unpredictable actions– Provide an accurate, real-time, live-virtual-constructive environment that interacts with the

system under test’s autonomy in unpredictable ways.

– Reliable command and control (C2) of the systems under test (SUT) and robust 2-way data links are needed to stimulate the SUT, to record behaviors, and to maintain SAFE/LIVE test control

9

Safe Testing of Autonomy in Complex, Interactive Environments (TACE) – Key Elements

Capability TACE is a 3-phase/36-month program (started in April 2013) where the open/general

architecture will be at TRL6 with initial capabilities for the testing (black and white box) of autonomous unmanned vehicles (AUVs) of all types. TACE is “portable” but will rely upon a “thin client” interface to the SUT and the inclusion on the SUT of a minimal “TACE test applique.” There are 3 levels of control/override within TACE: SUT autonomy, TACE automatic experiment controls, test range safety officer control

Relevance to T&E Scenarios The TACE architecture will use rigorous techniques to prevent unsafe AUV operations or

actions while stimulating, measuring, monitoring, and recording the AUV’s autonomous response/performance in complex environments. TACE assurance algorithms provide mathematically rigorous, on-board, real-time, safety guarantees, while TACE dual simulation system provides a complex live-virtual constructive test environment.

Cost Benefits A critical element, as always, is selecting the critical tests. TACE is not a planning tool.

A recent award was made to JHU/APL under the Unmanned and Autonomous System Test (UAST) program for an autonomy planning tool, Rapid Adversarial Planning Tool (RAPT). The “cost” of conducting tests where the “Achilles heel” of the embedded autonomous behavior is unknown would involve very high risk to the loss of the SUT and the delay of system fielding. The combination of tools such as RAPT and TACE should prevent unnecessary and expensive testing

10

TACE System Architecture

11

TACE System Under Test (SUT )Payload

TACE’s payload for the Boeing ScanEagle is a repurposing of JHU/APL’s Autonomy Toolkit (ATK) software, which has been used on more than a dozen types of UAVs in Boeing, Lockheed, Aerovironment manufactured vehicles, and hardware previously developed jointly under internal research funds by Boeing and JHU/APL

TACE SUT payload installed in a Boeing ScanEagle payload bay

ATK payload with Persistent Systems Wave Relay Wireless

Local Area Network Card

12

TACE Flight Tests at Aberdeen Test Center

APL Test Team on the tarmac at Phillips Army Airfield (PAAF) Aberdeen Test Center (ATC)

Hand launch of the Procerus research AUV controlled by

JHU/APL’s Autonomy Tool Kit (ATK)

Five Test Events with Multiple Sorties Were Executed during January/February 2014

13

Sample Results from TACE Testing at ATC

Constraint Violation - Range Safety Executive User Interface

SUT Moving NoGo Virtual Target Safe Loiter Point Fixed NoGo Area

14

Path to a Testing Capability – Phase 2 OV1

Test Director Test

Ground Station

Intruder C2

Restricted Airspace(s)

UnicornUAV

Range Safety

ScanEagle SUT-System Under Test

Native UHF Link

Wave Relay

2400MHz

Live target track to SUT

Intruder GS

UHF Safety Link

Land Lines

Onboard Test

Payload

Autonomous BehaviorsSearch, Track, Avoid

Watchdog Behaviors3D Range BoundariesPlatform Safety Limits

Virtual JIMM Entities 1 to N

Virtual Translated Air Traffic

Live Unicorn UAV

Defense Research & Engineering Network Connection

Tactical EO/IR

UHF PNT Downlink

Test Range Topology

ScanEagle as the SUT at Yuma Test Center

15

10 Miles

1.0Mile

2.5 Miles

7.5 Miles

GeographicConstraintBoundaries

1.0Mile

1.0 Mile

2.0 Miles

Notional Playbox at a Large Test Range

Flight Test Operations

Base

16

Proximity Violation (Virtual Entity)

TrackEstablished

Violation

LiveGroundEntity

VirtualAircraft

Not

e: p

aths

sho

wn

are

not

actu

al p

aths

)

17

Summary for Autonomy and T&E Perspectives

TACE Phase 3 will provide some basic operational functions at TRL 6 – this is an “initial architecture” but not a “range capability”

To our knowledge, there are no “active” programs to extend beyond TRL6 – highly understandable since there are no PORs with autonomous behaviors

At the recent NDIA Annual T&E conference – SHIFT LEFT is coming RFPs will include CONOPS and initial TEMPs - will “autonomy” be a “shift left” – Insights from Dr. Brown, Director TRMC - cited “cyber, big data, hypersonic, and

autonomy” as the T&E challenge areas for the near future

The NATO Science and Technology Organization (STO) promotes and ... Validation and Verification of Autonomous Systems (SCI-274)

18

Acknowledgements

• JHU/APL TACE Team: Robert Chalmers, Robert Bamberger, Kristine Ramachandran, Dean Kleissas, Brendan John, Subodh Harmalkar, William Van Besien, Michael Biggins

• TACE Subcontractors to JHU/APL :

– Trideum: Michael O’Connor

– Boeing Corporation: Gabriel Santander

– InSitu Inc.: Jerry McWithey

• Government Flight Test Support:

– Aberdeen Test Center: John Wiley

– Yuma Test Center: Mary Beth Weaver and Bob Vondell

• Unmanned and Autonomous Systems Test Test Technology Area

– Executive Agent: Vernon Panei

– Deputy Executive Agent: Stephanie Riddle

– Subject Matter Experts: William Hamel, Kirk Bonnevier