Presentation ‘Benchmarking Autonomous Robotics Controllers’ by Pablo Muñoz (University of Alcalá)

     07/12/2016

Introduction by TO (M. van Winnendael, TEC­MMA) Title of the contract: Cooperative Systems for Autonomous Exploration MissionsBudget + Program: 90 k€, NPI (Networking/Partnering Initiative) Duration: 3 yearsContractor: University of Alcalá, UAH (Spain)Subject of the activity: On­board autonomy of space robotics systems, in the first place planetary roversMain Objectives:­   Metrics for measuring planning and execution performance­   Path planning algorithms to define navigation routes ­   Algorithms for controlling the reactive behavior of ESA’s Goal­Oriented    Autonomous Controller (GOAC) in uncertain situations

ESA UNCLASSIFIED -For Official Use

Benchmarking Autonomous Robotics Controllers

Pablo Muñoz Martínez TEC-ED & TEC-SW Final Presentation Days ESA/ESTEC December 7, 2016

PhD funded by ESA NPITechnical officer : Mr. Michel van Winnendael

PhD program on space researchPhD advisor :

Dra. María Dolores Rodríguez Moreno

PhD co-advisor: Dr. Amedeo Cesta Dr. Andrea Orlandini

Contents

1. Introduction

2. Objectives

3. Planning for planetary rovers

4. Evaluating autonomous controllers

5. OGATE Demo

5/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Motivation

Photos: ESA

Space exploration is a challengeNew missions carry more science instrumentsGo deeper into the spaceOn-board autonomy can be helpful to

Maximize science returnMinimize risksReduce operators workload

6/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Autonomy levels

Decision making capability (AI)According to ESA standards

E1 On-ground control (teleoperation)E2 On-board scheduler for on-ground commandsE3 Event based autonomous operationE4 Goal oriented and mission replanning

7/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Autonomous controllers

Integration of P&S (Planning & Scheduling) in roboticsP&S is used for on-ground operationsMost common schema: 3T (3-Tired)Wide research field

ESA: GOACNASA: IDEA, RAUAH: MoBAr

Deliberation

Execution

Functional

8/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Open issues

No appropriate software to facilitate operationLack of experimentationHard to extract meaningful dataApproaches are validated with few scenariosDifficult to generate reproducible experiments

→ Tests can be seen as proof of concept → Difficult to compare approaches → Hard to trust in autonomy

Contents

1. Introduction

2. Objectives

3. Planning for planetary rovers

4. Evaluating autonomous controllers

5. OGATE Demo

10/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Objectives

Create a framework for autonomous controllers characterization and evaluation

Testbench methodologyPerformance metricsCompare at least two controllersEase deployment and operationImprove trust in autonomy for robotics

Contents

1. Introduction

2. Objectives

3. Planning for planetary rovers

4. Evaluating autonomous controllers

5. OGATE Demo

12/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Planetary exploration case study

Perform scientific tasks in different locationsLimited communication opportunitiesE4 autonomy: user provides initial goalsCommon scenario for autonomous controllers testingRequires

Task planning

Path planning

13/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Path planning

Obtain feasible and optimal paths2D binary grid map

Used in real missionsDeterministic algorithms

A*A*PSTheta*

Path planning

Navigation

More about 2D path planning: P. Muñoz and M.D. R-Moreno. On Heading Change Measurement: Improvements for Any Angle Path Planning. Novel Applications of Intelligent Systems, ch. 6 (2015)

14/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Path planningPlanetary robotics require more realistic terrains with

Terrain characteristics (cost maps)Altitude (DTM)

Field D* (MER/MSL operations) exploits cost mapsModified A* can use different DTM representations

15/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Path planning – 3Dana

3Dana: path planning on realistic surfacesAccurate lineal interpolationAllow DTM and/or cost mapPath cost = path length × cell costHeading changes heuristicTerrain slope Known altitude

Interpolated altitude

cell cost

16/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Path planning – 3Dana

Evaluation on Mars DTM3Dana provides safer pathsSlope limitationReachability map

More results: P. Muñoz, M.D. R-Moreno and Bonifacio Castaño. 3Dana: Path Planning on 3D Surfaces. In procs. of the 36th SGAI International Conference on Artificial Intelligence (2016)

17/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Task planning – The UP2TA planner

Exploration requires to achieve multiple goalsUP2TA (Unified Path planning & Task planning Architecture)

PDDL planner : FFPath planner : 3Dana

Objective: optimize the planMinimize travelled distanceBest task ordering

18/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Task planning – The UP2TA planner

More results: P. Muñoz, M.D. R-Moreno and D. F. Barrero. Unified Framework for Path Planning and Task Planning for Autonomous Robots. In Robotics and Autonomous Systems, vol. 82, pp. 1—14 (2016)

Contents

1. Introduction

2. Objectives

3. Planning for planetary rovers

4. Evaluating autonomous controllers

5. OGATE Demo

20/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

An evaluation framework

Hard to compare autonomous controllersObjective

Generation of meaningful execution dataAnalysis integration of P&S in roboticsGeneration of reproducible/comparable testbenchs

Based onA methodology to guide the testing phaseA set of metrics to assess P&S integrationA software tool to

Automate testbenchsImprove autonomy support for users

21/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Methodology

Evaluation consists on 3 sequential phases

1. Evaluation design

2. Tests execution

3. Performance report

22/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Methodology

1. Evaluation design

Objective of the testApplication scenario and robotic platformControllers configurationsOperative conditions

NominalContingencyGoal injection

Normalized performance metrics

23/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Methodology

2. Tests execution

Scenario and controller instantiationControllers executionMonitoring of metrics dataCollection of average behaviours

24/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Methodology

3. Performance report

Data synthesisGeneration of objective evaluationsMeaningful data representation

25/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

General metrics

Characterize autonomous behavioursEnable different controllers comparisonFour groups

Model

Controller

Plan accuracyPlanner model adequacyPlanner performancePlanning & Execution (P&E) integration

These groups fit in the graphical reportEasy to focus on particular characteristics

26/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

General metrics

Characterization of plan accuracy

Planning performanceTimeMemory

Lower diff

Higher diff

Execution ends

Minimum duration

Maximum duration

27/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

General metrics

Integration of P&S with underlying layers of the controllerAnalysis of information flow Characterization of reaction time to unexpected events

Deliberation

Execution

Functional

Plan

Sensor dataCommands

Unexpected event

28/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

The OGATE software

OGATE (On-Ground Autonomy Test Environment)Designed to support autonomy

Simplify controllers deployment/operationMonitor P&S featuresPerform automatic testbenchsPlug-ins

Implements the methodologyand metrics collection

Allows user-defined metrics

More info: P. Muñoz, A. Cesta, A. Orlandini and M.D. R-Moreno. First Steps on an On-Ground Autonomy Test Environment. In procs. of the 5th IEEE Conference on Space Mission Challenges for Information Technology (2014)

29/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

The OGATE software

GOAC (GMV)

GOAC (PST) OGATE Beta

No data available

OGATE Alpha Automated tests, data gathering

30/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

The OGATE software

Current deployment: mission planning

31/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

The OGATE software

Current deployment: mission monitoringCurrent statusPast status

Map with rover and goals positions

Goals status

32/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

The OGATE software

Current deployment: performance assessment

33/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

A comparison testbench

Two controllersGOAC (Goal Oriented Autonomous Controller)

ESA effort for autonomy in space (E1-E4)Timelines based planning (APSI-TRF)

MoBAr (Model Based Architecture)UAH E4 controllerPredicate based planning (UP2TA)

Exploration domain, simulated TurtleBot 2OGATE carries out automated tests

Nominal / Goal injection / Execution failureReports summarize average behaviours

34/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

A comparison testbench

Report after 30 executions MoBAr GOAC

More results: P. Muñoz, A. Cesta, A. Orlandini and M.D. R-Moreno. A Framework for Performance Assessment of Autonomous Robotic Controllers. In procs. of the 2nd Workshop on Planning and Robotics – ICAPS (2015)

Contents

1. Introduction

2. Objectives

3. Planning for planetary rovers

4. Evaluating autonomous controllers

5. OGATE Demo

36/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016

Introduction · Objectives · Plan · Eval. Controllers · Demo

Demo

Thanks for your attention

Pablo Muñoz Martíneze-mail: pmunoz@aut.uah.es