Ivano MalavoltaAssistant professor

Vrije Universiteit Amsterdam

Mission planning of autonomous quadrotors

VRIJEUNIVERSITEITAMSTERDAM

Hello

Empirical software engineering+ Software Architecture + MDE

+applied to

Complex systemsAutonomous robots

Mobile-enabled systems

If you think good architecture is expensive, try bad architecture.

... Brian Foote and Joseph Yoder

Roadmap

Background

The platform

Flight plan synthesis

Adaptation at run-time

Support for any kind of robot

Conclusions

Background

High coststeam training and transportation

operating costs

Safetysignificant risks (e.g., fire, earthquake, etc.)

Timing and enduranceexhausting shiftsactivities stopped at night

Civilian missions today

INTUITION

A team of autonomous quadrotors might work

together with staff

Special kind of helicopter with:

• high stability

• omni-directional

• smaller fixed-pitch rotors

à safer than classical helicopters

• simple to design and construct

• relatively inexpensive

http://goo.gl/FJFS5l

What is a drone for us?

Many civilian missions can be executed either by flying, ground or water robots

Using robots for civilian missions

Civilian missions can be executed by multiple robots

à lower mission completion timeà fault-toleranceà highly-specialized robots

All the robots perform their actions to fulfil the common goal of the mission

however...

common goal

Multi-robot missions

On-site operators must be expert of all the types of used robots

in terms of dynamics, hardware capabilities, etc.

On-site operators have to simultaneously control a large number of robots during the mission execution

Robots provide very low-level APIs and very basic primitives

error-prone development task-specific robotsno reuse

These issues ask for • abstraction• automation

Challenges

Application scenario

Mission

to monitor the CO2 level in a geographical area by monitoring it along a grid composed of cells of size 12x12 meters

Contextual entities

obstacles

emergency areas

no fly zones

home NO FLY ZONE

MDE allows all stakeholders to focus on models of the mission with concepts that are:

• closer to the application domain • independent from the specific robot technologies

• enabling automation à autonomous robots

http://mdse-book.com

Model-driven engineering for robotics

The platform

Mission modeling

Mission

Context

Map

MML

BL

Behavior specification

Detailed flight plan synthesis

Robot typemodels

MissionExecution

Engine

this layer is extensible

RL

Mask complexity• usable by non-technical experts• domain-specific concepts• separation of concerns

Independence w.r.t. the types of robots• behaviour generation capability

Reuse• context models• robot models

Robots must be autonomous

Principles Mission

Context

Map

MML

BL

Behavior specification

Detailed flight plan synthesis

Robot typemodels

MissionExecution

EngineRL

Modeling languages Mission

Context

Map

MML

BL

Behavior specification

Detailed flight plan synthesis

Robot typemodels

MissionExecution

EngineRL

Operator

in-the-field stakeholder specifying the mission

Robot engineermodels a specific kind of robot

develops the controller that instructs the robot on how to perform BL basic operations

Platform extenderextends the MML metamodel with new kinds of tasks

develops a synthesizer for transforming each new task to its corresponding BL operations

MML

RL + software controller

MML + synthesizer

Involved stakeholders

Mission layer

sequence of tasks executed by a team of robots

Extension point

Monitoring mission language (MML)

Photogrid task

• identifies a virtual grid within an area• each cell of the grid has a size of n meters

• each drone flies over each cell of the grid at an altitude of z meters and takes a picture of the ground

Example of extension

Context layer

geographical areas that can influence the execution of the mission

The focus is on spatial context

Monitoring mission language (MML)

Hardware and low-level configuration of each type of robot

Robot language (RL)

This makes RL models reusable and shared across missions, projects, and organizations

Atomic movements

and actions performedby each robot of the

swarm

Behaviour language (BL)

Example (1)MML model (in the tool)

NF1

NF2

home

RT

PGT

Example (2)

Robot model (Parrot)

Example (3)

Behavioural model

Drone&D1&

Drone&D2&

Drone&D3&

Start&(ε,&ε)& Start&(ε,&ε)& Start&(ε,&ε)&

TakeOff&(ε,&ε)& TakeOff&(ε,&ε)& TakeOff&(ε,&ε)&

GoTo&(ε,&ε)&GoTo&(ε,&ε)& GoTo&(ε,&ε)&

GoTo&(ε,&{Photo})&GoTo&(ε,&{Photo})& GoTo&(ε,&{Photo})&

GoTo&(ε,{Photo,BroadCast(D3.R1.Done)})&

GoTo&(ε,&ε)&

Land&(ε,&ε)&

Stop&(ε,&ε)&

GoTo&(ε,&ε)&

Land&(ε,&ε)&

Stop&(ε,&ε)&

0GoTo&(ε,&{Photo,&&BroadCast&(D2.PG1.Done)})&

0GoTo&(ε,&ε)&

Land&(ε,&ε)&

Stop&(ε,&ε)&

GoTo(ε,&{Photo,&&BroadCast&(D1.PG1.Done)})&

PG1 PG1 R1

Tool support…

ROS

HTTPS and web sockets

Web interface

Editor for MML models

M2M transformation+ models validation+ mission workflow

manager

Layer of controllers that interpret BL models at run-time

HTML5, CSS3, JavaScript Java + OCL + Rosbridge

Any technology supported by ROS

Drone driver

any

C1

C2

C3

…

Any protocol

Web UI

Mission Design and

BehaviourSynthesis

request MML &drones configurations

1

2 MML concepts & drones

3 mission design

4 submit MML models

6 BL model and analysis results

5behaviour analysisand synthesis

7 start mission

Mission ExecutionEngine

Mission modeling workflow

Mission execution

Mission ExecutionEngine

6 send status*(continuous flow)

start mission

setup drone controllers

* steps executed in parallel

MonitoringEngine

tasks executioncommands*

feedback*

feedback*

25

1

3

4 C1

D1

Mission simulation

Simulation-specific controller • Software-In-The-Loop (SITL) platform• Same components of real deployment Mission Execution

Engine

navigationcommands(MAVLink)

telemetry(MAVLink)

C1

MAVProxy

DroneKit

Arducopterphysics

simulator

physics data

Google Earth

drones positions at run-time

DEMO

Flight plan synthesis

OVERVIEW OF THE SYNTHESIS APPROACH

BL model automatic synthesis

BL

Mission entering

Mission leaving

Mission tasks execution

……

dn

…

d2

…

d1

State transitionState

Synchronization and communication message

Overview of the synthesis approach

MML

Each MML task is constrained by:• geometry• behavioural strategy• actions

pointline polygon

volume

Geometries supported by MML

STRATEGIES AND ACTIONS SUPPORTED BY MML

sweep(d) – full coverage of an area discretized by a grid whose cells dimension is d

search(target, d) – visit performed on a d-sized grid towards the discovery of target (e.g., represented by a PNG image)

track(m, m’, d) – it is like sweep(d) where m and m’arethe restart and stop messages, respectively

Actions:• taking a picture• making a video• detecting the presence of CO2• etc.

Strategies supported by MML

SYNTHESIS METHOD AT WORK ON THE PEM SCENARIOLet’s go back to the example…

NF1

NF2

home

RT

PGT

SYNTHESIS METHOD AT WORK ON THE PEM SCENARIO

Context = ( (NF1,NF2), (OB) )

PGT = ( (p1,p2,p3,p4,p1), sweep(10), (doPhoto(2480×3508), i) )

Swarm ={ (d1, {PGT}), (d2, {PGT}), (d3, {RT}) }

Task dependency graph( (t0, PGT, RT, t3), ∅ ∪ { (t0, {PGT, RT}) } ∪ { ({PGT, RT}, t3) } )

t0

PGT RT

t3

Synthesis method on the example

q4= p2

q1= p1

q3

q14

q5

q6

q15

q16

q2

q13

q12

q11

q7

q8

q9

q10

p3

p4

c1

c2

p1

Synthesis method on the example

SYNTHESIS METHOD AT WORK ON THE PEM SCENARIO

s10 TakeOff(p1.z) s1

Mission entering

s2s'''2

Goto(c1)

s''2

Goto(c2)s'2

Goto(p1)Goto(home1)Land

s1f land1

Mission leaving

non-fluid transition fluid transition Transition label syntax = <OP> |<ACT> | <OP> / <ACT>

s'1 Goto(c2) s''1 Goto(p1) v1

v11

Goto(q1) / DoPhoto(…)

v21v161

…

NoOp

Mission tasks execution

Goto(q16) / DoPhoto(…)

NoOpr1

NoOp

Goto(c1)

v171

Synthesis method on the example

SYNTHESIS AUXILIARY FUNCTIONS

Divide – distributes the geographical area of a task into a set of (sub-)areasoverlapping between no-fly zones, obstacle, and areasno “cross-cutting” no-fly zonessmallest distance assignment criterion1 point to 1 drone, 1 line to n drones (l/n)polygon/volume partitioning algorithm (rif. H. Bast and S. Hert. The area partitioning problem. In Proceedings of the 12th Canadian

Conference on Computational Geometry, 2000)

Appr – generates the obstacle- and collision-free path that a drone d must travel to reach a geographical area a

path planning problem in 3-dimensional world (rif. S. S. Skiena. The algorithm design manual, 1997. Stony Brook, NY: TelosPr, 504)

spherical obstacle enlargementtarget points identification (vertexes of the sub-area)trajectory definition = well-known visibility graph

Cover – takes a starting position s of a drone d, a geographical area a, and a real number rrepresenting the resolution of the grid that implicitly discretizes a

returns a set of <point,angle>coverage path planning problem (rif. R. Mannadiar and I. Rekleitis. Optimal coverage of a known arbitrary environment. In Robotics

and Automation (ICRA), 2010 IEEE International Conference on, pages 5525–5530. IEEE, 2010)

minimum visit planbased on the Boustrophedon cellular decomposition

Auxiliary functions

PROPERTIES OF THE SYNTHESIZED QBL MODEL

Safety• avoidance of collisions (P1), and of traversing no-fly

zones (P2)• no concurrency issues (P3)

P1&P2• correct and complete specification• Appr• Cover

P3• Divide• no message lost• sound mechanisms for sequencing, join, and fork• a drone cannot be involved in concurrent tasks

Properties of the synthesized BL model

Adaptation at run-time

• Drone d1 of the team T must reach a target geographical position p

• Drone d1 identifies an obstacle along its trajectory towards p

• if the obstacle is avoidable (e.g., a tree):

• then d1 adapts its trajectory so that it avoids the obstacle to reach p

• otherwise (e.g., a large building):

• the behaviour of d1 and other drones in T are adapted so that:

• the position p is still covered by another drone di∈ T

• d1 can cover some other points within the area

Scenario

• Safety as first-class element in the design of the system

• Clear separation of concerns between the generic safety-specific mechanisms and the functional behavior of the robots while defining the mission

• Decentralized and collective adaptation

• Multiple entities are adapted simultaneously so that

• critical runtime condition are properly addressed

• the working consistency and the collaboration of the ensemble arepreserved

Overview

Sustainable safety via collective adaptationGOAL

Software architecture Key features:• collective adaptation in a

decentralized fashion • managed at run-time• new solvers can be

introduced at any time• separation of concerns

• Entities - basic building blocks representing the actors and components ofthe system

• Ensembles - set of roles that can be played by participating entities

Ensemble role specification

• Issue - definition of a critical situation that can happen to a role of an ensemble

• Solver - ability of a role to handle certain types of issues

Issues resolution

Support for any kind of robot

• Modeling languages• special care has been put in MML and BL

• Modeling infrastructure• overall software architecture• model transformations

What is independent of the used robots?

• Definition of a generic editing environment• flexible w.r.t. the use of geographical concepts

à we need to relax the constraint of always referencing to a map

• Evaluate alternative architectures• decentralized• no assumption of continuous connectivity with ground station• P2P communication between robots

• Experimentation with other kinds of physical robots• e.g., underwater autonomous robots

What needs to be done?

Step 1 - check the expressiveness of the languages

• we reverse engineered the AUV used in the RALF3 project

Step 2 - extend the modelling languages

• Mission tasks

• Context

• Behaviour

• Robot

Extension for underwater robots

no extension needed Winning features:• geo and relative coordinates• behavioural movements driven by

specific conditions (e.g., vision)• objects as behavioural targets

Extended robot language for UAVs

Conclusions

EXPERIMENTATION

models@runtime to control the mission execution and manage adaptation

Support other types of robots

We need you!Mission ExecutionEngine

navigationcommands(MAVLink)

telemetry(MAVLink)

C1

MAVProxy

DroneKit

Arducopterphysics

simulator

physics data

Google Earth

drones positions at run-time

Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione, and Massimo Tivoli. Automatic Gen-eration of detailed Flight Plans from High-level Mission Descriptions. In ACM/IEEE 19thInternational Conference on Model Driven Engineering Languages and Systems (MODELS),pages 45–55. ACM/IEEE, Oct 2016.

Federico Ciccozzi, Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione (2016) Adopting MDEfor Specifying and Executing Civilian Missions of Mobile Multi-Robot Systems IEEE AccessJournal. http://dx.doi.org/10.1109/ACCESS.2016.2613642

Darko Bozhinoski, Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione, Massimo Tivoli (2015).FLYAQ: Enabling Non-Expert Users to Specify and Generate Missions of AutonomousMulticopters. In 30th IEEE/ACM International Conference on Automated Software Engineering(ASE 2015).

Darko Bozhinoski, Ivano Malavolta, Antonio Bucchiarone, Annapaola Marconi (2015). SustainableSafety in Mobile Multi-Robot Systems via Collective Adaptation. In Ninth IEEE InternationalConference on Self-Adaptive and Self-Organizing Systems, SASO 2015, Cambridge,Massachusetts, USA, September 21-25, 2015,

Darko Bozhinoski (2015). Managing Safety and Adaptability in Mobile Multi-Robot Systems. InProceedings of the 11th International ACM SIGSOFT Conference on Quality of SoftwareArchitectures, pp. 135–140.

References

Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione (2014). A family of Domain-SpecificLanguages for specifying Civilian Missions of Multi-Robot Systems. In Proceedings of the 1stInternational Workshop on Model-Driven Robot Software Engineering (MORSE), pp. 13–26.

Davide Di Ruscio and Ivano Malavolta and Patrizio Pelliccione (2014). The Role of Parts in theSystem Behaviour. In Software Engineering for Resilient Systems - 6th International Workshop,SERENE 2014, Budapest, Hungary, October 15-16, 2014. Proceedings, pp. 24–39.

Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione (2013). Engineering a Platform for MissionPlanning of Autonomous and Resilient Quadrotors. In Software Engineering for ResilientSystems - Fifth International Workshop, SERENE 2013, pp. 33–47.

References

ContactIvano Malavolta |

Assistant professorVrije Universiteit Amsterdam

iivanoo

i.malavolta@vu.nl

www.ivanomalavolta.com