A Mosaic of Models as the Driver for Domain-Specific ...2 Robotics and DSL in Research Conferences...

1© 2019 The MathWorks, Inc.

A Mosaic of Models as the Driver for

Domain-Specific Modeling in the Robotics Industry

Advanced Research & Technology Office

MathWorks

Anastasia Mavrommati, PhD

[email protected]

stacymav.github.io

mailto:[email protected]

stacymav.github.io

2

Robotics and DSL in Research Conferences

A. Nordmann, N. Hochgeschwender, D. Wigand and S. Wrede, “A Survey on Domain-Specific Modeling Languages in Robotics”, Journal of Software Engineering for Robotics (JOSER), 2016

Robotics

Software

Hybrid

3

What about Robotics in the manufacturing industry?

4

Where are we and how did we get

here?

A mosaic of models for Robot System

Design

Robot System Design for pick-and-place applications

Summary

Outline

5

Source: Kotsanas Museum of Ancient

Greek Technology, Athens, Greece

www.kotsanas.com

The Automatic Servant of Philon

~300 BC

6

UNIMATE robot (serving liquor)

1954 AD

G. DevolJ. Engelberger

~300 BC

Source: https://robots.ieee.org/robots/unimate/

7

When did the need for programming languages in Robotics appear?

1954

First industrial robotFirst robot programming

language in industry

1979

First robot programming

language in academia

VALWAVE

1973

Teach the robot by guiding it through motion

• Physical guide through

• Using the teaching pendant

Source: “Introduction to Robotics: Mechanics and Control”, John J. Craig, 1985

PUMA Teaching Pendant

~20 years

8

First desideratum: Move robot!

Move to location PART.

Then, move to location BOX.

ROBOT USERS

9

First solution: Textual programming languages

Variable Assembly Language

1979

ROBOT USERS

10

First solution: Textual programming languages

Teaching Pendant Language

1979 2019

>> L P[1:PART] 100%

>> J P[2:BOX] 50%

40 years!

✓ Programmed on the robot

computer

✓ Robot operation stopped

✓ Similar semantics

(assembler-like)

✓ More GPL-like syntax

ROBOT USERS

11

Target groups for Robotics applications

ROBOT USERS

ROBOT BUILDERS

SOFTWARE

DEVELOPERS

12


ROBOT USERS

ROBOT BUILDERS

ROBOTICS

RESEARCHERS

SOFTWARE

DEVELOPERS

13


ROBOT USERS

ROBOT

BUILDERS

SOFTWARE

DEVELOPERS

ROBOTICS

RESEARCHERS

14

New desideratum: Move robot at the office!

ROBOT USERSMove to location PART. Then,

move to location BOX.

SOFTWARE

DEVELOPERS

15

Solution: Off-line programming languages

✓ No access to robot required

✓ Similar to General Purpose Languages with abstractions for Robotics

L P[1:PART] 100%

J P[2:BOX] 50%

Assignment

Declaration

1979 2019 2019

TP VAL KAREL

ROBOT USERS

SOFTWARE

DEVELOPERS

16

Solution: Off-line programming languages

ROBOT COMPANY PENDANT TEACHING OFF-LINE

FANUC TP KAREL

KUKA KUKA Robot Language (KRL) KRL

ABB RAPID RAPID

Universal Robots URScript URScript

Staubli VAL3 VAL3

YASKAWA INFORM MotoCom C++ Library

DENSO PacScript PacScript

More than 30 different robot programming languages

ROBOT USERS

SOFTWARE

DEVELOPERS

17

All languages are specific to robot manufacturers.

“A Model-Driven Approach to Support Engineering Changes in Industrial Robotics Software”

Yu Sun, Jeff Gray, Karlheinz Bulheller, Nicolaus von Baillou

18

Robotics developers need non-robot specific tools

…circa 2007

Not a modeling language but an enabling platform for domain-specific applications

Publisher-subscriber communication middleware

Source: https://robohub.org/ros-101-intro-to-the-robot-operating-system/

SOFTWARE

DEVELOPERS

https://robohub.org/ros-101-intro-to-the-robot-operating-system/

19

Development of ROS

…circa 2007

From languages specific to robots, to languages specific to Robotics.

…circa 2012

SOFTWARE

DEVELOPERS

20


here?

A mosaic of models for Robot System Design

Outline

21

Actuation

Dynamics

Motor Control

Sensing

Kinematics

Component Modeling

Rigid Body Modeling

Motion Planning

Motion Modeling

World Modeling

Collision modeling Vision modeling

Perception Modeling

Camera Depth LiDAR

Intelligence Modeling

Deep Learning Reinforcement Learning

Task Modeling

Task Planning Task Scheduling Task Description

Layers of technologies for Robotic System Design

Trajectory Optimization

Physics

One modeling language for all?

Human-Machine Interface Modeling

Speech Recognition Gesture Recognition Graphical User Interface

22

Physics

Actuation

Dynamics

Motor Control

Sensing

Kinematics

Component Modeling

Rigid Body Modeling

Motion Planning Motion Modeling

World ModelingCollision modeling Vision modeling

Perception ModelingCamera Depth LiDAR

Intelligence ModelingDeep Learning Reinforcement Learning

Task ModelingTask Planning Task Scheduling Task Description

Layers of technologies for Robotic System Design


Or a pandaisia* of languages?

Speech Recognition Gesture Recognition Graphical User Interface Human-Machine Interface Modeling

* Pronounced p-aa-n-dh-eh-s-EE-aa, loosely translated as “feast”

23

Layers of language abstractions for Robotic System Design

Robot interface DSL KINOVA robots Users

Robotics System Toolbox DSL Robotics Developers

MATLAB DSL Math/Engineering

Researchers

Builders

C/C++ GPL

Example technology: Simulate robot motion

Source: www.kinovarobotics.com

24

Robot interface DSL KINOVA robots Users

Robotics System Toolbox DSL Robotics Developers

MATLAB DSL Math/Engineering

Researchers

Builders

C/C++ GPL

Languages applied to Robotics

Languages adapted to Robotics

Languages inherent to robots

Layers of language abstractions for Robotic System Design

25

Example: Modeling robot kinematics

“Model the kinematic structure of a robot arm with 1 rigid link and 1 rotational joint”

26

Example: Languages for robot kinematics

GAZEBO / ROS Robotics System Toolbox

✓ Textual

✓ Declarative✓ Textual

✓ Imperative

27

More languages for robot kinematics

Robotics System Toolbox

✓ Graphical

✓ Declarative

GAZEBO / ROS

Simscape Multibody

28

More languages for robot kinematics

Unreal Engine

✓ Graphical

✓ Declarative

GAZEBO / ROS

Use third party 3D computer graphics software*

Robotics System Toolbox

Simscape Multibody

* Source: Christopher Yonge – YouTube ChannelSource: https://developer.nvidia.com/isaac-sdk

29


here?


Design


Outline

30

Pick and place workflow

31

Pick and place workflowspeed x12

What formalisms do we need to implement this workflow?

32

Task ModelingRigid Body Modeling

Rigid Body & Motion Modeling

HMI Modeling

Environment Modeling

Perception & Intelligence

Modeling

A pandaisia of modeling formalisms!

33


Rigid Body & Motion Modeling

HMI Modeling



Modeling


❑ A single developer must be able to switch among modeling

formalisms with intuitive semantics

❑ There must be mechanisms in place to integrate all

modeling formalisms into one system

What are some formalisms that enable these requirements?

34



35

Task Modeling

Object detection

Initialize

Pick

Place

Home position

Use Stateflow as high-level task-planner

The task modeling language:

✓ Allows the developer to visually match their

hand-drawn flowchart to syntactic states

36

Pick

Place

Task Modeling

Use Stateflow as high-level task-planner

The task modeling language:

✓ Allows the developer to visually match their

hand-drawn flowchart to syntactic states

✓ renders the task complexity manageable

37

Rigid Body Modeling



38


GAZEBO is a simulation platform

essential for vision and contact modeling

Source: gym-gazebo

Simulink is a simulation platform

essential for systems modeling

Co-simulation allows

systems integration for

Robotics with combination of

most intuitive languages.

Simulation results are

✓ Synchronous

✓ Reproducible

39

Environment modeling: Co-simulation necessities

Composite ports• width

• height

• data

• data_type

Data structure transformation• From GAZEBO to Simulink

Multi-formalism semantics• Integrate with MATLAB scripts for data processing

40

Rigid Body Modeling



Modeling

41

Intelligence Modeling: System Object

AI algorithms can be embedded to execution systems to be used in the real world.

Inherits from MATLAB System Object

Constructor definition

Step implementation

42

Reinforcement Learning Example: Co-simulation and System Object

Reinforcement Learning Agent as System Object• Inputs

• Observations

• Rewards

• Output

• Action

GAZEBO Co-simulation

43


here?


Design


Summary

Outline

44

Robotics models span a large range of modeling formalisms.

From languages specific to robots, to languages specific to Robotics.

“Robotics is a science of integration rather

than a fundamental science, and the

integration becomes ever more complex.”Herman Bruyninckx

Bruyninckx, H. , “Robotics Software: The Future Should Be Open,” IEEE Robotics & Automation Magazine, vol. 15(1), pp. 9–11, 2008.

47

Can I order a “Robotics-specific modeling language”?

Pick…

1. …. a target group

Robot Users

Software

Developers

Robotics

Researchers

Robot

Builders

2. …. a technology

Actuation

Dynamics

Sensing

Kinematics

Motion Planning


Depth Camera


Task Scheduling Task Description


3. …. an abstraction

Languages applied to Robotics

Languages adapted to Robotics


Speech Recognition Gesture Recognition

4. …. a target

ROS node

Embedded MCU

48

A mosaic of models and languages

Actuation

Dynamics

Motor Control

Sensing

Kinematics

Motion Planning


Camera Depth LiDAR




Robot Users

Software

Developers

Robotics

Researchers

Robot Builders


Languages applied to RoboticsLanguages adapted to Robotics


ROS nodeEmbedded MCU

49

A mosaic of models and languages!

Actuation

Dynamics

Motor Control

Sensing

Kinematics

Motion Planning


Camera Depth LiDAR




Robot Users

Software

Developers

Robotics

Researchers

Robot Builders


Languages applied to RoboticsLanguages adapted to Robotics


ROS nodeEmbedded MCU

50

Thank you!

A Mosaic of Models as the Driver for Domain-Specific ...2 Robotics and DSL in Research Conferences...

Documents

Transcript of A Mosaic of Models as the Driver for Domain-Specific ...2 Robotics and DSL in Research Conferences...