© TESIS GmbH ◼ www.tesis.de

Dipl.-Ing. Ronnie Dessort – Senior Simulation Consultant at TESIS GmbH

Autonomous Vehicle Software Symposium, Stuttgart (June 6, 2018)

Object detection based on virtually trained neural networks

2

Motivation

1https://medium.com/@Synced/the-humans-behind-artificial-intelligence-3ff578cfcc60

1. Reduce manual work by using virtual ground truth data.

◼ AI algorithms need labelled data as ground truth for training.

◼ This work is tedious and costly: a person labels 10-40 images

per day1.

→ Increased frontloading for object detection by using virtual

objects

2. Complex traffic scenarios are crucial for testing vehicle

automation.

◼ Real-world test drives are expensive and inefficient for early

development stages.

◼ Small scenario numbers and deterministic behavior can

introduce an undesired bias.

→ Increased frontloading by testing in complex virtual traffic

scenarios

3

◼ Resources for training and testing

◼ Virtual training approach

◼ Evaluation of different detection models

◼ Testing object detection in complex virtual

traffic scenarios

◼ Summary & Conclusion

Agenda

4

Resources for virtual object detection training and testing

DYNA4

Simulation framework Visualization

DYNAanimation

Object catalogue for training Arbitrary scenes for testing

Publicly available image databases

for object detection and recognition

1https://arxiv.org/pdf/1405.0312.pdf2http://www.cvlibs.net/publications/Geiger2012CVPR.pdf3http://host.robots.ox.ac.uk/pascal/VOC/pubs/everingham15.pdf4https://arxiv.org/pdf/1409.0575.pdf5https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research

1

2

3

4

… and many more …5

▪ Vehicles

▪ Pedestrians

▪ Wildlife

▪ Road signs

▪ Buildings

▪ Vegetation

▪ Weather

▪ … and much more …

How to link real

and virtual

world?

5

Virtual training approach

Creating images under clinical conditions

En

vir

on

men

t

Automatic image database

generation by varying

▪ object type:

sedan, sports car, SUV, etc.

male, female, adult, child

▪ object color:

red, green, blue, etc.

different clothing (e.g.

doctor, construction worker)

▪ zoom:

object details, full size,

stamp size

▪ perspective:

360 degree panoramic view,

inclined camera

environment without relevant

objects to provide negligible

image background information

Automatic generation of annotation files based on

PASCAL VOC definition

Veh

icle

sP

ed

estr

ian

s

Training and

evaluation

with

TensorFlow

class:

Vehicle

class:

dontcare

class:

Pedestrian

6

◼ All self-trained models …

◼ … are based on Faster R-CNN + ResNet-101

◼ … used a pre-trained model (KITTI dataset) as initial solution

◼ … were trained for > 300,000 epochs

◼ Global training database consists of 14,568 virtual images:

◼ 4,488 vehicle images (14 types from sports car to SUV)

◼ 5,400 pedestrian images (12 types of male/female child/adult)

◼ 4,680 environment images (urban, rural, parking garage)

◼ Investigation of models trained on and detecting various sets of classes:

◼ V only based on vehicle images

◼ VE considering also environment images

◼ PE recognizing pedestrians in arbitrary environments

◼ VPE recognizing vehicles and pedestrians in arbitrary environments

Detector models

https://arxiv.org/pdf/1506.01497.pdf

7

◼ Detector model based on training with virtual data of Vehicles

Results of detector model „V“

Virtual test set data Validation with realistic scenes

8

◼ Detector model based on training with virtual data of Vehicles and Environment

Results of detector model „VE“

Virtual test set data Validation with realistic scenes

9

◼ Detector model based on training with virtual data of Pedestrians and Environment

Results of detector model „PE“

Virtual test set data Validation with realistic scenes

10

◼ Detector model based on training with virtual data of Vehicles, Pedestrians and Environment

Results of detector model „VPE“

Virtual test set data Validation with realistic scenes

11

Testing detectors in a virtual environment

Creation of virtual ground truth data with DYNAanimation

◼ Intrinsically available information: object class per pixel of each frame

◼ Semantic segmentation can be used for creating image annotations

automatically

◼ Scenes with arbitrary complexity can be analyzed and relevant objects

marked with ground-truth labels

Complex traffic scenarios are crucial for testing advanced algorithms for vehicle automation

Solution: Co-Simulation of DYNA4 and SUMO1

1http://sumo.dlr.de/

Change of seed

and

environment settings

12

◼ Traffic sign detection and recognition is influenced by

◼ weather conditions → fog, rain, snow

◼ color fading → long sun and rain exposure

◼ illumination variations → daytime sunshine vs. car light at night

◼ vehicle motion → image blur

Outlook: road sign detection

Apply weather

and lighting

13

Summary & Conclusion

DYNA4 SUMO

DYNAanimation

TensorFlow

◼ Object detection with deep neural networks

◼ Training with virtual data avoiding manual labelling

◼ Creation of virtual ground truth data for evaluation of

detection quality

◼ Co-Simulation of the virtual vehicle in virtual traffic

with complex traffic scenarios

◼ One-click scenario variation for stochastic, but

reproducible testing

Validation of conventionally trained object

detection in virtual traffic and environment &

virtual training for use with real-world data

Facilitate frontloading for ADAS/AD functions

by virtual testing

14

Thank you for your attention!

Object detection based on virtually

trained neural networks.

For more information visit our booth #AV801

and our talk

Virtual testing by coupling system simulation with SUMO traffic flow simulation

at theTest & Development Symposium (today at 17:30)

Dipl.-Ing. Ronnie Dessort

e-mail: dessort@tesis.de

phone: +49 89 747377-58