NVIDIA DRIVE UPDATE€¦ · Sensor Complexity - More higher resolution cameras New DNN Models-CNN,...

December 2019

NVIDIA DRIVE

2NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE PLATFORMMass production

SAFETY

TECHNOLOGY REGULATIONS

INDUSTRY / MARKET


THE DRIVE INITIATIVEEnd-to-End, Open Platform for Building Autonomous Vehicles

DGX Saturn V Constellation Xavier

DRIVE AV Re-SIMDRIVE IX

The NVIDIA DRIVE™ Platform provides all major AI technologies for Driving, Development and Validation -from driving & cockpit computers & software to data collection & DNN training, to simulation tools. The system is an open platform with multiple programming points. Car makers are free to use or develop as much as they like and take advantage of an ecosystem of hundreds of partners.


SIMULATETRAIN MODELSCOLLECT DATA DRIVE

NVIDIA DRIVEEND-TO-END PLATFORM

Lanes Lights

Path

Signs

PedestriansCars

Lanes Lights

Path

Signs

PedestriansCars

5

DRIVE COMPUTER / ARCHITECTURESCALABLE NOW & INTO THE FUTURE

NCAP TO ROBOTAXI

6

DRIVE ROADMAP

DRIVE Parker

1x Perf

1x Efficiency

Samples – Q4’2015

Production – Q1’2017

DRIVE PX2

20x Perf

1x Efficiency



ORIN

DRIVE Orin-X

>100x Perf

~20x Efficiency

Samples – Mid 2021

Production - Q3’2022

DRIVE Xavier

23x Perf

8x Efficiency



DRIVE Pegasus

~230x Perf

5x Efficiency



Auto-Grade

Super Energy-Efficient

7

ORIN – HIGHEST PERF SOC EVERWhat is driving the need for increasing performance in AV?

Sensor

Complexity -

More higher

resolution

cameras

New DNN Models-

CNN, RCNN, RNN

DNN GOPs/Frame

increasing

20+ DNN Models

required for most

demanding ODD

DNN Precision /

Accuracy

OTA new features

/ SW Patch

Integration of

functions

Centralized

Computer

Future Proof

EUNCAP

With 360°

Perception

More perf equates to faster time to market


NVIDIA DRIVEOne Architecture

NVIDIA DRIVE AGX platform is an open platform; allows partners to do their own innovation

PEGASUS (L4/RoboTaxis)XAVIER (NCAP, L2+/L3)

One AV Architecture for L2 -> Robo Taxi

ORIN

ORIN


DRIVE ARCHITECTUREAUTOMOTIVE GRADE – INDUSTRIALIZED ECU’S

10

NVIDIA DRIVESENSOR ECOSYSTEM


SENSOR ECOSYSTEM

https://developer.nvidia.com/drive/drive-core-sensors

GNSS/IMURADAR

LIDARIMAGING SENSOR / CAMERA

AR0820, AR0144, AR0231, IMX390, IMX490, IMX424

ARS430

SC13S, MTi-G-710, SPAN-IGM A1

https://developer.nvidia.com/drive/drive-core-sensors

12

NVIDIA DRIVEDRIVE SOFTWARE


NVIDIA DRIVE SOFTWARE ARCHITECTUREOpen and modular software for AV development

DRIVE AV

DRIVE Planning

DRIVE Mapping

DRIVE Perception

DRIVE IX

Visualization AI CoPilot AI Assistant

DRIVE Networks DRIVE Calibration

DRIVE OS

DRIVE AGX DEVELOPER KITS(Xavier/Pegasus)

DRIVE HYPERION(Reference Architecture)

DRIVE Core

DRIVEWORKS

14

NVIDIA DRIVEDRIVE OS


DRIVE AGX AV Software Platform

Linux BSP

OS Drivers, USB,

File System, Network

Linux

Hardware

Software

I/O

OS/3rd Party SoftwareNVIDIA SoftwareHardware

Hypervisor

Radar Lidar GPS IMU

XavierMCU

CAN &

Flexray10G/1G/BR Ethernet

DRIVE OS

CUDA

TensorRTcuDNN

NvM

edia

NvStr

eam

s

Open G

L

ES 3

.2

L1 S

afe

ty S

erv

ices

Foundati

on S

erv

ices

Securi

ty S

erv

ices

Com

munic

ati

on S

erv

ices

L3 S

afe

ty S

erv

ices

Dia

gnost

ics

Cameras

RS

232

GMSL

(via CSI-2)NOR UFSeMMCPCIe NVMEPMIC

EG

L 1

.5

NVLINK

Pow

er

Managem

ent

L2 S

afe

ty S

erv

ices

Data

Dis

trib

uti

on (

DD

S)1

Notes: 1) For development only.

DriveWorks

OEM/T1 Application

TU104

Optional Services

Scalability;+Xavier, NvME, or Optional Processor


DRIVE AGX AV Software Platform

Hardware

Software

I/O

OS/3rd Party SoftwareNVIDIA SoftwareHardware

Hypervisor

Radar Lidar GPS IMU

XavierMCU

CAN &

Flexray10G/1G/BR Ethernet

DRIVE OS

CUDA

TensorRTcuDNN1

QNX BSP

OS Drivers, USB1,

File System, Network

NvM

edia

NvStr

eam

s

Open G

L1

ES 3

.2

L1 S

afe

ty S

erv

ices

Foundati

on S

erv

ices

Securi

ty S

erv

ices1

Com

munic

ati

on S

erv

ices1

L3 S

afe

ty S

erv

ices

Dia

gnost

ics

Cameras

RS

232

GMSL

(via CSI-2)NOR UFSeMMCPCIe NVMEPMIC

Scalability w/dGPU

EG

L 1

.51

NVLINK

Scalability;+Xavier, NvME, or Optional Processor

Pow

er

Managem

ent

L2 S

afe

ty S

erv

ices

DRIVE AV

Data

Dis

trib

uti

on (

DD

S)1

Optional ServicesNotes: 1) Not for use in production in a safety context.

17

NVIDIA DRIVEAUTONOMOUS DRIVING

APPLICATION


NVIDIA DRIVEWORKS SDKData Abstraction | Calibration | Compute Modules

DRIVE AV DRIVE IX

DRIVEWORKS

DRIVE Core

DRIVE OS

DRIVE AGX DEVELOPER KITS DRIVE HYPERION

Sensor Abstraction

Layer + Plugins

Vehicle IO

DNN Frameworks

Tools (Recorder)

Image Processing

Point Cloud Processing

DRIVE Networks

DriveNet OpenRoadNet

SignNetLightNet

ClearSightNet

PathNet

WaitNet

MapNet LaneNet

DRIVE Calibration

Egomotion

Self-

Calibration

Calibration

Tools


NVIDIA DRIVEWORKS SDKFoundation for AV SW Development on NVIDIA DRIVE

- Open, modularized library of functions

- Optimized for DRIVE AGX

- Easy to integrate and build upon

SOFTWARE MODULES

- Software dev tools ranging from sensor

data capture to calibration and

visualization

TOOLS

- Sample apps

- Documentation, tutorials

- Active developer forum and community

DEVELOPER SUPPORT

- Architected for automotive safety

- Compliant with industry standards, (ISO

26262/21448, MISRA)

PRODUCTION GRADE


DriveWorks

Drive AV

RoadRunner

MapWorks

DNNs

Self-Calibration

Obstacle Perception Path Perception Wait Perception

Egomotion Estimation

Localization

Sensor Abstraction

Low Level Processing

Actuation

Cloud Mapping

Map Perception

DriveNet

OpenRoadNet

Motion/Structure

PilotNet

LaneNet

PathNet

Frame Packaging and

Serialization

Map Creation

and updatesDriveNet

WaitNet

HD Map

MapNet

Route Planner

Control

World Model

Lane

Planner

Behavior Planner

Safety Force Field

Search Based Planning AI Based Prediction

LSA ImageSafety Checks

Learned Planning

SubModule

DRIVE AV

Local World Model


SCALABLE PLATFORMOne architecture – Highway and Urban AV Driving

ON RAMP TO OFF RAMP URBAN: ADDRESS TO ADDRESS


ADVANCED FUNCTIONSAuto High Beam Clear Sight

NCAPParking


DriveWorks

Drive AV

RoadRunner

MapWorks

DNNs

Self-Calibration



Localization

Sensor Abstraction


Actuation

Cloud Mapping

Map Perception

DriveNet

OpenRoadNet

Motion/Structure

PilotNet

LaneNet

PathNet

Frame Packaging and

Serialization

Map Creation

and updatesDriveNet

WaitNet

HD Map

MapNet

Route Planner

Control

World Model

Lane

Planner

Behavior Planner

Safety Force Field



Learned Planning

SubModule

DRIVE AV

Local World Model


SENSOR SELF-CALIBRATION


EGOMOTION

IMU/Odometry Ego-motion

Camera Ego-motion (SLAM) LIDAR Ego-motion

Radar Ego-motion


DriveWorks

Drive AV

RoadRunner

MapWorks

DNNs

Self-Calibration



Localization

Sensor Abstraction


Actuation

Cloud Mapping

Map Perception

DriveNet

OpenRoadNet

Motion/Structure

PilotNet

LaneNet

PathNet

Frame Packaging and

Serialization

Map Creation

and updatesDriveNet

WaitNet

HD Map

MapNet

Route Planner

Control

World Model

Lane

Planner

Behavior Planner

Safety Force Field



Learned Planning

SubModule

DRIVE AV

Local World Model


CAMERA OBSTACLE PERCEPTION

OBJECT DETECTION

SURROUND OBJECT TRACKING

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DNN DISTANCE

FUTURE MOTION PREDICTION (RNN)


VELOCITY ESTIMATION


LIDAR – OBSTACLE PERCEPTION


LIDAR FREESPACE


LIDAR MOTION DETECTION


RADAR TRACKING


PATH REPRESENTATION (THE LANE GRAPH)


LANENET + TRACKING + FUSION

Tracking and Fusion Lane Assignment (OIPA)


PATHNET


FACTORED PILOTNET TURNS

PATH PERCEPTION ENSEMBLE


WAIT CONDITIONS

46

INTERSECTION DETECTION

47

SIGN AND TRAFFIC LIGHT CLASSIFICATION

CAMERA BLINDNESS DETECTION

PARKING SPACE DETECTION

LIGHT SOURCE PERCEPTION


DRIVE NETWORKS Fully trained DNN solutions for advanced situational awareness Obstacle

Perception

DriveNetDetects objects including vehicles, pedestrians,

bicycles, traffic lights & signs

DepthNet Dense depth inference at pixel level

OpenRoadNet Detects drivable freespace around the vehicle

PredictionNet RNN to predict dynamic objects

LidarNet Lidar point cloud classification

Path

Perception

LaneNet Detects and classifies lanes

MapNet Detects visual landmarks such as lanes & crosswalks

PilotNet Learn trajectories from human driver

PathNet Predicts full geometry of drivable paths

Wait

Condition

Perception

SignNetClassifies traffic signs detected by DriveNet, for US and

EU

LightNetClassifies traffic lights (color, solid, and arrows)

detected by DriveNet

WaitNet Detects intersections

Assistance

Functions

ClearSightNet Determines if the camera view is blocked

AhbNet Auto high beam detection


NVIDIA DRIVE MAPPINGDrive, Update and Create HD Maps

Map LocalizationUsing DRIVE Localization API

Map UpdateUsing DRIVE MapStream API

Map CreationUsing DRIVE MapStream/MapServices API


LOCALIZATION STEPSHERE example

DetectMapNet

ProjectHERE HD Live Map

MatchDLA and CUDA

DriveDRIVE Software


CAMERA LOCALIZATION


LIDAR/RADAR MAPPING+LOCALIZATION


MAP UPDATE STEPSTomTom example on I-85 in CA

DetectMapNet

Create MapStreamsMap update primitives

ConvertMapStreams to Roadgrams

Update MapHD map in the Cloud

TODO from TomTom


MAPSTREAM CREATION IN THE CAR

Traffic sign

Trajectory

Road boundaryDashed

Solid

Pole


VISUALIZING A CREATED MAPSTREAM

http://localhost:3000/?map=a48dc9bf-72ac-5d50-b91a-032369e90afd&branch=ego


Route Planner

Lane Planner

Advanced Behavior Planner

Basic Behavior Planner

Behavior Planner

Route Planner

Latest Safe

Arrival (LSA)

Image

Lane Planner

Safety Force

Field

Search Based

PlanningPrediction

Learned

Planning

Safety

Checks

PLANNING


SAFETY FORCE FIELDHigh-Level Architecture


SAFETY FORCE FIELD

Our vehicle is always perceiving other actors and obstacles.

The Safety Force Field is a computational module that monitors for overlap between our safety procedure and that

of other actors.

The Safety Force Field provides a repelling constraint that

prevents us from contributing to a collision.


SAFETY FORCE FIELD

Longitudinal

Lateral

65

NVIDIA DRIVEDRIVE EXPERIENCE

66NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE IX – CONNECT TO SELF DRIVINGUsing AR and VR

Confidence View Perception view


EXAMPLE USE CASES

Convenience Cockpit MMI Guardian Angel

Side traffic detect

Drowsiness detectDistraction detect

Driver/ Passenger Int face ID/

Greeting/ Settings

Front cross traffic detect

User Ext Face ID /

Trunk opening /

Car start

Safe back-up

PA: Personal Assistant / NLU Hand Gestures detection

Auto adjust mirrors/seats

Respond to sign commandsChild /Pet detect

Emotion detect Health/Wellness detect

PA: AV Driving modes


AI Toolkit for Intelligent Experience

Plugin Manager

TTS EmotionsFace ID

API ManagerIX Server

Inference Hub

IX Core Capabilities

Remote

Experience

Surround

View

Surround

Monitoring

Driver

Monitoring

Head Pose, Gaze,

Distraction,

Drowsiness

Risk AssessmentStitching, Pixel

Level

Segmentation

AVPerceptions

IX

Sensor

Hub

AI CoPilotAI Assistant

IX Speech Engine Partner plugins

CabinSensors

Partner Technology

OTA

SurroundCamera

IX

Sensor

Hub

NLUASR

Remote Rendering Engine(Rich Graphics, Video, Audio)

DRIVE IX

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

AR Viz

69

NVIDIA DRIVEDRIVE SIM

70

DRIVE CONSTELLATION

Hardware in the-Loop System Level Simulator

Simulate Rare and Difficult Conditions

Scalable Platform | Data Center Solution

Timing Accurate and Bit Accurate

Scenario Based Coverage

Virtual Reality AV Simulator

71

DRIVE CONSTELLATION ARCHITECTURE

DRIVE Constellation Simulator

DRIVE Sim API

DRIVE Constellation Vehicle

Environment

Model

DRIVE Constellation OS

DRIVE Sim

Traffic

Model

Vehicle

Model

Sensor

Model

Scenario

Model

DRIVE OS

DRIVE CORE | DRIVE NETWORKS

DRIVE AV

DRIVE IXPerception Mapping Planning

72

CONSTELLATION DATA CENTER WORKFLOW – CONT.


DRIVE PLATFORMMass production

SAFETY

TECHNOLOGY REGULATIONS

INDUSTRY / MARKET

NVIDIA DRIVE UPDATE€¦ · Sensor Complexity - More higher resolution cameras New DNN Models-CNN,...

Documents

Transcript of NVIDIA DRIVE UPDATE€¦ · Sensor Complexity - More higher resolution cameras New DNN Models-CNN,...