NVIDIA DRIVE UPDATE€¦ · Sensor Complexity - More higher resolution cameras New DNN Models-CNN,...

December 2019

NVIDIA DRIVE

2NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE PLATFORMMass production

SAFETY

TECHNOLOGY REGULATIONS

INDUSTRY / MARKET

THE DRIVE INITIATIVEEnd-to-End, Open Platform for Building Autonomous Vehicles

DGX Saturn V Constellation Xavier

DRIVE AV Re-SIMDRIVE IX

The NVIDIA DRIVE™ Platform provides all major AI technologies for Driving, Development and Validation -from driving & cockpit computers & software to data collection & DNN training, to simulation tools. The system is an open platform with multiple programming points. Car makers are free to use or develop as much as they like and take advantage of an ecosystem of hundreds of partners.

SIMULATETRAIN MODELSCOLLECT DATA DRIVE

NVIDIA DRIVEEND-TO-END PLATFORM

Lanes Lights

PedestriansCars

Lanes Lights

PedestriansCars

DRIVE COMPUTER / ARCHITECTURESCALABLE NOW & INTO THE FUTURE

NCAP TO ROBOTAXI

DRIVE ROADMAP

DRIVE Parker

1x Perf

1x Efficiency

Samples – Q4’2015

Production – Q1’2017

DRIVE PX2

20x Perf

1x Efficiency

DRIVE Orin-X

>100x Perf

~20x Efficiency

Samples – Mid 2021

Production - Q3’2022

DRIVE Xavier

23x Perf

8x Efficiency

DRIVE Pegasus

~230x Perf

5x Efficiency

Auto-Grade

Super Energy-Efficient

ORIN – HIGHEST PERF SOC EVERWhat is driving the need for increasing performance in AV?

Sensor

Complexity -

More higher

resolution

cameras

New DNN Models-

CNN, RCNN, RNN

DNN GOPs/Frame

increasing

20+ DNN Models

required for most

demanding ODD

DNN Precision /

Accuracy

OTA new features

/ SW Patch

Integration of

functions

Centralized

Computer

Future Proof

EUNCAP

With 360°

Perception

More perf equates to faster time to market

NVIDIA DRIVEOne Architecture

NVIDIA DRIVE AGX platform is an open platform; allows partners to do their own innovation

PEGASUS (L4/RoboTaxis)XAVIER (NCAP, L2+/L3)

One AV Architecture for L2 -> Robo Taxi

DRIVE ARCHITECTUREAUTOMOTIVE GRADE – INDUSTRIALIZED ECU’S

NVIDIA DRIVESENSOR ECOSYSTEM

SENSOR ECOSYSTEM

https://developer.nvidia.com/drive/drive-core-sensors

GNSS/IMURADAR

LIDARIMAGING SENSOR / CAMERA

AR0820, AR0144, AR0231, IMX390, IMX490, IMX424

ARS430

SC13S, MTi-G-710, SPAN-IGM A1

NVIDIA DRIVEDRIVE SOFTWARE

NVIDIA DRIVE SOFTWARE ARCHITECTUREOpen and modular software for AV development

DRIVE AV

DRIVE Planning

DRIVE Mapping

DRIVE Perception

DRIVE IX

Visualization AI CoPilot AI Assistant

DRIVE Networks DRIVE Calibration

DRIVE OS

DRIVE AGX DEVELOPER KITS(Xavier/Pegasus)

DRIVE HYPERION(Reference Architecture)

DRIVE Core

DRIVEWORKS

NVIDIA DRIVEDRIVE OS

DRIVE AGX AV Software Platform

Linux BSP

OS Drivers, USB,

File System, Network

Hardware

Software

OS/3rd Party SoftwareNVIDIA SoftwareHardware

Hypervisor

Radar Lidar GPS IMU

XavierMCU

Flexray10G/1G/BR Ethernet

DRIVE OS

TensorRTcuDNN

Open G

Foundati

Securi

Cameras

(via CSI-2)NOR UFSeMMCPCIe NVMEPMIC

NVLINK

Managem

Notes: 1) For development only.

DriveWorks

OEM/T1 Application

Optional Services

Scalability;+Xavier, NvME, or Optional Processor

DRIVE AGX AV Software Platform

Hardware

Software

OS/3rd Party SoftwareNVIDIA SoftwareHardware

Hypervisor

Radar Lidar GPS IMU

XavierMCU

Flexray10G/1G/BR Ethernet

DRIVE OS

TensorRTcuDNN1

QNX BSP

OS Drivers, USB1,

File System, Network

Open G

Foundati

Securi

Cameras

(via CSI-2)NOR UFSeMMCPCIe NVMEPMIC

Scalability w/dGPU

NVLINK

Scalability;+Xavier, NvME, or Optional Processor

Managem

DRIVE AV

Optional ServicesNotes: 1) Not for use in production in a safety context.

NVIDIA DRIVEAUTONOMOUS DRIVING

APPLICATION

NVIDIA DRIVEWORKS SDKData Abstraction | Calibration | Compute Modules

DRIVE AV DRIVE IX

DRIVEWORKS

DRIVE Core

DRIVE OS

DRIVE AGX DEVELOPER KITS DRIVE HYPERION

Sensor Abstraction

Layer + Plugins

Vehicle IO

DNN Frameworks

Tools (Recorder)

Image Processing

Point Cloud Processing

DRIVE Networks

DriveNet OpenRoadNet

SignNetLightNet

ClearSightNet

PathNet

WaitNet

MapNet LaneNet

DRIVE Calibration

Egomotion

Calibration

NVIDIA DRIVEWORKS SDKFoundation for AV SW Development on NVIDIA DRIVE

- Open, modularized library of functions

- Optimized for DRIVE AGX

- Easy to integrate and build upon

SOFTWARE MODULES

- Software dev tools ranging from sensor

data capture to calibration and

visualization

- Sample apps

- Documentation, tutorials

- Active developer forum and community

DEVELOPER SUPPORT

- Architected for automotive safety

- Compliant with industry standards, (ISO

26262/21448, MISRA)

PRODUCTION GRADE

DriveWorks

Drive AV

RoadRunner

MapWorks

Self-Calibration

Obstacle Perception Path Perception Wait Perception

Egomotion Estimation

Localization

Sensor Abstraction

Low Level Processing

Actuation

Cloud Mapping

Map Perception

DriveNet

OpenRoadNet

Motion/Structure

PilotNet

LaneNet

PathNet

Frame Packaging and

Serialization

Map Creation

and updatesDriveNet

WaitNet

HD Map

MapNet

Route Planner

Control

World Model

Planner

Behavior Planner

Safety Force Field

Search Based Planning AI Based Prediction

LSA ImageSafety Checks

Learned Planning

SubModule

DRIVE AV

Local World Model

SCALABLE PLATFORMOne architecture – Highway and Urban AV Driving

ON RAMP TO OFF RAMP URBAN: ADDRESS TO ADDRESS

ADVANCED FUNCTIONSAuto High Beam Clear Sight

NCAPParking

DriveWorks

Drive AV

RoadRunner

MapWorks

Self-Calibration

Localization

Sensor Abstraction

Actuation

Cloud Mapping

Map Perception

DriveNet

OpenRoadNet

Motion/Structure

PilotNet

LaneNet

PathNet

Frame Packaging and

Serialization

Map Creation

and updatesDriveNet

WaitNet

HD Map

MapNet

Route Planner

Control

World Model

Planner

Behavior Planner

Safety Force Field

Learned Planning

SubModule

DRIVE AV

Local World Model

SENSOR SELF-CALIBRATION

EGOMOTION

IMU/Odometry Ego-motion

Camera Ego-motion (SLAM) LIDAR Ego-motion

Radar Ego-motion

DriveWorks

Drive AV

RoadRunner

MapWorks

Self-Calibration

Localization

Sensor Abstraction

Actuation

Cloud Mapping

Map Perception

DriveNet

OpenRoadNet

Motion/Structure

PilotNet

LaneNet

PathNet

Frame Packaging and

Serialization

Map Creation

and updatesDriveNet

WaitNet

HD Map

MapNet

Route Planner

Control

World Model

Planner

Behavior Planner

Safety Force Field

Learned Planning

SubModule

DRIVE AV

Local World Model

CAMERA OBSTACLE PERCEPTION

OBJECT DETECTION

SURROUND OBJECT TRACKING

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DNN DISTANCE

FUTURE MOTION PREDICTION (RNN)

VELOCITY ESTIMATION

LIDAR – OBSTACLE PERCEPTION

LIDAR FREESPACE

LIDAR MOTION DETECTION

RADAR TRACKING

PATH REPRESENTATION (THE LANE GRAPH)

LANENET + TRACKING + FUSION

Tracking and Fusion Lane Assignment (OIPA)

PATHNET

FACTORED PILOTNET TURNS

PATH PERCEPTION ENSEMBLE

WAIT CONDITIONS

INTERSECTION DETECTION

SIGN AND TRAFFIC LIGHT CLASSIFICATION

CAMERA BLINDNESS DETECTION

PARKING SPACE DETECTION

LIGHT SOURCE PERCEPTION

DRIVE NETWORKS Fully trained DNN solutions for advanced situational awareness Obstacle

Perception

DriveNetDetects objects including vehicles, pedestrians,

bicycles, traffic lights & signs

DepthNet Dense depth inference at pixel level

OpenRoadNet Detects drivable freespace around the vehicle

PredictionNet RNN to predict dynamic objects

LidarNet Lidar point cloud classification

Perception

LaneNet Detects and classifies lanes

MapNet Detects visual landmarks such as lanes & crosswalks

PilotNet Learn trajectories from human driver

PathNet Predicts full geometry of drivable paths

Condition

Perception

SignNetClassifies traffic signs detected by DriveNet, for US and

LightNetClassifies traffic lights (color, solid, and arrows)

detected by DriveNet

WaitNet Detects intersections

Assistance

Functions

ClearSightNet Determines if the camera view is blocked

AhbNet Auto high beam detection

NVIDIA DRIVE MAPPINGDrive, Update and Create HD Maps

Map LocalizationUsing DRIVE Localization API

Map UpdateUsing DRIVE MapStream API

Map CreationUsing DRIVE MapStream/MapServices API

LOCALIZATION STEPSHERE example

DetectMapNet

ProjectHERE HD Live Map

MatchDLA and CUDA

DriveDRIVE Software

CAMERA LOCALIZATION

LIDAR/RADAR MAPPING+LOCALIZATION

MAP UPDATE STEPSTomTom example on I-85 in CA

DetectMapNet

Create MapStreamsMap update primitives

ConvertMapStreams to Roadgrams

Update MapHD map in the Cloud

TODO from TomTom

MAPSTREAM CREATION IN THE CAR

Traffic sign

Trajectory

Road boundaryDashed

VISUALIZING A CREATED MAPSTREAM

Route Planner

Lane Planner

Advanced Behavior Planner

Basic Behavior Planner

Behavior Planner

Route Planner

Latest Safe

Arrival (LSA)

Lane Planner

Safety Force

Search Based

PlanningPrediction

Learned

Planning

Safety

Checks

PLANNING

SAFETY FORCE FIELDHigh-Level Architecture

SAFETY FORCE FIELD

Our vehicle is always perceiving other actors and obstacles.

The Safety Force Field is a computational module that monitors for overlap between our safety procedure and that

of other actors.

The Safety Force Field provides a repelling constraint that

prevents us from contributing to a collision.

SAFETY FORCE FIELD

Longitudinal

Lateral

NVIDIA DRIVEDRIVE EXPERIENCE

66NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE IX – CONNECT TO SELF DRIVINGUsing AR and VR

Confidence View Perception view

EXAMPLE USE CASES

Convenience Cockpit MMI Guardian Angel

Side traffic detect

Drowsiness detectDistraction detect

Driver/ Passenger Int face ID/

Greeting/ Settings

Front cross traffic detect

User Ext Face ID /

Trunk opening /

Car start

Safe back-up

PA: Personal Assistant / NLU Hand Gestures detection

Auto adjust mirrors/seats

Respond to sign commandsChild /Pet detect

Emotion detect Health/Wellness detect

PA: AV Driving modes

AI Toolkit for Intelligent Experience

Plugin Manager

TTS EmotionsFace ID

API ManagerIX Server

Inference Hub

IX Core Capabilities

Remote

Experience

Surround

Monitoring

Driver

Monitoring

Head Pose, Gaze,

Distraction,

Drowsiness

Risk AssessmentStitching, Pixel

Segmentation

AVPerceptions

Sensor

AI CoPilotAI Assistant

IX Speech Engine Partner plugins

CabinSensors

Partner Technology

SurroundCamera

Sensor

NLUASR

Remote Rendering Engine(Rich Graphics, Video, Audio)

DRIVE IX

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

AR Viz

NVIDIA DRIVEDRIVE SIM

DRIVE CONSTELLATION

Hardware in the-Loop System Level Simulator

Simulate Rare and Difficult Conditions

Scalable Platform | Data Center Solution

Timing Accurate and Bit Accurate

Scenario Based Coverage

Virtual Reality AV Simulator

DRIVE CONSTELLATION ARCHITECTURE

DRIVE Constellation Simulator

DRIVE Sim API

DRIVE Constellation Vehicle

Environment

DRIVE Constellation OS

DRIVE Sim

Traffic

Vehicle

Sensor

Scenario

DRIVE OS

DRIVE CORE | DRIVE NETWORKS

DRIVE AV

DRIVE IXPerception Mapping Planning

CONSTELLATION DATA CENTER WORKFLOW – CONT.

DRIVE PLATFORMMass production

SAFETY

TECHNOLOGY REGULATIONS

INDUSTRY / MARKET

NVIDIA DRIVE UPDATE€¦ · Sensor Complexity - More higher resolution cameras New DNN Models-CNN,...

Documents

Transcript of NVIDIA DRIVE UPDATE€¦ · Sensor Complexity - More higher resolution cameras New DNN Models-CNN,...

Packaging DNN extensions

DNN Module Development Company - DNN Extension

Dnn Tutorial

AIIA DNN benchmark

A Southern Fried Buffet of DNN Goodness - DNN Community, User Groups & More

FASTER-RCNN BASED DEEP LEARNING MODEL FOR POMEGRANATE …

Lecture 10: Faster RCNN

DNN Intro v2.1

Draft White Paper - nerc.com · or requirements that apply to GOs and/or GOPs. The goal of the proposed applicability changes is to provide the GOs and GOPs of dispersed generation

Technical Communication and DNN - DotNetNuke - DNN Connect 2014

A 110 GOPS/W 16-bit Multiplier and Reconfigurable PLA Loop in 90 ...

DNN Accelerator Architectures

NOTE-RCNN: NOise Tolerant Ensemble RCNN for Semi ...€¦ · training samples using the detectors and then train detec-tors with the updated training samples. The detector can be

An application of Residual Network and Faster - RCNN for Medico ...ceur-ws.org/Vol-2283/MediaEval_18_paper_11.pdf · An application of Residual Network and Faster - RCNN for Medico:

Multiple Scale Faster-RCNN Approach to Driver's Cell-Phone ... · Multiple Scale Faster-RCNN Approach to Driver’s Cell-phone Usage and Hands on Steering Wheel Detection ... detector

Multi-adversarial Faster-RCNN for Unrestricted Object Detection · 2019-09-10 · Multi-adversarial Faster-RCNN for Unrestricted Object Detection Zhenwei He Lei Zhang School of Microelectronics

Accelerators DNN Designing

Pr057 mask rcnn

Dnn for beginners

DNN Sentinel 2_2_final.pdf