THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare...
Transcript of THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare...
![Page 1: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/1.jpg)
Ming Li | Senior Software Manager – Autonomous Machines
THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER
![Page 2: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/2.jpg)
COMPUTERS WRITING SOFTWARE
DATA
PRE
PO
ST
PROGRAMDEEP NEURAL NETWORK
![Page 3: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/3.jpg)
ACCELERATED COMPUTING1000X EVERY 10 YEARS
1980 1990 2000 2010 2020
103
105
107
GPU PERFORMANCECPU PERFORMANCE
+
![Page 4: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/4.jpg)
BILLIONS OF AUTONOMOUS MACHINES
Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City
Retail Logistics Inventory Mgmt Delivery Inspection Service
![Page 5: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/5.jpg)
ROBOTICS CONTROL LOOP
REASON& PLAN
PERCEPTION
ACT
![Page 6: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/6.jpg)
EXAMPLE — AI DELIVERYTotal: 20-30 TOPS
Sensor Fusion
HMI
Stereo Depth
Obstacle Detection
Tracking
Path Planning
Controller
Localization Mapping
![Page 7: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/7.jpg)
VISION NETWORKS
Input Size GOPs/Frame GOPs@30Hz
IMAGE RECOGNITION
MobileNet 224x224 0.6 17
AlexNet 227x227 0.7 22
GoogleNet 224x224 2 60
ResNet-50 224x224 4 120
VGG19 224x224 20 600
OBJECT DETECTION
YOLO-v3 416x416 65 1,950
SSD-VGG 512x512 91 2,730
Faster-RCNN 600x850 172 5,160
Compute Demand
Input Size GOPs/Frame GOPs@30Hz
SEGMENTATION
FCN-8S 384x384 125 3,750
DeepLab-VGG 513x513 202 6,060
SegNet 640x360 286 8,580
POSE ESTIMATION
PRM 256x256 46 1,380
Multipose 368x368 136 4,080
STEREO DEPTH DNN 1280x640 260 7,800
![Page 8: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/8.jpg)
RECENT DEPLOYMENTS OF AI IN ROBOTS
Optical InspectionMusashi Seimitsu
ConstructionKomatsu
Agriculture — WeedingBlue River
Agriculture — HarvestingAgrobot
Last Mile DeliveryRobby Technologies
Inventory ManagementFellow Robots
Drone Aerialtronics
LogisticsIAM Robotics
![Page 9: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/9.jpg)
GROUNDBREAKING RESEARCH
Object Detection & Pose Estimation Trained in Simulation
NVIDIA
Complex In-hand Manipulation Trained in Simulation
OpenAI
Reactive Path Planning in Dynamic Environment
NVIDIA, University of Washington
Physics Based Simulation of Flexible Gripper
NVIDIA
![Page 10: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/10.jpg)
10
JETSON AGX XAVIERWorld’s first AI computer for Autonomous Machines
AI Server Performance in 30W 15W 10W
512 Volta CUDA Cores 2x NVDLA
8 core CPU
32 DL TOPS
![Page 11: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/11.jpg)
XAVIERWorld’s First Autonomous Machine Processor
Volta Tensor Core GPUFP32 / FP16 / INT8 Multi-Precision
512 CUDA Tensor Cores2.8 CUDA TFLOPS (FP16)
22.6 Tensor Core DL TOPS
ISP2.4 GPIX/sNative Full-Range HDRTile-based Processing
Vision Accelerator1.7 TOPS
Stereo & Optical Flow Engine2x 3.1 TOPS
Multimedia Engines1.2 GPIX/s Encode | 1.8 GPIX/s Decode
4 GPIX/s Video Image Compositor
16 Lane CSI109 Gbps CPHY 1.1
1Gb Ethernet
DLA5.7 TFLOPS FP1611.4 TOPS INT8
Carmel ARM64 CPU8 Cores10-wide Superscalar21 SpecInt2K6
Industry Standard High-Speed IOPCle Gen4 Root and EndpointUSB 3.1 Gen2 Host and DeviceUFS 2.1 Embedded Storage
256-Bit LPDDR4X137 GB/s
Most Complex SOC Ever Made | 9 Billion Transistors, 350mm2, 12nFFN | ~8,000 Engineering Years
![Page 12: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/12.jpg)
12
JETSON AGX XAVIER
JETSON TX2 JETSON AGX XAVIER
GPU 256 Core Pascal 512 Core Volta
DL Accelerator - NVDLA x 2
Vision
Accelerator- VLA – 7 way VLIW Processor
CPU 6 core Denver and A57 CPUs 8 core Carmel CPUs
Memory8 GB 128 bit LPDDR4
58.4 GB/s
16 GB 256 bit LPDDR4x
137 GB/s
Storage 32 GB eMMC 32 GB eMMC
Video Encode2x 4K @30
HEVC
2x 4K @ 60 / 4x 4K @30
HEVC
Video Decode2x 4K @30
12 bit support
2x 8K @ 30 / 8x 4K @30
12 bit support
CameraUp to 6 cameras
CSI2 D-PHY 1.2 2.5Gbps/lane
Up to 8 cameras
CSI2 D-PHY 1.2 2.5 Gbps/lane
Mechanical50mm x 87mm
400 pin connector
100mm x 87mm
699 pin connector
New!
+2
x4
x2
x2
New!
New!
New!
![Page 13: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/13.jpg)
13
JETSON AGX XAVIER20X PERFORMANCE IN 18 MONTHS
55
112
JetsonTX2
JetsonAGX Xavier
1.3
11
JetsonTX2
JetsonAGX Xavier
1.3
32
Jetson TX2 Jetson AGX Xavier
24x DL / AI 8x CUDA 2x CPU
58
137
JetsonTX2
JetsonAGX Xavier
2.4x DRAM BW
2
8
JetsonTX2
JetsonAGX Xavier
4x CODEC
TO
PS TFLO
PS (
FP16)
Cum
. D
MIP
S
GB/s
4K E
ncode a
nd D
ecode
![Page 14: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/14.jpg)
JETSON AGX XAVIER
0
400
800
1200
1600
Core i7 + GTX 1070 Jetson AGX Xavier
GPU Workstation Perf 1/10th Power, Size
ResN
et-
50 Im
ages/
sec
1.4x
0
10
20
30
40
50
60
70
Core i7 + GTX 1070 Jetson AGX XavierResN
et-
50 Im
ages/
sec/W
14x
AI Inference Performance AI Inference Efficiency
![Page 15: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/15.jpg)
15
Intel Core i7 Skylake + GTX 1070
>200 W
28 TOPS
4000 cm3
Consumer grade
AUTONOMOUS MACHINES PROTOTYPE
![Page 16: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/16.jpg)
16
1 Jetson AGX Xavier
30 W
32 TOPS
600 cm3
Commercial grade
JETSON AGX XAVIER PRODUCTION PLATFORM
![Page 17: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/17.jpg)
NVIDIA ISAAC — PLATFORM FOR ROBOTICS
SIMULATION DEPLOYMENTDEVELOPMENT
ISAAC SIM
VIRTUAL SENSORS
PLUGINS
PHYSICS
ENGINE
RENDERERNVIDIA
GPU CLOUD
ISAAC SIM
MAPPING &
LOCALIZATION
PHYSICS
ENGINE
SITUATION
UNDERSTANDINGPERCEPTION
ISAAC SIM
32 DL TOPSHI-BANDWIDTH
SERDES
DIVERSITY &
REDUNDANCY
7 DIFFERENT
PROCESSORS
PATH &
TASK PLANNING
![Page 18: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/18.jpg)
NVIDIA ISAAC SDK
Perception
Isaac Framework
Drivers and APINavigation Visualization
Jetpack SDK ∙ CUDA ∙ TensorRT ∙ TensorFlow ∙ ONNX ∙ ROS
Stereo DNN
3D Pose Estimation
Object Detection
Model-based Tracking
Human Pose Detection
Path planning
Obstacle Avoidance
Mapping
Plots
Custom Drawings
3D Renderings
Inspection
Camera
LIDAR
IMU
Pan/Tilt
Sound Source Loc.
Applications
Message
Passing
Jetson AGX Xavier
ConfigurationNode/Component
Model Behaviors
Record /
Replay
Task
Scheduling
Distributed
Compute
![Page 19: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/19.jpg)
NVIDIA ISAAC GEMS
Object / People DetectionGlobal Localization LQR Path Planner Depth Estimation Human Pose Estimation
Map Editor Visual Odometry Physical Simulation Gesture Recognition ASR
![Page 20: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/20.jpg)
![Page 21: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/21.jpg)
21
JETPACK SDK
FOR AI AT THE EDGE
JETSON SDKS OVERVIEW
JETSON AGX XAVIER
DEEPSTREAM SDK
FOR VIDEO ANALYTICS
ISAAC SDK
FOR ROBOTICS
![Page 22: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/22.jpg)
22
EXAMPLE – VIDEO ANALYTICS
Typical application: 30+ TOPS
![Page 23: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/23.jpg)
23
NVIDIA DEEPSTREAMZero Memory Copies
Typical multi-stream application: 30+ TOPS
![Page 24: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/24.jpg)
JETSON AGX XAVIER ECOSYSTEM
SENSORS
SYSTEM DESIGN
SYSTEM SOFTWARE/TOOLS
MuJoCo
QUICK START PLATFORMS
AI SOFTWARE
RESEARCH
DISRIBUTORS WORLDWIDE
![Page 25: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/25.jpg)
![Page 26: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/26.jpg)
JETSON AGX XAVIER DEVKITWorld’s First AI Computer for Autonomous Machines
AVAILABLE NOW
AI Server Performance in 30W | 15W | 10W
512 Volta CUDA Cores | 2x NVDLA
8 Core CPU
32 DL TOPS
![Page 27: THE FUTURE OF ROBOTICS WITH JETSON AGX XAVIER · Industrial Aerospace/Defense Healthcare Construction Agriculture Smart City ... Physics Based Simulation of Flexible Gripper NVIDIA.](https://reader035.fdocuments.in/reader035/viewer/2022071215/604577b71185e02a9d3641db/html5/thumbnails/27.jpg)
27