DEEPSTREAM SDK 2 - NVIDIA€¦ · 3rd party IP plugin (Face recognition) facedetect To jump start...
Transcript of DEEPSTREAM SDK 2 - NVIDIA€¦ · 3rd party IP plugin (Face recognition) facedetect To jump start...
Presenter : Ettikan Karuppiah (Ph.D), Director Developers Ecosystem
AI Conference Sydney, Australia. 3rd September 2018
DEEPSTREAM SDK 2.0 “ACCELERATES VIDEO ANALYTICS DEVELOPMENT”
Deepstream application Intelligent Video Analytics
Traffic Engineering Self-service Store
Access control Parking Management
3NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
DeepStream 2.0 SDK
NVIDIA Platforms
Linux, CUDA
DeepStream 2.0 SDK
Sample
DNNs &
plug-ins
3rd party
application
plugins
Modular,
Scalable,
Architecture
Development
Tools
Retail analytics & checkout
Intelligent traffic systems
Law
enforcemen
t
Smart parking
Access control
User Applications
Solution
adaptation
guide
* Early release and limited availability.
TensorRT
cuDNN
Multimedia
Multi-stream Encode &
Decode
Imaging
Image capture &
processing
Deep Learning
4
INTELLIGENT VIDEO ANALYTICS
Cameras Decode Detectors/Classifiers/Trackers Composite Encode Metadata
TESLA JETSON
5
END-TO-END PRODUCT FAMILY
HPC/TRAINING INFERENCE
EMBEDDED
Jetson TX1
DATA CENTER
Tesla P4
AUTOMOTIVE
Drive PX2
Tesla P100Tesla V100Titan V
DATA CENTERDESKTOP
FULLY INTERGRATED DL SUPERCOMPUTER
Tesla V100TITAN V
DESKTOP WORKSTATIONDATA
CENTER
Tesla V100DGX StationTITAN Quadro
DGX-1 DGX-2
6
DEEPSTREAM SDK
7
BUILT ON WIDELY USED SOFTWARE
JETSON / TESLA PLATFORM
DEEPSTREAM PLUGINS
LOW LEVEL API
APPLICATION
NVCUVID
TENSORRT
CUDA
CUDNN
8
WHAT’S IN DEEPSTREAM 2.0
Reference Applications
Accelerated Plugins & Utilities
RESNET Models
Sample examples
Reference Video clips
Documentation
CUDA 9.2 , TensorRT 4.0, GStreamer 1.8.3
9
GSTREAMER
PLUGINS – Basic building block for media processing pipeline.
PADS – Pads are plugin’s input and output, where you can connect other plugins. They are used to negotiate links and data flow.
BINS - A bin is a container for a collection of plugins.
PIPELINE – Top level bin. It provides a bus for the application and manage the synchronization for its child plugins.
10
GSTREAMER
Buffers – objects for passing streaming data between plugins in the pipeline.
Events – events are objects sent between plugins or from the application to plugins.
Messages – objects posted by plugins on the pipeline's message bus, where they will be held for collection by the application.
Queries - allow applications to request information from the pipeline.
11
DEEPSTREAM PIPELINE ARCHITECTURE
Jetson
GPU CPU GPU
VIC VIC VIC
GPU DMANVDEC CPU
VIC
GPU
CPU
NVDC
Tesla
NVDEC
CPU
GPU DMA GPU CPU
GPU
GPU CPU
GPU
NVDisplay
GST-uridecodebin
GST-NvStreamMux
GST-NvInferGST-
NvTracker GST-NvInferGST-
nvvidconv GST-NvOsdGST-
NvEglGlesSink
libnvtensor-rt
(TensortRT)libnvtracker
(visionworks)
Low Level APIs
DeepStream
Libnvtensor-rt
(TensortRT) libnvosdlibnvvidconv
VIC
GPU
decode
12
LIST OF PLUGINS
Plugin Type Functionality
Decoder H265 / H264 Video decoders
Format Convertor Scaling, Format conversion, rotation
Aggregator StreamMux and Batching
Inferencing Primary / Secondary detector / classifier
Tracker Reference object tracker
Tiler Video wall output
On Screen Display For drawing text, rectangles lines etc.
Template Example plugin for 3rd party IP / models
13
VIDEO AGGREGRATOR
Plugin that accepts “n” inputs streams and converts to sequential batch frames
Scaling support – Incase video input resolution differs with the model resolution or vice-a-versa
Name - nvstreammux
nvstreammux
NPP
CPU + GPU
YUV
batched YUV +
metadata
Low Level API
HW
14
VIDEO AGGREGRATOR
Contiguous buffer of
size W x H x N
allocated by
GstNvStreamMuxer
Async Cuda
Memcpy
Queue for S1 buffers
Queue for S2 buffers
Queue for SM buffers
Batching algorithm
Batch of N
buffers of size (W
x H)
Decoder / Camera 1
Decoder / Camera 2
Decoder / Camera M
GST - NvInfer
Round Robin
then FCFS
15
SCREEN TILER
Used for creating video wall effect
Arranges multiple input sources into complete video tiled output
Configurable window size
Name - nvstreamtiler
nvstreamtiler
CUDA
GPU
Low Level API
HW
YUV + All Meta YUV + All Meta
16
ON SCREEN DISPLAY
17
… AND MANY MORE PLUGINSDESCRIPTION NAME
Read from arbitrary point in a file filesrc
Receive data over the network via RTSP rtspsrc
Reads frames from a Video4Linux2 device v4l2src
X11 based videosink xvimagesink
H264 Encoder x264Enc
Decode / Encode in / from JPEG format jpegdec/enc
Dewarp fish eye video Dewarp
3rd party IP plugin (License plate recognition) ALPR
3rd party IP plugin (Face recognition) facedetect
To jump start development for adding custom IP ds-example
18
DEEPSTREAM REFERNCE APPLICATION
1) Supports N simultaneously input sources.
• File Input
• RTSP input
• Camera input
2) Supports H265 / H264 decode.
3) 1 Primary object detector + 3 Attribute classifiers + Tracker.
4) Application generates metadata in KITTI format.
19
DEEPSTREAM REFERENCE APPLICATION
20
DEEPSTREAM REFERENCE APP OUTPUT
21
DEEPSTREAM REFERENCE APP OUTPUT
22
DEEPSTREAM APPLICATIONMap to DeepStream plugins
Gst-rtspsrc
IMAGES or VIDEO
Gst-
decodebinGst-infer Gst-tracker Gst-nvosd
Gst-
nveglglessink
IP CAMERA DECODE INFER OBJECT
TRACKERDISPLAY
Analytics
plugin
(Counting)
23
(Car-LPR)
(Car-Color)
(Car-Model)
VEHICLE DETECTION
GST-NvInfer
(Car-Detect)
Gst-
uridecode
GST-
NvTracker
GST-NvInfer
(Car-Make)
GST-
NvEglglessin
k
GST-OSD GST-Tiler
Video
Decode
Primary
Detector
Object
Tracker
Secondary Classifiers On Screen
Display
Tiler Renderer
24
VEHICLE DETECTION
Inserting video: Insert/Video/Video from File.Insert video by browsing your directory and selecting OK.
File types that works best in PowerPoint are mp4 or wmv
1) Describe/Design your use case and work flow
2) Understand required plugins for the defined workflow and
associated input and output for each stage/plugins
3) Check if you can use the reference plugins provided in DS SDK2.0
4) No, create new plugin (refer to plugin implementation guide)
5) Link the plugins and test your application
6) If you are happy with the apps, review the performance
HOW DO I DEVELOP MY DS APPS?