A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF...
Transcript of A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF...
![Page 1: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/1.jpg)
Thomas True
A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF 360 VIDEO AND AUDIO
![Page 2: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/2.jpg)
2
AGENDA
Stereo Panoramas
VRWorks 360 Video SDK Overview
Calibration
Video Stitching
Audio Processing
Multi-GPU Optimization
VRWorks SDK Usage Examples
Q&A
![Page 3: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/3.jpg)
3
STEREO PANORAMAS
![Page 4: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/4.jpg)
4
STEREO PANORAMAS
� 360 Video
� Immersive
� Provide sense of depth and scale
� Capture reality
Stereo Video in VR
![Page 5: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/5.jpg)
5
STEREO PANORAMASWorkflow
Ingest Decode Calibrate Encode
VRWORKS 360 VIDEO
OutputStitch
![Page 6: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/6.jpg)
High-Level API
360 STEREO STITCHING PIPELINE
Low-Level
API
Core
Video
Core
Audio
Calibration
Input:
Decode
Demux
Output:
Encode
Mux/Sync
Generate
rig
XML
Save
to
XML
(GPU)
(CPU) (CPU)
(GPU)
Rig Parameters(Estimated)
MP4/MOV(H.264 with audio)
UncompressedVideo (RGBA)
Separate video andaudio streams
Rig XML
Stitcher XML
RGBAFrames
Rig
Audio Frames(PCM or AAC)
StitchedPanoramas
(RGBA)
Blended Audio Frames
Rig Parameters(Calibrated) Calibrated Rig XML
MP4/MOV (H.264with blended audio)
UncompressedVideo (RGBA)
Elementary H.264 streamblended audio stream
![Page 7: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/7.jpg)
7
VRWORKS 360 VIDEO SDK
![Page 8: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/8.jpg)
8
VRWORKS 360 VIDEO SDK
�GPU-optimized stereo pipeline
�GPU-accelerated calibration
�GPU-accelerated encode and decode
�Audio mix down and stretch
�Audio encode / decode
�Audio resampling
�Realtime and offline
�Stereo and mono stitching
�Compressed and uncompressed inputs and outputs
MP4, H.264 and RGBA8
PCM and AAC audio
Features
https://developer.nvidia.com/vrworks/vrworks-360video
![Page 9: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/9.jpg)
9
VRWORKS 360 VIDEO SDKWorks With Multiple Rigs
![Page 10: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/10.jpg)
10
VRWORKS 360 VIDEO SDKWorkflow Flexibility
DemuxMP4
Video
Audio
H.264
Audio
+
RGBA8UI
+
Audio
Stitch
MuxVideo
Audio
MP4
H.264
Audio
+
RGBA8UI
+
Audio
File Based Workflow
Stream Based Workflow
Buffer Based Workflow
![Page 11: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/11.jpg)
11
VRWORKS 360 VIDEO SDKHigh-Level Video and Audio Pipeline
Stitch
RGB8
DemuxMP4
H.264
Audio
Decode
H.264
Audio
Decode RGBA8
Encode
Encode
RGBA8
MuxRGBA8 H.264
Audio
RGBA8
Audio
H.264
RGBA8
![Page 12: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/12.jpg)
12
VRWORKS 360 VIDEO SDK
Advantages:
� GPU-accelerated demux, mux, decode and encode
� File and stream oriented API
� Includes calibration API
� Support for additional audio and video formats
High-Level Video and Audio Pipeline
![Page 13: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/13.jpg)
13
VRWORKS 360 VIDEO SDKLow-Level Video and Audio Pipeline
RGBA8UI
Stitch
FP32 PCM
RGBA8
FP32 PCM
![Page 14: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/14.jpg)
14
VRWORKS 360 VIDEO SDK
Advantages:
� Works directly with CUDA device memory
� Plugs directly into a larger CUDA pipeline
� Interops with graphics API to keep frames on GPU
� Application flexibility to control threading and data transfers
Low-Level Video and Audio Pipeline
![Page 15: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/15.jpg)
15
CALIBRATION
![Page 16: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/16.jpg)
16
CALIBRATIONGPU-Based Feature Extraction and Correspondence Matching
Calibration
RigSpecs
Sampleimages
Stitcher
UpdatedRigSpecs
1. Input initial rig specs and a set of images from the cameras
2. GPU-accelerated feature extraction
3. Construct feature correspondence pairs from overlapped regions
4. Optimize rig parameters to minimize the correspondence distances(on the virtual sphere)
![Page 17: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/17.jpg)
17
CALIBRATIONGPU-Based Feature Extraction and Correspondence Matching
![Page 18: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/18.jpg)
18
CALIBRATION
� Works with any rig configuration
� High error tolerance in input instances.
� Accommodates lenses with up to or greater that 180° FOV
� Support standard distortion characteristics as well as custom fisheye lenses
� Provides accuracy metric based upon cross correlation [0:1]
� GPU-accelerated
Features
![Page 19: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/19.jpg)
19
VIDEO STITCHING
![Page 20: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/20.jpg)
20
VIDEO STITCHING
1. Compute pixel motion between adjacent cameras
� Lambertian: Appearance not view dependent
� Cameras photometrically consistent
� Rig calibrated
� Surface point same when viewed by adjacent cameras
� Compute disparity
2. Interpolate to the virtual camera position, reprojection the virtual sphere and blend.
3. Apply post processing
Creating Virtual Cameras Between Physical Cameras
![Page 21: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/21.jpg)
21
IMAGE ALIGNMENT CHALLENGES
texturelessregions
occlusionperiodic textures
specular surfaces
![Page 22: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/22.jpg)
22
VIDEO STITCHING
� Detect occlusion boundaries
� Reduce pixel motion noise between occlusion boundaries
� Fill holes (textureless regions) with plausible motion
� Enforce temporal coherence
Overcoming Pixel Alignment Challenges
VideoStitch
RigSpecs
StitcherSpecs
![Page 23: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/23.jpg)
23
AUDIO
![Page 24: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/24.jpg)
24
AUDIO PROCESSING
� Stereo mixdown with adjustable spread effect
� MP4 embedded audio, separate audio streams or individual audio buffers
� PCM and AAC audio streams
� Audio resampling
� Performed in parallel on the CPU while the video is stitching on the GPU
� Unlimited number of internal and external audio sources
� Push-pull model
Overview
![Page 25: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/25.jpg)
25
AUDIO PROCESSING
Audio Mixer
L
R
Stereo Mixdown With Spread Effect
1. Convert all inputs to mono.2. Add together3. Spread to stereo.
Application programs depth of spread
4. Add gain
Result: Output volume same as input
![Page 26: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/26.jpg)
26
MULTI-GPU OPTIMIZATIONS
![Page 27: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/27.jpg)
27
CHALLENGE
� Need to decode 8 separate 4K streams at 30 fps (similar to 240 fps!)
� After getting frames to GPU, will we have any time left to stitch?
How can we decode and stitch in real-time?
![Page 28: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/28.jpg)
28
PIPELINING FOR PERFORMANCEPipelining Ingest, Stitch, and Output for Better Throughput
DecodeFrame N
StitchFrame N
OutputFrame N
DecodeFrame N+1
StitchFrame N+1
DecodeFrame N+2
OutputFrame N+1
StitchFrame N+2
OutputFrame N+2
Time
![Page 29: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/29.jpg)
29
CHALLENGE
� Must copy input/output between GPUs and CPU
� Synchronous memory copy injects bubbles in compute workload
Minimizing Memory Copy Latency
![Page 30: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/30.jpg)
30
HIDING MEMORY LATENCYUsing CUDA Streams and Asynchronous Memory Copies
![Page 31: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/31.jpg)
31
CHALLENGE
� We can stitch lower-quality on one GPU
� We want more cameras, higher quality, higher output resolution
Maximum Quality Achievable in Real-Time
ProjectDisparity
Map
Post-
Process
Interpolate
& Blend
![Page 32: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/32.jpg)
32
MULTI-GPU SCALINGDistribute Stereo Pairs Among Available Devices
![Page 33: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/33.jpg)
33
STITCHING PERFORMANCE8x 4K input streams
STITCH TYPE GPU OUTPUT STITCHING RATE (FPS)
Stereo 2x Quadro P6000 Uncompressed 32
Stereo 2x Quadro P6000 MP4 30
Mono Quadro P6000 Uncompressed 40
Mono GTX 1060/1080 Uncompressed 40
![Page 34: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/34.jpg)
34
SAMPLE APPLICATIONS
![Page 35: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/35.jpg)
35
VRWORKS 360 VIDEO SDK
// Initialize stitcher instanceRETURN_NVSS_ERROR(nvssVideoCreateInstance(&stitcher_props, ¶ms->rig_properties, &stitcher));
// Load image frames for each camerafor (uint32_t camera = 0; camera < params->rig_properties.num_cameras; camera++){
nvstitchImageBuffer_t input_image;RETURN_NVSS_ERROR(nvssVideoGetInputBuffer(stitcher, camera, &input_image));
// Read image from filegetRgbaImage(image_file_path, &rgba_bitmap_ptr, image_width, image_height);
// Copy image data to CUDA device memorycudaMemcpy2D(input_image.dev_ptr, input_image.pitch, rgba_bitmap_ptr, input_image.row_bytes,
input_image.row_bytes, input_image.height, cudaMemcpyHostToDevice);}
// StitchRETURN_NVSS_ERROR(nvssVideoStitch(stitcher));
Low-Level Video API Example
Create stitcher from rig
parameters.
Get video input buffer
Copy image to CUDA
device memory
Execute stitcher
![Page 36: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/36.jpg)
36
VRWORKS 360 VIDEO SDK
for (int eye = 0; eye < num_eyes; eye++){
// Fetch stitched panorama(s)if (stereo)
RETURN_NVSS_ERROR(nvssVideoGetOutputBuffer(stitcher, nvstitchEye(eye), &output_image));else
RETURN_NVSS_ERROR(nvssVideoGetOutputBuffer(stitcher, NVSS_EYE_MONO, &output_image));
if (out_stacked == nullptr)out_stacked = (unsigned char *)malloc(output_image.row_bytes * output_image.height * num_eyes);
// Copy to host memorycudaMemcpy2D(out_stacked + out_offset, output_image.row_bytes, output_image.dev_ptr,
output_image.pitch, output_image.row_bytes, output_image.height, cudaMemcpyDeviceToHost);
out_offset += output_image.height * output_image.row_bytes;}// Write to fileputRgbaImage(params->out_file, out_stacked, output_image.width, output_image.height * num_eyes);
// Clean upRETURN_NVSS_ERROR(nvssVideoDestroyInstance(stitcher));
Low-Level Video API Example (continued)
Get stitched
panoramas
Copy to host memory
Destory stitcher
![Page 37: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/37.jpg)
37
VRWORKS 360 VIDEO SDK
// Create video rigRETURN_NVSTITCH_ERROR(nvstitchCreateVideoRigInstance(&rig_properties, &stitcher_properties.video_rig));
// Create audio rigRETURN_NVSTITCH_ERROR(nvstitchCreateAudioRigInstance(&audio_rig_properties,
&stitcher_properties.audio_rig, NULL));
// Create stitcher instanceRETURN_NVSTITCH_ERROR(nvstitchCreateStitcher(¶ms->stitcher_properties, &stitcher));
// Start stitcherRETURN_NVSTITCH_ERROR(nvstitchStartStitcher(stitcher, nullptr, nullptr));
// Load video inputsfor (uint32_t i{}; i < (uint32_t)params->rig_properties.num_cameras; ++i)
RETURN_NVSTITCH_ERROR(nvstitchFeedStitcherAudioVideo(0, stitcher, i, i, ¶ms->payloads[i], false));
// Stop stitcherRETURN_NVSTITCH_ERROR(nvstitchStopStitcher(stitcher));
// Clean upRETURN_NVSTITCH_ERROR(nvstitchDestroyStitcher(stitcher));
High-Level API ExampleCreate video
rig instance
Create audio rig instance
Create stitcher
Start stitcher
Feed stitcher
Stop stitcher
Destroy stitcher
![Page 38: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/38.jpg)
38
VRWORKS 360 VIDEO SDK
// Create calibration instanceRETURN_NVSTITCH_ERROR(nvstitchCreateCalibrationInstance(calib_prop, &h_calib));
// Read and load input images for calibrationfor (uint32_t cam_index = 0; cam_index < camera_count; cam_index++){
getRgbaImage(image_file_path, &rgba_bitmap_ptr, image_width, image_height);nvstitchPayload_t calib_payload = nvstitchPayload_t{ calib_prop.input_form,{ width, height } };calib_payload.payload.buffer.ptr = rgba_bitmap_ptr;calib_payload.payload.buffer.pitch = rig_properties.cameras[cam_index].image_size.x * num_channels;
RETURN_NVSTITCH_ERROR(nvstitchFeedCalibrationInput(frame_index, h_calib, cam_index, &calib_payload));}
// CalibrateRETURN_NVSTITCH_ERROR(nvstitchCalibrate(h_calib, &h_calibrated_video_rig));
// Destroy calibration instanceRETURN_NVSTITCH_ERROR(nvstitchDestroyCalibrationInstance(h_calib));
Calibration
Feed images
Create calibration
instance
Calibrate. Return updated video rig
Destroy calibration instance
![Page 39: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/39.jpg)
39
VRWORKS 360 VIDEO SDKXML Based Configuration
rig.xml stitcher.xml footage.xml
CreateVideoRig Calibrate CreateStitcher
rig_calibrated.xml
CreateVideoRig
Stitch
CreateAudioRig
rig.xml
CreateAudioStitcher
![Page 40: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/40.jpg)
40
VRWORKS 360 VIDEO SDK
// Read from Xmlbool readCameraRigXml(const std::string& xmlPath,
std::vector<nvstitchCameraProperties_t>&cameraProperties,nvstitchVideoRigProperties_t* videoRigProperties);
// Read filenames from Xml// For nvcalibbool readInputCalibFilenamesXml(const std::string& xmlPath,
std::vector<std::vector<std::string>>& filenames);
// Read media filenames Xmlbool readInputMediaFeedFilenamesXml(const std::string& xmlPath, std::vector<std::string>& filenames);
bool readInputMediaFeedXml(const std::string& inputFeedXmlPath, std::vector<nvstitchPayload_t>& mediaPayloadArray, int* arrayCount);
bool readInputAudioFeedXml(const std::string& inputFeedXmlPath, std::vector<nvstitchAudioPayload_t>& audioPayloadArray, int* arrayCount);
bool readStitcherPropertiesXml(const std::string& stitcherPropertiesXmlPath, const std::string& inputFeedXmlPath,nvstitchStitcherProperties_t* stitcherProperties);
XML Examples
![Page 41: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/41.jpg)
41
VRWORKS 360 VIDEO SDK
bool getRgbImage(const string& imgPath, unsigned char** imgRgb, int& imgWidth, int& imgHeight)
bool getRgbaImage(const string& imgPath, unsigned char** imgRgb, int& imgWidth, int& imgHeight)
bool putRgbaImage(const string& imgPath, unsigned char* imgRgba, int imgWidth, int imgHeight)
bool showImageRgb(int width, int height, unsigned char* imgRgb)
Utilities
Image I/O
![Page 42: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/42.jpg)
42
EXHIBITIONCome See Our Demo in the VR Village!
![Page 43: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/43.jpg)
43
VRWORKS 360 VIDEO SDK
�GPU-optimized stereo pipeline
�GPU-accelerated calibration
�GPU-accelerated encode and decode
�Audio mix down and stretch
�Audio encode / decode
�Audio resampling
�Realtime and offline
�Stereo and mono stitching
�Compressed and uncompressed inputs and outputs
MP4, H.264 and RGBA8
PCM and AAC audio
Available Now
https://developer.nvidia.com/vrworks/vrworks-360video
![Page 44: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/44.jpg)
44
Q & A
![Page 45: A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING OF ...on-demand.gputechconf.com/gtc-eu/2017/presentation/23449-thomas... · A MULTI-GPU SCALABLE SDK FOR REAL-TIME STEREO STITCHING](https://reader030.fdocuments.in/reader030/viewer/2022020120/5b94d2ef09d3f272648b7ce2/html5/thumbnails/45.jpg)