TENSOR-TRAIN LSTM FOR CONVOLUTIONAL MIXED PRECISION …
20
Wonmin Byeon, August 23, 2020 MIXED PRECISION TRAINING FOR CONVOLUTIONAL TENSOR-TRAIN LSTM
Transcript of TENSOR-TRAIN LSTM FOR CONVOLUTIONAL MIXED PRECISION …
Wonmin Byeon, August 23, 2020
MIXED PRECISION TRAINING FOR CONVOLUTIONAL TENSOR-TRAIN LSTM
2
APPLICATIONS
Early Activity Recognition
Video Prediction
Spatio-Temporal Learning
3
CONVOLUTIONAL LSTM
4
CONVOLUTIONAL TENSOR-TRAIN LSTM
5
OPTIMIZATION TRICKS
• Speed-up tricks to train Convolutional LSTM/Convolutional Tensor-Train LSTM
#1. Enabling mixed precision training using NVIDIA AMP
#2. GPU Optimization: Fused Adam, Fused kernels in LSTM cell, Thread affinity binding
#3. LSTM Activation Checkpointing
• Fast convergence without performance change
• Model parallelism using multi-streams
6
#1. ENABLING AMP
# save the checkpoint
# load the checkpoint
7
#2. GPU Optimization● Fused Adam: Adam optimizer
8
#2. GPU Optimization● Fused optimizer: Adam optimizer
● Fused kernel with JIT: LSTM cell
9
#2. GPU Optimization● Fused optimizer: Adam optimizer
● Fused kernel with JIT: LSTM cell
● Thread affinity binding: distributed computing
10
TRAININGConvolutional LSTM
Application: video prediction Machine: V100 x 8, 16GBBatch Size: 16 videos12 Conv. LSTM layersInput/output image resolution: 128x128
No performance change!
7.8x faster6.6x faster
1.5x less
Convergence time GPU Memory
11
Convolutional Tensor-Train LSTM
5.5x faster
Application: video prediction Machine: V100 x 8, 16GBBatch Size: 16 videos12 Conv. LSTM layersInput/output image resolution: 128x128 TRAINING
No performance change!
1.5x less
Convergence time GPU Memory
12
#3. ACTIVATION CHECKPOINTING
13
TRAININGConvolutional LSTM
Convergence time GPU Memory
Application: video prediction Machine: V100 x 8, 16GBBatch Size: 16 videos12 Conv. LSTM layersInput/output image resolution: 128x128
No performance change!
7.8x faster
1.8x less memory usage
14
Conv. Tensor-Train LSTM
5.5x faster
2.3x less memory usage
Application: video prediction Machine: V100 x 8, 16GBBatch Size: 16 videos12 Conv. LSTM layersInput/output image resolution: 128x128 TRAINING
No performance change!
Convergence time GPU Memory
15
MODEL PARALLEL: MULTI-STREAMS
example model:
• Use multiple cuda-streams• Execute multiple kernels that do not have
data dependency in parallel
16
• Use multiple cuda-streams• Execute multiple kernels that do not have
data dependency in parallel
Operations of layers at different branches are overlapped.example model:
MODEL PARALLEL: MULTI-STREAMS
17
MODEL PARALLEL: MULTI-STREAMS
single-stream
multi-stream
2.8x faster
Computation time comparison
18
SUMMARY
Speed-up summary to train Convolutional Tensor-Train LSTM
• Enabling AMP
• GPU Optimization
• Activation Checkpointing
• ConvLSTM: 7.8x speed-up, 1.8x less memory usage
• Conv. Tensor-Train LSTM: 5.5x speed-up, 2.3x less memory usage.
• Fast convergence without performance change
Model parallelism using multi-streams
19
CONCLUSION
• Paper: Jiahao Su* (UMD), Wonmin Byeon* (NVIDIA), Jean Kossaifi (NVIDIA), Furong Huang (UMD), Jan Kautz
(NVIDIA), Animashree Anandkumar (NVIDIA), 'Convolutional Tensor-Train LSTM for Spatio-Temporal Learning',
under submission, 2020.
• Project page: https://sites.google.com/nvidia.com/conv-tt-lstm/home
• Code Optimization: Sangkug Lym (NVIDIA)
• The code with the optimization tricks will be available soon: https://github.com/NVlabs/conv-tt-lstm
![arXiv:1902.09130v2 [cs.CV] 29 Mar 2019 · arXiv:1902.09130v2 [cs.CV] 29 Mar 2019. Figure 2. The architecture of the proposed attention enhanced graph convolutional LSTM network (AGC-LSTM).](https://static.fdocuments.in/doc/165x107/5f1287e71b26904837666219/arxiv190209130v2-cscv-29-mar-arxiv190209130v2-cscv-29-mar-2019-figure.jpg)
![Deep Multi-Kernel Convolutional LSTM Networks and an ... · 3D convolutions [15], and they demonstrated state-of-the-art performance by pretraining on large image and video datasets.](https://static.fdocuments.in/doc/165x107/603f411aa3cee365aa3a607d/deep-multi-kernel-convolutional-lstm-networks-and-an-3d-convolutions-15-and.jpg)
![IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, … · networks [11], Convolutional Neural Networks (CNNs) [12], DeepBeliefNetworks(DBNs)[13]ormemoryenhancedneural networkmodelssuchasLongShort-TermMemory(LSTM)[14]](https://static.fdocuments.in/doc/165x107/600e68800e5b0701d41b6b04/ieee-journal-of-selected-topics-in-signal-processing-networks-11-convolutional.jpg)
![arXiv:1811.10899v1 [cs.CV] 27 Nov 2018 · pletely remove the recurrent layers, relying on simple feed-forward convolutional only architectures. The most ... perspectives. 1 Here LSTM](https://static.fdocuments.in/doc/165x107/5f05485d7e708231d412314b/arxiv181110899v1-cscv-27-nov-2018-pletely-remove-the-recurrent-layers-relying.jpg)