Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et...
Transcript of Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et...
![Page 1: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/1.jpg)
Deep Neural Networks for Mobile Platforms
Oleksandr Baiev
PhD, Sr. EngineerSamsung R&D Instutute Ukraine
AI Ukraine, 2016
![Page 2: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/2.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 1 / 27
![Page 3: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/3.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 2 / 27
![Page 4: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/4.jpg)
Neural networks
Cutting edge results for CV, NLP, Signal Processing,Recommendations.
Unified solution for different problems.
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 3 / 27
![Page 5: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/5.jpg)
Neural networksConvolution neural network
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 4 / 27
![Page 6: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/6.jpg)
Deployment stage problemModels quality
Alex Krizhevsky, et al. ImageNet Classification with Deep Convolutional Neural Networks, 2012Christian Szegedy, et al. Going Deeper with Convolutions, 2014K. Simonyan, et al. Very Deep Convolutional Networks for Large-Scale Image Recognition, 2014Kaiming He, et al. Deep Residual Learning for Image Recognition, 2015
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 5 / 27
![Page 7: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/7.jpg)
Deployment stage problemModels size
Application more than 100MB requires WiFi for downloading via app stores
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 6 / 27
![Page 8: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/8.jpg)
Deployment stage problemModels execution
Test with https://github.com/sh1r0/caffe-android-lib
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 7 / 27
![Page 9: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/9.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 8 / 27
![Page 10: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/10.jpg)
Problem overviewApproximate layers distribution
Fully Connected are bigger thanConvolution layers in terms ofMB
Convolution takes much moretime for forward pass
Target device have to storelayer’s feature maps in RAM forat least one layer
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 9 / 27
![Page 11: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/11.jpg)
Problem overviewExecution time and size by layer
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 10 / 27
![Page 12: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/12.jpg)
Problem overviewParameters importance
Feature map’s width, height influence on execution time
Feature map’s depth influences on model size
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 11 / 27
![Page 13: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/13.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 12 / 27
![Page 14: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/14.jpg)
Bottlenecks
Min Lin, et al. Network In Network, 2013
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 13 / 27
![Page 15: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/15.jpg)
Inception module
Compose different kernel sizes
Christian Szegedy, et. al., Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, 2016
Christian Szegedy, et. al., Rethinking the Inception Architecture for Computer Vision, 2015
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 14 / 27
![Page 16: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/16.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 15 / 27
![Page 17: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/17.jpg)
Quantization
Pete Warden, How to Quantize Neural Networks with TensorFlow, 2016
Matthieu Courbariaux, et. al., BinaryConnect: Training Deep Neural Networks with binary weights during propagations,2015
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 16 / 27
![Page 18: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/18.jpg)
Quantization Schema
All operations use precalculatedMin and Max values which areused for rescaling
Min and Max values selectedfrom function behavior and realnumber values
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 17 / 27
![Page 19: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/19.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 18 / 27
![Page 20: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/20.jpg)
Pruning basics
The idea of pruning is removing unimportant weights. The one question ishow to define “unimportant”.
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 19 / 27
![Page 21: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/21.jpg)
Pruning basicsUnimportant criteria
Yann Le Cun, et al. Optimal Brain Damage, 1990
Babak Hassibi, et al. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon, 1992
Song Han, et al., Learning both Weights and Connections for Efficient Neural Networks, 2015
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 20 / 27
![Page 22: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/22.jpg)
Pruning basicsIterative Process
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 21 / 27
![Page 23: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/23.jpg)
Outline
Problems overviewKey problemsDeeper problem overview
TopologyOverview
QuantizationOverview
PruningOverviewExamples
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 22 / 27
![Page 24: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/24.jpg)
Pruning example VGG
Song Han, DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow, 2016
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 23 / 27
![Page 25: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/25.jpg)
Pruning example VGG
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 24 / 27
![Page 26: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/26.jpg)
Pruning example GoogLeNet
Hao Li, et.al., Pruning Filters for Efficient ConvNets, 2016
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 25 / 27
![Page 27: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/27.jpg)
Pruning example ResNet-152
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 26 / 27
![Page 28: Deep Neural Networks for Mobile Platforms · Pruning basics Unimportant criteria Yann Le Cun, et al. Optimal Brain Damage, 1990 Babak Hassibi, et al. Second Order Derivatives for](https://reader033.fdocuments.in/reader033/viewer/2022060222/5f078d757e708231d41d8d80/html5/thumbnails/28.jpg)
Summary
Deep neural networks provides excellent quality, but requires powerfulcomputation instances
There are several simple and useful approaches for reducing requiredmemory size and execution time without increasing hardware cost
Oleksandr Baiev, PhD, Sr. Engineer Samsung R&D Instutute Ukraine DNN for Mobile Platforms 27 / 27