Leveraging Deep Learning Applications with CNTK

Dr. Miguel González-Fierro

@miguelgfierrohttps://miguelgfierro.com

LEVERAGING DEEP LEARNING APPLICATIONS WITH CNTK

1. CNTK FRAMEWORK

2. TRANSFER LEARNING WITH DEEP NETWORKS

3. IMAGE RECOGNITION API

1. CNTK FRAMEWORK

MICROSOFT’S COGNITIVE TOOLKIT: CNTK

Dr. Miguel González-Fierro - @miguelgfierro

MPIBrainScript

https://github.com/Microsoft/CNTK

CNTK STRENGHTS: RNN & MULTISERVER GPU

Source:

https://arxiv.org/abs/1608.07249

Multi-server algorithms:

1-bit SGD

Block momentum

Recurrent Neural Networks

CNTK TUTORIALS & EXAMPLES

https://github.com/Microsoft/CNTK/tree/master/Tutorialshttps://github.com/Microsoft/CNTK/tree/master/Examples

GPU VIRTUAL MACHINE

$5.3/h$2.65/h$1.32/h

as of May 2017

DATA SCIENCE VIRTUAL MACHINE (DSVM)

2. TRANSFER LEARNING WITH DEEP NETWORKS

KAGGLE DATA SCIENCE BOWL 2017

$1 million in prizes !

CT scan: horizontal section of lung

Determine

whether a

patient has

cancer or notcompetition

OBJECTIVE: BASELINE SOLUTION

END TO END PROCESS

DSVM & CNTK

less than 1h

Top 10% (Jan 19th)

LUNG SCANS DATASET

1595 patients with a diagnostic

200-500 scans per patient

Images of 512x512

AUTOMATIC FEATURE GENERATOR

Source:

http://vision03.csail.mit.edu/cnn_art/

https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

TRANSFER LEARNING

ImageNet dataset Lung cancer dataset

weight transference

STANDARD TRAINING

input hidden hidden hidden hidden hidden output

Forward and backward propagation

TRANSFERENCE 1: FREEZE N-1 LAYERS

Frozen layers

TRANSFERENCE 2: FREEZE INITIAL LAYERS

Frozen layers

TRANSFERENCE 3: MODEL INITIALIZATION

Weight initialization

PRETRAINED RESNET MODEL

ImageNet ResNet N layers

penultimate

SOLUTION: CNN FEATURIZER

ResNet N-1 layers

penultimate

(53min)k batch

of images= 1 patient

features

SOLUTION: BOOSTED TREE TRAINING

ResNet N-1 layers

penultimate

cancer

(53min)

LightGBM

(2min)

Boosted tree

k batch

of images= 1 patient

features

KAGGLE SUBMISSION

top 10% as of Jan 19th

RESOURCES

Kaggle script Cortana Gallery notebook

Blog in TechNet

https://aka.ms/dsb2017-cntk-notebookhttps://aka.ms/dsb2017-cntk-script https://aka.ms/dsb2017-cntk-blog

+6000 views +1500 views+300 shares on SN

+1500 views+300 downloads

TechNet

3. IMAGE RECOGNITION API

SUCCESS IN IMAGE CLASSIFICATION

3.6%3.1%

5.1% (human)

error (%)

ImageNet (image recognition competition) top-5 error

AlexNet(2012)

VGG(2014)

Inception(2015)

ResNet(2015)

Inception-ResNet(2016)

ARCHITECTURE

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

ARCHITECTURE

Dr. Miguel González-Fierro

@miguelgfierrohttps://miguelgfierro.com

QUESTIONS?

Leveraging Deep Learning Applications with CNTK

Technology

Transcript of Leveraging Deep Learning Applications with CNTK

CNTK: deep learning framework - Nvidiaimages.nvidia.com/events/sc15/pdfs/CNTK-Overview-SC150-Kamanev.pdfmeanFile=$WorkDir$/ImageNet1K_mean.xml ] labels=[ labelDim=1000 ] ] ] Next steps:

I Understand What You Are Saying: Leveraging Deep ... › bitstream › 10125 › ...2.3. Deep Learning in Text Analytics collection. In addition to Traditional machine learning techniques

Abstract · exible deep learning framework, The Microsoft Cognitive Toolkit (formerly CNTK). In this work we rst provide background on deep learning with the Cognitive Toolkit (section

NVIDIA CUDA バージョン 8.0 の インストール · nvidia cuda はcntkというソフトでも使用される．今後cntkを使う可能性があるので，次 のウェブページでCNTKが指定しているバージョンである8.0をインストールする．

Leveraging Machine Learning for Advancing Smart-X Systems ... · 1.Deep Reinforcement Learning: Use deep neural networks to approximate Q (DQN) 2.Experience replay to reuse past data

Deep Learning, Microsoft Cognitive Toolkit (CNTK) and Azure Machine Learning Services

Accelerating Deep Learning with MVAPICHmvapich.cse.ohio-state.edu/static/media/talks/slide/awan...Performance Benefits: Microsoft CNTK DL framework (25% avg. improvement ) Performance

PATTERN RECOGNITION AND MACHINE · PDF file · 2018-02-05PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 6: Neural Networks and Deep Learning ... MS Cognitive TK (CNTK): Microsoft

Deep3: Leveraging Three Levels of Parallelism for ...lepsucd.com/wp-content/uploads/2017/10/001_Deep3.pdf · Deep3: Leveraging Three Levels of Parallelism for Efficient Deep Learning

Scalpel-CD: Leveraging Crowdsourcing and Deep ... · Debugging training data; crowdsourcing; deep probabilistic models ACM Reference Format: Jie Yang, Alisa Smirnova, Dingqi Yang,

Innovation. interaction. inspiration. Leveraging Middleware Jeff Wofford Deep Red Games jeffwofford@yahoo.com.

SFO17-509 Deep Learning on ARM Platforms - from the ...connect.linaro.org.s3.amazonaws.com/sfo17/Presentations...Keras: a Python DL library supporting TensorFlow, CNTK, Theano and

Leveraging Deep Learning Techniques for Malaria Parasite ...

Leveraging deep learning hyperparameter tuning frameworks · Leveraging deep learning hyperparameter tuning frameworks Derek Jensen, Donald Lucas, Cli˙ord Anderson-Bergman, Sonia

Image Classification, Deep Learning and Convolutional ...1111144/FULLTEXT02.pdfTensorFlow and Microsoft CNTK, primarily based on their performance in the training time of neural networks.

Comparative Study of Deep Learning Framework in HPC ......2) CNTK The Microsoft Cognitive Toolkit (CNTK) is a unified deep-learning toolkit developed by Microsoft [2]. It provides

Scalpel-CD: Leveraging Crowdsourcing and Deep ... · argue that debugging noisy labels is of key importance for deep learning-based systems, both in terms of performance and security.

DEEP LEARNING WITH MICROSOFT COGNITIVE TOOLKIT CNTK … · Microsoft Cognitive Toolkit library trains LSTM-models up to 4 times faster than TensorFlow, it parallelizes the processes

High-Performance Training for Deep Learning and …mvapich.cse.ohio-state.edu/static/media/talks/slide/cvpr...Research Challenges to Exploit HPC Technologies CPU InfiniBand GPU CNTK

Optimizing Data Aggregation by Leveraging the Deep Memory ...

NVIDIA CUDA バージョン 8.0 のインストール · nvidia cuda はcntkというソフトでも使用される．今後cntkを使う可能性があるので，次のウェブページでCNTKが指定しているバージョンである8.0をインストールする．