Leveraging Deep Learning Applications with CNTK

Post on 12-Apr-2017

266 views 2 download

Transcript of Leveraging Deep Learning Applications with CNTK

Dr. Miguel González-Fierro

@miguelgfierrohttps://miguelgfierro.com

LEVERAGING DEEP LEARNING APPLICATIONS WITH CNTK

1. CNTK FRAMEWORK

2. TRANSFER LEARNING WITH DEEP NETWORKS

3. IMAGE RECOGNITION API

1. CNTK FRAMEWORK

MICROSOFT’S COGNITIVE TOOLKIT: CNTK

Dr. Miguel González-Fierro - @miguelgfierro

MPIBrainScript

https://github.com/Microsoft/CNTK

CNTK STRENGHTS: RNN & MULTISERVER GPU

Dr. Miguel González-Fierro - @miguelgfierro

Source:

https://arxiv.org/abs/1608.07249

Multi-server algorithms:

1-bit SGD

Block momentum

Recurrent Neural Networks

CNTK TUTORIALS & EXAMPLES

Dr. Miguel González-Fierro - @miguelgfierro

https://github.com/Microsoft/CNTK/tree/master/Tutorialshttps://github.com/Microsoft/CNTK/tree/master/Examples

GPU VIRTUAL MACHINE

Dr. Miguel González-Fierro - @miguelgfierro

$5.3/h$2.65/h$1.32/h

NOTE:

as of May 2017

DATA SCIENCE VIRTUAL MACHINE (DSVM)

Dr. Miguel González-Fierro - @miguelgfierro

2. TRANSFER LEARNING WITH DEEP NETWORKS

KAGGLE DATA SCIENCE BOWL 2017

Dr. Miguel González-Fierro - @miguelgfierro

$1 million in prizes !

CT scan: horizontal section of lung

Determine

whether a

patient has

cancer or notcompetition

OBJECTIVE: BASELINE SOLUTION

Dr. Miguel González-Fierro - @miguelgfierro

END TO END PROCESS

DSVM & CNTK

less than 1h

Top 10% (Jan 19th)

LUNG SCANS DATASET

Dr. Miguel González-Fierro - @miguelgfierro

1595 patients with a diagnostic

200-500 scans per patient

Images of 512x512

AUTOMATIC FEATURE GENERATOR

Dr. Miguel González-Fierro - @miguelgfierro

Source:

http://vision03.csail.mit.edu/cnn_art/

https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

TRANSFER LEARNING

ImageNet dataset Lung cancer dataset

weight transference

Dr. Miguel González-Fierro - @miguelgfierro

STANDARD TRAINING

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Forward and backward propagation

TRANSFERENCE 1: FREEZE N-1 LAYERS

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Frozen layers

TRANSFERENCE 2: FREEZE INITIAL LAYERS

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Frozen layers

Forward and backward propagation

TRANSFERENCE 3: MODEL INITIALIZATION

Dr. Miguel González-Fierro - @miguelgfierro

input hidden hidden hidden hidden hidden output

Forward and backward propagation

Weight initialization

PRETRAINED RESNET MODEL

Dr. Miguel González-Fierro - @miguelgfierro

3

224

224

last

layer

ImageNet ResNet N layers

penultimate

layer

tabby

cat

SOLUTION: CNN FEATURIZER

3

224

224

ResNet N-1 layers

penultimate

layer

CNTK

(53min)k batch

of images= 1 patient

features

Dr. Miguel González-Fierro - @miguelgfierro

SOLUTION: BOOSTED TREE TRAINING

3

224

224

ResNet N-1 layers

penultimate

layer

no

cancer

CNTK

(53min)

LightGBM

(2min)

Boosted tree

k batch

of images= 1 patient

features

Dr. Miguel González-Fierro - @miguelgfierro

KAGGLE SUBMISSION

Dr. Miguel González-Fierro - @miguelgfierro

top 10% as of Jan 19th

RESOURCES

Kaggle script Cortana Gallery notebook

Blog in TechNet

https://aka.ms/dsb2017-cntk-notebookhttps://aka.ms/dsb2017-cntk-script https://aka.ms/dsb2017-cntk-blog

+6000 views +1500 views+300 shares on SN

+1500 views+300 downloads

TechNet

Dr. Miguel González-Fierro - @miguelgfierro

3. IMAGE RECOGNITION API

SUCCESS IN IMAGE CLASSIFICATION

Dr. Miguel González-Fierro - @miguelgfierro

15.4%

7.3%

6.7%

3.6%3.1%

5.1% (human)

error (%)

ImageNet (image recognition competition) top-5 error

AlexNet(2012)

VGG(2014)

Inception(2015)

ResNet(2015)

Inception-ResNet(2016)

ARCHITECTURE

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

ARCHITECTURE

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

SUCCESS IN IMAGE CLASSIFICATION

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

SUCCESS IN IMAGE CLASSIFICATION

Dr. Miguel González-Fierro - @miguelgfierro

Ilia Karmanov: https://github.com/ilkarman/CV_end_to_end

Dr. Miguel González-Fierro

@miguelgfierrohttps://miguelgfierro.com

QUESTIONS?