360|iDev 2017

Shuichi Tsutsumi @shu223

Deep Learning on iOS

Overview• How to implement “Deep Learning” on iOS

Metal Performance Shaders (MPSCNN) Accelerate (BNNS)

Core ML

Vision

Your App

Why so exciting? !

AIphaGo

Cancer Detection

Self-driving Car

AutoDraw

Pose Estimation

http://qiita.com/nnn112358/items/a4490d85dac5827db53b

Frontal View Synthesis

AutoHair

AutoDraw Pose Estimation

Frontal View AutoHair

AutoDraw Pose Estimation

Frontal View AutoHair

This evolutional tech works on iOS"

Deep Learning ON iOS

= works on iOS devices

(Demo)

Image

Result60 times / sec

xUsers

#

#

TOOTHPASTE

!

Train InferenceTrained Params

iOS ML frameworks

Implementation

"""Core ML"""

The largest hall was 100% full for the Core ML session.

✋

Metal Performance Shaders (MPSCNN) Accelerate (BNNS)

Core ML

Vision

Your App

iOS 11

iOS 10

Metal Performance Shaders (MPSCNN) Accelerate (BNNS)

Core ML

Vision

Your App

iOS 11

iOS 10

Your App

Metal Performance Shaders (MPSCNN) Accelerate (BNNS) iOS 10

• Optimized for GPU (by ) and CPU (by Accelerate) • Available for iOS 10, too • Basically any ML tools can be used to train the models

Still works!

Metal Performance Shaders (MPSCNN) Accelerate (BNNS)

Your App

GPU CPU

MPSCNN 900 results

Core ML 160,000 results

3 steps to implement w/ MPSCNN

How to implement w/ MPSCNN

Step 1: Create the model

Train InferenceTrained Params

-Which tools can be used for the training? -What kind of formats can be used to pass the pre-trained params?

Which ML tools can be used? : ANY What model format can be used? : ANY

Model (Trained Params)

dat

Train

ML Tools

Which ML tools can be used? : ANY What model format can be used? : ANY

Model (Trained Params)

dat

Train

ML Tools

hdf5

•The “.dat” files are common binary files. - Not specific for iOS or MPS.

•Contains the trained params - Weights / Biases

•Any other format can be used as long as it can be read by the iOS app. - c.f. hdf5

Which ML tools can be used? : ANY What model format can be used? : ANY

Model (Trained Params)

dat

Train

ML Tools

hdf5

•Any tools which can train CNN, and export the params

How to implement w/ MPSCNN

Step 2: Implement the network

MPSCNNConvolution

MPSCNNFullyConnected

MPSCNNPooling

MPSCNNConvolution

MPSCNNConvolution

MPSCNNPooling

MPSCNNConvolution

MPSCNNFullyConnected

MPSCNNPooling

MPSCNNConvolution

MPSCNNConvolution

MPSCNNPooling

• Almost same name -> Easy to find• Complicated Maths or GPU optimization are encapsulated

Classes corresponding to each CNN layers are provided

How to implement w/ MPSCNN

Step 3: Implement the inference

Input image

MPSImage

Result (UInt, etc.)

CNN • Implemented in Step 2 • Trained params (created in Step 1)

are loaded.

Do something

• Step 1: Create the model - Any ML tools can be used

- Any file format can be used for the trained params

• Step 2: Implement the network - Classes corresponding to each CNN layer are provided

• Step 3: Implement the inference - Input to the CNN, and output from the CNN

Demo : Logo detection

Increased by 70 times

Trained Params

dat

f.write(session.run(w_conv1_p).tobytes())

(Demo)

⚠

CNN for the logo detection

GoogLeNet (Inception v3)

GoogLeNet (Inception v3) Apple’s implementation w/ MPSCNN

Inception3Net.swift

2,000 lines&Only for the CNN

Core ML

Development Flow w/ MPSCNNML Tools

Train

some format

Trained Params

dat

Extract

Parse

dat

MPSCNNConvolution

MPSCNNFullyConnected

Implement Network 2,000

lines&App

Development Flow w/ Core MLML Tools

Train

some format

Trained Params

dat

Extract

Parse

dat

MPSCNNConvolution

MPSCNNFullyConnected

Implement NetworkApp 2,000

lines&

1) Convert w/ coremltools

2) Drag & Drop

some format

Generate

Input image

MPSImage

Result (UInt, etc.)

CNN

Do something

let size = MTLSize(width: inputWidth, height: inputHeight, depth: 1) let region = MTLRegion(origin: MTLOrigin(x: 0, y: 0, z: 0), size: size) network.srcImage.texture.replace( region: region, mipmapLevel: 0, slice: 0, withBytes: context.data!, bytesPerRow: inputWidth, bytesPerImage: 0)

Need to know Metal to use MPSCNN

let origin = MTLOrigin(x: 0, y: 0, z: 0) let size = MTLSize(width: 1, height: 1, depth: 1) finalLayer.texture.getBytes(&(result_half_array[4*i]), bytesPerRow: MemoryLayout<UInt16>.size*1*4, bytesPerImage: MemoryLayout<UInt16>.size*1*1*4, from: MTLRegion(origin: origin, size: size), mipmapLevel: 0, slice: i)

MPSCNN Accelerate (BNNS)

Core ML

Vision

Your App

Input image

MPSImage

Results

CNN

Do something

let ciImage = CIImage(cvPixelBuffer: imageBuffer)let handler = VNImageRequestHandler(ciImage: ciImage)try! handler.perform([self.coremlRequest])

Don’t need to touch Metal to use Vision

guard let results = request.results as? [VNClassificationObservation] else { return }guard let best = results.first?.identifier else { return }

MPSCNN Accelerate (BNNS)

Your App

MPSCNN Accelerate (BNNS)

Core ML

Vision

Your App

MPSCNN Accelerate (BNNS)

Core ML

Vision

Your App

How about BNNS?

Deep Learning on CPU?!

CPU: 40 days GPU: 6 days

Deep Learning on CPU?!

Asked @ Lab in WWDC17

“How can we utilize both MPSCNN and BNNS?”

' “Basically use MPSCNN.”

“OK, but, when should I choose BNNS?”

' “I don’t know.”

• He added “watchOS might be a case on that you should use BNNS.”

• Because watchOS doesn’t support MPSCNN, but supports BNNS. (I haven’t tried yet.)

My current understanding:• The cost for passing data between CPU and GPU is

not small • When the network is small, the CPU <-> GPU cost

might be bigger than the benefit of parallel processing.

BNNS might be better when the network is small.

Recap• Why is “Deep Learning on iOS” exciting? • How to implement “Deep Learning” on iOS

- w/ MPSCNN (iOS 10)

- w/ Core ML & Vision (iOS 11)

- When to choose BNNS

MPSCNN BNNS

Core ML

Vision

Thank you!

https://github.com/shu223