Machine Intelligence at Google Scale

ML APIs, TensorFlow and Cloud ML

+Kazunori Sato @kazunori_279

Kaz Sato

Staff Developer Advocate

Tech Lead for Data & Analytics

Cloud Platform, Google Inc.

What we’ll cover

What is Neural Network and Deep Learning

Machine Learning use cases at Google services

Externalizing the power with ML APIs

TensorFlow: the open source library for ML

TensorFlow in the Wild

Distributed training and prediction with Cloud ML

What is Neural Network

and Deep Learning

Neural Network is a function that can learn

xn

> b?

w1

wn

x2

x1

Inspired by the behavior of biological neurons

How do you classify them?

weights

bias (threshold)

Programmers need to specify the parameters

Let’s see how neural network solves the problem

The computer tries to find the best parameters

A neuron classifies a data point into two kinds

Gradient Descent: adjusting the params gradually to reduce errors

How do you classify them?

What we see What the computer “sees”

28 x 28 gray scale image =

784 numbers

input vector (pixel data)

output vector (probability)

How do you classify them?

More neurons = More features to extract

Hidden Layers:

mapping inputs to

a feature space,

classifying with

a hyperplane From: Neural Networks, Manifolds, and Topology, colah's blog

From: Neural Networks, Manifolds, and Topology, colah's blog

How about this?

More hidden layers = More hierarchies of features

How about this?

We need to go deeper neural network

From: Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations, Honglak Lee et al.

From: mNeuron: A Matlab Plugin to Visualize Neurons from Deep Models, Donglai Wei et. al.

How CNN works

Machine Learning use cases

at Google services

27

signal for Search ranking,

out of hundreds

improvement to ranking quality

in 2+ years

#3 #1

Search

machine learning for search engines

RankBrain: a deep neural network for search ranking

WaveNet by

DeepMind

30

[glacier]

Google Photos

30

31

Smart reply in Inbox by Gmail

10% of all responses sent on mobile

Google Translate with Neural Machine Translation

Saved Data Center cooling energy for 40% Improved Power Usage Effectiveness (PUE) for 15%

34

Android

Apps

Gmail

Maps

Photos

Speech

Search

Translation

YouTube

and many others ...

Used across products:

2012 2013 2014 2015

Deep Learning usage at Google

Externalizing the power

with ML APIs

TensorFlow Cloud Machine Learning ML API

Easy-to-Use, for non-ML engineers

Customizable, for Data Scientists

Machine Learning products from Google

Image analysis with pre-trained models

No Machine Learning skill required

REST API: receives an image and returns a JSON

$1.50 per 1,000 units

GA - cloud.google.com/vision

Cloud Vision API

Confidential & Proprietary Google Cloud Platform 38

Faces Faces, facial landmarks, emotions

OCR Read and extract text, with support for > 10 languages

Label Detect entities from furniture to transportation

Logos Identify product logos

Landmarks & Image Properties Detect landmarks & dominant color of image

Safe Search Detect explicit content - adult, violent, medical and spoof

39 39

Demo

Pre-trained models. No ML skill required

REST API: receives audio and returns texts

Supports 80+ languages

Streaming or non-streaming

Public Beta - cloud.google.com/speech

Cloud Speech API

Confidential & Proprietary Google Cloud Platform 41

Features

Automatic Speech Recognition

(ASR) powered by deep learning

neural networking to power your

applications like voice search or

speech transcription.

Recognizes over 80

languages and variants

with an extensive

vocabulary.

Returns partial

recognition results

immediately, as they

become available.

Filter inappropriate

content in text results.

Audio input can be captured by an application’s

microphone or sent from a pre-recorded audio

file. Multiple audio file formats are supported,

including FLAC, AMR, PCMU and linear-16.

Handles noisy audio from many

environments without requiring

additional noise cancellation.

Audio files can be uploaded in the

request and, in future releases,

integrated with Google Cloud

Storage.

Automatic Speech Recognition Global Vocabulary Inappropriate Content Filtering

Streaming Recognition

Real-time or Buffered Audio Support Noisy Audio Handling Integrated API

42 42

Demo

Pre-trained models. No ML skill required

REST API: receives text and returns analysis results

Supports English, Spanish and Japanese

GA - cloud.google.com/natural-language

Cloud Natural Language API

Confidential & Proprietary Google Cloud Platform 44

Features

Extract sentence, identify parts of

speech and create dependency parse

trees for each sentence.

Identify entities and label by types such

as person, organization, location, events,

products and media.

Understand the overall sentiment of a

block of text.

Syntax Analysis Entity Recognition

Sentiment Analysis

45 45

Demo

Pre-trained models. No ML skill required

REST API: receives text and returns translated text

8 languages: English to Chinese, French, German,

Japanese, Korean, Portuguese, Spanish, Turkish

Public Beta - cloud.google.com/translate

Cloud Translation API Premium

47 47

Demo

TensorFlow:

An open source library for

Machine Intelligence

Google's open source library for

machine intelligence

tensorflow.org launched in Nov 2015

Used by many production ML projects

What is TensorFlow?

50

Sharing our tools with researchers and developers around the world

repository for “machine learning”

category on GitHub

#1

Released in Nov.

2015

From: http://deliprao.com/archives/168

51

Before

Hire Data Scientists ↓

Understand the math model ↓

Impl with programming code ↓

Train with single GPU ↓

Build a GPU cluster ↓

Train with the GPU cluster ↓

Build a prediction server or Impl mobile/IoT prediction

After

Easy network design and impl

↓

Train with single machine

↓

Train on the cloud

↓

Prediction on the cloud

or mobile/IoT devices

many people stuck here

# define the network import tensorflow as tf x = tf.placeholder(tf.float32, [None, 784]) W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([10])) y = tf.nn.softmax(tf.matmul(x, W) + b) # define a training step y_ = tf.placeholder(tf.float32, [None, 10]) xent = -tf.reduce_sum(y_*tf.log(y)) step = tf.train.GradientDescentOptimizer(0.01).minimize(xent)

TensorBoard: visualization tool

Portable and Scalable

Training on:

Mac/Windows

GPU server

GPU cluster / Cloud

Prediction on:

Android and iOS

RasPi and TPU

Distributed Training with TensorFlow

TensorFlow in the Wild (or democratization of deep learning)

TensorFlow powered Cucumber Sorter

From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/

TensorFlow powered Cucumber Sorter

TensorFlow

powered

Fried Chicken

Nugget Server

From: http://www.rt-net.jp/karaage1/

TV popstar face

generator with DCGAN

From: http://memo.sugyan.com/entry/2016/10/12/084751

TensorFlow +

Drones

for counting trucks

From: http://www.brainpad.co.jp/news/2016/09/02/3454

TensorFlow＋

RasPi for

sorting garbages

From: https://techcrunch.com/2016/09/13/auto-trash-sorts-garbage-automatically-at-the-techcrunch-disrupt-hackathon/

Autonomous

Driving

of RasPi car

with Inception 3

on TensorFlow

From: https://github.com/zxzhijia/GoPiGo-Driven-by-Tensorflow

From: http://otoro.net/

Generative Arts with TensorFlow

Distributed Training and

Prediction with Cloud ML

From: Andrew Ng

The Bigger, The Better

The Challenge: Computing Power

DNN requires large training datasets

Large models doesn't fit into a GPU

Requires try-and-errors to find the best design, configs and params

↓

Need to spend a few days or weeks to finish a training

GPUs run at nanoseconds GPU cluster needs microsec network

Enterprise

Google Cloud is

The Datacenter as a Computer

Jupiter network

10 GbE x 100 K = 1 Pbps

Consolidates servers with

microsec latency

Borg

No VMs, pure containers

10K - 20K nodes per Cell

DC-scale job scheduling

CPUs, mem, disks and IO

Distributed Training

with TensorFlow

by data parallelism

split data,

share model

CPU/GPU scheduling

Communications

Local, RPC, RDMA

32/16/8 bit quantization

Cost-based optimization

Fault tolerance

Distributed Systems for Large Neural Network

Distributed Training with TensorFlow

on Google Cloud

"Large Scale Distributed Systems for Training Neural

Networks", NIPS 2015

Inception / ImageNet: 40x with 50 GPUs

RankBrain: 300x with 500 nodes

Fully managed distributed training and prediction

Supports custom TensorFlow graphs

HyperTune for hyper-parameter tuning automation

Integrated with Cloud Dataflow and Cloud Datalab

Public Beta - cloud.google.com/ml

Cloud Machine Learning (Cloud ML)

Cloud ML at Work: AUCNET

The largest real-time car auction service in Japan

For 30K used car dealers

The auction volume overs $3.7B every year

Problem: auction entry is time consuming task for dealers

Classifying parts of car for thousands of photos

Identifying the exact car model

Solution: Custom Image Classification with TensorFlow/Cloud ML

Used 5,000 training images for 500 car models

Inception v3 + Transfer Learning

Cloud ML: increased training performance for 6x faster

Predicting "large loss" cases in car insurance

Old method (Random Forest): 38% accuracy

New method (TensorFlow): 73% accuracy

A Global Insurance Firm

81 81

Demo

82 82

Ready to use Machine Learning models

Use your own data to train models

Cloud Vision API

Cloud Speech API

Cloud Translation API

Cloud Machine Learning

Develop - Model - Test

Google BigQuery

Cloud Storage

Cloud Datalab

Beta

GA Beta GA

Beta GA

GA GA

Cloud Natural Language API

Tensor Processing Unit

ASIC for TensorFlow

Designed by Google

10x better perf / watt

latency and efficiency

bit quantization

TPU on Production

RankBrain

AlphaGo

Google Photos

Speech

and more

Thank you!