Persistent RNNs: Stashing Recurrent Weights On-Chip

21
Persistent RNNs (stashing recurrent weights on-chip) Presenter: Gregory Diamos Silicon Valley AI Lab Baidu Jun 20, 2016 Presenter: Gregory Diamos Persistent RNNs

Transcript of Persistent RNNs: Stashing Recurrent Weights On-Chip

Page 1: Persistent RNNs: Stashing Recurrent Weights On-Chip

Persistent RNNs(stashing recurrent weights on-chip)

Presenter: Gregory Diamos

Silicon Valley AI LabBaidu

Jun 20, 2016

Presenter: Gregory Diamos Persistent RNNs

Page 2: Persistent RNNs: Stashing Recurrent Weights On-Chip

Machine learning has benefited greatly from faster computer systems.

GPUs in particular, have delivered a step forward.

Presenter: Gregory Diamos Persistent RNNs

Page 3: Persistent RNNs: Stashing Recurrent Weights On-Chip

Imagine the problems that you could solve

with even faster systems.

Presenter: Gregory Diamos Persistent RNNs

Page 4: Persistent RNNs: Stashing Recurrent Weights On-Chip

HPC is an opportunity

10,000x

TitanX GPU

Fastest superco

mputer

Presenter: Gregory Diamos Persistent RNNs

Page 5: Persistent RNNs: Stashing Recurrent Weights On-Chip

Limits of data-parallelism

Presenter: Gregory Diamos Persistent RNNs

Page 6: Persistent RNNs: Stashing Recurrent Weights On-Chip

Hardware limits

wal

l-cl

ock

tim

e to

con

verg

ence

mini-batch size

inefficient hardware

Hardware becomes less efficient at small batch sizes.

Presenter: Gregory Diamos Persistent RNNs

Page 7: Persistent RNNs: Stashing Recurrent Weights On-Chip

Optimization limits

wal

l-cl

ock

tim

e to

con

verg

ence

mini-batch size

inefficient optimization

Optimization algorithms perform more work at large batch sizes.

Presenter: Gregory Diamos Persistent RNNs

Page 8: Persistent RNNs: Stashing Recurrent Weights On-Chip

Mini-batch limits

wal

l-cl

ock

tim

e to

con

verg

ence

mini-batch size

inefficient hardware inefficient optimization

These effects combine to limit the maximum number of GPUs.

Presenter: Gregory Diamos Persistent RNNs

Page 9: Persistent RNNs: Stashing Recurrent Weights On-Chip

Persistent RNNs

Open source CUDA implementation:

https://github.com/baidu-research/persistent-rnns

Presenter: Gregory Diamos Persistent RNNs

https://github.com/baidu-research/persistent-rnns
Page 10: Persistent RNNs: Stashing Recurrent Weights On-Chip

Persistent RNN Details

Presenter: Gregory Diamos Persistent RNNs

Page 11: Persistent RNNs: Stashing Recurrent Weights On-Chip

Persistent RNNs

weights

GEMM GEMM GEMM GEMM

Persistent RNN

weights

weights weights weights

data0 data1 data2 data3 data4

data0 data1 data2 data3 data4

RNNs built on GEMM routines reload the weights each timestep.

However, the weights are constant, and this is wasteful.

Presenter: Gregory Diamos Persistent RNNs

Page 12: Persistent RNNs: Stashing Recurrent Weights On-Chip

Cache weights in registers

weights

GPU thread

registers

datapath

Presenter: Gregory Diamos Persistent RNNs

Page 13: Persistent RNNs: Stashing Recurrent Weights On-Chip

A global barrier

data0 GPU data1 GPUbarrier

Presenter: Gregory Diamos Persistent RNNs

Page 14: Persistent RNNs: Stashing Recurrent Weights On-Chip

Experiments

Presenter: Gregory Diamos Persistent RNNs

Page 15: Persistent RNNs: Stashing Recurrent Weights On-Chip

Scaling to 128 GPUs

Presenter: Gregory Diamos Persistent RNNs

Page 16: Persistent RNNs: Stashing Recurrent Weights On-Chip

Exploring deep residual RNNs

Presenter: Gregory Diamos Persistent RNNs

Page 17: Persistent RNNs: Stashing Recurrent Weights On-Chip

Pascal and future

Future GPUs will enable bigger and faster RNN layers.

Presenter: Gregory Diamos Persistent RNNs

Page 18: Persistent RNNs: Stashing Recurrent Weights On-Chip

Three challenges

Presenter: Gregory Diamos Persistent RNNs

Page 19: Persistent RNNs: Stashing Recurrent Weights On-Chip

Close the gap with the fastest supercomputers.

Presenter: Gregory Diamos Persistent RNNs

Page 20: Persistent RNNs: Stashing Recurrent Weights On-Chip

Do not settle for inefficient algorithms.

Presenter: Gregory Diamos Persistent RNNs

Page 21: Persistent RNNs: Stashing Recurrent Weights On-Chip

Push performance to the edge of physical limits.

10 PetaFlops in 300 Watts.

150 ExaFlops in 25 MegaWatts.

Presenter: Gregory Diamos Persistent RNNs


Recurrent Neural Networks - Welcome to CEDARsrihari/CSE676/10.2.0 RecurrentNets.pdf · Recurrent Neural Networks 3. Bidirectional RNNs 4. Encoder-Decoder Sequence-to-Sequence Architectures

Recurrent Neural Networks - Welcome to CEDARsrihari/CSE676/10.2.0 RecurrentNets.pdf · Recurrent Neural Networks 3. Bidirectional RNNs 4. Encoder-Decoder Sequence-to-Sequence Architectures

13-recurrent neural network - KAISTmac.kaist.ac.kr/.../slides/13-recurrent_neural_network.pdf · 2019-05-16 · The resulting network is called a recurrent neural network (RNN). RNNs

13-recurrent neural network - KAISTmac.kaist.ac.kr/.../slides/13-recurrent_neural_network.pdf · 2019-05-16 · The resulting network is called a recurrent neural network (RNN). RNNs

Recurrent Neural Networks (RNNs) - GitHub Pages · 2020-03-25 · Recurrent Neural Networks (RNNs) Dr. Benjamin Roth, Nina Poerner CIS LMU Munchen Dr. Benjamin Roth, Nina Poerner

Recurrent Neural Networks (RNNs) - GitHub Pages · 2020-03-25 · Recurrent Neural Networks (RNNs) Dr. Benjamin Roth, Nina Poerner CIS LMU Munchen Dr. Benjamin Roth, Nina Poerner

jason.williams@microsoft.com kavosh@brown.edu ...gmail.com Abstract End-to-end learning of recurrent neural networks (RNNs) is an attractive solu-tion for dialog systems; however,

[email protected] [email protected] ...gmail.com Abstract End-to-end learning of recurrent neural networks (RNNs) is an attractive solu-tion for dialog systems; however,

DA-RNN: Semantic Mapping with Data Associated Recurrent ... · work, we introduce Data Associated Recurrent Neural Networks (DA-RNNs), a novel framework for joint 3D scene mapping

DA-RNN: Semantic Mapping with Data Associated Recurrent ... · work, we introduce Data Associated Recurrent Neural Networks (DA-RNNs), a novel framework for joint 3D scene mapping

Statistical Natural Language Processing - Recurrent and ...DeepANNs RNNs CNNs Whydeepnetworks? • Wesawthatafeed-forwardnetworkwithasinglehiddenlayerisauniversal approximator •

Statistical Natural Language Processing - Recurrent and ...DeepANNs RNNs CNNs Whydeepnetworks? • Wesawthatafeed-forwardnetworkwithasinglehiddenlayerisauniversal approximator •

Sequence Modelling and Recurrent Neural Networks (RNNs) · •Neural Networks (recap) and Deep Learning •Improving DNN: Hyperparameter tuning, regularization, optimization •Convolutional

Sequence Modelling and Recurrent Neural Networks (RNNs) · •Neural Networks (recap) and Deep Learning •Improving DNN: Hyperparameter tuning, regularization, optimization •Convolutional

Lecture 8: Recurrent Neural NetworksRecurrent neural networks (RNNs) •Recurrent •Perform the same task for every element of a sequence, with the output being dependent on the previous

Lecture 8: Recurrent Neural NetworksRecurrent neural networks (RNNs) •Recurrent •Perform the same task for every element of a sequence, with the output being dependent on the previous

Sequence Modeling: Recurrent and Recursive Netssrihari/CSE676/10.0 SequenceModeling.pdfRecurrent Neural Networks process sequential data •RNNs are a family of neural nets for sequential

Sequence Modeling: Recurrent and Recursive Netssrihari/CSE676/10.0 SequenceModeling.pdfRecurrent Neural Networks process sequential data •RNNs are a family of neural nets for sequential

TopicsPlan for Today • Model – Recurrent Neural Networks (RNNs) • Learning – BackPropThrough Time (BPTT) (C) Dhruv Batra and Zsolt Kira 2

TopicsPlan for Today • Model – Recurrent Neural Networks (RNNs) • Learning – BackPropThrough Time (BPTT) (C) Dhruv Batra and Zsolt Kira 2

Recurrent Neural Networks (RNNs) - idi.ntnu.no of Recurrent NNs 1 Language Modeling - Given a sequence of words, predict the next one. ... "surrender" U, V, W = Weights ¶L ¶H1 =

Recurrent Neural Networks (RNNs) - idi.ntnu.no of Recurrent NNs 1 Language Modeling - Given a sequence of words, predict the next one. ... "surrender" U, V, W = Weights ¶L ¶H1 =

Introduction to Deep Learning - Robot Learning Laboratorycpslab.snu.ac.kr/courses/deep-learning-2018/files/04_dl... · Recurrent Neural Networks : Typical RNNs Prof. Songhwai Oh (ECE,

Introduction to Deep Learning - Robot Learning Laboratorycpslab.snu.ac.kr/courses/deep-learning-2018/files/04_dl... · Recurrent Neural Networks : Typical RNNs Prof. Songhwai Oh (ECE,

Recurrent Neural Networks - UPC Universitat Politècnica ...bejar/DLMAI/RNN.pdf · Augmented recurrent neural networks RNNs are Turing complete, but it is di cult to achieve it in

Recurrent Neural Networks - UPC Universitat Politècnica ...bejar/DLMAI/RNN.pdf · Augmented recurrent neural networks RNNs are Turing complete, but it is di cult to achieve it in

Recurrent Neural Networksbhiksha/courses/deeplearning/...The Backpropagation Through Time (BTT) Algorithm Different Recurrent Neural Network (RNN) paradigms How Layering RNNs works

Recurrent Neural Networksbhiksha/courses/deeplearning/...The Backpropagation Through Time (BTT) Algorithm Different Recurrent Neural Network (RNN) paradigms How Layering RNNs works

Introduction to Deep Learning - RLLAB @ SNUrllab.snu.ac.kr/courses/deep-learning-2018/files/04_dl... · Recurrent Neural Networks : Typical RNNs Prof. Songhwai Oh (ECE, SNU) Introduction

Introduction to Deep Learning - RLLAB @ SNUrllab.snu.ac.kr/courses/deep-learning-2018/files/04_dl... · Recurrent Neural Networks : Typical RNNs Prof. Songhwai Oh (ECE, SNU) Introduction

Learning visual motion with recurrent neural networksmarius/slides/RedwoodTalk.pdf · Marius Pachitariu Learning visual motion with RNNs 12/48. Learning visual motion Statistical

Learning visual motion with recurrent neural networksmarius/slides/RedwoodTalk.pdf · Marius Pachitariu Learning visual motion with RNNs 12/48. Learning visual motion Statistical

Introduction - ISIP€¦  · Web viewThis system integrates convolutional neural networks (CNNs) with recurrent neural networks (RNNs) to deliver state of the art performance. This

Introduction - ISIP€¦  · Web viewThis system integrates convolutional neural networks (CNNs) with recurrent neural networks (RNNs) to deliver state of the art performance. This

RNNs: Teacher Forcing

RNNs: Teacher Forcing

Deep Learning - Recurrent Neural Network (RNNs) · Deep Learning Recurrent Neural Network (RNNs) Ali Ghodsi University of Waterloo October 23, 2015 ... in preparation, Deep Learning

Deep Learning - Recurrent Neural Network (RNNs) · Deep Learning Recurrent Neural Network (RNNs) Ali Ghodsi University of Waterloo October 23, 2015 ... in preparation, Deep Learning

RNNs and Tensorflow

RNNs and Tensorflow