DRAWDeepRecurrentA-en.veWriter

KarolGregor,IvoDanihelka,AlexGraves,DaniloJimenezRezende,DaanWierstraGoogleDeepmind–(ICML2015)

TranQuocHoanPaperalert@2015-12-07

A-en.onPlease!!!(???)

Wheretolook?

Wheretoread?

Wheretowrite?

Imagegenera.ngwitha-en.on

Trainedgenera.ngsequen.alMNISTdigitwithouta)en,on

Trainedgenera.ngsequen.alMNISTdigitwitha)en,on

Introduc.on

•  Addsana-en.onmechanismovertheinputtodefineasequen.alprocess

•  Augmentstheencoderanddecoderwithrecurrentneuralnetworks

•  Inferenceandgenera.ondefinedbyasequen.alprocess,evenfornon-sequen,aldata

4/25

Varia.onalAutoencoderRecap

•  KingmaandWelling,Auto-EncodingVaria,onalBayes,(ICLR)2014

•  Rezende,MohamedandWierstra,Stochas,cback-propaga,onandvaria,onalinferenceindeeplatentGaussianmodels,(ICML)2014

We’llintroduceVAEfirst!!!

5/25

VAEApproach•  Leverageneuralnetworkstolearnalatentvariablemodel

Imagefrom:Ward,A.D.,Hamarneh,G.:3DSurfaceParameteriza,onUsingManifoldLearningforMedialShapeRepresenta,on,ConferenceonImageProcessing,Proc.ofSPIEMedicalImaging,2007

Latentvariable

6/25

Theinferenceproblem•  Wheredoeszcomefrom?->Theposteriorp(z|x)isintractable

•  VAEapproach:findinferencemachinethatlearnstoapproximatetheposteriorbyop.mizingthevaria.onallowerbound

7/25

Encoder Decoder

Reparametriza.ontrick•  Consider

•  Parametrizezaswhere

•  Parametrizexaswhere

8/25

Trainingwithbackpropaga.on•  Duetoreparametriza.ontrick,wecansimultaneously

trainboththegenera.vemodelandtheinferencemodelbyop.mizingthevaria.onalboundusinggradientbackpropaga.on

•  Maximum

9/25

BuildingasimpleVAE

10/25

Fromh-p://mul.threaded.s.tchfix.com/blog/2015/09/17/deep-style/

Encoding

11/25

Encoding

12/25

Encoding

13/25

Varia.onalstep

14/25

Varia.onalstep

15/25

Decoding

16/25

Decoding

17/25

Update

18/25

Update

19/25

DRAWModelCanvasatt

Couldbeasubsetofx(witha-en.on) 20/25

DRAWMNISTGenera.on

•  Simplestinstan.a.onofDRAWiswithoutana-en.onmechanism

•  En.reinputispassedtotheencoderatevery.me-step

•  Decoderwritestoen.recanvasatevery.me-step

No.cehowthenetworkfirstgeneratesaveryblurryimagethatissubsequentlyrefined

21/25

DRAWA-en.onMechanism

•  DRAWusesadifferen.ablea-en.onmechanism

•  A-en.onusesrecurrence(viadecoder)toselectsubsetsofxforreadingandwri.ng

•  A-en.oncontrolstheextractedpatchloca.on,scaleandblur

22/25

DRAWA-en.onMechanism

23/25

DRAW:Clu-eredMNISTClassifica.on

•  Drawa-en.onbeingappliedtoaclassifica.ontask:clu-eredMNIST

•  A-en.onlearnstofocusonthedigitinthescene

Time→24/25

DRAWMNISTGenera.onwithA-en.on

Figure6.GeneratedMNISTimages.AlldigitsweregeneratedbyDRAWexceptthoseintherightmostcolumn,whichshowsthetrainingsetimagesclosesttothoseinthecolumnsecondtoright.Notethatthenetworkwastrainedonthebinarysamples,whilethegeneratedimagesaremeanprobabili.es. 25/25

DRAWMNISTGenera.onwithA-en.on

h-ps://www.youtube.com/watch?v=Zt-7MI9eKEo

26/25

DRAWDeepRecurrentA-en.veWriter

KarolGregor,IvoDanihelka,AlexGraves,DaniloJimenezRezende,DaanWierstraGoogleDeepmind–(ICML2015)

TranQuocHoanPaperalert@2015-12-07

Manyslidesareborrowedfromh)p://videolectures.net/deeplearning2015_courville_autoencoder_extension/

h)p://mul,threaded.s,tchfix.com/blog/2015/09/17/deep-style/