Deep Advances in Generative Modeling
-
Upload
indico-data -
Category
Data & Analytics
-
view
2.696 -
download
2
Transcript of Deep Advances in Generative Modeling
![Page 1: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/1.jpg)
Deep Advances in Generative Modeling
Alec Radford@AlecRad
March 5th 2016
![Page 2: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/2.jpg)
Generative modelingModeling complex high dimensional data is an open problem.
Deep generative models are currently making progress here.
Various areas of study/application:
unsupervised/representation/manifold learning
generative counterparts of discriminative models
density/likelihood estimation
conditional generation
![Page 3: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/3.jpg)
Examples of Generative Modeling
![Page 4: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/4.jpg)
CNNs and RNNs
![Page 5: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/5.jpg)
Useful Generative Model - Skipthought [1506.06726]
![Page 6: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/6.jpg)
Two promising approaches
Variational Autoencoders (VAE) Kingma and Welling [1312.6114]
Generative Adversarial Networks (GAN) Goodfellow et al. [1406.2661]
encoder Z decoder x̂X
z generator x̂discriminato
r
X
![Page 7: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/7.jpg)
Variational Autoencoder
from Kingma and Welling [1312.6114]
● Theoretically elegant autoencoder● Straightforward to implement● Impose a prior on code space
○ regularization○ allows for sampling
● Optimizes variational lower bound on likelihood
encoder Z decoder x̂X
![Page 8: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/8.jpg)
Generative Adversarial Networks z x̂
discriminator
X
generator
![Page 9: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/9.jpg)
Generative Adversarial Networks z x̂
discriminator
X
generator
![Page 10: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/10.jpg)
VAE Extensions
from Kingma et al. [1406.5298]
Semi-Supervised Learning
from Gregor et al. [1502.04623]
DRAW
![Page 11: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/11.jpg)
GAN Extensions - LAPGAN
![Page 12: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/12.jpg)
Deep convolutional GANs (DCGAN) [1511.06434]
Luke Metz Soumith ChintalaAlec Radford
tl;dr add more layers
indico indico FAIR
![Page 13: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/13.jpg)
Deep convolutional GANs (DCGAN) [1511.06434]
![Page 14: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/14.jpg)
DCGAN Architecture tricks
No fully connected layers
Batch Normalization Ioffe and Szegedy [1502.03167]
Leaky Rectifier in the discriminator
Use Adam Kingma and Ba [1412.6980]
Tweak Adam hyperparameters a bit (lr=0.0002, b1=0.5)
![Page 15: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/15.jpg)
Really really really ridiculously good looking samples
on constrained image distributions :(
![Page 16: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/16.jpg)
Interpolation suggests non-overfitting behavior
![Page 17: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/17.jpg)
Vector arithmetic properties of generator
![Page 18: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/18.jpg)
Generator disentangles objects from scene?
![Page 19: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/19.jpg)
Discriminator learns generalizing object detectors
These are responses on validation examples!
![Page 20: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/20.jpg)
Results on standard supervised tasks
![Page 21: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/21.jpg)
Conditional DCGAN
![Page 22: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/22.jpg)
Conditional DCGAN (unpublished)
Sunrise over the ocean
Beautiful falls and stream
sahara desert sand dunes
Tropical rainforest brazil
Stars of the milkyway at night
![Page 23: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/23.jpg)
IssuesStill not completely stable
especially for deep and higher res
Unconstrained natural images
Even the biggest models underfit
Hard to evaluate
no reliable/straightforward metrics
No inference model
limits kinds of analysis
Little work on conv VAE equivalents
makes comparison difficult
Some funky stuff going on
separate data/sample batchnorm statistics
train with heuristic cost not GAN theory
![Page 24: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/24.jpg)
Hybridizing VAEs and GANs (best of both worlds?)
from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]
![Page 25: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/25.jpg)
Hybridizing VAEs and GANs (best of both worlds?)
from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]
![Page 26: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/26.jpg)
Thanks!
Questions?
![Page 27: Deep Advances in Generative Modeling](https://reader036.fdocuments.in/reader036/viewer/2022062401/58f9a937760da3da068b6c4e/html5/thumbnails/27.jpg)
indico.io