Practice: VAE and GAN - Deep Generative Models

37
Practice: VAE and GAN Hao Dong Peking University 1

Transcript of Practice: VAE and GAN - Deep Generative Models

Page 1: Practice: VAE and GAN - Deep Generative Models

Practice: VAE and GANHao Dong

Peking University

1

Page 2: Practice: VAE and GAN - Deep Generative Models

2

Practice: VAE + GAN

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 3: Practice: VAE and GAN - Deep Generative Models

3

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 4: Practice: VAE and GAN - Deep Generative Models

4

Hello World: MNIST Classification

28x28x1 = 784 binary values/image

28

28

MNIST dataset

• Image X is a list of row vectors:>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 784))>>> print(X_train.shape)… (50000, 784)

• Image X is a list of images:>>> X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 28, 28, 1))>>> print(X_train.shape)… (50000, 28, 28, 1)

If RGB image, we will have 3 channels

(height, width, channels)

Page 5: Practice: VAE and GAN - Deep Generative Models

5

Hello World: MNIST Classification

• Simple Iteration

Page 6: Practice: VAE and GAN - Deep Generative Models

6

Hello World: MNIST Classification

• Dataset API

Page 7: Practice: VAE and GAN - Deep Generative Models

7

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 8: Practice: VAE and GAN - Deep Generative Models

8

Introduction of VAE

!𝑿!𝒁X

L2

KLD

• Two network architectures

• Two loss functions

• Reparameterization trick

Page 9: Practice: VAE and GAN - Deep Generative Models

9

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 10: Practice: VAE and GAN - Deep Generative Models

10

VAE Architecture

!𝑿!𝒁X

L2

KLD

28

28

28x28x1 or 784

𝑥~𝑝𝑑𝑎𝑡𝑎

Page 11: Practice: VAE and GAN - Deep Generative Models

11

VAE Architecture

!𝑿!𝒁X

L2

KLD

• Architecture of Encoder

Page 12: Practice: VAE and GAN - Deep Generative Models

12

VAE Architecture

!𝑿!𝒁X

L2

KLD

• Architecture of Decoder/Generator

Page 13: Practice: VAE and GAN - Deep Generative Models

13

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 14: Practice: VAE and GAN - Deep Generative Models

14

VAE Training

!𝑿!𝒁X

L2

KLD

Reconstruction Loss

Page 15: Practice: VAE and GAN - Deep Generative Models

15

VAE Training

!𝑿!𝒁X

L2

KLD

KL-Divergence loss

Page 16: Practice: VAE and GAN - Deep Generative Models

16

VAE Training

!𝑿!𝒁X

L2

KLD

Training Pipeline

Page 17: Practice: VAE and GAN - Deep Generative Models

17

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 18: Practice: VAE and GAN - Deep Generative Models

18

VAE Interpolation

Page 19: Practice: VAE and GAN - Deep Generative Models

19

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 20: Practice: VAE and GAN - Deep Generative Models

20

Sampling

!𝑿Z

Page 21: Practice: VAE and GAN - Deep Generative Models

21

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 22: Practice: VAE and GAN - Deep Generative Models

22

Introduction of DCGAN

• Two network architectures

• Two loss functions!𝑿

X

real

fakeZ

Page 23: Practice: VAE and GAN - Deep Generative Models

23

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 24: Practice: VAE and GAN - Deep Generative Models

24

DCGAN Architecture

!𝑿

X

real

fakeZ

28

28

28x28x1 or 784

𝑧~𝑁(0,1)

𝑥~𝑝𝑑𝑎𝑡𝑎

Page 25: Practice: VAE and GAN - Deep Generative Models

25

DCGAN Architecture

!𝑿

X

real

fakeZ

Architecture of generator

Page 26: Practice: VAE and GAN - Deep Generative Models

26

DCGAN Architecture

!𝑿

X

real

fakeZ

Architecture of discriminator

Page 27: Practice: VAE and GAN - Deep Generative Models

27

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 28: Practice: VAE and GAN - Deep Generative Models

28

DCGAN Training

!𝑿

X

fake

realZ

Loss of discriminator

Page 29: Practice: VAE and GAN - Deep Generative Models

29

DCGAN Training

!𝑿 realZ

Loss of generator

Page 30: Practice: VAE and GAN - Deep Generative Models

30

DCGAN Training

Training pipeline

!𝑿

X

real

fakeZ

Page 31: Practice: VAE and GAN - Deep Generative Models

31

• Hello World: MNIST Classification

• Introduction of VAE• VAE Architecture• VAE Training• VAE Interpolation• Sampling

• Introduction of DCGAN• DCGAN Architecture• DCGAN Training• DCGAN Interpolation

Page 32: Practice: VAE and GAN - Deep Generative Models

32

DCGAN Interpolation

!𝑿

Z1

Z2

(1-a) Z1+a Z2

Page 33: Practice: VAE and GAN - Deep Generative Models

33

More

Page 34: Practice: VAE and GAN - Deep Generative Models

34

Improved GANLSGANWGAN

WGAN-GPBiGAN

VAE-GAN…

with MNIST

Page 35: Practice: VAE and GAN - Deep Generative Models

35

Pix2PixCycleGAN

SRGAN…

with other datasets

Page 36: Practice: VAE and GAN - Deep Generative Models

36

Proposal Your Projects

Page 37: Practice: VAE and GAN - Deep Generative Models

Thanks

37


Generative Adversarial Networks (GAN)hanxiaol/slides/gan.pdf · Outline I Implicit Generative Models I GAN I Variants I f-GAN I Wasserstein GAN I Use cases I Conditional generation

Generative Adversarial Networks (GAN)hanxiaol/slides/gan.pdf · Outline I Implicit Generative Models I GAN I Variants I f-GAN I Wasserstein GAN I Use cases I Conditional generation

Social GAN: Socially Acceptable Trajectories with Generative … · 2018. 12. 5. · Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks AgrimGupta,

Social GAN: Socially Acceptable Trajectories with Generative … · 2018. 12. 5. · Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks AgrimGupta,

Capsule Architecture as a Discriminator in Generative Adversarial …€¦ · Variational Autoencoder VAE Deep Learning DL Convolutional Neural Network CCN Generative Adversarial

Capsule Architecture as a Discriminator in Generative Adversarial …€¦ · Variational Autoencoder VAE Deep Learning DL Convolutional Neural Network CCN Generative Adversarial

MD-GAN: Multi-Discriminator Generative Adversarial ...

MD-GAN: Multi-Discriminator Generative Adversarial ...

Generative Adversarial Networks for Hyperspectral Image ...static.tongtianta.site/paper_pdf/488448f6-1a35-11e9-89a1-00163e08bb86.pdfAbstract—A generative adversarial network (GAN)

Generative Adversarial Networks for Hyperspectral Image ...static.tongtianta.site/paper_pdf/488448f6-1a35-11e9-89a1-00163e08bb86.pdfAbstract—A generative adversarial network (GAN)

Deep Generative Models - Columbia Universitystat.columbia.edu/~cunningham/teaching/GR8201/STAT_GR8201_20… · STAT G8201: Deep Generative Models 12 / 29 -VAE: qualitative assessment

Deep Generative Models - Columbia Universitystat.columbia.edu/~cunningham/teaching/GR8201/STAT_GR8201_20… · STAT G8201: Deep Generative Models 12 / 29 -VAE: qualitative assessment

Introduction of Generative Adversarial Network (GAN)yvchen/f106-adl/doc/171211+171214_GAN.pdf · GAN as structured learning algorithm Conditional Generation by GAN •Modifying input

Introduction of Generative Adversarial Network (GAN)yvchen/f106-adl/doc/171211+171214_GAN.pdf · GAN as structured learning algorithm Conditional Generation by GAN •Modifying input

RIS-GAN: Explore Residual and Illumination with Generative ...

RIS-GAN: Explore Residual and Illumination with Generative ...

MC-GAN: Multi-conditional Generative Adversarial Network for Image ... · MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis Hyojin Park 1 wolfrun@snu.ac.kr

MC-GAN: Multi-conditional Generative Adversarial Network for Image ... · MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis Hyojin Park 1 [email protected]

GAN-OPC: Mask Optimization with Lithography-guided Generative … · 2018. 6. 29. · GAN-OPC: Mask Optimization with Lithography-guided Generative Adversarial Nets Haoyu Yang, Shuhe

GAN-OPC: Mask Optimization with Lithography-guided Generative … · 2018. 6. 29. · GAN-OPC: Mask Optimization with Lithography-guided Generative Adversarial Nets Haoyu Yang, Shuhe

GraphRNN: A Deep Generative Model for Graphs · Recent advances in deep generative models, such as varia-tional autoencoders (VAE) (Kingma & Welling,2014) and generative adversarial

GraphRNN: A Deep Generative Model for Graphs · Recent advances in deep generative models, such as varia-tional autoencoders (VAE) (Kingma & Welling,2014) and generative adversarial

Deep Learning: A Statistical Perspectivestat.snu.ac.kr/mcp/Lecture_9_VAE.pdf · 2019-10-30 · Variational autoencoder (VAE) and Generative adversarial network (GAN) attempt to estimate

Deep Learning: A Statistical Perspectivestat.snu.ac.kr/mcp/Lecture_9_VAE.pdf · 2019-10-30 · Variational autoencoder (VAE) and Generative adversarial network (GAN) attempt to estimate

Distilling portable Generative Adversarial Networks for ... AAAI GAN Distillation.pdf · Distilling portable Generative Adversarial Networks for Image Translation Hanting Chen1, Yunhe

Distilling portable Generative Adversarial Networks for ... AAAI GAN Distillation.pdf · Distilling portable Generative Adversarial Networks for Image Translation Hanting Chen1, Yunhe

Sub-GAN: An Unsupervised Generative Model via Subspacesliangjie.xyz/LjHomepageFiles/paper_files/SubGAN_eccv.pdf · Sub-GAN: An Unsupervised Generative Model via Subspaces Jie Liang

Sub-GAN: An Unsupervised Generative Model via Subspacesliangjie.xyz/LjHomepageFiles/paper_files/SubGAN_eccv.pdf · Sub-GAN: An Unsupervised Generative Model via Subspaces Jie Liang

MSG-GAN: Multi-Scale Gradients for Generative Adversarial …openaccess.thecvf.com/content_CVPR_2020/papers/Karnewar_MSG … · MSG-GAN: Multi-Scale Gradients for Generative Adversarial

MSG-GAN: Multi-Scale Gradients for Generative Adversarial …openaccess.thecvf.com/content_CVPR_2020/papers/Karnewar_MSG … · MSG-GAN: Multi-Scale Gradients for Generative Adversarial

f-GAN: Training Generative Neural Samplers using Variational Divergence ...papers.nips.cc/paper/6066-f-gan-training-generative-neural... · f-GAN: Training Generative Neural Samplers

f-GAN: Training Generative Neural Samplers using Variational Divergence ...papers.nips.cc/paper/6066-f-gan-training-generative-neural... · f-GAN: Training Generative Neural Samplers

Sub-GAN: An Unsupervised Generative Model via Subspacesliangjie.xyz/LjHomepageFiles/paper_files/SubGAN_eccv_slides.pdf · 14 Sub-GAN: An Unsupervised Generative Model via Subspaces

Sub-GAN: An Unsupervised Generative Model via Subspacesliangjie.xyz/LjHomepageFiles/paper_files/SubGAN_eccv_slides.pdf · 14 Sub-GAN: An Unsupervised Generative Model via Subspaces

ST-GAN: Spatial Transformer Generative Adversarial Networks ......ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing Chen-Hsuan Lin1* Ersin Yumer2,3*

ST-GAN: Spatial Transformer Generative Adversarial Networks ......ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing Chen-Hsuan Lin1* Ersin Yumer2,3*

Networkscampar.in.tum.de/files/teaching/2019ws/MLMI/protected/GANs/GAN… · Discriminative vs. Generative Models Generative models • Can generate new data instances. • Capture

Networkscampar.in.tum.de/files/teaching/2019ws/MLMI/protected/GANs/GAN… · Discriminative vs. Generative Models Generative models • Can generate new data instances. • Capture

SPI-GAN: Towards Single-Pixel Imaging through Generative ...

SPI-GAN: Towards Single-Pixel Imaging through Generative ...