GenerativeAdversarialNetworks

Yoav Orlev - Deep Image Processing Seminar

Generative ModelsAnd why we want them

Discriminative vs Generative

Discriminative vs Generative

What are generative models?

Given a training set drawn from some distribution,

Tries to fit a model to best represent the data probability.

What are generative models?

Given a training set drawn from some distribution,

Tries to fit a model to best represent the data probability.

Why do we want generative models?

Why do we want generative models?

“What I cannot create

I can not understand”*Richard Feynman

Why do we want generative models?

● Understanding and Compressing Knowledge

Why do we need generative models?

Radford et al (2015).

100 Million Parameters

● Understanding and Compressing Knowledge

DCGAN Radford et al.

200GB -> 10MB

Generated Results From The ImageNet Dataset

Why do we want generative models?

● Understanding and Compressing Knowledge

● Semi supervised Learning

Why do we want generative models?

● Semi supervised Learning

Assumes some

Structure to the underlying

distribution of the data.

Why do we want generative models?

● Semi supervised Learning

99.14% Accuracy on MNIST

Only 10 labels per class.

OpenAI Research

Why do we need generative models?

● Understanding and Compressing Knowledge

● Semi supervised Learning

● Multi modal outputs

Why do we need generative models?

● Multi modal outputs

Lotter et al. (2015)

Why do we need generative models?

● Understanding and Compressing Knowledge

● Semi supervised Learning

● Multi modal outputs

● Generating Data!

Why do we need generative models?

● Understanding and Compressing Knowledge

● Semi supervised Learning

● Multi modal outputs

● Generating Data!

● And more...

Generative Model Types

● Naive Bayes

● Variational Autoencoders

● Hidden Markov Models

● Generative Adversarial Network

● And more...

Generative Model Types

● Naive Bayes

● Variational Autoencoders

● Hidden Markov Models

● Generative Adversarial Network

● And more...

Generative adversarial networksQuick Introduction

End Goal

Training set

End Goal

Input Output

1

2

7

5

2

1

code

End Goal

Input Output

8

2

7

9

3

4

code

How do we learn to generate?

Every Month Different Show.

Every Month Different Show.

Concert name 1

You can’t come in!The ticket color should be red!

Concert name 2

You can’t come in!The ticket date should be in Arial font!

Concert name 3

You can’t come in!The ticket … should be ...!

Concert name x

Concert name 1011

Come In!

Credits

DiscriminatorGenerator Training Data

Notice!

Generator Training Data

X

Generative adversarial networksIan J. Goodfellow et el. [2014]

GAN has 2 players

D Discriminator

GeneratorG

GAN has 2 players

D Discriminator

GeneratorG

Given a sample x, Outputs the probability of x coming from the training set

GAN has 2 players

D Discriminator

GeneratorG

Given a sample x, Outputs the probability of x coming from the training set

Given a random z, Output an imageWhich look as if it came from the training

set

GAN Architecture

Gz

G(z)

x

D

D(x)

D(G(z)

Slide taken from Kevin Mcguinness -

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

GAN Architecture

Gz

G(z)

x

D

D(x)

D(G(z)

Slide taken from Kevin Mcguinness -

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

1

0

GAN Training

Gz

G(z)

x

D

D(x)

D(G(z)

Slide taken from Kevin Mcguinness -

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

GAN Training

Gz

G(z)

x

D

D(x)

D(G(z)

Slide taken from Kevin Mcguinness -

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

GAN Process

GAN Process

GAN Process

GAN Process

GAN Process

GAN Architecture

Slide taken from :

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

Loss Function - MiniMax Game

Loss Function - MiniMax Game

Mini-Max Game

What is the max value V(D, G) can take?

Mini-Max Game

What is the max value V(D, G) can take? 0

Mini-Max Game

What is the max value V(D, G) can take? 0

What is the min value V(D, G) can take?

Mini-Max Game

What is the max value V(D, G) can take? 0

What is the min value V(D, G) can take? -

8

Why D wants to maximize

Why D wants to maximize

training set

generated sample

Why D wants to maximize

training set

generated sample

Why G wants to Minimize

generated sample

Why G wants to Minimize

generated sample

Loss - In practice

Discriminator Generator

Given (x, 1) the loss is :

Given (x=G(z), 0) the loss is :

Given (G(z), 1) the loss is :

So basically we have cross entropy loss :

-

-

In practice we would maximize :

Game Equilibrium

log(x)

0.5

log(1-x)

GAN Recap

Gz

G(z)

x

D

D(x)

D(G(z)

Slide taken from Kevin Mcguinness -

https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

GAN Algorithm

Process

Generative Adversarial NetworksResults

Original GAN (2014)

Goodfellow et el. (2014)

DCGAN (2015)

Radford et el. (2014)

PPGN (2016)

Nguyen et el. (2016)

Progressive Growing of GANs (2018)

Karras et el. (2018)

Generative adversarial networksProblems

Non Convergence

Unlike traditional Deep Learning approaches GAN involve two players

● D is trying to maximize its reward● G is trying to minimize D’s reward

● SGD was not designed to find the NE of a game● Might not converge!

Non Convergence example

Non Convergence example

● State 1: [x > 0 | y > 0 | V > 0]

Increase y | Decrease x

Non Convergence example

● State 1: [x > 0 | y > 0 | V > 0]

Increase y | Decrease x

● State 2: [x < 0 | y > 0 | V < 0]

Decrease y | Decrease x

Non Convergence example

● State 1: [x > 0 | y > 0 | V > 0]

Increase y | Decrease x

● State 2: [x < 0 | y > 0 | V < 0]

Decrease y | Decrease x

● State 3: [x < 0 | y < 0 | V > 0]

Non Convergence example

● State 1: [x > 0 | y > 0 | V > 0]

Increase y | Decrease x

● State 2: [x < 0 | y > 0 | V < 0]

Decrease y | Decrease x

● State 3: [x < 0 | y < 0 | V > 0]

Non Convergence example

● State 1: [x > 0 | y > 0 | V > 0]

Increase y | Decrease x

● State 2: [x < 0 | y > 0 | V < 0]

Decrease y | Decrease x

● State 3: [x < 0 | y < 0 | V > 0]

Mode collapse

Mode collapse

Mode collapse

Progressive growing of GANs for improved quality, Stability and variation.Karras, Tero, et al. (2018)

Progressive Growing

Smooth Transition

Network architecture and learning

Network architecture and learning

Network architecture and learning

Network architecture and learning

Network architecture and more

Minibatch Standard Deviation

Results

Results

Results

Thank You!

References

● Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016).

● Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint

arXiv:1703.10593 (2017).

● Karras, Tero, et al. "Progressive growing of gans for improved quality, stability, and variation." arXiv preprint arXiv:1710.10196

(2017).

● Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014.