Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation...
Transcript of Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation...
![Page 1: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/1.jpg)
GENERATIVE ADVERSARIAL
NETWORKS (GAN)Presented by Omer Stein and Moran Rubin
![Page 2: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/2.jpg)
GENERATIVE MODEL
Given a training dataset, x, try to estimate the
distribution , Pdata(x)
Explicitly or Implicitly (GAN)
Explicitly estimation:
Implicitly estimation:
![Page 3: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/3.jpg)
GENERATIVE MODEL
Given a training dataset, x, try to estimate the
distribution , Pdata(x)
Explicitly or Implicitly (GAN)
Explicitly estimation:
Implicitly estimation:
Variational
Autoencoders
(VAE)
![Page 4: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/4.jpg)
GENERATIVE MODEL
Given a training dataset, x, try to estimate the
distribution , Pdata(x)
Explicitly or Implicitly (GAN)
Explicitly estimation:
Implicitly estimation:
Variational
Autoencoders
(VAE)
Generative
Adversarial
Networks
(GAN)
![Page 5: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/5.jpg)
GENERATIVE MODEL
Given a training dataset, x, try to estimate the
distribution , Pdata(x)
Explicitly or Implicitly (GAN)
Explicitly estimation:
Implicitly estimation:
Variational
Autoencoders
(VAE)
Generative
Adversarial
Networks
(GAN)
![Page 6: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/6.jpg)
WHY GENERATIVE MODEL?
Realistic samples for artwork, colorization, etc.
Training GM can enable inference of latent
representation that can be useful as general
features
GM can be trained with missing data and can
provide predictions on inputs that are missing
data
GM enable machine learning to work with multi-
model outputs (produce multiple different correct
answers)
![Page 7: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/7.jpg)
GENERATIVE ADVERSARIAL NETWORKS (GAN)
Problem:
Want to sample from complex, high dimensional
training distribution.
No direct way to do this!
Solution:
Sample from a simple distribution e.g. random noise.
Learn complex transformation to training
distribution.
![Page 8: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/8.jpg)
GENERATIVE ADVERSARIAL NETWORKS (GAN)
Problem:
Want to sample from complex, high dimensional
training distribution.
No direct way to do this!
Solution:
Sample from a simple distribution e.g. random noise.
Learn complex transformation to training
distribution.
Complex
transformation =
neural network!
![Page 9: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/9.jpg)
GENERATIVE ADVERSARIAL NETWORKS (GAN)
Problem:
Want to sample from complex, high dimensional
training distribution.
No direct way to do this!
Solution:
Sample from a simple distribution e.g. random noise.
Learn complex transformation to training
distribution.
Complex
transformation =
neural network!
![Page 10: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/10.jpg)
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
𝑥(1), … ,𝑥(𝑚)~𝑃𝑑𝑎𝑡𝑎
𝐺(𝑧)~𝑃𝑔
𝐷 𝑥𝐷(𝐺(𝑧))Generator
DiscriminatorReal
FakeLoss
Late
nt
ran
dom
va
riable
Sample
SampleReal World
Images
𝑥~𝑃𝑑𝑎𝑡𝑎
![Page 11: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/11.jpg)
Generator Goal:
Try to fool the discriminator by
generating real looking images
Discriminator Goal:
Try to distinguish between real and
fake images
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
𝑥(1), … ,𝑥(𝑚)~𝑃𝑑𝑎𝑡𝑎
𝑥~𝑃𝑑𝑎𝑡𝑎
𝐺(𝑧)~𝑃𝑔
𝐷 𝑥𝐷(𝐺(𝑧))Generator
DiscriminatorReal
FakeLoss
Late
nt
ran
dom
va
riable
Sample
SampleReal World
Images
![Page 12: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/12.jpg)
TRAINING GAN : TWO PLAYER GAME
Real
FakeLoss
Late
nt
ran
dom
va
riable
Sample
SampleReal World
Images
counterfeiter
police
![Page 13: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/13.jpg)
Generator
DiscriminatorReal
FakeLoss
La
ten
t ra
nd
om
vari
able
Sample
SampleReal Images
𝑥(1), … ,𝑥(𝑚)~𝑃𝑑𝑎𝑡𝑎
𝑥~𝑃𝑑𝑎𝑡𝑎
𝐺(𝑧)~𝑃𝑔
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
𝐷 𝑥𝐷(𝐺(𝑧))
Generator Goal:
generate an image G(z) such that:
𝑃𝑔=𝑃𝑑𝑎𝑡𝑎 → 𝐷 𝐺 𝑍 = 1
Discriminator Goal:
Discriminate between real and generated images:
𝐷 𝑥 = 1 , 𝑤ℎ𝑒𝑟𝑒 𝑥 𝑖𝑠 𝑎 𝑟𝑒𝑎𝑙 𝑖𝑚𝑎𝑔𝑒𝐷 𝐺(𝑧) = 0 ,𝑤ℎ𝑒𝑟𝑒 𝐺(𝑧) 𝑖𝑠 𝑎 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑖𝑣𝑒 𝑖𝑚𝑎𝑔𝑒
TRAINING GAN : TWO PLAYER GAME
![Page 14: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/14.jpg)
G(z) – Generator network.
Output in the dimensions of x.
D(x) – Discriminator network. Output in [0,1] where 1 means x is from 𝒑𝒅𝒂𝒕𝒂. (𝐥𝐨𝐠{𝑫 𝒙 } [-inf,0] )
𝒎𝒊𝒏𝑮𝒎𝒂𝒙𝑫𝑽 𝑫,𝑮 = 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]
Discriminator output
for real dataDiscriminator
output for
generated fake
data G(z)
TRAINING GAN : THE MINMAX GAME
![Page 15: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/15.jpg)
Alternate between:
Gradient ascent on discriminator:
Gradient descent on generator:
𝒎𝒊𝒏𝑮𝒎𝒂𝒙𝑫𝑽 𝑫,𝑮 = 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]
TRAINING GAN : THE MINMAX GAME
𝒎𝒂𝒙𝑫[ 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]]
𝒎𝒊𝒏𝑮[𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]]
![Page 16: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/16.jpg)
Equilibrium is a saddle point of the discriminator
loss
Estimating this ratio using supervised learning is
the key approximation mechanism used by GAN
𝑫𝑮∗ 𝒙 =
𝒑𝒅𝒂𝒕𝒂(𝒙)
𝒑𝒅𝒂𝒕𝒂 𝒙 + 𝒑𝒈(𝒙)
𝒑𝒈 𝒙 =𝒑𝒅𝒂𝒕𝒂(𝒙)𝟎. 𝟓
Model
distributionData
Discriminator
TRAINING GAN : THE MINMAX GAME
![Page 17: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/17.jpg)
Alternate between:
Gradient ascent on discriminator:
Gradient descent on generator:
𝒎𝒊𝒏𝑮𝒎𝒂𝒙𝑫𝑽 𝑫,𝑮 = 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]
TRAINING GAN : THE MINMAX GAME
𝒎𝒂𝒙𝑫[ 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]]
𝒎𝒊𝒏𝑮[𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]]
In practice, optimizing this
generator objective not
work well!
Generator Loss
![Page 18: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/18.jpg)
Alternate between:
Gradient ascent on discriminator:
Gradient ascent on generator:
𝒎𝒊𝒏𝑮𝒎𝒂𝒙𝑫𝑽 𝑫,𝑮 = 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]
TRAINING GAN : THE MINMAX GAME
𝒎𝒂𝒙𝑫[ 𝑬𝒙~𝒑𝒅𝒂𝒕𝒂(𝒙)[𝐥𝐨𝐠{𝑫 𝒙 }]+ 𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝟏 − 𝑫 𝑮 𝒛 }]]
𝒎𝒂𝒙𝑮[𝑬𝒛~𝒑𝒛(𝒛)[𝐥𝐨𝐠{𝑫 𝑮 𝒛 }]]
Maximizing likelihood of
discriminator being wrong!
Higher gradient for bad samples
Generator Loss
![Page 19: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/19.jpg)
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
𝑥(1), … ,𝑥(𝑚)~𝑃𝑑𝑎𝑡𝑎
𝑥~𝑃𝑑𝑎𝑡𝑎
𝐺(𝑧)~𝑃𝑔
𝐷 𝑥𝐷(𝐺(𝑧))Generator
DiscriminatorReal
FakeLoss
Late
nt
ran
dom
va
riable
Sample
SampleReal World
Images
![Page 20: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/20.jpg)
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
Generator
Late
nt
ran
dom
va
riable
![Page 21: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/21.jpg)
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
Generator
Late
nt
ran
dom
va
riable
![Page 22: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/22.jpg)
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
Generator
Late
nt
ran
dom
va
riable
![Page 23: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/23.jpg)
TRAINING GAN : TWO PLAYER GAME
𝑧(1), … ,𝑧(𝑚)~𝑃𝑧
Generator
Late
nt
ran
dom
va
riable
![Page 24: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/24.jpg)
Architecture key insights:
Use batch normalization layers in both discriminator and
generator.
Use convolution with a stride greater than 1 instead of
pooling layer.
Use Adam optimizer instead of SGD.
Use ReLU activation in generator for all layers except for
the output, which uses Tanh.
Use LeakyReLU activation in the discriminator for all
layers.
DEEP CONVOLUTION GAN (DCGAN)
![Page 25: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/25.jpg)
Results:
DEEP CONVOLUTION GAN (DCGAN)
Trained filters
Images of bedrooms generated by a DCGAN
![Page 26: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/26.jpg)
Results:
DEEP CONVOLUTION GAN (DCGAN)
Vector space arithmetic
Samples
from the
model
![Page 27: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/27.jpg)
Results:
DEEP CONVOLUTION GAN (DCGAN)
Vector space arithmetic
Samples
from the
model
Average Z
vectors, do
arithmetic
![Page 28: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/28.jpg)
Results:
DEEP CONVOLUTION GAN (DCGAN)
Vector space arithmetic
![Page 29: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/29.jpg)
Our Results:
DEEP CONVOLUTION GAN (DCGAN)
![Page 30: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/30.jpg)
Our Results:
DEEP CONVOLUTION GAN (DCGAN)
![Page 31: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/31.jpg)
Our Results:
DEEP CONVOLUTION GAN (DCGAN)
![Page 32: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/32.jpg)
Our Results:
DEEP CONVOLUTION GAN (DCGAN)
![Page 33: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/33.jpg)
Our Results:
DEEP CONVOLUTION GAN (DCGAN)
The Generator never saw
these faces ! Only the
Discriminative network !
![Page 34: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/34.jpg)
Single image super-resolution
Image to image translation
Text to image
Style transfer
Video generation
Semi-supervised learning
And more…
GAN : APPLICATIONS
![Page 35: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/35.jpg)
Single image super-resolution
Image to image translation
Text to image
Style transfer
Video generation
Semi-supervised learning
And more…
GAN : APPLICATIONS
![Page 36: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/36.jpg)
Single image super-resolution
From the paper “Photo-Realistic Single Image Super-
Resolution Using a Generative Adversarial Network“
GAN : APPLICATIONS
![Page 37: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/37.jpg)
Single image super-resolution
From the paper “Photo-Realistic Single Image Super-
Resolution Using a Generative Adversarial Network“
GAN : APPLICATIONS
![Page 38: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/38.jpg)
Single image super-resolution
Image to image translation
Text to image
Style transfer
Video generation
Semi-supervised learning
And more…
GAN : APPLICATIONS
![Page 39: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/39.jpg)
IMAGE TO IMAGE TRANSLATION
![Page 40: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/40.jpg)
A class of vision and graphics problems
where the goal is to learn the mapping
between an input image and an output
image
IMAGE TO IMAGE TRANSLATION
![Page 41: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/41.jpg)
We will focus on 2 papers from 2017
dealing with unsupervised image to
image translation Unpaired Image-to-Image Translation using Cycle-
Consistent Adversarial Networks (UC Berkeley) -
cycleGAN
Unsupervised Image-to-Image Translation Networks
(NVIDIA) – UNIT network
IMAGE TO IMAGE TRANSLATION
![Page 42: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/42.jpg)
IMAGE TO IMAGE TRANSLATION
Data {𝑥𝑖 , 𝑦𝑖}
Goal: Learn mapping
from xi → 𝑦𝑖 across all
pairs
Data ( 𝑥 , {𝑦})
Goal: learn mapping
from group 𝑥 to group
{𝑦} by learning
underlying structure
Supervised approach Unsupervised approach
![Page 43: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/43.jpg)
IMAGE TO IMAGE TRANSLATION
Examples:
![Page 44: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/44.jpg)
IMAGE TO IMAGE TRANSLATION
Examples:
![Page 45: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/45.jpg)
Papers use similar approach to solving this
problem:
Use GAN generator – discriminator model
Use VAE latent space encoding as part generator
architecture (though only UNIT trains VAE
explicitly)
Learn translation in both directions at once
IMAGE TO IMAGE TRANSLATION
![Page 46: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/46.jpg)
assumes a shared latent space: pair of
corresponding images can be mapped to same
latent representation
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 47: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/47.jpg)
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 48: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/48.jpg)
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 49: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/49.jpg)
Encoder-generator pair is a VAE
E1(E2) encodes image 𝑥1 (𝑥2) to a latent vector z
G1 (G2) decodes a randomly perturbed version of the code z
VAE1 and VAE2 share weights: high level layers of E1 and
E2, and same for G1 and G2
VAE training ensure the reconstructed image and the
original image are similar
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 50: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/50.jpg)
Generator-discriminator pair is a GAN
Reconstruction is already trained by VAE, so D is only
applied to translated images
G1(G2) generates images for domain 1 (2) from a latent
vector z
D1 is trained to output True for 𝑥1, and False for 𝑥22→1
(opposite for D2)
GAN training ensures translated images resemble images
in the target domain
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 51: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/51.jpg)
Shared-latent space implies cycle consistency
Added weight sharing to regularize training
Forward translation: x*2 = G2(E1(x1))
Backward translation: x**1 = G1(E2(x*2))
Impose that original image x1 and cycle-translated image
x*1 are equal: x1 = x**1 (and similarly: x2 = x**2)
IMAGE TO IMAGE TRANSLATION
UNITCYCLE CONSISTENCY
![Page 52: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/52.jpg)
Explicit loss function:
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 53: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/53.jpg)
Explicit loss function:
IMAGE TO IMAGE TRANSLATION
UNIT
![Page 54: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/54.jpg)
IMAGE TO IMAGE TRANSLATION
CycleGAN
![Page 55: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/55.jpg)
IMAGE TO IMAGE TRANSLATION
CycleGAN
1
![Page 56: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/56.jpg)
IMAGE TO IMAGE TRANSLATION
CycleGAN
1
1
![Page 57: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/57.jpg)
IMAGE TO IMAGE TRANSLATION
CycleGAN
1
1
![Page 58: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/58.jpg)
Explicit loss function:
IMAGE TO IMAGE TRANSLATION
CycleGAN
![Page 59: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/59.jpg)
IMAGE TO IMAGE TRANSLATION
CycleGAN
Generator architecture:
![Page 60: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/60.jpg)
Discriminator architecture
IMAGE TO IMAGE TRANSLATION
CycleGAN
![Page 61: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/61.jpg)
Results:
IMAGE TO IMAGE TRANSLATION
CycleGAN
![Page 62: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/62.jpg)
Results: video
IMAGE TO IMAGE TRANSLATION
CycleGAN
![Page 63: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/63.jpg)
Results:
IMAGE TO IMAGNE TRANSLATION
UNIT
𝑛𝑖𝑔ℎ𝑡 ↔ 𝑑𝑎𝑦
𝑟𝑎𝑖𝑛 ↔ sunshine
![Page 64: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/64.jpg)
Results:
IMAGE TO IMAGNE TRANSLATION
UNIT
![Page 65: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/65.jpg)
Results:
IMAGE TO IMAGNE TRANSLATION
UNIT
![Page 66: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/66.jpg)
SOME MORE RESULTS CycleGAN
![Page 67: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/67.jpg)
REFERENCES
Ian Goodfellow. “NIPS 2016 Tutorial: Generative
Adversarial Networks” (2016)
Jun-Yan Zhu, Tasung Park. “Unpaired Image-to-Image
Translation using Cycle-Consistent Adversarial Networks”
(2017)
Ming-Yu Liu, Thomas Breuel. “Unsupervised Image-to-
Image Translation Networks” (2017)
Ian Goodfellow. “Generative adversarial nets” (2014)
Christian Ledig. “Photo-Realistic Single Image Super-
Resolution Using a Generative Adversarial Network“
(2017)
![Page 68: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/68.jpg)
Thanks for listening
![Page 69: Generative Adversarial Networks (GAN)web.eng.tau.ac.il/deep_learn/wp-content/uploads/...Translation using Cycle-Consistent Adversarial Networks” (2017) Ming-Yu Liu, Thomas Breuel.](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed401cc8d46b66d22634336/html5/thumbnails/69.jpg)
Labels improve subjective sample quality: (Denton et al 2015)
Learning a conditional model p(y|x) often gives much better
samples from all classes than learning p(x) does.
One-sided label smoothing: (Salimans et al 2016)
D_cost = cross_entropy(1., D(data)) + cross_entropy(0., D(samples))
D*_cost = cross_entropy(.9, D(data)) + cross_entropy(0., D(samples))
Prevent extreme extrapolation behavior in the discriminator.
Excellent regularizer that doesn’t encourage misclassification.
TIPS AND TRICKS