SIMPLe : Simple Idea Meaningful Performance Level up*

ISL Lab Seminar

Hansol Kang

* Mao, Xudong, et al. "Least squares generative adversarial networks." Proceedings of the IEEE International Conference on Computer Vision. 2017.

Contents

2019-04-09

2

Introduction

Research Trend

Review

LSGAN

Concept

Objective Function

Global Optimality

Parameter Selection

Results Experiment

Source Code

Celeb-A(DCGAN, LSGAN)Summary

I. Introduction

Introduction

Research Trend, Review(Concept, Vanilla GAN, DCGAN, InfoGAN)

Introduction• Research Trend

2019-04-09

4

Introduction• Research Trend

2019-04-09

5

• 70,000 high-quality PNG images at 1024×1024 resolution

• Considerable variation in terms of age, ethnicity and image background

• Good coverage of accessories such as eyeglasses, sunglasses, hats, etc.

Flickr-Faces-HQ (FFHQ)

Introduction• Research Trend

2019-04-09

6

Introduction• Concept of GAN

2019-04-09

7

VsDGF1

F1

F1

F1

FakeR1

Introduction• Concept of GAN

2019-04-09

8

VsDG

Fake?R1

@

F1

Introduction• Vanilla GAN : Adversarial Nets

2019-04-09

9

)))]((1[log()]([log),(maxmin )(~)(~ zGDExDEGDV zpzxpxDG zdata

Smart D

)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdataReal case

Fake case

1

)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata

0

should be 0

should be 0

1

Log(x)cf.

Stupid D

)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdataReal case

Fake case

0

)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata

1

should be negative infinity

should be negative infinity

D perspective,

it should be maximum.

Introduction• Vanilla GAN : Adversarial Nets

2019-04-09

10

)))]((1[log()]([log),(maxmin )(~)(~ zGDExDEGDV zpzxpxDG zdata

Generator

)))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata

1

should be negative infinity

1

Log(x)cf.

G perspective,

it should be minimum.

Smart G

Stupid G )))]((1[log()]([log )(~)(~ zGDExDE zpzxpx zdata

0

should be 0

Introduction

2019-04-09

11

• Vanilla GAN : Mathematical Proof

1) Global Optimality of datag pp

2) Convergence of Algorithm

D GVs

x

)(xpdata

“Generative Adversarial Networks”

Goal Method

Introduction

2019-04-09

12

• DCGAN : Network

DG

“쟤들 뭐하냐?”

“CNN이 MLP보다 훨씬 낫지롱”D

“우리가 짱이야”

GVanilla GAN DCGAN

“VAE 죽어요 ㅠㅠ”

Introduction

2019-04-09

13

• DCGAN : Latent Space

0 1

0.1

0.15

0.18

0.143

0.5

0.45

0.47

0.473

0.9

0.95

0.96

0.937

0.607±

고차원(Image)에서의미 x

저차원(Latent code)에서의미 o

Introduction

2019-04-09

14

• InfoGAN - Network

D“우리가 짱이야”

G D

DCGAN InfoGAN

Gz “나도 신경 써줘…”

zc

z“우리는 Z(Latent code)를 더 세분화해서 조작이 가능해!”

Introduction

2019-04-09

15

• InfoGAN - Latent Code

GNoise

Latent code

0.001

0.008

1.000

0.007

…

0.005

? : 실제 latent code의 구조는 복잡하여해석이어려움(entangled).

Let's make the latent code simple.

The proper generation is difficult.

[0.001, 0.008, …, 0.005] c

Latent code

0.001

0.008

1.000

0.007

…

0.005

0

0

0

0

…

1

Z C

Z C : Condition

How about adding latent code?

해석이 가능한 Condition을 제공.

Idea

Introduction

2019-04-09

16

• InfoGAN - Latent Code

G

Latent code

Z C

“뭐야? 그러면 C를 Z 옆에 바로붙이면 되는 거야?”

[0.001, 0.008, …, 005 | 0, 0, … 1]

z c

[0.001, 0.008, …, 005 | 1, 0, … 0]

z c

[0.001, 0.008, …, 005 ]

z

[0.001, 0.008, …, 005 ]

z

Ignore the additional latent code c

Cost function을 수정하여 c의 영향을 만듦.),(maxmin GDVDG

(Mutual Information)

Introduction

2019-04-09

17

• InfoGAN - Latent Code

: Generator와 c 사이의연관성을 cost로정의 ),(;),(),(maxmin czGcIGDVGDVIDG

Maximize

Hard to maximize directly as it requires access to the posterior )|( xcP

)(||)|()(|log),,( )|( zpxzqKLzgxExL xzq

),,(min xL

Reconstruction Error Regularization

VAE Seminar (18.07.23)

Introduction

2019-04-09

18

• InfoGAN - Results

Introduction

2019-04-09

19

GAN

Concept

Performance

Manipulability

Stability

Concept

• GAN

Performance

• DCGAN, LSGAN

Stability

• DCGAN, LSGAN

Manipulability

• DCGAN, InfoGAN

II. LSGAN

LSGAN

Concept, Objective Function, Global Optimality, Parameter Selection, Results

LSGAN

2019-04-09

21

• Concept

D학습이 잘되었다(=50:50)

학습이 잘되었다(=Good representation)

GReal

Fake

F5

R1R2

R5

F1

F5

R1

F1

여전히너무나도가짜같은데이터가존재

If 60:40 then stupid G

If 40:60 then stupid D

LSGAN

2019-04-09

22

• Concept

GReal

Fake

F5

R1R2

R5

F1

F5

R1

F1

경계근처 ≈ Real? Fake?

Fake -> 경계근처

F5

F1

훨씬헷갈리는(Real에가까운)

데이터생성

LSGAN

2019-04-09

23

• Concept

LSGAN• Objective function

2019-04-09

24

2

)(~

2

~ ))((2

1)(

2

1)(min

)(azGDEbxDEDV zpxpxLSGAN

D zxdata

Smart D

2

)(~

2

~ ))((2

1)(

2

1)(

azGDEbxDE zpxpx zxdata

2

)(~

2

~ ))((2

1)(

2

1)(

azGDEbxDE zpxpx zxdataReal case

Fake case

1

0

(b=1), should be 0

(a=0), should be 0

Stupid D

D perspective,

it should be minimum. 2

)(~

2

~ ))((2

1)(

2

1)(

azGDEbxDE zpxpx zxdata

2

)(~

2

~ ))((2

1)(

2

1)(

azGDEbxDE zpxpx zxdataReal case

Fake case

0 1

(b=1), should be 1

(a=0), should be 1

a : fake label.

b : real label.

c : G wants to make D believe for fake data

2

~ ))((2

1)(min

)(czGDEGV

zzpzLSGANG

LSGAN• Objective function

2019-04-09

25

2

)(~

2

~ ))((2

1)(

2

1)(min

)(azGDEbxDEDV zpxpxLSGAN

D zxdata

Generator

2

~ ))((2

1)(

czGDEzzpz Smart G

Stupid G

1

(c=1), should be 0

(c=1), should be 1 2

~ ))((2

1)(

czGDEzzpz

0

G perspective,

it should be minimum.

2

~ ))((2

1)(min

)(czGDEGV

zzpzLSGANG

a : fake label.

b : real label.

c : G wants to make D believe for fake data

LSGAN• Objective function

2019-04-09

26

a : fake label.

b : real label.

c : G wants to make D believe for fake data

조금더직관적으로생각해보면,

D = Classifier

2

)(~

2

~ ))((2

1)(

2

1)(min

)(azGDEbxDEDV zpxpxLSGAN

D zxdata 2

~ ))((2

1)(min

)(czGDEGV

zzpzLSGANG

Prediction - Label

LSGAN

2019-04-09

27

• Global Optimality – Vanilla GAN

LSGAN : 𝝌𝑷𝒆𝒂𝒓𝒔𝒐𝒏𝟐 를 통한 증명

Vanilla GAN : JSD를 통한 증명

LSGAN

2019-04-09

28

• Global Optimality – LSGAN

2*

~

2*

~ )()()(2 cxDEcxDEGCgd pxpx

2

)(~

2

~ ))((2

1)(

2

1)(min

)(azGDEbxDEDV zpxpxLSGAN

D zxdata

2

~ ))((2

1)(min

)(czGDEGV

zzpzLSGANG

)()(

)()()(*

xpxp

xapxbpxD

gdata

gdata

2

~

2

~)()(

)()(

)()(

)()(c

xpxp

xapxbpEc

xpxp

xapxbpE

gdata

gdata

px

gdata

gdata

px gd

2

~ )(2

1)(

cxDExdatapx This term does not contain parameters of G

LSGAN

2019-04-09

29

• Global Optimality – LSGAN

22

)()(

)()()()()(

)()(

)()()()()(

xpxp

xpcaxpcbxpdx

xpxp

xpcaxpcbxp

gdata

gdata

x

g

gdata

gdata

x

data

2

~

2

~)()(

)()(

)()(

)()(c

xpxp

xapxbpEc

xpxp

xapxbpE

gdata

gdata

px

gdata

gdata

px gd

gdata

gdata

gdata

gdata

gdata

gdata

pp

xpcapcb

xpxp

xcpxcp

xpxp

xapxbp

)()()(

)()(

)()(

)()(

)()(

x gdata

gdatadx

xpxp

xpcaxpcb

)()(

)()()()(2

x gdata

gdata

gdata dxxpxp

xpcaxpcbxpxp

2

)()(

)()()()()()(

x gdata

ggdatadx

xpxp

xpabxpxpcb

)()(

)()()()()(2

LSGAN

2019-04-09

30

• Global Optimality – LSGAN

x gdata

ggdatadx

xpxp

xpabxpxpcb

)()(

)()()()()(2

dx

xpxp

xpxpxpGC

x gdata

gdatag

)()(

))()(()(2)(2

2

)2||(2

ggdataPearson ppp

datag pp

)(

)()(2

2

xp

xpxqPearson

21 abandcbIf we set

If minimum

LSGAN

2019-04-09

31

• Parameters Selection

21 abandcb

01,1 candba

2

)(~

2

~ ))((2

1)(

2

1)(min

)(azGDEbxDEDV zpxpxLSGAN

D zxdata 2

~ ))((2

1)(min

)(czGDEGV

zzpzLSGANG

2

)(~

2

~ 1))((2

11)(

2

1)(min

)( zGDExDEDV zpxpxLSGAN

D zxdata

i)

2

~ ))((2

1)(min

)(zGDEGV

zzpzLSGANG

bcandba 1,0

2

)(~

2

~ 1))((2

11)(

2

1)(min

)( zGDExDEDV zpxpxLSGAN

D zxdata

ii)

2

~ ))((2

1)(min

)(zGDEGV

zzpzLSGANG

->조건을따르지않는경우

성능은비슷하며, 큰차이가없음!

LSGAN

2019-04-09

32

• Results

LSGAN

2019-04-09

33

• Results

LSGAN

2019-04-09

34

• Results

Example of mode collapse

LSGAN

2019-04-09

35

• Results

III. Experiment

Experiment

Source Code, Celeb-A, Korean Idol

Experiment• Source Code

2019-04-09

37

https://github.com/messy-snail/GAN_PyTorch

Experiment• Celeb-A

2019-04-09

38

DCGAN LSGAN

ep=1

ep=2

ep=10

ep=16

Experiment• Celeb-A(without BN)

2019-04-09

39

DCGAN LSGAN

ep=1

ep=2

ep=3

ep=5

Experiment

2019-04-09

40

• Korean Idol(DCGAN)

ep 1 ep 6

ep 51ep 21

ep 201ep 101

Experiment

2019-04-09

41

• Korean Idol(LSGAN)

ep 1 ep 6

ep 51ep 21

ep 201ep 101

Experiment

2019-04-09

42

• Gueess#1 Optimizer problem?

Experiment

2019-04-09

43

• Gueess#2 Domain problem?

Experiment• Celeb-A

2019-04-09

44

https://www.slideshare.net/NaverEngineering/1-gangenerative-adversarial-network

“저자들이 source code를공개했으면한다.”

“같은구조라면, LSGAN이훨씬잘동작한다.”

IV. Summary

Summary

Summary, Future Work

Summary

2019-04-09

46

• 기존의 GAN보다 Real에 가까운 데이터를 생성하고, 안정성도 확보함.

• Pearson Chi square divergence으로 global optimality를 증명함. (기존 GAN은 JSD로 증명)

• 클래스가 많은 데이터에 대해서도 정상적으로 데이터를 생성함.

• 기존의 코드에서 단순히 loss만을 변경하기에 손쉽게 적용이 가능함.

GAN Research

Vanilla GAN

DCGAN

InfoGAN

LSGAN

BEGAN

Cycle GAN

Style GAN

SRGAN

Tools

Document

Programming

PyTorch

Python executable & UI

I Know What You Did Last Faculty

C++ Coding Standard

Mathematical theory

LSM applications

Other Research

Level Processor

Ice Propagation

Future work

2019-04-09

47

&