By Hatice Oncel TOEFL VOCAB-1. v. End, eliminate I think bullfighting should be abolished abolish.
Unsupervised Conditional Generation -...
Transcript of Unsupervised Conditional Generation -...
![Page 1: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/1.jpg)
Unsupervised Conditional Generation
![Page 2: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/2.jpg)
Domain X Domain Y
male female
It is good.
It’s a good day.
I love you.
It is bad.
It’s a bad day.
I don’t love you.
Unsupervised Conditional Generation
Transform an object from one domain to another without paired data (e.g. style transfer)
GDomain X Domain Y
![Page 3: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/3.jpg)
Unsupervised Conditional Generation• Approach 1: Direct Transformation
• Approach 2: Projection to Common Space
?𝐺𝑋→𝑌
Domain X Domain Y
For texture or color change
𝐸𝑁𝑋 𝐷𝐸𝑌
Encoder of domain X
Decoder of domain Y
Larger change, only keep the semantics
Domain YDomain X FaceAttribute
![Page 4: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/4.jpg)
?
Direct Transformation
𝐺𝑋→𝑌
Domain X
Domain Y
𝐷𝑌
Domain Y
Domain X
scalar
Input image belongs to domain Y or not
Become similar to domain Y
![Page 5: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/5.jpg)
Direct Transformation
𝐺𝑋→𝑌
Domain X
Domain Y
𝐷𝑌
Domain Y
Domain X
scalar
Input image belongs to domain Y or not
Become similar to domain Y
Not what we want!
ignore input
![Page 6: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/6.jpg)
Direct Transformation
𝐺𝑋→𝑌
Domain X
Domain Y
𝐷𝑌
Domain X
scalar
Input image belongs to domain Y or not
Become similar to domain Y
Not what we want!
ignore input
[Tomer Galanti, et al. ICLR, 2018]
The issue can be avoided by network design.
Simpler generator makes the input and output more closely related.
![Page 7: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/7.jpg)
Direct Transformation
𝐺𝑋→𝑌
Domain X
Domain Y
𝐷𝑌
Domain X
scalar
Input image belongs to domain Y or not
Become similar to domain Y
EncoderNetwork
EncoderNetwork
pre-trained
as close as possible
Baseline of DTN [Yaniv Taigman, et al., ICLR, 2017]
![Page 8: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/8.jpg)
Direct Transformation
𝐺𝑋→𝑌
𝐷𝑌
Domain Y
scalar
Input image belongs to domain Y or not
𝐺Y→X
as close as possible
Lack of information for reconstruction
[Jun-Yan Zhu, et al., ICCV, 2017]
Cycle consistency
![Page 9: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/9.jpg)
Direct Transformation
𝐺𝑋→𝑌 𝐺Y→X
as close as possible
𝐺Y→X 𝐺𝑋→𝑌
as close as possible
𝐷𝑌𝐷𝑋scalar: belongs to domain Y or not
scalar: belongs to domain X or not
![Page 10: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/10.jpg)
Cycle GAN –Silver Hair
• https://github.com/Aixile/chainer-cyclegan
![Page 11: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/11.jpg)
Cycle GAN –Silver Hair
• https://github.com/Aixile/chainer-cyclegan
![Page 12: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/12.jpg)
Issue of Cycle Consistency
• CycleGAN: a Master of Steganography (隱寫術)[Casey Chu, et al., NIPS workshop, 2017]
𝐺Y→X𝐺𝑋→𝑌
The information is hidden.
![Page 13: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/13.jpg)
Cycle GAN
Dual GAN
Disco GAN
[Jun-Yan Zhu, et al., ICCV, 2017]
[Zili Yi, et al., ICCV, 2017]
[Taeksoo Kim, et
al., ICML, 2017]
![Page 14: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/14.jpg)
StarGANFor multiple domains, considering starGAN
[Yunjey Choi, arXiv, 2017]
![Page 15: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/15.jpg)
StarGAN
![Page 16: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/16.jpg)
StarGAN
![Page 17: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/17.jpg)
StarGAN
![Page 18: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/18.jpg)
Unsupervised Conditional Generation• Approach 1: Direct Transformation
• Approach 2: Projection to Common Space
?𝐺𝑋→𝑌
Domain X Domain Y
For texture or color change
𝐸𝑁𝑋 𝐷𝐸𝑌
Encoder of domain X
Decoder of domain Y
Larger change, only keep the semantics
Domain YDomain X FaceAttribute
![Page 19: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/19.jpg)
Domain X Domain Y
𝐸𝑁𝑋
𝐸𝑁𝑌 𝐷𝐸𝑌
𝐷𝐸𝑋image
image
image
imageFaceAttribute
Projection to Common Space
Target
![Page 20: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/20.jpg)
Domain X Domain Y
𝐸𝑁𝑋
𝐸𝑁𝑌 𝐷𝐸𝑌
𝐷𝐸𝑋image
image
image
image
Minimizing reconstruction error
Projection to Common SpaceTraining
![Page 21: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/21.jpg)
𝐸𝑁𝑋
𝐸𝑁𝑌 𝐷𝐸𝑌
𝐷𝐸𝑋image
image
image
image
Minimizing reconstruction error
Because we train two auto-encoders separately …
The images with the same attribute may not project to the same position in the latent space.
𝐷𝑋
𝐷𝑌
Discriminator of X domain
Discriminator of Y domain
Minimizing reconstruction error
Projection to Common SpaceTraining
![Page 22: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/22.jpg)
Sharing the parameters of encoders and decoders
Projection to Common SpaceTraining
𝐸𝑁𝑋
𝐸𝑁𝑌
𝐷𝐸𝑋
𝐷𝐸𝑌
Couple GAN[Ming-Yu Liu, et al., NIPS, 2016]
UNIT[Ming-Yu Liu, et al., NIPS, 2017]
![Page 23: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/23.jpg)
𝐸𝑁𝑋
𝐸𝑁𝑌 𝐷𝐸𝑌
𝐷𝐸𝑋image
image
image
image
Minimizing reconstruction error
The domain discriminator forces the output of 𝐸𝑁𝑋 and 𝐸𝑁𝑌 have the same distribution.
From 𝐸𝑁𝑋 or 𝐸𝑁𝑌
𝐷𝑋
𝐷𝑌
Discriminator of X domain
Discriminator of Y domain
Projection to Common SpaceTraining
DomainDiscriminator
𝐸𝑁𝑋 and 𝐸𝑁𝑌 fool the domain discriminator
[Guillaume Lample, et al., NIPS, 2017]
![Page 24: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/24.jpg)
𝐸𝑁𝑋
𝐸𝑁𝑌 𝐷𝐸𝑌
𝐷𝐸𝑋image
image
image
image
𝐷𝑋
𝐷𝑌
Discriminator of X domain
Discriminator of Y domain
Projection to Common SpaceTraining
Cycle Consistency:
Used in ComboGAN [Asha Anoosheh, et al., arXiv, 017]
Minimizing reconstruction error
![Page 25: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/25.jpg)
𝐸𝑁𝑋
𝐸𝑁𝑌 𝐷𝐸𝑌
𝐷𝐸𝑋image
image
image
image
𝐷𝑋
𝐷𝑌
Discriminator of X domain
Discriminator of Y domain
Projection to Common SpaceTraining
Semantic Consistency:
Used in DTN [Yaniv Taigman, et al., ICLR, 2017] and XGAN [Amélie Royer, et al., arXiv, 2017]
To the same latent space
![Page 26: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/26.jpg)
世界二次元化
• Using the code: https://github.com/Hi-king/kawaii_creator
• It is not cycle GAN, Disco GAN
input output domain
![Page 27: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/27.jpg)
Voice Conversion
![Page 28: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/28.jpg)
In the past
Today
Speaker A Speaker B
How are you? How are you?
Good morning Good morning
Speaker A Speaker B
天氣真好 How are you?
再見囉 Good morning
Speakers A and B are talking about completely different things.
![Page 29: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/29.jpg)
Speaker A Speaker B
我
感謝周儒杰同學提供實驗結果
![Page 30: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/30.jpg)
Reference
• Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV, 2017
• Zili Yi, Hao Zhang, Ping Tan, Minglun Gong, DualGAN: Unsupervised Dual Learning for Image-to-Image Translation, ICCV, 2017
• Tomer Galanti, Lior Wolf, Sagie Benaim, The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings, ICLR, 2018
• Yaniv Taigman, Adam Polyak, Lior Wolf, Unsupervised Cross-Domain Image Generation, ICLR, 2017
• Asha Anoosheh, Eirikur Agustsson, Radu Timofte, Luc Van Gool, ComboGAN: Unrestrained Scalability for Image Domain Translation, arXiv, 2017
• Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, Kevin Murphy, XGAN: Unsupervised Image-to-Image Translation for Many-to-Many Mappings, arXiv, 2017
![Page 31: Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial](https://reader034.fdocuments.in/reader034/viewer/2022051912/60029d517ad00075d315e728/html5/thumbnails/31.jpg)
Reference
• Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, LudovicDenoyer, Marc'Aurelio Ranzato, Fader Networks: Manipulating Images by Sliding Attributes, NIPS, 2017
• Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, Jiwon Kim, Learning to Discover Cross-Domain Relations with Generative Adversarial Networks, ICML, 2017
• Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial Networks”, NIPS, 2016
• Ming-Yu Liu, Thomas Breuel, Jan Kautz, Unsupervised Image-to-Image Translation Networks, NIPS, 2017
• Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, JaegulChoo, StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation, arXiv, 2017