Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation...
Transcript of Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation...
![Page 1: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/1.jpg)
Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. EfrosBerkeley AI Research (BAIR) Laboratory, UC Berkeley
Presenters: Terry Yang and Mingyang ZhouECS 289G: Visual Recognition
![Page 2: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/2.jpg)
Agenda
● Motivating the problem● Introduction to GANs● Method● Dataset & tasks● Evaluation Metrics● Results● Strength & Weakness● Future Extension
![Page 3: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/3.jpg)
Introduction: Image-to-Image Translation
Isola, Phillip, et al. "Image-To-Image Translation With Conditional Adversarial Networks." CVPR, 2017.
![Page 4: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/4.jpg)
Previous Methods
A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. CVPR, 2015.
R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. ECCV, 2016.
T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu.Sketch2photo: internet image montage. ACM, 2009
Required loss functions and architectures designed specifically for the task at hand.
![Page 5: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/5.jpg)
MotivationGoal: A general-purpose solution to image-to-image translation problems.
Garcia, Victor. "Image-to-Image Translation with Conditional Adversarial Networks." 25 Nov 2016. UPC Computer Vision Reading Group, Universitat Politècnica de Catalunya, Microsoft Powerpoint presentation
This paper: Use the same architecture and objective for each image-to-image translation task.
![Page 7: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/7.jpg)
Generative Adversarial Networks (GANs)
Garcia, Victor. "Image-to-Image Translation with Conditional Adversarial Networks." 25 Nov 2016. UPC Computer Vision Reading Group, Universitat Politècnica de Catalunya, Microsoft Powerpoint presentation
![Page 8: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/8.jpg)
Conditional Generative Adversarial Networks (cGANs)
Garcia, Victor. "Image-to-Image Translation with Conditional Adversarial Networks." 25 Nov 2016. UPC Computer Vision Reading Group, Universitat Politècnica de Catalunya, Microsoft Powerpoint presentation
![Page 9: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/9.jpg)
Objective
Isola, Phillip, et al. "Image-To-Image Translation With Conditional Adversarial Networks." CVPR, 2017.
![Page 10: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/10.jpg)
Generator
Garcia, Victor. "Image-to-Image Translation with Conditional Adversarial Networks." 25 Nov 2016. UPC Computer Vision Reading Group, Universitat Politècnica de Catalunya, Microsoft Powerpoint presentation
U-NetEncoder-Decoder Network
![Page 11: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/11.jpg)
Discriminator
Garcia, Victor. "Image-to-Image Translation with Conditional Adversarial Networks." 25 Nov 2016. UPC Computer Vision Reading Group, Universitat Politècnica de Catalunya, Microsoft Powerpoint presentation
PatchGAN
![Page 12: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/12.jpg)
Discriminator
Garcia, Victor. "Image-to-Image Translation with Conditional Adversarial Networks." 25 Nov 2016. UPC Computer Vision Reading Group, Universitat Politècnica de Catalunya, Microsoft Powerpoint presentation
![Page 13: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/13.jpg)
Optimization and Inference
● Alternate between one gradient descent step on D and one gradient descent step on G
● Minibatch SGD and Adam optimization● At inference time, the generator net is run in exactly
the same manner as during training phase
Isola, Phillip, et al. "Image-To-Image Translation With Conditional Adversarial Networks." CVPR, 2017.
![Page 14: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/14.jpg)
Datasets & Tasks
CityScape
CMP Facades Map to Aerial
![Page 15: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/15.jpg)
Datasets & Tasks
Edge to Photo Sketch to Photo Day to Light
![Page 16: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/16.jpg)
Evaluation Metrics
AMT Perceptual Studies
Left or Right?
1 2
![Page 17: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/17.jpg)
Evaluation Metrics
FCN-score
![Page 18: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/18.jpg)
Experiment ResultsAnalysis of the Objective Function
![Page 19: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/19.jpg)
Experiment ResultsAnalysis of generator architecture
![Page 20: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/20.jpg)
Experiment ResultsReceptive Field of Discriminator
![Page 21: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/21.jpg)
Experiment ResultsPerceptual Validation
![Page 22: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/22.jpg)
Experiment ResultsRich Community Driven Results
Edges2Cats Background Masking
![Page 23: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/23.jpg)
Strength & WeaknessesStrength:
1. An easily generalizable structure to conduct image-to-image translation tasks.
2. A good combination of existing techniques and well-designed loss function to ensure the generation of high-quality synthesized image.
Weakness:
1. Required large number of 1-to-1 paired images for training, which are expensive to collect.
2. Does not provide decent comparisons between this network with the other task-specific model to defend its claim for good generalizability.
![Page 24: Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aa80bd278d405c141dcb/html5/thumbnails/24.jpg)
Future Extension
1. Modify the structure such that it can learn the translation mapping with unpaired images in two different domains.
2. Further improvement to generate photorealistic image.