Automatic Image Colorization - University of Wisconsin...
Transcript of Automatic Image Colorization - University of Wisconsin...
![Page 1: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/1.jpg)
Automatic Image ColorizationSrinivas Tunuguntla, Adithya Bhat
![Page 2: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/2.jpg)
Problem Definition
![Page 3: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/3.jpg)
Previous Work ● User assisted
● Fully automated
![Page 4: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/4.jpg)
User assisted Scribble based
● User provides scribbles that give
reference colors.
● The system extends the colors to
the rest of the image.
● However, the colors have to be
given as input, they are not learnt.
Photo: Levin et. al (2004)
![Page 5: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/5.jpg)
User assisted Reference based
● User provides reference image with
similar features.
● The system transfers colors to the
given image.
● However, providing/selecting
suitable reference images is vital to
this method.
Photos: Welsh et. al (2002)
![Page 6: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/6.jpg)
Fully AutomatedFeature Engineering
● Image features created via pre-processing.
● Feature engineering plays a major role.
Neural Networks
● Convolutional Neural Networks are the current state of the art.
● Features are automatically learned.
![Page 7: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/7.jpg)
SetupInfrastructure
● Google Cloud
● Nvidia Tesla K80 GPU
Framework
● Keras
Training Data
● Imagenet
○ Geological formation synset
![Page 8: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/8.jpg)
Design
![Page 9: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/9.jpg)
Lab Color Space● Euclidean distance in the a-b space corresponds to visually perceived differences
● L is the lightness channel
○ 0 - black, 100 - white
● a, b are color components
○ Range from -128 to +127 (or scaled to -100 to +100)
Our task : predict the channels a, b using input L
![Page 10: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/10.jpg)
Input
Grayscale
Conv 1 Conv 2 Conv 3 Conv 4
Probabilities of
each of 484 a-b
bins
Conv 6Conv 5 Conv 7 Conv 8
Soft
max
Convolution
ReLu
Activation
Conv Unit
Batch
Normalization
Conv Unit
Conv Unit
Conv Layer
64
(112, 112)
128
(56, 56)
256
(28, 28)
512
(28, 28)
512
(28, 28)
512
(28, 28)
512
(28, 28)
484
(14, 14)(224, 224)
Network architecture
![Page 11: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/11.jpg)
Loss Function MSE
● Not robust to inherent ambiguity
● Predicts mean of possible
colorizations
● Results in desaturated color
predictions
![Page 12: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/12.jpg)
Loss Function Multinomial classification
● Divide a, b space into K bins
● Predict probability distribution over
these bins
![Page 13: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/13.jpg)
Problem : Desaturated output
![Page 14: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/14.jpg)
Solution :Color Rebalancing
● Distribution of a,b values strongly
biased towards low (desaturated)
values
● Reweight each pixel loss based on
pixel color rarity
● Integrated into the loss function
![Page 15: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/15.jpg)
Desaturated output
![Page 16: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/16.jpg)
Color rebalanced
![Page 17: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/17.jpg)
Problem : Color patches
![Page 18: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/18.jpg)
Solution :Segmentation
● Intuition: Neighboring pixels that
have similar intensities tend to
have similar colors
● Introduce the notion of spatial
locality into the loss function
![Page 19: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/19.jpg)
Color patches
![Page 20: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/20.jpg)
Segmented
![Page 21: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/21.jpg)
Results
![Page 22: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/22.jpg)
![Page 23: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/23.jpg)
![Page 24: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/24.jpg)
![Page 25: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/25.jpg)
Failure cases
![Page 26: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/26.jpg)
Future WorkTune the trade-off of vibrancy vs. realism in color re-balancing weights.
Explicitly control the colors used in a picture
Loss function - ambiguity invariant
Visualize network’s intermediate layers
Applications to other underconstrained problems? E.g. Depth of field etc.
![Page 27: Automatic Image Colorization - University of Wisconsin ...pages.cs.wisc.edu/~stunuguntla/ImageColorization_ppt.pdf · References [1] Zhang, Richard, Phillip Isola, and Alexei A. Efros.](https://reader034.fdocuments.in/reader034/viewer/2022042219/5ec5aab6bd278d405c141e6d/html5/thumbnails/27.jpg)
References[1] Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization."
European Conference on Computer Vision. Springer International Publishing, 2016.
[2] Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic
colorization. European Conference on Computer Vision (2016)
[3] Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be Color!: Joint End-to-end
Learning of Global and Local Image Priors for Automatic Image Colorization with
Simultaneous Classification. ACM transactions on Graphics (Proc. of SIGGRAPH
2016) 35(4) (2016)