Learning to Color from Language

Varun ManjunathaF♠© Mohit IyyerF†‡ Jordan Boyd-Graber♠♣©♦ Larry Davis♠©

University of Maryland : Computer Science♠, Language Science♦, iSchool♣, UMIACS©

Allen Institute of Artificial Intelligence† University of Massachusetts, Amherst‡

{varunm@cs, jbg@umiacs, lsd@umiacs}.umd.edu miyyer@cs.umass.edu

Abstract

Automatic colorization is the process ofadding color to greyscale images. We condi-tion this process on language, allowing endusers to manipulate a colorized image by feed-ing in different captions. We present two dif-ferent architectures for language-conditionedcolorization, both of which produce moreaccurate and plausible colorizations than alanguage-agnostic version. Through thislanguage-based framework, we can dramati-cally alter colorizations by manipulating de-scriptive color words in captions.

1 Introduction

Automatic image colorization (Cheng et al., 2015;Larsson et al., 2016; Zhang et al., 2016; Iizukaet al., 2016; Deshpande et al., 2017)—the processof adding color to a greyscale image—is inherentlyunderspecified. Unlike background scenery suchas sky or grass, many common foreground objectscould plausibly be of any color, such as a person’sclothing, a bird’s feathers, or the exterior of a car.Interactive colorization seeks human input, usuallyin the form of clicks or strokes on the image with aselected color, to reduce these ambiguities (Levinet al., 2004; Huang et al., 2005; Endo et al., 2016;Zhang et al., 2017). We introduce the task of col-orization from natural language, a previously unex-plored source of color specifications.

Many use cases for automatic colorization in-volve images paired with language. For example,comic book artwork is normally first sketched inblack-and-white by a penciller; afterwards, a col-orist selects a palette that thematically reinforcesthe written script to produce the final colorized art.Similarly, older black-and-white films are often col-orized for modern audiences based on cues fromdialogue and narration (Van Camp, 1995).

FAuthors contributed equally

Language is a weaker source of supervision forcolorization than user clicks. In particular, lan-guage lacks ground-truth information about thecolored image (e.g., the exact color of a pixel orregion). Given a description like a blue motorcy-cle parked next to a fleet of sedans, an automaticcolorization system must first localize the motorcy-cle within the image before deciding on a context-appropriate shade of blue to color it with. The chal-lenge grows with abstract language: a red colorpalette likely suits an artistic rendering of the boythrew down his toy in a rage better than it does theboy lovingly hugged his toy.

We present two neural architectures forlanguage-based colorization that augment an exist-ing fully-convolutional model (Zhang et al., 2016)with representations learned from image captions.As a sanity check, both architectures outperform alanguage-agnostic model on an accuracy-based col-orization metric. However, we are more interestedin whether modifications to the caption properlymanifest themselves in output colorizations (e.g.,switching one color with another); crowdsourcedevaluations confirm that our models properly local-ize and color objects based on captions (Figure 1).

2 Model

This section provides a quick introduction to colorspaces (Sec. 2.1) and then describes our baselinecolorization network (Sec. 2.2) alongside two mod-els (Sec. 2.3) that colorize their output on represen-tations learned from language.

2.1 Images and color spaces

An image is usually represented as a three dimen-sional tensor with red, green and blue (RGB) chan-nels. Each pixel’s color and intensity (i.e., light-ness) are jointly represented by the values of thesethree channels. However, in applications such as

arX

iv:1

804.

0602

6v1

[cs

.CV

] 1

7 A

pr 2

018

Figure 1: Three pairs of images whose colorizations are conditioned on corresponding captions by ourFILM architecture. Our model can localize objects mentioned by the captions and properly color them.

colorization, it is more convenient to use represen-tations that separately encode lightness and color.These color spaces can be obtained through mathe-matical transformations of the RGB color space; inthis work, following Zhang et al. (2016), we usethe CIE Lab space (Smith and Guild, 1931). Here,the first channel (L) encodes only lightness (i.e.,black-and-white). The two color channels a andb represent color values between green to red andblue to yellow, respectively. In this formulation, thetask of colorization is equivalent to taking the light-ness channel of an image as input and predictingthe two missing color channels.

2.2 Fully-convolutional networks forcolorization

Following Zhang et al. (2016), we treat coloriza-tion as a classification problem in CIE Lab space:given only the lightness channel L of an image (i.e.,a greyscale version), a fully-convolutional networkpredicts values for the two color channels a and b.For efficiency, we deviate from Zhang et al. (2016)by quantizing the color channels into a 25×25 grid,which results in 625 labels for classification. Tofurther speed up training, we use a one-hot encod-ing for the ab channels instead of soft targets asin Zhang et al. (2016); preliminary experimentsshowed no qualitative difference in colorizationquality with one-hot targets. The contribution ofeach label to the loss is downweighted by a factorinversely proportional to its frequency in the train-ing set, which prevents desaturated ab values. Ourbaseline network architecture (FCNN) consists ofeight convolutional blocks, each of which containsmultiple convolutional layers followed by batchnormalization (Ioffe and Szegedy, 2015).1 Next,we propose two ways to integrate additional text

1See Zhang et al. (2016) for complete architectural de-tails. Code and pretrained models are available at https://github.com/superhans/colorfromlanguage.

ab Accuracy Human Experiments

Model acc@1 acc@5 plaus. qual. manip.

FCNN 15.4 45.8 20.4 32.6 N/ACONCAT 17.9 50.3 39.0 34.1 77.4

FILM 23.7 60.5 40.6 32.1 81.2

Table 1: While FILM is the most accurate modelin ab space, its outputs are about as contextuallyplausible as CONCAT’s according to our plausi-bility task, which asks workers to choose whichmodel’s output best depicts a given caption (how-ever, both models significantly outperform thelanguage-agnostic FCNN). This additional plau-sibility does not degrade the output, as shown byour quality task, which asks workers to distinguishan automatically-colorized image from a real one.Finally, our caption manipulation experiment, inwhich workers are guided by a caption to selectone of three outputs generated with varying colorwords, shows that modifying the caption signifi-cantly affects the outputs of CONCAT and FILM.

input into FCNN.

2.3 Colorization conditioned on languageGiven an image I paired with a unit of text T, wefirst encode T into a continuous representationh using the last hidden state of a bi-directionalLSTM (Hochreiter and Schmidhuber, 1997). Weintegrate h into every convolutional block of theFCNN, allowing language to influence the compu-tation of all intermediate feature maps.

Specifically, say Zn is the feature map of thenth convolutional block. A conceptually simpleway to incorporate language into this feature mapis to concatenate h to the channels at each spatiallocation i, j in Zn, forming a new feature map

Z′ni,j = [Zni,j ;h]. (1)

While this method of integrating language with

a green pickup truck next to trees

x8

+CONV

batch norm

FiLM

Figure 2: FILM applies feature-wise affine transfor-mations (conditioned on language) to the output ofeach convolutional block in our architecture.

images (CONCAT) has been successfully used forother vision and language tasks (Reed et al., 2016;Feichtenhofer et al., 2016), it requires consider-ably more parameters than the FCNN due to theadditional language channels.

Inspired by recent work on visual question an-swering, we also experiment with a less parameter-hungry approach, feature-wise linear modula-tion (Perez et al., 2018, FILM), to fuse the languageand visual representations. Since the activationsof FILM layers have attention-like properties whentrained on VQA, we also might expect FILM tobe better at localizing objects from language thanCONCAT on colorization (see Figure 4 for heatmapvisualizations).

FILM applies a feature-wise affine transforma-tion to the output of each convolutional block,where the transformation weights are conditionedon language (Figure 2). Given Zn and h, we firstcompute two vectors γn and βn through linearprojection,

γn = Wnγh βn = Wnβh, (2)

where Wnγ and Wnβ are learned weight matrices.The modulated feature map then becomes

Z′ni,j = (1 + γn) ◦ Zni,j + βn, (3)

where ◦ denotes the element-wise product. Com-pared to CONCAT, FILM is parameter-efficient, re-quiring just two additional weight matrices per fea-ture map.

3 Experiments

We evaluate FCNN, CONCAT, and FILM using ac-curacy in ab space (shown by Zhang et al. (2016)

to be a poor substitute for plausibility) and withcrowdsourced experiments that ask workers tojudge colorization plausibility, quality, and thecolorization flexibly reflects language manipula-tions. Table 1 summarizes our results; while thereis no clear winner between FILM and CONCAT,both rely on language to produce higher-qualitycolorizations than those generated by FCNN.

3.1 Experimental setup

We train all of our models on the 82,783 imagesin the MSCOCO (Lin et al., 2014) training set, eachof which is paired with five crowdsourced cap-tions. Training from scratch on MSCOCO results inpoor quality colorizations due to a combination ofnot enough data and increased image complexitycompared to ImageNet (Russakovsky et al., 2015).Thus, for our final models, we initialize all convolu-tional layers with a FCNN pretrained on ImageNet;we finetune both FILM and CONCAT’s convolu-tional weights during training. To automaticallyevaluate the models, we compute top-1 and top-5 accuracy in our quantized ab output space2 onthe MSCOCO validation set. While FILM achievesthe highest ab accuracy, FILM and CONCAT donot significantly differ on crowdsourced evaluationmetrics.

3.2 Human experiments

We run three human evaluations of our models onthe Crowdflower platform to evaluate their plausi-bility, overall quality, and how well they conditiontheir output on language. Each evaluation is runusing a random subset of 100 caption/image pairsfrom the MSCOCO validation set,3 and we obtainfive judgments per pair.

Plausibility given caption: We show workersa caption along with three images generated byFCNN, CONCAT, and FILM. They choose the im-age that best depicts the caption; if multiple im-ages accurately depict the caption, we ask them tochoose the most realistic. FCNN does not receivethe caption as input, so it makes sense that its out-put is only chosen 20% of the time; there is nosignificant difference between CONCAT and FILMin plausibility given the caption.

2We evaluate accuracy at the downsampled 56×56 resolu-tion at which our network predicts colorizations. For humanexperiments, the prediction is upsampled to 224×224.

3We only evaluate on captions that contain one of ten“color” words (e.g., red, blue purple).

Figure 3: The top row contains successes from our caption manipulation task generated by FILM andCONCAT, respectively. The second row shows examples of how captions guide FILM to produce moreaccurate colorizations than FCNN (failure cases outlined in red). The final row contains, from left to right,particularly eye-catching colorizations from both CONCAT and FILM, a case where FILM fails to localizeproperly, and an image whose unnatural caption causes artifacts in CONCAT.

Colorization quality: Workers receive a pair ofimages, a ground-truth MSCOCO image and a gen-erated output from one of our three architectures,and are asked to choose the image that was notcolored by a computer. The goal is to fool workersinto selecting the generated images; the “foolingrates” for all three architectures are comparable,which indicates that we do not reduce colorizationquality by conditioning on language.

Caption manipulation: Our last evaluation mea-sures how much influence the caption has on theCONCAT and FILM models. We generate three dif-ferent colorizations of a single image by swappingout different colors in the caption (e.g., blue car,red car, green car). Then, we provide workers witha single caption (e.g., green car) and ask them tochoose which image best depicts the caption. If ourmodels cannot localize and color the appropriateobject, workers will be unable to select an appro-priate image. Fortunately, CONCAT and FILM areboth robust to caption manipulations (Table 1).

4 Discussion

Both CONCAT and FILM can manipulate imagecolor from captions (further supported by the toprow of Figure 3). Here, we qualitatively examinemodel outputs and identify potential directions forimprovement.

Language-conditioned colorization depends oncorrespondences between language and color statis-tics (stop signs are always red, and school busesare always yellow). While this extra informationhelps us produce more plausible colorizations com-pared to language-agnostic models (second rowof Figure 3), it biases models trained on naturalimages against unnatural colorizations. For exam-ple, the yellow sky produced by CONCAT in thebottom right of Figure 3 contains blue artifacts be-cause skies are usually blue in MSCOCO. Addition-ally, our models are limited by the lightness chan-nel L of the greyscale image, which prevents dra-matic color shifts like black-to-white. Smaller ob-jects are also problematic; often, colors will “leak”

Figure 4: Examples of intermediate layer activations while generating colorized images using the FILMnetwork. These activation maps correspond to the mean activation immediately after the FILM layers ofthe sixth, seventh, and eighth blocks. Interestingly, the activations after the FILM layer of Block 6 alwaysseems to focus on the object that is to be colorized, while those of Block 8 focus almost exclusivelyon the background. The activation maps do not significantly differ when color words in the caption aremanipulated; therefore, we show maps only for the first color word in these examples.

into smaller objects from larger ones, as shownby FILM’s colorizations of purple plants (Figure 3,bottom-middle) and yellow tires (middle-left).

Figure 4 shows activation maps from interme-diate layers generated while colorizing images us-ing the FILM network. Each intermediate layer iscaptured immediately after the FILM layer and isof dimension h × w × c (e.g., 112 × 112 × 64,28× 28× 512, etc.), where h is the height of thefeature map, w is its width, and c is the numberof channels.4 On inspection, the first few activa-tion maps correspond to edges and are not visuallyinteresting. However, we notice that the sixth acti-vation map usually focuses on the principal subjectof the image (such as a car or a horse), while theeighth activation map focused everywhere but onthat subject (i.e., entirely on the background). Thisanalysis demonstrates that the FILM layer emulatesvisual attention, reinforcing similar observationson visual QA datasets by Perez et al. (2018).

5 Future Work

While these experiments are promising, that thereare many avenues to improve language-conditionedcolorization. From a vision perspective, we wouldlike to more accurately colorize parts of objects(e.g., a person’s shoes); moving to more complex ar-

4We compute the mean across the c dimension and scalethe resulting h× w feature map between the limits [0, 255].

chitectures such as variational autoencoders (Desh-pande et al., 2017) or PixelCNNs (Guadarramaet al., 2017) might help here, as could increasingtraining image resolution. We also plan on usingrefinement networks (Shrivastava et al., 2017) tocorrect for artifacts in the colorized output image.On the language side, moving from explicitly spec-ified colors to abstract or emotional language isa particularly interesting. We plan to train ourmodels on dialogue/image pairs from datasets suchas COMICS (Iyyer et al., 2017) and visual story-telling (Huang et al., 2016); these models couldalso help learn powerful joint representations ofvision and language to improve performance ondownstream prediction tasks.

Acknowledgement

Manjunatha and Davis are partially supportedby the Office of Naval Research under GrantN000141612713: Visual Common Sense Reason-ing. Boyd-Graber is supported by NSF Grant IIS-1652666. We thank Richard Zhang for helpfulcomments on our ideas to make training the col-orization more efficient.

ReferencesZezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015.

Deep colorization. In International Conference onComputer Vision.

Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, MinJin Chong, and David Forsyth. 2017. Learning di-verse image colorization. In Computer Vision andPattern Recognition.

Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, andJun Mitani. 2016. Deepprop: Extracting deep fea-tures from a single image for edit propagation. InEurographics.

Christoph Feichtenhofer, Axel Pinz, and Andrew Zis-serman. 2016. Convolutional two-stream networkfusion for video action recognition. In Computer Vi-sion and Pattern Recognition.

Sergio Guadarrama, Ryan Dahl, David Bieber, Moham-mad Norouzi, Jonathon Shlens, and Kevin Murphy.2017. Pixcolor: Pixel recursive colorization. arXivpreprint arXiv:1705.07208 .

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Longshort-term memory. Neural computation 9:1735–1780.

Ting-Hao Kenneth Huang, Francis Ferraro, NasrinMostafazadeh, Ishan Misra, Aishwarya Agrawal, Ja-cob Devlin, Ross Girshick, Xiaodong He, PushmeetKohli, Dhruv Batra, et al. 2016. Visual storytelling.In Conference of the North American Chapter of theAssociation for Computational Linguistics.

Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005. An adaptiveedge detection based colorization algorithm and itsapplications. In Proceedings Annual ACM Interna-tional Conference on Multimedia.

Satoshi Iizuka, Edgar Simo-Serra, and HiroshiIshikawa. 2016. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priorsfor Automatic Image Colorization with Simultane-ous Classification. In ACM Transactions on Graph-ics.

Sergey Ioffe and Christian Szegedy. 2015. Batch nor-malization: Accelerating deep network training byreducing internal covariate shift. In Proceedings ofthe International Conference of Machine Learning.

Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yoga-rshi Vyas, Jordan Boyd-Graber, Hal Daumé III, andLarry Davis. 2017. The amazing mysteries of thegutter: Drawing inferences between panels in comicbook narratives. In Computer Vision and PatternRecognition.

Gustav Larsson, Michael Maire, and GregoryShakhnarovich. 2016. Learning representationsfor automatic colorization. In Proceedings of theEuropean Conference on Computer Vision.

Anat Levin, Dani Lischinski, and Yair Weiss. 2004.Colorization using optimization. In ACM Transac-tions on Graphics.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie,Lubomir D. Bourdev, Ross B. Girshick, James Hays,Pietro Perona, Deva Ramanan, Piotr Dollár, andC. Lawrence Zitnick. 2014. Microsoft COCO: Com-mon objects in context. In Proceedings of the Euro-pean Conference on Computer Vision.

Ethan Perez, Florian Strub, Harm de Vries, Vincent Du-moulin, and Aaron C. Courville. 2018. FiLM: Vi-sual reasoning with a general conditioning layer. InAssociation for the Advancement of Artificial Intelli-gence.

Scott E Reed, Zeynep Akata, Santosh Mohan, SamuelTenka, Bernt Schiele, and Honglak Lee. 2016.Learning what and where to draw. In Advances inNeural Information Processing Systems.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause,Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An-drej Karpathy, Aditya Khosla, Michael Bernstein,Alexander C. Berg, and Li Fei-Fei. 2015. Imagenetlarge scale visual recognition challenge. Interna-tional Journal of Computer Vision .

Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, JoshuaSusskind, Wenda Wang, and Russell Webb. 2017.Learning from simulated and unsupervised imagesthrough adversarial training. In Computer Visionand Pattern Recognition.

Thomas Smith and John Guild. 1931. The C.I.E. col-orimetric standards and their use. Transactions ofthe Optical Society 33(3):73.

Julie Van Camp. 1995. The colorization controversy.The Journal of Value Inquiry 29(4).

Richard Zhang, Phillip Isola, and Alexei A Efros. 2016.Colorful image colorization. In Proceedings of theEuropean Conference on Computer Vision.

Richard Zhang, Jun-Yan Zhu, Phillip Isola, XinyangGeng, Angela S Lin, Tianhe Yu, and Alexei A Efros.2017. Real-time user-guided image colorizationwith learned deep priors. ACM Transactions onGraphics 9(4).