1

MW-GAN: Multi-Warping GAN for CaricatureGeneration with Multi-Style Geometric

ExaggerationHaodi Hou, Jing Huo, Jing Wu, Yu-Kun Lai, and Yang Gao

Abstract—Given an input face photo, the goal of caricature generation is to produce stylized, exaggerated caricatures that share thesame identity as the photo. It requires simultaneous style transfer and shape exaggeration with rich diversity, and meanwhilepreserving the identity of the input. To address this challenging problem, we propose a novel framework called Multi-Warping GAN(MW-GAN), including a style network and a geometric network that are designed to conduct style transfer and geometric exaggerationrespectively. We bridge the gap between the style and landmarks of an image with corresponding latent code spaces by a dual waydesign, so as to generate caricatures with arbitrary styles and geometric exaggeration, which can be specified either through randomsampling of latent code or from a given caricature sample. Besides, we apply identity preserving loss to both image space andlandmark space, leading to a great improvement in quality of generated caricatures. Experiments show that caricatures generated byMW-GAN have better quality than existing methods.

Index Terms—Caricature Generation, Generative Adversarial Nets, Multiple Styles, Warping

F

1 INTRODUCTION

C ARICATURES are artistic drawings of faces with exaggerationof facial features to emphasize the impressions of or inten-

tions towards the subject. As an art form, caricatures have variousdepiction styles, such as sketching, pencil strokes, oil painting,and various exaggeration styles to express different impressionsand emphasize different aspects of the subject. Artists have theirown subjectivity and different skills which also contribute to thediversity of caricatures. These varieties in caricature generationmake caricatures a fascinating art form with long-lasting popular-ity.

However, such diversity has not been achieved in computer-ized generation of caricatures. Early works generate caricaturesthrough amplifying the difference from the mean face [1], [2], [3]or automatically learning rules from paired photos and caricatures.However, these methods can only generate caricatures with aspecific style. The recent style transfer methods [4], [5], [6],[7] and image translation methods [8], [9], [10], [11], [12], [13]based on Convolutional Neural Networks (CNNs) and GenerativeAdversarial Nets (GANs) [14] have achieved appealing resultson image style translation in texture and color. However, thesemethods are not designed to deal with geometric shape exagger-ation in caricatures. The recent GAN-based caricature generationmethods [15], [16], [17] can generate caricatures with reasonableexaggerations, but still lack variety in exaggeration styles, leavinga gap between computer generated and real caricatures.

To tackle this issue, in this paper, we propose Multi-WarpingGAN for generating caricatures from face photos with a focuson generating various exaggeration styles. Multi-Warping GANconsists of a style network and a geometric network. The style

H. Hou, J. Huo and Y. Gao are with the State Key Laboratory for NovelSoftware Technology, Nanjing University, No. 163 Xianlin Avenue, Nanjing,210023, Jiangsu, China. E-mail: hhd@smail.nju.edu.cn, huojing@nju.edu.cn,gaoy@nju.edu.cn.J. Wu and Y.-K. Lai are with the School of Computer Science & Informatics,Cardiff University. E-mail: {WuJ11,LaiY4}@cardiff.ac.uk.

network is trained to render images with different texture andcoloring styles, while the geometric network learns the exag-geration in the landmark space and warps images accordingly.In both networks, we propose to use latent codes to controlthe texture and exaggeration styles respectively. The diversity isachieved by random sampling of the latent codes or extractingthem from sample caricatures. To correlate the latent codes withmeaningful texture styles and shape exaggerations, we propose adual way architecture, i.e. to simultaneously translate photos intocaricatures and caricatures into photos, with the aim to providemore supervision on the latent code. With the dual way design,cycle consistency loss on latent code can be introduced. Thisallows us to not only get more meaningful latent codes, but also toget better generation results compared with using the single waydesign.

In addition to diversity, another challenge is the identity preser-vation in generated caricatures. Observing that caricaturizationinvolves both style translation and shape deformation, to preservethe identity of the subject in the input photo, we deploy identityrecognition loss in both image space and landmark space whentraining the networks, which leads to remarkable improvement inthe quality of the generated caricatures.

We conducted ablation studies to verify the effectiveness of thedual way architecture in comparison with the single way design,and the introduction of the landmark constraints in the identityrecognition loss. We compare our method with the state-of-the-art caricature generation methods in terms of the quality of thegenerated caricatures. And we demonstrate the diversity of boththe texture and exaggeration styles in the generated caricaturesusing our method. Results show both the effectiveness of ourmethod and its superiority to the state-of-the-arts.

In summary, the contributions of our work are as follows:

1) Our method is the first to focus on geometric diversity incaricature generation, and propose a GAN-based frame-

arX

iv:2

001.

0187

0v1

[cs

.CV

] 7

Jan

202

0

2

work that can generate caricatures with arbitrary textureand exaggeration styles.

2) Our framework can unsupervisedly learn the relationsbetween image styles, shape exaggerations and the corre-sponding latent code spaces in a dual way design, so thatthe style and exaggeration of generated caricatures canalso be set through caricature samples.

3) To preserve the identity of the subject in the photo,we also deploy identity recognition loss in both imagespace and landmark space when training the network,which leads to remarkable improvement in the qualityof generated caricatures.

We compare our results with those from the state-of-the-artmethods, and demonstrate the superiority of our method in termsof both quality and diversity of the generated caricatures.

2 RELATED WORK

2.1 Style TransferSince CNNs have achieved great success in understanding thesemantics in images, it is widely studied to apply CNNs to styletransfer. The ground-breaking work of Gatys et al. [4] presenteda general neural style transfer method that can transfer the texturestyle from a style image to a content image. Following this work,many improved methods [5], [6] have been proposed to speedup the transfer process by learning a specific style with a feed-forward network and transfer an arbitrary style in real time throughadaptive instance normalization [7]. Despite the achievements intransferring images with realistic artistic styles, these methods canonly change the texture rendering of images, but are not designedto make the geometric exaggeration required in caricature genera-tion. In our MW-GAN, a style network together with a geometricnetwork are used to simultaneously render the image’s texturestyle and exaggerate its geometric shape, with the aim to generatecaricatures with both realistic texture styles and meaningful shapeexaggerations.

2.2 Image TranslationThe success of Generative Adversarial Nets (GANs) [14] hasinspired a series of work on cross-domain image translation. Thepix2pix network [18] is trained with a conditional GAN, andneeds supervision from paired images which are hard to get.Triangle GAN [19] achieved semi-supervised image translationby combining a conditional GAN and a Bidirectional GAN [20]with a triangle framework. There have been efforts to achieveimage translation in a totally unsupervised manner through sharedweights and latent space [8], [9], cycle consistency [10], andmaking use of semantic features [12]. The above methods treatimage translation as a one-to-one mapping. Recently more meth-ods have been proposed to deal with image translation withmultiple styles. Augmented Cycle GAN [21] extends cycle GANto multiple translations by adding a style code to model variousstyles. MUNIT [11] and CDAAE [13] disentangle image into acontent code and a style code, so that one input image can betranslated to various output images by sampling different stylecodes. These methods can successfully translate images betweendifferent domains, and can render with various texture stylesin one translation. However, these translations mostly keep theimage’s geometric shapes unchanged, which is not suitable forcaricature generation. By contrast, we separately model the two

aspects, texture rendering and geometric exaggeration, and achieveboth translations in a multiple style manner. That is, our modelcan generate caricatures with various texture styles and diversegeometric exaggerations for a given input.

2.3 Caricature GenerationCaricature generation has been studied for a long time. Traditionalmethods translate photos to caricatures using computer graphicstechniques. The first interactive caricature generator was presentedby Brennan et al. [22]. The caricature generator allows users tomanipulate photos interactively to create caricatures. Followinghis work, rule-based methods were proposed [1], [2], [3] to au-tomatically amplify the difference from the mean face. Example-based methods [23], [24] can automatically learn rules from photo-caricature pairs. Though these methods can generate caricatureautomatically or semi-automatically, they suffer from some limi-tations, such as the need of human interactive manipulation andpaired data collection. Moreover, caricatures generated by theseearly methods are often unrealistic and lack diversity.

Since GANs have made great progress in image genera-tion, many GAN-based methods for caricature generation werepresented recently. Some of these methods translate photos tocaricatures with a straightforward network [12], [15], while otherstranslate the texture style and geometric shapes separately [16],[17]. For the straightforward methods, DTN [12] uses a pretrainedneural network to extract semantic features from input so thatsemantic content can be preserved during translation. CariGAN byLi et al. [15] adopts facial landmarks as an additional condition toenforce reasonable exaggeration and facial deformation. As thesemethods translate both texture and shape in a single network,it is hard for them to achieve meaningful deformation or tobalance identity preservation and shape exaggeration. By contrast,warpGAN [16] and CariGANs by Cao et al. [17] separatelyrenders the image’s texture and exaggerates its shape. Thoughthey can generate caricatures with realistic texture styles andmeaningful exaggerations, warpGAN and CariGANs [17] stillsuffer from lacking exaggeration variety. Specifically, if the inputis specified, the exaggeration style is also fixed, while in realworld, it is common that different artists draw caricatures withdifferent exaggeration styles for the same photo. In this paper,we design a framework that is able to model the variety of bothtexture styles and geometric exaggerations and propose the firstmodel that can generate caricatures with diverse styles in bothtexture and exaggeration for one input photo.

3 MULTI-WARPING GANIn this section, we describe the network architecture of theproposed Multi-Warping GAN and the loss functions used fortraining.

3.1 NotationsLet xp ∈ Xp denote an image in the photo domain Xp, andxc ∈ Xc denote an image in the caricature domain Xc. Givenan input face photo xp ∈ Xp, the goal is to generate a caricatureimage in the space Xc, while sharing the same identity as xp.This process involves two types of transition, texture style transferand geometric shape exaggeration. Previous works [16], [17] canonly generate caricatures with fixed geometric exaggeration stylewhen an input is given. In this paper, we focus on the problem of

3

𝐸𝑝𝑐

𝐸𝑝𝑠

𝐺𝑝𝑠

𝐸𝑐𝑠

𝐸𝑐𝑐

𝐺𝑐𝑠

𝑥𝑝

𝑥𝑐

𝑧𝑝𝑠

𝑧𝑐𝑠

𝑧𝑐𝑐

𝑧𝑝𝑐

𝑥𝑐′

𝑥𝑝→𝑐′

𝑥𝑝′

𝑥𝑐→𝑝′

𝑧𝑐𝑐

𝑧𝑝𝑐

𝑧𝑝→𝑐𝑙

Δ𝑙𝑝→𝑐

𝐺𝑝→𝑐𝑙

warp

𝑥𝑝→𝑐

𝑧𝑐→𝑝𝑙

𝐺𝑐→𝑝𝑙

Δ𝑙𝑐→𝑝

warp

𝑥𝑐→𝑝

Style Network Geometric Network

Fig. 1. The network architecture of the proposed multi-warping GAN. The left part is the style network and the right part is the geometric network.The black solid arrows denote the flow of two auto-encoders, with the upper one being the auto-encoder of photos and the lower one for caricaturereconstruction. The orange dashed lines denote the flow of photo to caricature generation and the blue dashed lines for caricature to photogeneration. In our dual way framework, we assume the caricature and the photo share the same content feature space but have separate stylespaces. Ec

p and Ecc are two encoders to encode the content of photos and caricatures respectively. Similarly defined, Es

p and Esc are two encoders

to encode the style of photos and caricatures. Gaussian distribution is imposed on their outputs zsp and zsc , so that we can sample style codesfrom Gaussian when translating photos to caricatures or vice versa. To model the geometric transformation procedure, we assume the geometrictransformation can be learned by a network (Gl

p→c or Glc→p) with the image’s content code (zcp or zcc ) as input. Besides, by adding a random code

(zlp→c or zlc→p) which represents different geometric transformation style, the network can output various geometric transformations for the inputimage. With the learned geometric transformation, by warping the image with its texture style changed, we get the generated caricature or photo.

caricature generation with multiple geometric exaggeration styles,and propose the first framework to deal with it.

The notations used in this paper are as follows. We usex, z, l, y to denote image sample, latent code, landmark andidentity label respectively. Subscripts p and c refer to photo andcaricature respectively, while superscripts s and c represent styleand content. Encoder, generator and discriminator are representedby capital letters E, G and D.

3.2 Multi-Warping GAN

The network architecture of Multi-Warping GAN is shown inFigure 1. It consists of a style network and a geometric network.The style network is designed to render images with differenttexture and color styles, while the geometric network aims toexaggerate the face shapes in the input images. The style networkworks in the image space, while the geometric network is builton landmarks and exaggerates geometric shapes through warping.Both style and geometric networks are designed in a dual way,i.e., there is one way to translate photos to caricatures and alsothe other way to translate caricatures to photos. Although forour purpose, we are mainly interested in translating photos tocaricatures, which can also be achieved with a single way network.However, the dual way design allows us to incorporate cycleconsistency loss to help with learning of meaningful latent codes,

which is impossible in the single way design. We experimentallyverified that the dual way framework is more effective comparedwith the single way design. In our dual way design, the style andthe shape exaggeration are represented by latent codes zs andzl respectively. Both latent codes can be sampled from Gaussiandistribution or extracted from sample caricature images to achievethe diversity in both style and exaggeration. To train this network,we design a set of loss functions to tighten the correspondingrelations between the latent code space and the image space, andto keep identity consistency. In the following, we will explain thedetails of our style network and geometric network along with theloss functions accordingly.

3.2.1 Style Network

During the texture style transfer, the face shape in the imageshould be preserved. We thus assume that there is a joint shapespace, referred to as “content” space) shared by both photos andcaricatures, while their style spaces are independent. FollowingMUNIT [11], the style network is composed of two autoencodersfor content and style respectively, and is trained to satisfy theconstraints in both the image reconstruction process and the styletranslation process.

The image reconstruction process is shown in Figure 1 with

4

Δ𝑙𝑝→𝑐 ሚ𝑙𝑝→𝑐

𝑧𝑝𝑐

𝑙𝑝

𝐺𝑝→𝑐𝑙

⨁ 𝐸𝑐𝑙

𝐸𝑝𝑙

𝑧𝑝→𝑐𝑙

ǁ𝑧𝑐→𝑝𝑙

𝑧′𝑝→𝑐𝑙

Δ𝑙𝑐→𝑝 ሚ𝑙𝑐→𝑝

𝑧𝑐𝑐

𝑙𝑐

𝐺𝑐→𝑝𝑙

⨁ 𝐸𝑝𝑙

𝐸𝑐𝑙

𝑧𝑐→𝑝𝑙

ǁ𝑧𝑝→𝑐𝑙

𝑧′𝑐→𝑝𝑙

a: Translation from photo to caricature b: Translation from caricature to photo

Fig. 2. Geometric Network. The left part is the network for learning a transformation from a photo’s landmarks to a caricature’s landmarks. Theright part is the network for the reverse transformation. For the left network, a generator Gl

p→c with the content code of a photo zcp and a landmarktransformation latent code zlp→c (which can be randomly sampled from a Gaussian distribution) as input will output landmark displacement vectors∆lp→c. By adding the displacement vectors to the photo’s landmark positions, we get the transformed caricature landmarks lp→c. To make therandomly sampled zlp→c correlate to meaningful shape transformation styles, we introduce two encoders and force cycle consistency loss on theencoded latent code and sampled latent code. For example, z

′lp→c = El

c(lp→c, lp) is the encoded latent code, we will force z′lp→c to be as close as

possible to zlp→c.

black arrows, and can be formulated as follows:

x′p = Gsp(Ec

p(xp), Esp(xp)),

x′c = Gsc(E

cc(xc), E

sc (xc)).

(1)

where Ecp and Es

p are content and style encoders for photos.Similarly, Ec

c and Esc are content and style encoders for cari-

catures. Gsp and Gs

c are two decoders for photos and caricaturesrespectively. The image reconstruction loss is defined as the `1difference between the input and reconstructed images:

Lrec x = ‖x′p − xp‖1 + ‖x′c − xc‖1. (2)

The style translation process is shown in Figure 1 with dashedarrows, and can be formulated as:

x′p→c = Gsc(E

cp(xp), zsc),

x′c→p = Gsp(Ec

c(xc), zsp),

(3)

where zsc and zsp are style codes sampled from Gaussian distribu-tions for the two modalities. x′p→c is a generated image with thecontent from the input photo and a caricature style, while x′c→p isa generated image with the content from the input caricature anda photo style.

The constraints in the style transfer process are based on threeaspects. Firstly, after transfer, the style code of the transferredimage should be consistent with the style code of input, i.e.,

Lrec s = ‖zsp − Esp(x′c→p)‖1 + ‖zsc − Es

c (x′p→c)‖1 (4)

where x′c→p and x′p→c are defined in Eq. (3). Secondly, theimage content should keep unchanged during the transfer. Acycle consistency loss on the content codes of the input and thetransferred images is used, as shown below:

Lcyc c =‖Ecp(xp)− Ec

c(x′p→c)‖1+‖Ec

c(xc)− Ecp(x′c→p)‖1

(5)

Thirdly, the transferred image should be able to convert backwhen passing through the same encoder-decoder and using theoriginal style code. Again a cycle consistency loss on the inputand transferred images is used for this constraint:

Lcyc x =‖xp −Gsp(Ec

c(x′p→c), Esp(xp))‖1

+‖xc −Gsc(E

cp(x′c→p), Es

c (xc))‖1.(6)

Please note, the second terms in Eq. (5) and Eq. (6) constrainthe style transfer from caricatures to photos. It is only possibleto impose this cycle consistency in our dual way design. It isexpected that the cycle consistency can help build the relationbetween the latent code space and the image space. A singleway network from photos to caricatures only is also implementedas a baseline (Section 3.3) which is trained without these twoterms. Experimental results, as in Section 4.1, demonstrate thesuperior generation results using the dual way design with thecycle consistency loss.

Eqs. (2), (4), (5) and (6) give all the loss functions to train thestyle network. With the above network architecture, we can gen-erate caricatures with various texture styles by sampling differentstyle codes.

3.2.2 Geometric NetworkGeometric exaggeration is an essential feature of caricatures.There are two aspects to consider when modeling caricatureexaggerations. One is that exaggerations usually emphasize thesubject’s characteristics. The other is that they also reflect the skillsand preference of the artists. Therefore, in caricature generation,it is natural to model geometric exaggeration based on both theinput photo and an independent exaggeration style. By changingthe exaggeration style, we can mimic different artists to generatecaricatures that have different shape exaggerations for a giveninput photo. In our design, we build a latent code space on land-marks (landmark transformation latent code space) to represent thedifferent exaggeration styles from artists, and use the content codeto represent the input photo. The geometric network is designedand trained to learn the mapping from a combination of a codein the landmark transformation latent space and a content codeto a landmark displacement map which defines the geometricdeformation between the input photo and the caricature to begenerated. The output landmark displacements thus capture boththe input subject’s characteristics and a specific geometric exag-geration style. Then geometric exaggeration is finally achievedby warping the input photo according to the learned landmarkdisplacements [25].

Following the above assumption, the design of our geometricnetwork is as shown in Figure 2. It consists of one generator(Gl

p→c or Glc→p) in each way of translation and two encoders

(Elp, El

c) whose functions will be explained later. Taking the

5

translation from photos to caricatures as an example (shown inFigure 2a), the generator Gl

p→c takes the content code zcp and thelandmark transformation latent code zlp→c as input, and outputsthe landmark displacements ∆lp→c which are then added to theinput landmarks lp to get the target caricature landmarks lp→c.

The landmark transformation latent code zlp→c encodes ashape exaggeration style. To make it correlate to meaningful shapeexaggerations, we follow the idea of Augmented CycleGAN [21]and introduce the two encoders El

p, Elc into the geometric net-

work. Again, taking the translation from photos to caricaturesfor example, the encoder El

c takes the landmarks of the photolp and the landmarks of the corresponding caricature lp→c asinput, and extracts the difference between them and reconstruct thelandmark latent code z′lp→c. By introducing the encoder, it enablesus 1) to enforce cycle consistency between the randomly sampledlatent code zlp→c and the encoded latent code z′lp→c to correlatezlp→c to meaningful exaggerations; and 2) to extract zlp→c fromexample caricatures to perform sample guided shape exaggeration.The same applies for the encoder El

p used in translation fromcaricatures to photos (Figure 2b).

Basically, to train the geometric network, we have the land-mark transformation latent code reconstruction loss:

Lrec z l = ‖zlc→p −Elp(lc→p, lc)‖1 + ‖zlp→c −El

c(lp→c, lp)‖1(7)

where the first term is the reconstruction loss of zlc→p, thelandmark transformation latent code from photos to caricatures.The second term is the reconstruction loss of zlp→c and is definedin a similar way.

Besides the above loss, we use LSGAN [26] to match thegenerated landmarks with real ones:

LGgan l =‖1−Dl

p(lc→p)‖2

+ ‖1−Dlc(lp→c)‖

2, (8)

LDgan l =‖1−Dl

p(lp)‖2 + ‖Dlp(lc→p)‖

2

+‖1−Dlc(lc)‖

2+ ‖Dl

c(lp→c)‖2. (9)

where Eq. (8) is the loss for generators and Eq. (9) is the lossfor discriminators. The objective of generators is to make thegenerated caricature landmarks lp→c or photo landmarks lc→p

indistinguishable from real landmarks, i.e., the output of the dis-criminator with the generated landmarks as input becomes 1. Onthe other hand, the objective of discriminators is to discriminatebetween the real photo landmarks lp and the generated photolandmarks lc→p, as well as to discriminate between the realcaricature landmarks lc and the generated caricature landmarkslp→c.

Similar to above loss for generated landmarks, we define theloss for generated images:

LGgan x =‖1−Dx

p (xc→p)‖2 + ‖1−Dxc (xp→c)‖2, (10)

LDgan x =‖1−Dx

p (xp)‖2 + ‖Dxp (xc→p)‖2

+‖1−Dxc (xc)‖2 + ‖Dx

c (xp→c)‖2. (11)

The definition of the above two losses are in the same way as thelosses in Eqs. (8) and (9), except that the generated landmarks arenow changed to images.

We also use LSGAN [26] to match all the latent codes(including both landmark transformation latent code and the style

latent code, except content code) to Gaussian:

LGgan z =‖1−Dz(z)‖2, (12)

LDgan z =‖1−Dz(z)‖2 + ‖Dz(z)‖2. (13)

Here, z is latent codes encoded by neural encoders, while z islatent codes sampled from Gaussian distribution. Eq. (12) is theloss for the generator and Eq. (13) is the loss for the discriminator.The objective of the generator is to make the discriminator unableto tell if the encoded latent code z is sampled from Gaussianor not. And the discriminator’s objective is to try to discriminatebetween these two kinds of codes.

3.2.3 Identity PreservationIdentity preservation in the generated caricatures becomes morechallenging with the explicit geometric deformation introduced.As a result, in addition to preserving identity in the image spaceas in [16], we add further constraints on identity in the landmarkspace. Two discriminators are added to classify the identity fromboth the image and the landmarks.

Lid x =− log(Dxid(yp, xp))− log(Dx

id(yp, xp→c))

− log(Dxid(yc, xc))− log(Dx

id(yc, xc→p)) (14)

Lid l =− log(Dlid(yp, lp))− log(Dl

id(yp, lp→c))

− log(Dlid(yc, lc))− log(Dx

id(yc, lc→p)). (15)

Here, yp and yc are the identity labels for photos and caricatures.

3.2.4 Overall LossIn summary, training the proposed MW-GAN is to minimize thefollowing types of loss functions: 1) the reconstruction loss ofthe image Lrec x, the style latent code Lrec s, and the landmarktransformation latent code Lrec z l, and the cycle consistencyloss of the content code Lcyc c and of the image Lcyc x; 2)the generative adversarial loss pairs on images LG

gan x, LDgan x,

landmarks LGgan l, L

Dgan l, and latent codes LG

gan z , LDgan z; and

3) the identify loss on the image Lid x, and on the landmarksLid l.

Our framework is trained by optimizing the following overallobjective functions on encoders, generators, and discriminators:

minE,G

λ1Lrec x + λ2(Lrec s + Lrec z l + Lcyc c + Lcyc x)

+λ3Lid x + λ4Lid l + λ5(LGgan x + LG

gan l + LGgan z),

(16)

minD

λ3Lid x +λ4Lid l +λ5(LDgan x +LD

gan l +LDgan z). (17)

Here, E and G denote the encoder and generator networks,whereasD denotes the discriminator networks. λ1, λ2, λ3, λ4 andλ5 are weight parameters to balance the influence of the differentloss terms in the overall objective function.

3.3 Degradation to Single Way BaselineNotice for the caricature generation task, we can also degrade theabove framework to a single way network, i.e. only from photos tocaricatures as shown in Figure 3. However, we found that withoutthe cycle consistency loss, the single way network for caricaturegeneration performs worse than the dual way design. Here, wegive details of the single way network and this forms a baselinemethod in our experiments.

6

𝐸𝑝𝑐 𝐺𝑐

𝑠

𝐸𝑝𝑠

Warp

𝐺𝑝→𝑐𝑙

𝑥𝑝

𝑧𝑝→𝑐𝑙

𝑧𝑠

Δ𝑙𝑝→𝑐

𝑙𝑝

𝑥𝑝→𝑐′

𝑥𝑝→𝑐

Fig. 3. Framework of baseline single way GAN. The upper two rowsdenote the style network and the lower two rows denote the geometricnetwork.

As shown in Figure 3, the degraded single way network alsoconsists of a style network and a geometric network. The stylenetwork is used to render an input photo with a caricature stylewhile preserving the geometric shapes. It consists of two encodersEs

p , Ecp and one generator Gs

c. The content encoder Ecp extracts

the feature map zcp that contains the geometric information ofthe input photo, while the style encoder Es

p extracts texture stylezsp from the input. The style code zsp is adapted to a Gaussiandistribution and affects the image’s style through adaptive instancenormalization [7]. The geometric network exaggerates the facein the rendered image by warping it according to the landmarkdisplacements ∆lp→c. To achieve multi-style exaggerations, weassume that ∆lp→c is controlled by not only the photo’s content zcpbut also a landmark transformation latent code zlp→c that followsa Gaussian distribution.

This straightforward framework can increase the variety ofexaggeration styles through various landmark transformation la-tent codes. However, it suffers from some limitations. Firstly, asit is a one-way framework without cycle consistency, there lackssupervision to relate the generated caricatures with the landmarktransformation latent code. Then with the use of discriminator,the model may ignore the landmark transformation latent codein training which actually restricts the geometric deformation.Secondly, this one-way framework lacks supervision to bridge thegap between the latent space of zlp→c and the landmark space.Thus the learned landmark latent code may have no reference toreal landmarks. In experiments, we compare the generation resultsusing our dual way network with using the single way network,and demonstrate the advantages of our dual-way design.

4 EXPERIMENTS

We conducted experiments on the WebCaricature dataset [27],[28]. We will first describe the details of the dataset and thetraining process, and then demonstrate the effectiveness of the dualway design and the landmark based identity loss through ablationstudies. We will show the ability of our method to generatecaricatures with a variety of both texture and exaggeration styles.And finally, we will compare the caricatures generated using our

method with the previous state-of-the-art methods and show thesuperiority of our method in terms of the generation quality.

4.1 Experimental DetailsDataset preprocessing. We trained and tested our network on apublic dataset WebCaricature [27], [28]. There are 6,042 carica-tures and 5,974 photographs from 252 persons in this dataset. Wefirst pre-processed all the images by rotating the faces to makethe line between eyes horizontal, and cropping the face using abounding box which covers hair and ears. In detail, an initial boxis first created by passing through the centers of ears, the top ofhead and the chin. Then the bounding box used is the initial boxenlarged by a factor of 1.5. All processed images are resized to256×256. We randomly split the dataset into a training set of 202identities (4,804 photos and 4,773 caricatures) and a test set of 50identities (1,170 photos and 1,269 caricatures). All the imagespresented in this paper are from identities in the test set. Thelandmarks used in our experiments are the 17 landmarks providedin the WebCaricature dataset [27], [28].

Details of implementation. Our framework is implementedwith Tensorflow. The style network is modified based on MU-NIT [11]. We removed the discriminator on the stylized imagesand added a discriminator on the warped images at the end. Wealso added a discriminator on the style latent code generated bythe style encoder to match it with a Gaussian distribution. Inthe geometric network, we take Gl

p→c and Elp as an example to

explain the detailed structure, and Glc→p and El

c are implementedin the same way. For Gl

p→c, the content code is firstly down-sampled by max pooling with kernel size 3 and stride 2, andthen is fed into three blocks of 3 × 3 convolution with stride1 followed by leaky ReLU with α = 0.01 and 3 × 3 maxpooling with stride 2. After that, there is a fully connected layermapping this to a 32 dimensional vector. The landmark latent codeis also mapped to a 32 dimensional vector by a fully connectedlayer. Then the two vectors are concatenated and fed into a fullyconnected layer to output ∆lp→c. For El

p, the two sets of inputlandmarks (landmarks for photo and landmarks for caricature) arefirstly concatenated and then fed into four fully connected layersto give the estimated landmark latent code. All the fully connectedlayers in Gl

p→c and Elp are activated with leaky ReLU, except the

last layer. Discriminators for images are composed of 6 blocksof 4 × 4 convolution with stride 2 and a last layer with fullconnection, while discriminators for latent codes consist of sixlayers with full connection. Leaky ReLU is used as activation forall discriminators.

We empirically set λ1 = 10, λ2 = λ5 = 1.0, λ3 = 0.05,λ4 = 0.01. We used ADAM optimizers with β1 = 0.5 andβ2 = 0.999 to train the whole network. The network is trained for500,000 steps with batch size of 1. The learning rate is started with0.0001 and is decrease by half every 100,000 steps. The model istrained on a computer with an NVIDIA GeForce RTX 2080 TiGPU, and the training takes about three days.

4.2 Ablation StudyTo analyze our dual way design and identity recognition loss, weconducted experiments using the baseline method and three MW-GAN variants by respectively removing the identity recognitionloss in the image space, landmark space and both.

Study on different identity loss. Figure 4 shows the car-icatures generated using MW-GAN and its three variants with

7

Input 𝑤/𝑜 𝐿𝑖𝑑_𝑙 𝑤/𝑜 𝐿𝑖𝑑_𝑥 𝑤/𝑜( 𝐿𝑖𝑑_𝑥 & 𝐿𝑖𝑑𝑙) MW-GAN

Fig. 4. Ablation study. For MW-GAN’s variant methods, we generate one caricature for each input. For MW-GAN, we generate three caricatures foreach input by sampling different style latent codes and landmark transformation latent codes.

TABLE 1Ablation study. Comparison of FID scores of the three variants of MW-GAN, the single way baseline and MW-GAN.

Method w/o Lid l w/o Lid x w/o (Lid l&Lid x) baseline MW-GAN

FID 47.53 56.44 40.94 57.38 36.57

MW-GANBaselineInput

Fig. 5. Comparison with the single way baseline method.

different identity loss. From an overall view, it is obvious thatcaricatures generated by MW-GAN has much better visual qualitythan its other variants. At a closer look, the caricatures generatedwithout loss on landmarks or on images (w/o (Lid l&Lid x))tend to be unrecognizable. The caricatures generated using thevariant omitting the loss on images (w/o Lid x) suffer from badvisual quality, as it lacks adversarial supervision from image view.

As for the variant omitting the loss on landmarks (w/o Lid l),the generated caricatures have better visual quality, but theirexaggerations are not in the direction to emphasize the subjects’characteristics. By contrast, MW-GAN with both Lid x and Lid l

can exaggerate facial shapes to enlarge the characteristics of thesubjects, and meanwhile can render the caricatures with appealingtexture styles.

Comparison with the single way baseline. Figure 5 showsthe comparison of caricatures generated using our MW-GAN andusing the baseline method described in Section 3.3. For each inputwe randomly sample one style code and two landmark codes andgenerate two caricatures. From the results, we can see that withdifferent landmark transformation codes, MW-GAN can generatecaricatures with different exaggeration styles, while the single-way baseline method generates caricatures with almost the sameexaggeration for each input. Moreover, it is also obvious thatthe exaggerations from the single-way method are sometimes outof control with unrealistic distortions, while our MW-GAN cangenerate much more meaningful exaggerations with the addedcycle consistency supervision.

In addition to the above visual comparison, we also usedFrechet Inception Distance (FID) [29] to quantitatively compareour MW-GAN with its three variants and the one-way baselinemethod. As shown in Table 1, our identity recognition loss in bothlandmark space and image space, as well as our dual framework

8

Input Diverse style Diverse exaggeration

Fig. 6. Diversity in texture style and exaggeration of the proposed MW-GAN. The 1st column shows input images. The 2nd-4th columns arecorresponding generated caricatures with different texture styles with a fixed exaggeration. The last three columns are generated caricatures witha fixed texture style but different exaggerations.

Input photos

Guide caricatures

Fig. 7. Sample-guided caricature generation of the proposed method.

design can greatly improve the quality of the generated caricatures.

4.3 Diversity in Texture Style and Exaggeration

In our MW-GAN network, the generated caricatures have theirtexture styles and shape exaggerations controlled by the stylelatent code and the landmark transformation latent code respec-tively. To achieve the diversity in generated caricatures, we cansample different style codes and landmark transformation codes,and apply them to the input photo. Figure 6 shows generatedcaricatures with different texture and exaggeration styles. For eachinput, we generate three caricatures with fixed exaggeration butdifferent texture styles and another three caricatures with fixedtexture style but different exaggerations. The results meet ourexpectation that different style codes lead to different texture andcoloring, while different landmark codes lead to different shapeexaggerations.

Our dual-way design of MW-GAN enables unsupervisedlearning of the bidirectional mapping between image style andstyle latent space, geometric exaggeration and landmark trans-formation latent space. Therefore, MW-GAN can also generatecaricatures with a guide sample by applying its style and landmarktransformation codes to generators in the network. Figure 7 showsexample caricatures generated with different guide samples. Wecan see that the generated caricatures not only have similar textureand color styles as the guide caricatures, but also try to mimicthe exaggeration styles of the guide caricatures. From left to right,the generated caricatures have similar exaggeration styles of widecheeks and narrow forehead, high cheekbones and squeezed facialfeatures (eyes, nose and mouth), long face and pointed chin, andlaughing with wide open mouth.

9

CycleGAN MUNIT warpGAN MW-GANInput

Fig. 8. Comparison with state-of-the-art methods. CycleGAN and MUNIT can only generate images with style changed. Compared with WarpGAN,the proposed MW-GAN can generate caricatures with better visual quality. Beside, MW-GAN can generate caricatures with various styles andshape exaggerations.

TABLE 2Comparison with state-of-the-art methods on FID score.

Method CycleGAN MUNIT warpGAN MW-GAN

FID 49.82 52.33 40.69 36.57

4.4 Comparison with State-of-the-Art Methods

We qualitatively and quantitatively compare our MW-GAN withprevious state-of-the-art methods in image translation: CycleGAN [10], Multimodal UNsupervised Image-to-image Translation(MUNIT) [11], and in caricature generation: warpGAN [16].CariGANs [17] is not compared with as it is not open sourceand the re-implementation is complicated. All the networks aretrained using the same photo-caricature training set as describedin Section 4.1.

Figure 8 shows the generated caricatures using the differentmethods. When using MUNIT and warpGAN, we randomlysample the style code to generate one caricature for each input.When using our MW-GAN, we randomly sample style codes andlandmark transformation codes and generate three caricatures foreach input. CycleGAN is a deterministic method, and can onlygenerate a fixed caricature for each input. As shown in Figure 8,CycleGAN can only generate caricatures with limited changes intexture. Some results even look almost the same as the input.Caricatures generated by MUNIT have some changes in shape,but some results show clear artifacts, such as the speckles aroundthe nose (2nd row) and the dark and patchy appearance (3rd row).As MUNIT is not designed for shape deformation, we speculate

these artifacts arise from its attempt to achieve the appearanceof shape deformation by some texture disguise. Results fromwarpGAN can generate caricatures with more reasonable texturechanges and shape exaggerations. However, given the input photo,its exaggeration style is deterministic, which does not reflect thediverse skills and preferences among artists. In comparison, ourMW-GAN is designed to achieve diversity in both texture stylesand shape exaggerations. As can be seen in Figure 8, differentstyle codes and landmark codes lead to different texture styles andshape exaggerations in the generated caricatures.

We also calculated FID to quantitatively measure the qualityof generated caricatures (shown in Table 2). Since CycleGANand MUNIT are designed only for texture transformation but notgeometric exaggeration as required in caricature generation, theirFIDs are much higher than warpGAN and MW-GAN, indicatinglower quality. When it comes to warpGAN and MW-GAN, it isshown that MW-GAN has a lower FID than warpGAN, which webelieve are due to three reasons. Firstly, MW-GAN specificallyconsiders the exaggeration diversity, which is a closer assump-tion to the real distribution of caricatures. Secondly, the identityrecognition loss in both image space and landmark space enablesthe learning of more meaningful shape exaggerations using MW-

10

Input Output

Fig. 9. Translation of caricatures back to photos. The input includescaricatures drawn by artists. The outputs are photos generated by MW-GAN with randomly sampled style latent codes and landmark latentcodes.

GAN. Finally, our dual-way design enables the learning of a bidi-rectional translation between caricatures and photos, and bridgethe two with style latent codes and landmark latent codes. Thisdesign helps train the model as a whole.

4.5 Translation of Caricatures Back to PhotosAlthough not as our main goal, since MW-GAN is designed ina dual way, there is also a path to translate caricatures back tophotos. We conducted experiments by using caricatures and theirlandmarks as input, and randomly sampling style latent code zspand landmark latent code zlc→p. The generated photos are shownin Figure 9. The results show that MW-GAN can restore photosfrom caricatures by reversely deforming the exaggerated facesback to normal. As can be seen in the 3rd and 4th rows, in thecheek, chin and mouth area, the shapes of generated photos lookrealistic, which shows MW-GAN can moderately deform the car-icature shapes to photo shapes. However, as translating exagger-ated caricatures back to photos requires much more complicateddeformation, warping based on 17 landmarks sometimes cannotrepresent such deformation. When the input caricature has extremeexaggeration, MW-GAN may generate photos without necessaryshape deformation, resulting photos containing deformations ofthe input caricatures. Therefore, it should be further explored totranslate caricatures back to photos.

5 CONCLUSION

In this paper, we propose the first framework that can generatecaricatures with diversities in both texture styles and shape exag-gerations. In our design, we use style latent code and landmarktransformation latent code to capture the diversity in texture and

exaggeration respectively. We also design a dual way framework tolearn the bidirectional translation between photos and caricatures.This design helps the model to learn the bidirectional translationbetween the image style, face landmarks and their correspondinglatent spaces, which enables the generation of caricatures withsample-guided texture and exaggeration styles. We also introducedidentity recognition loss in both image space and landmark space,which enables the model to learn more meaningful exaggerationand texture styles for the input photo. Qualitative and quantitativeresults demonstrate that MW-GAN outperforms the state-of-the-art methods in image translation and caricature generation.

ACKNOWLEDGMENTS

This work is supported by Science and Technology Innovation2030 New Generation Artificial Intelligence Major Project No.(2018AAA0100905), and the National NSF of China (61432008,61806092), Jiangsu NSF(BK20180326), and the CollaborativeInnovation Center of Novel Software Technology and Industri-alization.

REFERENCES

[1] H. Koshimizu, M. Tominaga, T. Fujiwara, and K. Murakami, “Onkansei facial image processing for computerized facial caricaturingsystem picasso,” in IEEE SMC’99 Conference Proceedings. 1999 IEEEInternational Conference on Systems, Man, and Cybernetics (Cat.No.99CH37028), vol. 6, Oct 1999, pp. 294–299 vol.6.

[2] N. K. H. Le, Y. P. Why, and G. Ashraf, “Shape stylized face caricatures,”in Advances in Multimedia Modeling. Springer Berlin Heidelberg, 2011,pp. 536–547.

[3] P.-Y. C. W.-H. Liao and T.-Y. Li, “Automatic caricature generation byanalyzing facial features,” in Proceeding of 2004 Asia Conference onComputer Vision (ACCV2004), Korea, vol. 2, 2004.

[4] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer usingconvolutional neural networks,” in 2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), June 2016, pp. 2414–2423.

[5] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-timestyle transfer and super-resolution,” in Computer Vision – ECCV 2016.Springer International Publishing, 2016, pp. 694–711.

[6] D. Ulyanov, V. Lebedev, Andrea, and V. Lempitsky, “Texturenetworks: Feed-forward synthesis of textures and stylized images,”in Proceedings of The 33rd International Conference on MachineLearning, ser. Proceedings of Machine Learning Research, vol. 48.PMLR, 20–22 Jun 2016, pp. 1349–1357. [Online]. Available:http://proceedings.mlr.press/v48/ulyanov16.html

[7] X. Huang and S. Belongie, “Arbitrary style transfer in real-time withadaptive instance normalization,” in 2017 IEEE International Conferenceon Computer Vision (ICCV), Oct 2017, pp. 1510–1519.

[8] M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,”in Advances in Neural Information Processing Systems 29. CurranAssociates, Inc., 2016, pp. 469–477. [Online]. Available: http://papers.nips.cc/paper/6544-coupled-generative-adversarial-networks.pdf

[9] M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Advances in NeuralInformation Processing Systems 30. Curran Associates, Inc.,2017, pp. 700–708. [Online]. Available: http://papers.nips.cc/paper/6672-unsupervised-image-to-image-translation-networks.pdf

[10] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-imagetranslation using cycle-consistent adversarial networks,” in 2017 IEEEInternational Conference on Computer Vision (ICCV), Oct 2017, pp.2242–2251.

[11] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsuper-vised image-to-image translation,” in Computer Vision – ECCV 2018.Springer International Publishing, 2018, pp. 179–196.

[12] Y. Taigman, A. Polyak, and L. Wolf, “Unsupervised cross-domain imagegeneration,” CoRR, vol. abs/1611.02200, 2016. [Online]. Available:http://arxiv.org/abs/1611.02200

[13] H. Hou, J. Huo, and Y. Gao, “Cross-domain adversarial auto-encoder,” CoRR, vol. abs/1804.06078, 2018. [Online]. Available:http://arxiv.org/abs/1804.06078

11

[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio, “Generative adversarialnets,” in Advances in Neural Information Processing Systems 27.Curran Associates, Inc., 2014, pp. 2672–2680. [Online]. Available:http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

[15] W. Li, W. Xiong, H. Liao, J. Huo, Y. Gao, and J. Luo,“Carigan: Caricature generation through weakly paired adversariallearning,” CoRR, vol. abs/1811.00445, 2018. [Online]. Available:http://arxiv.org/abs/1811.00445

[16] Y. Shi, D. Deb, and A. K. Jain, “Warpgan: Automatic caricaturegeneration,” CoRR, vol. abs/1811.10100, 2018. [Online]. Available:http://arxiv.org/abs/1811.10100

[17] K. Cao, J. Liao, and L. Yuan, “CariGANs: Unpaired photo-to-caricaturetranslation,” ACM Trans. Graph., vol. 37, no. 6, pp. 244:1–244:14, Dec.2018. [Online]. Available: http://doi.acm.org/10.1145/3272127.3275046

[18] P. Isola, J. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translationwith conditional adversarial networks,” in 2017 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), July 2017, pp. 5967–5976.

[19] Z. Gan, L. Chen, W. Wang, Y. Pu, Y. Zhang, H. Liu, C. Li, andL. Carin, “Triangle generative adversarial networks,” in Advances inNeural Information Processing Systems 30. Curran Associates, Inc.,2017, pp. 5247–5256. [Online]. Available: http://papers.nips.cc/paper/7109-triangle-generative-adversarial-networks.pdf

[20] J. Donahue, P. Krahenbuhl, and T. Darrell, “Adversarial featurelearning,” CoRR, vol. abs/1605.09782, 2016. [Online]. Available:http://arxiv.org/abs/1605.09782

[21] A. Almahairi, S. Rajeshwar, A. Sordoni, P. Bachman, and A. Courville,“Augmented CycleGAN: Learning many-to-many mappings from un-paired data,” in Proceedings of the 35th International Conference onMachine Learning, ser. Proceedings of Machine Learning Research,vol. 80. PMLR, 10–15 Jul 2018, pp. 195–204.

[22] S. E. Brennan, “Caricature generator: The dynamic exaggeration of facesby computer,” Leonardo, vol. 18, no. 3, pp. 170–178, 1985.

[23] Lin Liang, Hong Chen, Ying-Qing Xu, and Heung-Yeung Shum,“Example-based caricature generation with exaggeration,” in 10th PacificConference on Computer Graphics and Applications, 2002. Proceed-ings., Oct 2002, pp. 386–393.

[24] R. N. Shet, K. H. Lai, E. A. Edirisinghe, and P. W. H. Chung, “Use ofneural networks in automatic caricature generation: An approach basedon drawing style capture,” in Pattern Recognition and Image Analysis.Springer Berlin Heidelberg, 2005, pp. 343–351.

[25] F. Cole, D. Belanger, D. Krishnan, A. Sarna, I. Mosseri, and W. T.Freeman, “Synthesizing normalized faces from facial identity features,”in 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), July 2017, pp. 3386–3395.

[26] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley, “Leastsquares generative adversarial networks,” in 2017 IEEE InternationalConference on Computer Vision (ICCV), Oct 2017, pp. 2813–2821.

[27] J. Huo, Y. Gao, Y. Shi, and H. Yin, “Variation robust cross-modal metriclearning for caricature recognition,” in Proceedings of the on ThematicWorkshops of ACM Multimedia 2017, 2017, pp. 340–348.

[28] J. Huo, W. Li, Y. Shi, Y. Gao, and H. Yin, “Webcaricature: a benchmarkfor caricature recognition,” in British Machine Vision Conference, 2018.

[29] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter,“Gans trained by a two time-scale update rule converge to a local nashequilibrium,” in Advances in Neural Information Processing Systems 30,2017, pp. 6626–6637. [Online]. Available: http://papers.nips.cc/paper/7240-gans-trained-by-a-two-time-scale-update-rule-converge-to-a-local-nash-equilibrium.pdf

Haodi Hou received the BSc from Nanjing Uni-versity, China in 2017. Currently, he is a MSccandidate in the Department of Computer Sci-ence and Technology at Nanjing University anda member of RL Group, which is led by professorYang Gao. His research interests are mainly onNeural Networks, Generative Adversarial Netsand their application in image translation andface recognition.

Jing Huo received the PhD degree from theDepartment of Computer Science and Technol-ogy, Nanjing University, Nanjing, China, in 2017.She is currently an Assistant Researcher withthe Department of Computer Science and Tech-nology, Nanjing University. Her current researchinterests include machine learning and computervision, with a focus on subspace learning, adver-sarial learning and their applications to hetero-geneous face recognition and cross-modal facegeneration.

Jing Wu received the BSc and MSc degreesfrom Nanjing University, China in 2002 and 2005respectively. She received the PhD degree incomputer science from the University of York, UKin 2009. She worked as a research associate inCardiff University subsequently, and is currentlya lecturer in computer science and informatics atCardiff University. Her research interests are onimage-based 3D reconstruction and its applica-tions. She is a member of ACM and BMVA.

Yu-Kun Lai received the bachelors and PhDdegrees in computer science from Tsinghua Uni-versity, in 2003 and 2008, respectively. He iscurrently a reader with the School of ComputerScience & Informatics, Cardiff University. His re-search interests include computer graphics, ge-ometry processing, image processing, and com-puter vision. He is on the editorial boards ofComputer Graphics Forum and The Visual Com-puter.

Yang Gao (M05) received the Ph.D. degree incomputer software and theory from Nanjing Uni-versity, Nanjing, China, in 2000. He is a Profes-sor with the Department of Computer Scienceand Technology, Nanjing University. He has pub-lished over 100 papers in top conferences andjournals. His current research interests includearticial intelligence and machine learning.