I Can Has Cheezburger?A Nonparanormal Approach to Combining Textual and Visual Information

for Predicting and Generating Popular Meme Descriptions

William Yang Wang and Miaomiao WenSchool of Computer ScienceCarnegie Mellon University

Pittsburgh, PA 15213

Abstract

The advent of social media has brought Inter-net memes, a unique social phenomenon, tothe front stage of the Web. Embodied in theform of images with text descriptions, little dowe know about the “language of memes”. Inthis paper, we statistically study the correla-tions among popular memes and their word-ings, and generate meme descriptions fromraw images. To do this, we take a multi-modal approach—we propose a robust non-paranormal model to learn the stochastic de-pendencies among the image, the candidatedescriptions, and the popular votes. In experi-ments, we show that combining text and visionhelps identifying popular meme descriptions;that our nonparanormal model is able to learndense and continuous vision features jointlywith sparse and discrete text features in a prin-cipled manner, outperforming various com-petitive baselines; that our system can gener-ate meme descriptions using a simple pipeline.

1 Introduction

In the past few years, Internet memes become a new,contagious social phenomenon: it all starts with animage with a witty, catchy, or sarcastic sentence, andpeople circulate it from friends to friends, colleaguesto colleagues, and families to families. Eventually,some of them go viral on the Internet.

Meme is not only about the funny picture, theInternet culture, or the emotion that passes along,but also about the richness and uniqueness of itslanguage: it is often highly structured with specialwritten style, and forms interesting and subtle con-notations that resonate among the readers. For ex-ample, the LOL cat memes (e.g., Figure 1) often

Figure 1: An example of the LOL cat memes.

include superimposed text with broken grammarsand/or spellings.

Even though the memes are popular over the In-ternet, the “language of memes” is still not well-understood: there are no systematic studies on pre-dicting and generating popular Internet memes fromthe Natural Language Processing (NLP) and Com-puter Vision (CV) perspectives.

In this paper, we take a multimodal approach topredict and generate popular meme descriptions. Todo this, we collect a set of original meme images,a list of candidate descriptions, and the correspond-ing votes. We propose a robust nonparanormal ap-proach (Liu et al., 2009) to model the multimodalstochastic dependencies among images, text, andvotes. We then introduce a simple pipeline for gen-erating meme descriptions combining reverse im-age search and traditional information retrieval ap-proaches. In empirical experiments, we show thatour model outperforms strong discriminative base-lines by very large margins in the regression/rankingexperiments, and that in the generation experiment,the nonparanormal outperforms the second-best su-pervised baseline by 4.35 BLEU points, and obtainsa BLEU score improvement of 4.48 over an unsu-pervised recurrent neural network language model

trained on a large meme corpus that is almost 90times larger. Our contributions are three-fold:

• We are the first to study the “language ofmemes” combining NLP, CV, and machinelearning techniques, and show that combiningthe visual and textual signals helps identifyingpopular meme descriptions;

• Our approach empowers Internet users to selectbetter wordings and generate new memes auto-matically;

• Our proposed robust nonparanormal modeloutperforms competitive baselines for predict-ing and generating popular meme descriptions.

In the next section, we outline related work. InSection 3, we introduce the theory of copula, andour nonparanormal approach. In Section 4, we de-scribe the datasets. We show the prediction and gen-eration results in Section 5 and Section 6. Finally,we conclude in Section 7.

2 Related Work

Although the language of Internet memes is a rel-atively new research topic, our work is broadly re-lated to studies on predicting popular social mediamessages (Hong et al., 2011; Bakshy et al., 2011;Artzi et al., 2012). Most recently, Tan et al. (2014)study the effect on wordings for Tweets. However,none of the above studies have investigated multi-modal approaches that combine text and vision.

Recently, there has been growing interests ininter-disciplinary research on generating image de-scriptions. Gupta el al. (2009) have studied the prob-lem of constructing plots from video understand-ing. The work by Farhadi et al. (2010) is amongthe first to generate sentences from images. Kulka-rni et al. (2011) use linguistic constraints and a con-ditional random field model for the task, whereasMitchell et al. (2012) leverage syntactic informationand co-occurrence statistics and Dodge et al. (2012)use a large text corpus and CV algorithms for detect-ing visual text. With the surge of interests in deeplearning techniques in NLP (Socher et al., 2013; De-vlin et al., 2014) and CV (Krizhevsky et al., 2012;Oquab et al., 2013), there have been several unref-ereed manuscripts on parsing images and generatingtext descriptions lately (Vinyals et al., 2014; Chen

and Zitnick, 2014; Donahue et al., 2014; Fang etal., 2014; Karpathy and Fei-Fei, 2014) using neuralnetwork models. Although the above studies haveshown interesting results, our task is arguably morecomplex than generating text descriptions: in ad-dition to the visual and textual signals, we have tomodel the popular votes as a third dimension forlearning. For example, we cannot simply train a con-volutional neural network image parser on billionsof images, and use recurrent neural networks to gen-erate texts such as “There is a white cat sitting nextto a laptop.” for Figure 1. Additionally, since notall images are suitable as meme images, collectingtraining images is also more challenging in our task.

In contrast to prior work, we take a verydifferent approach: we investigate copula meth-ods (Schweizer and Sklar, 1983; Nelsen, 1999), inparticular, the nonparanormals (Liu et al., 2009), forjoint modeling of raw images, text descriptions, andpopular votes. Copula is a statistical framework foranalyzing random variables from Statistics (Liu etal., 2012), and often used in Economics (Chen andFan, 2006). Only until very recently, researchersfrom the machine learning and information retrievalcommunities (Ghahramani et al., 2012; Han et al.,2012; Eickhoff et al., 2013). start to understand thetheory and the predictive power of copula models.Wang and Hua (2014) are the first to introduce semi-parametric Gaussian copula (a.k.a. nonparanormals)for text prediction. However, their approach maybe prone to overfitting. In this work, we generalizeWang and Hua’s method to jointly model text andvision features with popular votes, while scaling upthe model using effective dropout regularization.

3 Our Approach

A key challenge for joint modeling of text and visionis that, because textual features are often relativelysparse and discrete, while visual features are typi-cally dense and continuous, it is difficult to modelthem jointly in a principled way.

To avoid comparing “apple and oranges” in thesame probabilistic space, we propose the non-paranormal approach, which extends the Gaussiangraphical model by transforming its variables bysmooth functions. More specifically, for each di-mension of textual and visual features, instead of

Figure 2: Our nonparanormal method extends Gaussianby transforming each dimension with a smooth function,and jointly models the stochastic dependencies amongtextual and visual features, as well as the popular votesby the crowd.

using raw counts or histograms, we first use prob-ability integral transform to generate empirical cu-mulative density functions (ECDF): now instead ofthe probability density function (PDF) space, we areworking in the ECDF space where the value of eachfeature is based on the rank, and is strictly restrictedbetween 0 and 1. Then, we use kernel density esti-mation to smooth out the zeroing features1. Finally,now textual and visual features are compatible, andwe then build a parametric Gaussian copula modelto estimate the pair-wise correlations among the co-variate and the dependent variable.

In this section, we first explain the visual and tex-tual features used in this study. Then, we introducethe theory of copula, and describe the robust non-paranormal. Finally, we show a simple pipeline forgenerating meme descriptions.

3.1 FeaturesTextual Features To model the meme descriptions,we take a broad range of textual features into con-siderations:• Lexical Features: we extract unigrams and bi-

grams from meme descriptions as surface-levellexical features.

• Part-of-Speech Features: to model shallowsyntactic cues, we extract lexicalized part-of-speech features using the Stanford part-of-speech tagger (Toutanova et al., 2003).

• Dependency Triples: to better understand thedeeper syntactic dependencies of keywords in

1This is necessary for the normal inversion of the ECDFs,which we will describe in Section 3.2.

Figure 3: An example of the standard SIFT keypoints de-tected on the “doge” meme.

memes, we have also extracted typed depen-dency triples (e.g., subj(I,are)) using the Malt-Parser (Nivre et al., 2007).

• Named Entity Features: after browsing thedataset, we notice that certain names are of-ten mentioned in memes (e.g. “Drake”, “KenyeWest”, and “Justin Bieber”), so we utilize theStanford named entity recognizer (Finkel et al.,2005) to extract lexicalized named entities.

• Frame-Semantics Features: SEMAFOR (Daset al., 2010) is a state-of-the-art frame-semantics parser that produces FrameNet-stylesemantic annotation. We use SEMAFOR to ex-tract frame-level semantic features.

Visual Features A key insight on viral memes isthat the images producing a shared social signal aretypically inter-related in style. For example, LOL-cats are an early series of memes involving funny catphotos. Similarly, “Bieber memes” involve modifiedpictures of Bieber.

Therefore, we hypothesize that, by extracting vi-sual features, it is of crucial importance to capturethe entities, objects, and styles as visual words inthese inter-related meme images. The popular vi-sual bag-of-words representation (Sivic and Zisser-man, 2003) is used to describe images:

1. PHOW Features Extraction: unlike text fea-tures, SIFT first detects the Harris keypointsfrom an image, and then describes each key-point with a vector. An example of the SIFTframes are shown in Figure 3. PHOW (Boschet al., 2007) is a dense and multi-scale vari-ant of the Scale Invariant Feature Transform(SIFT) descriptors. Using PHOW, we obtainabout 20K keypoints for each image.

2. Elkan K-means Clustering is the clusteringmethod (Elkan, 2003) that we use to obtainthe vocabulary for visual words. Compar-ing to other variants of K-means, this methodquickly constructs the codebook from PHOWkeypoints.

3. Bag-of-Words Histograms are used to repre-sent each image. We match the PHOW key-points of each image with the vocabulary thatwe extract from the previous step, and generatea 1× 200 sized visual bag-of-words vector.

3.2 The Theory of CopulaIn the Statistics literature, copula is widely knownas a family of distribution function. The idea be-hind copula theory is that the cumulative distribu-tion function (CDF) of a random vector can be rep-resented in the form of uniform marginal cumula-tive distribution functions, and a copula that con-nects these marginal CDFs, which describes the cor-relations among the input random variables. How-ever, in order to have a valid multivariate distributionfunction regardless of n-dimensional covariates, notevery function can be used as a copula function. Thecentral idea behind copula, therefore, can be sum-marize by the Sklar’s theorem and the corollary.Theorem 1 (Sklar’s Theorem (1959)) Let F bethe joint cumulative distribution function of n ran-dom variables X1, X2, ..., Xn. Let the correspond-ing marginal cumulative distribution functions ofthe random variable be F1(x1), F2(x2), ..., Fn(xn).Then, if the marginal functions are continuous, thereexists a unique copula C, such that

F (x1, ..., xn) = C[F1(x1), ..., Fn(xn)]. (1)

Furthermore, if the distributions are continuous, themultivariate dependency structure and the marginalsmight be separated, and the copula can be consid-ered independent of the marginals (Joe, 1997; Parsaand Klugman, 2011). Therefore, the copula does nothave requirements on the marginal distributions, andany arbitrary marginals can be combined and theirdependency structure can be modeled using the cop-ula. The inverse of Sklar’s Theorem is also true inthe following:Corollary 1 If there exists a copula C : (0, 1)n

and marginal cumulative distribution func-tions F1(x1), F2(x2), ..., Fn(xn), then

C[F1(x1), ..., Fn(xn)] defines a multivariatecumulative distribution function.

3.3 The Nonparanormal

To model multivariate text and vision variables,we choose the nonparanormal (NPN) as the copulafunction in this study, which can be explained in thefollowing two parts.

The Nonparametric EstimationAssume we have n random variables of vision and

text features X1, X2, ..., Xn. The problem is thattext features are sparse, so we need to perform non-parametric kernel density estimation to smooth outthe distribution of each variable. Let f1, f2, ..., fnbe the unknown density, we are interested in deriv-ing the shape of these functions. Assume we havemsamples, the kernel density estimator can be definedas:

fh(x) =1

m

m∑i=1

Kh(x− xi) (2)

=1

mh

m∑i=1

K

(x− xih

)(3)

Here, K(·) is the kernel function, where in our case,we use the Box kernel2 K(z):

K(z) =1

2, |z| ≤ 1, (4)

= 0, |z| > 1. (5)

Comparing to the Gaussian kernel and other kernels,the Box kernel is simple, and computationally in-expensive. The parameter h is the bandwidth forsmoothing3.

Now, we can derive the empirical cumulative dis-tribution functions

FX1(f1(X1)), FX2(f2(X2)), ..., FXn(fn(Xn))

of the smoothed covariates, as well as the dependentvariable y (which is the reciprocal rank of the pop-ular votes of a meme) and its CDF Fy(f(y)). The

2It is also known as the original Parzen windows (Parzen,1962).

3In our implementation, we use the default h of the Boxkernel in the ksdensity function in Matlab.

empirical cumulative distribution functions are de-fined as:

F (ν) =1

m

m∑i=1

I{xi ≤ ν} (6)

where I{·} is the indicator function, and ν indicatesthe current value that we are evaluating. Note thatthe above step is also known as probability integraltransform (Diebold et al., 1997), which allows us toconvert any given continuous distribution to randomvariables having a uniform distribution. This is cru-cial for text: instead of using the raw counts, we arenow working with uniform marginal CDFs, whichhelps coping with the overfitting issue due to noiseand data sparsity. We also use the same procedure totransform the vision features into CDF space to becompatible with text features.

The Robust Estimation of CopulaNow that we have obtained the marginals, and

then the joint distribution can be constructed by ap-plying the copula function that models the stochasticdependencies among marginal CDFs:

F (f1(X1), ..., f1(Xn), f(y))

= C[FX1

(f1(X1)

), ..., FXn

(fn(Xn)

), Fy

(fy(y)

)](7)

In this work, we apply the parametric Gaussian cop-ula to model the correlations among the text featuresand the label. Assume xi is the smoothed version ofrandom variable Xi, and y is the smoothed label, wehave:F (x1, ..., xn, y)

= ΦΣ

(Φ−1[Fx1

(x1)], ..., ,Φ−1[Fxn(xn)],Φ−1[Fy(y)]

)(8)

where ΦΣ is the joint cumulative distribution func-tion of a multivariate Gaussian with zero mean andΣ variance. Φ−1 is the inverse CDF of a standardGaussian. In this parametric part of the model, theparameter estimation boils down to the problem oflearning the covariance matrix Σ of this Gaussiancopula. In this work, we perform standard maxi-mum likelihood estimation (MLE) for the Σ matrix,where we follow the details from prior work (Wangand Hua, 2014).

To avoid overfitting, traditionally, one resorts toclassic regularization techniques such as Lasso (Tib-

shirani, 1996). While Lasso is widely used, the non-differentiable nature of the L1 norm often make theobjective function difficult to optimize. In this work,we propose dropout training (Hinton et al., 2012)as copula regularization. Dropout was proposed byHinton et al. as a method to prevent feature co-adaptation in the deep learning framework, but re-cently studies (Wager et al., 2013) also show that itsbehaviour is similar to L2 regularization, and can beapproximated efficiently (Wang and Manning, 2013)in many other machine learning tasks. Another ad-vantage of dropout training is that, unlike Lasso, itdoes not require all the features for training, andtraining is “embarrassingly” parallelizable.

In Gaussian copula estimation context, we can in-troduce another dimension `: the number of dropoutlearners, to extend the Σ into a dropout tensor. Es-sentially, the task becomes the estimation of

Σ1,Σ2, ...,Σ`

where the input feature space for each dropout com-ponent is randomly corrupted by (1 − δ) percent ofthe original dimension. In the inference time, weuse geometric mean to average the predictions fromeach dropout learner, and generate the final predic-tion. Note that the final Σ matrix has to be symmet-ric and positive definite, so we apply tiny randomGaussian noise ε to maintain the property.Computational Complexity

One important question regarding the proposednonparanormal model is the corresponding compu-tational complexity. This boils down to the es-timation of the Σ matrix (Liu et al., 2012): oneonly needs to calculate the correlation coefficientsof n(n − 1)/2 pairs of random variables. Chris-tensen (2005) shows that sorting and balanced bi-nary trees can be used to calculate the correlationcoefficients with complexity of O(n log n). There-fore, the computational complexity of MLE for theproposed model is O(n log n).Efficient Approximate Inference

In this prediction task, in order to performthe exact inference of the conditional probabil-ity distribution p(Fy(y)|Fx1(x1), ..., Fxn(xn)),one needs to solve the mean responseE(Fy(y)|Fx1(x1), ..., Fx1(x1)) from a jointdistribution of high-dimensional Gaussian cop-ula. Unfortunately, the exact inference can be

Figure 4: Our pipeline for generating memes from rawimages.

intractable in the multivariate case, and approximateinference, such as Markov Chain Monte Carlosampling (Gelfand and Smith, 1990; Pitt et al.,2006) is often used for posterior inference. In thiswork, we propose an efficient sampling methodto derive y given the text features — we sample

ˆFy(y) s.t. it maximizes the joint high-dimensionalGaussian copula density:

arg maxˆFy(y)∈(0,1)

1√det Σ

exp

(−1

2∆T ·

(Σ−1 − I

)·∆)

(9)

where

∆ =

Φ−1(Fx1(x1))

...Φ−1(Fxn(xn))Φ−1(Fy(y))

This approximate inference scheme using max-

imum density sampling from the Gaussian copulasignificantly relaxes the complexity of inference. Fi-nally, to derive y, the last step is to compute theinverse CDF of ˆFy(y). A detailed description ofthe inference algorithm can be found in our priorwork (Wang and Hua, 2014).

3.4 A Simple Meme Generation PipelineNow after we train a nonparanormal model for rank-ing meme descriptions, we show the simple memegeneration pipeline in Figure 4.

Given a test image, we disguise as the InternetExplorer, and query Google’s “Search By Image”inverse image search service4. By comparing the

4http://www.google.com/imghp/

query image with all possible images with their cap-tions in Google’s database, a “Best Guess” of thekeywords in the image is then revealed.

Using the extracted image keywords, we furtherquery a TF-IDF based Lucene5 meme search en-gine, which we indexed with a large number of Web-crawled meme descriptions. After we obtain thecandidate generations, we then extract all the textand vision features that we described in Section 3.1.Finally, our nonparanormal model ranks all possiblecandidates, and selects the final generation with thehighest posterior.

4 DatasetsWe collected meme images and text descriptions6

from two popular meme websites7. In the predic-tion experiment, we use 3,008 image-descriptionpairs for training, and 526 image-description pairsfor testing. In the generation experiment, we use269,473 meme descriptions to index the memesearch engine, and 50 randomly selected images fortesting. During training, we convert the raw countsof popular votes into reciprocal ranks (e.g., the mostpopular text descriptions will all have a reciprocalrank of 1, and n-th popular one will have a score of1/n).

5 Prediction ExperimentsIn the first experiment, we compare the proposedNPN with various baselines in a prediction task,since prior literature (Hodosh et al., 2013) also sug-gests using ranking based evaluation for associatingimages with text descriptions. Throughout the ex-periment sections, we set ` = 10, and δ = 80 as thedropout hyperparameters.

Baselines:The baselines are standard squared-loss lin-

ear regression, linear kernel SVM, and non-linear(Gaussian) kernel SVM. In a recent empiricalstudy (Fernandez-Delgado et al., 2014) that evalu-ates 179 classifiers from 17 families on 121 UCIdatasets, the authors find that Gaussian SVM is oneof the top performing classifiers. We use the Sta-tistical Toolbox’s linear regression implementationin Matlab, and LibSVM (Chang and Lin, 2011) for

5http://lucene.apache.org/6http://www.cs.cmu.edu/˜yww/data/meme dataset.zip.7memegenerator.net and cheezburger.com

training and testing the SVM models. The hyperpa-rameter C in linear SVM, and the γ and C hyperpa-rameters in Gaussian SVM are tuned on the trainingset using 10-fold cross-validation.

Evaluation Metrics:Spearman’s correlation (Hogg and Craig, 1994)

and Kendall’s tau (Kendall, 1938) have been widelyused in many real-valued prediction (regression)problems in NLP (Albrecht and Hwa, 2007; Yo-gatama et al., 2011), and here we use them to mea-sure the quality of predicted values y by comparingto the vector of ground truth y. Kendall’s tau is anonparametric statistical metric that have shown tobe inexpensive, robust, and representation indepen-dent (Lapata, 2006). We use paired two-tailed t-testto measure the statistical significance.

5.1 Comparison with Various Baselines

The first two figures in Figure 5 show the learn-ing curve of our system, comparing other baselines.We see that when increasing the amount of trainingdata, our approach clearly dominates all other meth-ods by a large margin. Linear and Gaussian SVMsperform similarly, and have good performances withonly 25% of the training data, but the improvementsare not large when increasing the amount of trainingdata.

In the last two figures in Figure 5, we increasethe amount of features, and compare various mod-els. We see that the linear regression model overfitswith 600 features, and Gaussian SVM outperformsthe linear SVM. We see that our NPN model clearlyoutperforms all baselines by a big gap, and does notoverfit.

5.2 Combination of Text and Vision

In Table 1, we systematically compare the contribu-tions of each feature set. First, we see that bigramfeatures clearly improve the performance on top ofunigram features. Second, named entities are crucialfor further boosting the performance. Third, addingthe shallow part-of-speech features does not benefitall models, but the dependency triples are shown tobe useful for all methods. Finally, we see that usingsemantic features helps increasing the performancesfor most of the cases, and combining text and visionfeatures in our NPN framework doubles the perfor-

Feature Sets LR LSVM GSVM NPNUnigrams 0.152 0.158 0.176 0.241*+ Bigrams 0.163 0.248 0.279 0.318*+ Named Entities 0.188 0.296 0.312 0.339*+ Part-of-Speech 0.184 0.318 0.337 0.343+ Dependency 0.191 0.322 0.348 0.350+ Semantics 0.183 0.368 0.388 0.367All Text + Vision 0.413 0.415 0.451 0.754*

Unigrams 0.102 0.105 0.118 0.181*+ Bigrams 0.115 0.164 0.187 0.237*+ Named Entities 0.127 0.202 0.213 0.248*+ Part-of-Speech 0.125 0.218 0.232 0.239+ Dependency 0.130 0.223 0.242 0.255+ Semantics 0.124 0.257 0.270 0.270All Text + Vision 0.284 0.288 0.314 0.580*

Table 1: The Spearman correlation (top table) andKendall’s τ (bottom table) for comparing various text fea-tures and combining with vision features. The best resultsof each row are highlighted in bold. * indicates p < .001comparing to the second best result.

mance for associating popular votes, meme images,and text descriptions.

5.3 The Effects of Dropout Training forNonparanormals

As we mentioned before, because NPNs model thecomplex network of random variables, a key issuefor training NPN is to prevent the model from over-fitting to the training data. So far, none of the priorwork have investigated dropout training for regular-izing the nonparanormals or even copula in general.To empirical test the effects of dropout training fornonparanormals, in addition to our datasets, we alsocompare with the unregularized copula from Wangand Hua (2014) on predicting financial risks fromearnings calls. Table 2 clearly suggests that dropouttraining for NPNs significant improves the perfor-mances on various datasets.

5.4 Qualitative Analysis

Table 3 shows the top ranked text features that arehighly correlated with popular votes. We see that thenamed entity features are useful: Paul Walker, UPS,Bruce Willis, Pencil Guy, Amy Winehouse are rec-ognized as entities in the meme dataset. Dependencytriples, as a less-understood feature set, also performwell in this task. For example, xcomp(tell,mean)

Figure 5: Two figures on the left: varying the amount of training data. L(1): Spearman. L(2): Kendall. Two figures onthe right: varying the amount of features. R(1): Spearman. R(2): Kendall.

Datasets No Dropout With DropoutMeme 0.625 0.754*Finance (pre2009) 0.416 0.482*Finance (2009) 0.412 0.445*Finance (post2009) 0.377 0.409*Meme 0.491 0.580*Finance (pre2009) 0.307 0.349*Finance (2009) 0.302 0.318*Finance (post2009) 0.282 0.297*

Table 2: The effects of dropout training for NPNs onmeme and other datasets. The best results of each roware highlighted in bold. * indicates p < .001 comparingto the no dropout setting.

captures the dependency relation of the popularmeme series “You mean to tell me...”. Interestingly,the transitional dependency feature dep(when,but)plays an important role in the language of memes.The object of a preposition, such as pobj(vegas,in)and pobj(life,of), also made the list.

Bigrams are shown to be important features asusual. For example, “Yo daw” is a popular memebased on rapper Xzibit’s famous reality car show“Pimp My Ride”, where the rapper customizes peo-ple’s car according to personal preferences. This vi-ral meme follows the pattern8 of “Yo daw(g), I herdyou like X (noun), so I put an X in your Y (noun)so you can W (verb) while you Z (verb).”

The use of pronouns, captured by frame semanticsfeatures, is associated with popular memes. We hy-pothesize that by using pronouns such as “i”, “you”,“we”, and “they”, the meme recalls personal expe-riences and emotions, thus connects better with theaudience. Finally, we see that the punctuation bi-gram “... :” is an important feature in the language

8http://knowyourmeme.com/memes/xzibit-yo-dawg

Top 1-10 Top 11-20 Top 21-30paul/PER FE party you newxcomp(tell,mean) dep(when,but) FE Entity itpossessive(’s,it) ... : bruce/PERyo daw FE Theme i FE party wepobj(vegas,in) on a FE Food fatups/ORG FE Exp. they <start> makeinto FE Entity you so youso you’re <start> how penci/PERFE Cognizer i of the yyo . pobj(life,of) winehouse/PER

Table 3: Top-30 linguistic features that are highly corre-lated with the popular votes.

of memes, and Web dialect such as “y” (why) alsoexhibits high correlation with the popular votes.

6 Generation ExperimentsIn this section, we investigate the performance ofour meme generation system using 50 test memeimages. To quantitatively evaluate our system, wecompare with both unsupervised and supervisedbaselines. For the unsupervised baselines, we com-pare with a compact recurrent neural network lan-guage model (RNNLM) (Mikolov, 2012) trained onthe 3,008 text descriptions of our meme training set,as well as a full model of RNNLM trained on a largememe corpus of 269K sentences9. For the super-vised baselines, all models are trained on the 3,008training image-description pairs with labels. Allthese models can be viewed as different re-rankingmethods for the retrieved candidate descriptions. Weuse BLEU score (Papineni et al., 2002) as the evalu-ation metric, since the generation task can be viewedas translating raw images into sentences, and it is

9Note that there are no image features feeding to the unsu-pervised RNN models.

Figure 6: Examples from the meme generation exper-iment. First row: the chemistry cat meme. Secondrow: the forever alone meme. Third row: the Batmanslaps Robin meme. Left column: human generated top-voted meme descriptions on memegenerator.net at thetime of writing. Middle column: generated output fromRNNLM. Right column: generated output from NPNs.

used in many caption generation studies (Vinyalset al., 2014; Chen and Zitnick, 2014; Donahue etal., 2014; Fang et al., 2014; Karpathy and Fei-Fei,2014).

The generation result is shown in Table 4. Notethat when combining B-1 to B-4 scores, BLEU in-cludes a brevity penalty as described in the originalBLEU paper. We see that our NPN model outper-forms the best supervised baseline by 4.35 BLEUpoints, while also obtaining an advantage of 4.48

Systems BLEU B-1 B-2 B-3 B-4RNN-C 19.52 62.2 21.2 12.1 9.0RNN-F 23.76 72.2 31.4* 16.2 8.7LR 23.89 72.3 28.3 15.0 10.6LSVM 21.06 65.0 24.8 13.1 9.3GSVM 20.63 66.2 22.8 12.8 9.3NPN 28.24* 66.9 29.0 19.7* 16.6*

Table 4: The BLEU scores for generating memes fromimages. B-1 to B-4: BLEU unigram to four-grams. Thebest BLEU results are highlighted in bold. * indicatesp < .001 comparing to the second best system.

BLEU points over the full RNNLM, which is trainedon a corpus that is ∼90 times larger, in an unsuper-vised fashion. When breaking down the results, wesee that our NPN’s advantage is on generating longerphrases, typically trigrams and four-grams, compar-ing to the other models. This is very interesting, be-cause generating high-quality long phrases is diffi-cult, since the memes are often short.

We show some generation examples in Figure 6.We see that on the left column, the reference memesare the ones with top votes by the crowd. The firstchemistry cat meme includes puns, the second for-ever alone meme includes reference to the life sim-ulation video game, while the last Batman memehas interesting conversations. In the second col-umn, we see that the memes generated by the fullRNNLM model are short, which corresponds to thequantitative results in Table 4. In the third col-umn, our NPN meme generator was able to gen-erate longer descriptions. Interestingly, it also cre-ates a pun for the chemistry cat meme. Our genera-tion on the forever alone meme is also accurate. Inthe Batman example, we show that the NPN modelmakes a sentence-image-mismatch type of error: al-though the generated sentence includes the entitiesBatman and Robin, as well as their slapping activ-ity, it was originally created for the “overly attachedgirlfriend” meme10.

7 Conclusions

In this paper, we study the language of memesby jointly learning the image, the description, andthe popular votes. In particular, we propose a ro-bust nonparanormal approach to transform all vi-sion and text features into the cumulative densityfunction space. By learning the stochastic depen-dencies, we show that our model significantly out-performs various competitive baselines in the pre-diction experiments. In addition, we also proposea simple pipeline for generating memes from rawimages, drawing the wisdom from reverse imagesearch and traditional information retrieval perspec-tives. Finally, we show that our model obtains sig-nificant BLEU point improvements over an unsuper-vised RNNLM baseline trained on a larger corpus,as well as other strong supervised baselines.

10http://www.overlyattachedgirlfriend.com

References

Joshua Albrecht and Rebecca Hwa. 2007. Regression forsentence-level mt evaluation with pseudo references.In Proceedings of ACL.

Yoav Artzi, Patrick Pantel, and Michael Gamon. 2012.Predicting responses to microblog posts. In Proceed-ings of NAACL-HLT.

Eytan Bakshy, Jake M Hofman, Winter A Mason, andDuncan J Watts. 2011. Everyone’s an influencer:quantifying influence on twitter. In Proceedings ofWSDM, pages 65–74. ACM.

Anna Bosch, Andrew Zisserman, and Xavier Munoz.2007. Image classification using random forests andferns.

Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: alibrary for support vector machines. ACM TIST.

Xiaohong Chen and Yanqin Fan. 2006. Estimationof copula-based semiparametric time series models.Journal of Econometrics.

Xinlei Chen and C Lawrence Zitnick. 2014. Learning arecurrent visual representation for image caption gen-eration. arXiv preprint arXiv:1411.5654.

David Christensen. 2005. Fast algorithms for the calcu-lation of kendalls τ . Computational Statistics.

Dipanjan Das, Nathan Schneider, Desai Chen, andNoah A Smith. 2010. Probabilistic frame-semanticparsing. In Proceedings of NAACL-HLT.

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, ThomasLamar, Richard Schwartz, and John Makhoul. 2014.Fast and robust neural network joint models for statis-tical machine translation. In Proceedings of ACL.

Francis X Diebold, Todd A Gunther, and Anthony S Tay.1997. Evaluating density forecasts.

Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Men-sch, Margaret Mitchell, Karl Stratos, Kota Yamaguchi,Yejin Choi, Hal Daume III, Alexander C Berg, et al.2012. Detecting visual text. In Proceedings of theNAACL-HLT.

Jeff Donahue, Lisa Anne Hendricks, Sergio Guadar-rama, Marcus Rohrbach, Subhashini Venugopalan,Kate Saenko, and Trevor Darrell. 2014. Long-term re-current convolutional networks for visual recognitionand description. arXiv preprint arXiv:1411.4389.

Carsten Eickhoff, Arjen P. de Vries, and Kevyn Collins-Thompson. 2013. Copulas for information retrieval.In Proceedings of the 36th International ACM SIGIRConference on Research and Development in Informa-tion Retrieval.

Charles Elkan. 2003. Using the triangle inequality toaccelerate k-means. In ICML, volume 3, pages 147–153.

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Sri-vastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xi-aodong He, Margaret Mitchell, John Platt, et al. 2014.From captions to visual concepts and back. arXivpreprint arXiv:1411.4952.

Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi,Peter Young, Cyrus Rashtchian, Julia Hockenmaier,and David Forsyth. 2010. Every picture tells astory: Generating sentences from images. In Com-puter Vision–ECCV 2010, pages 15–29. Springer.

Manuel Fernandez-Delgado, Eva Cernadas, Senen Barro,and Dinani Amorim. 2014. Do we need hun-dreds of classifiers to solve real world classificationproblems? Journal of Machine Learning Research,15:3133–3181.

Jenny Rose Finkel, Trond Grenager, and ChristopherManning. 2005. Incorporating non-local informationinto information extraction systems by gibbs sampling.In Proceedings of the 43rd Annual Meeting on Associ-ation for Computational Linguistics, pages 363–370.Association for Computational Linguistics.

Alan Gelfand and Adrian Smith. 1990. Sampling-basedapproaches to calculating marginal densities. Journalof the American statistical association.

Zoubin Ghahramani, Barnabas Poczos, and Jeff Schnei-der. 2012. Copula-based kernel dependency mea-sures. In Proceedings of the 29th International Con-ference on Machine Learning.

Abhinav Gupta, Praveen Srinivasan, Jianbo Shi, andLarry S Davis. 2009. Understanding videos, con-structing plots learning a visually grounded storylinemodel from annotated videos. In Computer Vision andPattern Recognition, 2009. CVPR 2009. IEEE Confer-ence on, pages 2012–2019. IEEE.

Fang Han, Tuo Zhao, and Han Liu. 2012. Coda: Highdimensional copula discriminant analysis. Journal ofMachine Learning Research.

Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky,Ilya Sutskever, and Ruslan R Salakhutdinov. 2012.Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprintarXiv:1207.0580.

Micah Hodosh, Peter Young, and Julia Hockenmaier.2013. Framing image description as a ranking task:Data, models and evaluation metrics. J. Artif. Intell.Res.(JAIR), 47:853–899.

Robert V Hogg and Allen Craig. 1994. Introduction tomathematical statistics.

Liangjie Hong, Ovidiu Dan, and Brian D Davison. 2011.Predicting popular messages in twitter. In Proceedingsof WWW.

Harry Joe. 1997. Multivariate models and dependenceconcepts.

Andrej Karpathy and Li Fei-Fei. 2014. Deep visual-semantic alignments for generating image descrip-tions. Stanford University Technical Report.

Maurice Kendall. 1938. A new measure of rank correla-tion. Biometrika.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.2012. Imagenet classification with deep convolutionalneural networks. In Advances in neural informationprocessing systems, pages 1097–1105.

Girish Kulkarni, Visruth Premraj, Sagnik Dhar, SimingLi, Yejin Choi, Alexander C Berg, and Tamara L Berg.2011. Baby talk: Understanding and generating im-age descriptions. In Proceedings of the 24th CVPR.Citeseer.

Mirella Lapata. 2006. Automatic evaluation of informa-tion ordering: Kendall’s tau. Computational Linguis-tics.

Han Liu, John Lafferty, and Larry Wasserman. 2009.The nonparanormal: Semiparametric estimation ofhigh dimensional undirected graphs. The Journal ofMachine Learning Research, 10:2295–2328.

Han Liu, Fang Han, Ming Yuan, John Lafferty, and LarryWasserman. 2012. High-dimensional semiparamet-ric gaussian copula graphical models. The Annals ofStatistics.

Tomas Mikolov. 2012. Statistical language modelsbased on neural networks. Ph.D. thesis, Ph. D. the-sis, Brno University of Technology.

Margaret Mitchell, Xufeng Han, Jesse Dodge, AlyssaMensch, Amit Goyal, Alex Berg, Kota Yamaguchi,Tamara Berg, Karl Stratos, and Hal Daume III. 2012.Midge: Generating image descriptions from computervision detections. In Proceedings of EACL.

Roger B Nelsen. 1999. An introduction to copulas.Springer Verlag.

Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev,Gulsen Eryigit, Sandra Kubler, Svetoslav Marinov,and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency pars-ing. Natural Language Engineering, 13(02):95–135.

Maxime Oquab, Leon Bottou, Ivan Laptev, Josef Sivic,et al. 2013. Learning and transferring mid-level imagerepresentations using convolutional neural networks.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evalua-tion of machine translation. In Proceedings of ACL,pages 311–318. Association for Computational Lin-guistics.

Rahul A Parsa and Stuart A Klugman. 2011. Copularegression. Variance Advancing and Science of Risk.

Emanuel Parzen. 1962. On estimation of a probabilitydensity function and mode. The annals of mathemati-cal statistics.

Michael Pitt, David Chan, and Robert Kohn. 2006. Effi-cient bayesian inference for gaussian copula regressionmodels. Biometrika.

Berthold Schweizer and Abe Sklar. 1983. Probabilisticmetric spaces.

Josef Sivic and Andrew Zisserman. 2003. Video google:A text retrieval approach to object matching in videos.In Proceedings of ICCV, pages 1470–1477. IEEE.

Abe Sklar. 1959. Fonctions de repartition a n dimen-sions et leurs marges. Universite Paris 8.

Richard Socher, Alex Perelygin, Jean Y Wu, JasonChuang, Christopher D Manning, Andrew Y Ng, andChristopher Potts. 2013. Recursive deep models forsemantic compositionality over a sentiment treebank.In Proceedings of EMNLP, pages 1631–1642. Cite-seer.

Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The ef-fect of wording on message propagation: Topic- andauthor-controlled natural experiments on twitter. InProceedings of ACL.

Robert Tibshirani. 1996. Regression shrinkage and se-lection via the lasso. Journal of the Royal StatisticalSociety. Series B (Methodological), pages 267–288.

Kristina Toutanova, Dan Klein, Christopher D Manning,and Yoram Singer. 2003. Feature-rich part-of-speechtagging with a cyclic dependency network. In Pro-ceedings of NAACL-HLT.

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du-mitru Erhan. 2014. Show and tell: A neural imagecaption generator. arXiv preprint arXiv:1411.4555.

Stefan Wager, Sida Wang, and Percy Liang. 2013.Dropout training as adaptive regularization. In Ad-vances in Neural Information Processing Systems,pages 351–359.

William Yang Wang and Zhenhao Hua. 2014. A semi-parametric gaussian copula regression model for pre-dicting financial risks from earnings calls. In Proceed-ings of ACL.

Sida Wang and Christopher Manning. 2013. Fastdropout training. In Proceedings of ICML.

Dani Yogatama, Michael Heilman, Brendan O’Connor,Chris Dyer, Bryan R Routledge, and Noah A Smith.2011. Predicting a scientific community’s response toan article. In Proceedings of EMNLP.