Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4154–4164Brussels, Belgium, October 31 - November 4, 2018. c©2018 Association for Computational Linguistics

4154

MSMO: Multimodal Summarization with Multimodal Output

Junnan Zhu1,2, Haoran Li1,2, Tianshang Liu1,2, Yu Zhou1,2, Jiajun Zhang1,2 and Chengqing Zong1,2,3

1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China2 University of Chinese Academy of Sciences, Beijing, China

3 CAS Center for Excellence in Brain Science and Intelligence Technology, Beijing, China{junnan.zhu, yzhou, jjzhang, cqzong}@nlpr.ia.ac.cn

AbstractMultimodal summarization has drawn muchattention due to the rapid growth of multime-dia data. The output of the current multimodalsummarization systems is usually representedin texts. However, we have found through ex-periments that multimodal output can signifi-cantly improve user satisfaction for informa-tiveness of summaries. In this paper, we pro-pose a novel task, multimodal summarizationwith multimodal output (MSMO). To handlethis task, we first collect a large-scale datasetfor MSMO research. We then propose a multi-modal attention model to jointly generate textand select the most relevant image from themultimodal input. Finally, to evaluate mul-timodal outputs, we construct a novel multi-modal automatic evaluation (MMAE) methodwhich considers both intramodality salienceand intermodality relevance. The experimen-tal results show the effectiveness of MMAE.

1 Introduction

Text summarization is to extract the important in-formation from source documents. With the in-crease of multimedia data on the internet, someresearchers (Li et al., 2016b; Shah et al., 2016;Li et al., 2017) focus on multimodal summariza-tion in recent years. Existing experiments (Liet al., 2017, 2018a) have proven that, comparedto text summarization, multimodal summarizationcan improve the quality of generated summary byusing information in visual modality.

However, the output of existing multimodalsummarization systems is usually represented ina single modality, such as textual or visual (Liet al., 2017; Evangelopoulos et al., 2013; Madem-lis et al., 2016). In this paper, we argue thatmultimodal output1 is necessary for the follow-ing three reasons: 1) It is much easier and faster

1Note that in this work, the multimodal output refers to apictorial summary which contains one image (for the sake of

Researchers have discovered the fossilized remains of a small, lizard- like creature that is the missing ancestral link …

summarizeTiny was one of the firstfour-legged creatures tomove …

Figure 1: The illustration of our proposed task –Multimodal Summarization with Multimodal Output(MSMO). The image can help better understand thetext in the red font.

for users to get critical information from the im-ages (Li et al., 2017). 2) According to our ex-periments, the multimodal output (text+image) in-creases users’ satisfaction by 12.4% compared tothe single-modality output (text) (more details canbe found in Sec. 4.2). 3) Images help users tograsp events while texts provide more details re-lated to the events. Thus the images and textcan complement each other, assisting users to gaina more visualized understanding of events (Bianet al., 2013). We give an example in Fig. 1 to il-lustrate this phenomenon. For the output with onlythe text summary, user will be confused about thedescription of “four-legged creatures”; while witha relevant image, user will have a clearer under-standing of the text.

In recent years, some researchers(Bian et al.,2013, 2015; Wang et al., 2016) focus on incorpo-rating multimedia contents into the output of sum-marization which all treat the image-text pair asa basic summarization unit. But in our work, ourinput comes from a document and a collection ofimages where there is no alignment between textsand images. So our biggest challenge is how tobridge the semantic gaps between texts and im-ages. Based on the above discussion, in this work,we propose a novel task which we refer to as Mul-timodal Summarization with Multimodal Output(MSMO). To explore this task, we focus on the

simplicity, we first consider only one image) and a piece oftext. We leave the other multimodal content (like videos) asfuture work.

4155

following three questions: 1) how to acquire therelevant data; 2) how to generate the multimodaloutput; 3) how to automatically evaluate the qual-ity of the multimodal output in MSMO.

For the first question, similar to Hermannet al. (2015), we collect a large-scale multimodaldataset2 from Daily Mail website and annotatesome pictorial summaries. For the second ques-tion, we propose a multimodal attention modelto jointly generate text and the most relevant im-age, in which the importance of images is deter-mined by the visual coverage vector. For the lastquestion, we construct a novel multimodal auto-matic evaluation (MMAE) which jointly considerssalience of text, salience of image, and image-textrelevance.

Our main contributions are as follows:

• We present a novel multimodal summariza-tion task, which takes the news with imagesas input, and finally outputs a pictorial sum-mary. We construct a large-scale corpus forMSMO studying.

• We propose an abstractive multimodal sum-marization model to jointly generate sum-mary and the most relevant image.

• We propose a multimodal automatic evalua-tion (MMAE) method which mainly consid-ers three aspects: salience of text, salience ofimage, and relevance between text and im-age.

2 Our Models

2.1 OverviewWe begin by defining the MSMO task. The in-put of the task is a document and a collectionof images and the output is a pictorial summary.As shown in Fig. 2, our proposed model con-sists of four modules: text encoder, image en-coder, multimodal attention layer, and summarydecoder. The text encoder is a BiLSTM used toencode text. Our image encoder is VGG193 pre-trained on ImageNet (Simonyan and Zisserman,2015) used to extract global or local features. Themultimodal attention layer aims to fuse textual andvisual information during decoding. Our summary

2Our dataset has been released to the public, whichcan be found in http://www.nlpr.ia.ac.cn/cip/jjzhang.htm.

3http://www.robots.ox.ac.uk/˜vgg/research/very_deep

decoder, which is a unidirectional LSTM, makesuse of information from two modalities to gener-ate the text summary and select the most relevantimage according to visual coverage vector. Ourtext encoder and summary decoder are based onpointer-generator network which we will describein Sec. 2.2. We then describe image encoder andmultimodal attention layer in our multimodal at-tention model (Sec. 2.3).

2.2 Pointer-Generator Network

See et al. (2017) propose a pointer-generator net-work which allows both copying words from thesource text and generating words from a fixedvocabulary, achieving the best performance onCNN/Daily mail dataset. Their model consistsof an encoder (a single-layer bidirectional LSTM)and an attentive decoder (a unidirectional LSTM).The encoder maps the article to a sequence ofencoder hidden states hi. During decoding, thedecoder receives the embedding of the previousword and reaches a new decoder state st. Thenthe context vector ct is computed by the attentionmechanism (Bahdanau et al., 2015; Luong et al.,2015) as calculated in Eq. 1 and 2. To alleviatethe problem of repetition, See et al. (2017) main-tain a coverage vector covt, which is the sum ofattention distributions over all previous decodingtimesteps (initialized to zero vector at timestep 0):covt =

∑t−1t=0

αt. The coverage vector is used asan extra input to the attention vector (Eq. 1) andis also used to calculate the coverage loss (Eq. 6).Next, the attention distribution is used to calculatethe context vector as follows.

eti = vT tanh(Whhi + Wsst + Wccovt) (1)

αt = softmax(et) (2)

ct =∑i

αtihi (3)

The important part in this model is the calcula-tion of the generation probability pg. It representsthe probability of generating a word from the vo-cabulary distribution pv, and (1 − pg) representsthe probability of copying a word from the sourceby sampling from the attention distribution αt. pgis determined by ct, st, and the decoder input xt inEq. 4. The final probability distribution over theextended vocabulary, which denotes the union ofthe vocabulary and all words in the source, is cal-culated in Eq. 5. Finally, the loss for timestep t isthe sum of the negative log likelihood of the target

4156

Researchers have discovered the fossilized ...

Textual Attention Layer

𝑐"#"

…

Visual Attention Layer

𝑐$%&Multimodal Attention

Decoder

'Tiny' was one of the first four- legged creatures to move onto land …

Visual Coverage

Text Encoder Image Encoder

Final Output

𝑐%%

Image

features

𝛼"#" 𝛼$%&

Figure 2: The framework of our model.

word w∗t and the coverage loss (Eq. 6):

pg = σ(W∗hct + W∗sst + Wxxt) (4)

pw = pgpv(w) + (1− pg)∑wi=w

αti (5)

Lt = −logpw∗t+∑i

min(αti, cov

ti) (6)

2.3 Multimodal Attention ModelWe incorporate visual information into thepointer-generator network and propose a novelmultimodal attention model. As shown in Fig. 2,there are three main differences between ourmodel and pointer-generator network: 1) We havean extra image encoder and a corresponding visualattention layer; 2) To achieve the fusion of textualand visual information, we introduce a multimodalattention mechanism; 3) We add a visual cover-age (Li et al., 2018a) to both alleviate visual rep-etition and measure the salience of image. Moredetails are as follows.

Image Encoder. We apply the VGG19 to ex-tract global and local image feature vectors forall images. The global features g are 4096-dimensional activations of the pre-softmax fully-connected layer fc7. The local features l are the7× 7× 512 feature maps of the last pooling layer(pool5). We flatten the local feature into a ma-trix A = (a1, · · · , aL)(L = 49) where al ∈ R512

corresponds to a patch of an image.Visual Attention. The attention mechanism is

learned to focus on different parts of input textwhile decoding. Attention mechanisms have alsoshown to work with other modalities, like images,where they can learn to attend the salient parts ofan image (Xu et al., 2015). We then explore to useimages with a visual attention to learn text-imagealignment. Concretely, we extend attention mech-anism (Bahdanau et al., 2015; Luong et al., 2015)to visual attention mechanisms, which attend vi-

sual signals. There are three variants of our visualattention mechanisms: 1) attention on global fea-tures (ATG), 2) attention on local features (ATL),and 3) hierarchical visual attention on local fea-tures (HAN). We take the calculation of ATG as anexample. To attend to the salient parts of a collec-tion of images with size M , we flatten the globalfeature set g into a matrix g′ = (g1, · · · , gM ). Inaddition to calculating the text context vector inSec. 2.2, we also obtain a visual context vector.We first project the image feature into the samedimension as the text context vector. The visualattention is calculated as follows:

g∗ = W2I(W

1Ig + b1I) + b2I (7)

eta = vTa tanh(Wag∗i + Uast + covta) (8)

αta = softmax(eta) (9)

where W1I ∈ R4096×4096 and W2

I ∈ R4096×dh

are the image transformation matrices, b1I ∈ R4096

and b2I ∈ Rdh are bias vectors, and covta denotesthe visual coverage vector and is initialized to zerovector in the beginning. Then the visual attentiondistribution αt

a is used to obtain the visual contextvector ctimg through ctimg =

∑i α

ta,ig∗i . Similar

is the ATL, we flatten the local feature set A intoa matrix A′ = (a1, · · · , aM×49). The calculationof attention in ATL is the same as in ATG. Thereis a bit difference in the HAN model, which firstattend to the 49 image patches and get an interme-diate visual context vector to represent the image,and then attend to the intermediate visual contextvectors to get the visual context vector.

Multimodal Attention. To fuse the text and vi-sual context information, we add a multimodal at-tention layer (Li et al., 2018a), as shown in Fig. 2.And the attention distribution is calculated as fol-lows:

ettxt = vTtxt(Wtxtcttxt + Utxtst) (10)

etimg = vTimg(Wimgctimg + Uimgst) (11)

αttxt = softmax(ettxt) (12)

αtimg = softmax(etimg) (13)

ctmm = αttxtc

ttxt + αt

imgctimg (14)

where αttxt is the attention weight for text context

vector and αtimg is the attention weight for visual

context vector.Visual Coverage. In addition to the calculation

of the text coverage vector as in Sec. 2.2, we alsoobtain a visual coverage vector covtimg, which isthe sum of visual attention distributions. To help

4157

reduce repeated attention to multimodal informa-tion, we incorporate a text coverage loss and a vi-sual coverage loss into the loss function. The finalloss function is as follows:

Lt = −logpw∗t+∑i

min(αti, cov

ti)

+∑j

min(αtj , cov

timg,j) (15)

The attention mechanism can attend the salientparts of texts or images. Meanwhile, the coveragemechanism sums up all the historical attention dis-tributions. Therefore, we regard the coverage vec-tor as a global salience measure of the source be-ing attended. We then use the visual coverage vec-tor in the last decoding timestep to select the mostrelevant image. Concretely, we choose the imagewhose coverage score is the largest. The processis a bit different for the local features. An imagecorresponds to 49 patches, the coverage scores ofthese patches are summed up to get the saliencescore of the image as follows:

Sj =∑patch

covt∗patch,j (16)

where Sj denotes the salience of the j-th imageand covt

∗patch,j denotes the coverage score of each

corresponding image patch in the last decodingtimestep t∗. For the HAN, we introduce an ex-tra coverage vector for the image patches attentionand calculate coverage loss for it as follows:

Lt = −logpw∗t+∑k

min(αtk, cov

tpatch,k)

+∑i

min(αti, cov

ti) +

∑j

min(αtj , cov

timg,j)

(17)

3 Multimodal Automatic Evaluation

To evaluate the quality of a pictorial summary,we propose the MMAE method which is definedas y = f(m1,m2,m3). In this definition, m1,m2, and m3 denote scores measured by three met-rics which consider salience of text (Sec. 3.1),salience of image (Sec. 3.2), and image-text rele-vance (Sec. 3.3) respectively, f(·) denotes a map-ping function, and y denotes the score of the pic-torial summary.

In our experiments, the reference pictorial sum-mary consists of a text summary and a reference

image set4 ref img. In MMAE, m1 is obtained bycomparing the text summary in reference with thatin model output, m2 is obtained by comparing theimage set in reference with the image in modeloutput, and m3 considers the image-text similar-ity in model output. To learn MMAE, we choosethree simple methods to fit y with human judg-ment scores. These methods include Linear Re-gression (LR), and two nonlinear methods: Logis-tic Regression (Logis), and Multilayer Perceptron(MLP).

3.1 Salience of Text

ROUGE (Lin, 2004b) is widely used to automat-ically assess the quality of text summarizationsystems. It has been shown that ROUGE cor-relates well with human judgments (Lin, 2004a;Owczarzak et al., 2012; Over and Yen, 2004).Therefore, we directly apply ROUGE to assess thesalience of the text units.

3.2 Salience of Image

We propose a metric, namely, image precision(IP), to measure the salience of image. The im-age precision is defined as follows:

IP =|{ref img} ∩ {recimg}|

|{recimg}|(18)

where refimg, recimg denote reference images andrecommended images by MSMO systems respec-tively. The reasons for this metric are as follows.

A good summary should have good coverageof the events for both texts and images. The im-age in the output should be closely related to theevents. So we formulate the image selection pro-cess as an image recommendation —instead ofrecommending items to users as in a recommen-dation system, we recommend the most salientimage to an event. It can also be viewed as animage retrieval task, which retrieves the imagemost relevant to an event. Precision and recallare commonly used to evaluate recommendationsystems (Karypis, 2001) and information retrievaltask (Zuva and Zuva, 2012). However, we onlycare about whether the image appears in the ref-erence image set. Thus in our case, we are onlyinterested in calculating precision metric. There-fore, we adapt the precision here as IP to measureimage salience.

4More details can be found in Sec. 4.1

4158

3.3 Image-Text RelevanceA prerequisite for a pictorial summary to helpusers accurately acquire information is that the im-age must be related to the text. Therefore, we re-gard the image-text relevance as one of metrics tomeasure the quality of the pictorial summary. Weconsider using visual-semantic embedding (Faghriet al., 2018; Wang et al., 2018) to calculate thecosine similarity between visual feature and tex-tual feature, which we use as image-text relevance.Visual-semantic embedding has been widely usedin cross-modal retrieval (Kiros et al., 2014) andimage captioning (Karpathy and Fei-Fei, 2015).

We apply VSE0 model of Faghri et al. (2018),which achieves state-of-the-art performance forimage-caption retrieval task on the Flickr30Kdataset (Young et al., 2014). The difference isthat instead of training a CNN model to encodethe image, we use the pretrained VGG19 to ex-tract global features. The text is encoded by a uni-directional Gated Recurrent Unit (GRU) to a se-quence of vector representations. Then we applythe max-over-time pooling (Collobert et al., 2011)to get a single vector representation. Next, the vi-sual features and text features are projected to ajoint semantic space by two feed-forward neuralnetworks. The whole network is trained using amax-margin loss:

L =∑c

max(β − s(i, c) + s(i, c), 0)

+∑i

max(β − s(i, c) + s(i, c), 0) (19)

The loss comprises two symmetric terms, with iand c being images and captions repectively. Thefirst term is taken over negative captions c imagei in a batch. The second is over negative images igiven caption c. If i and c are closer to each otherin the joint embedding space than to any other neg-ative pairs, by a margin β, the loss is zero. Wechoose to use image-caption pairs in our dataset totrain the VSE0 model.

4 Experiments

We conduct the following five sets of experiments:1) To verify our motivation of the multimodal out-put (pictorial summary), we design an experimentfor user satisfaction test (Sec. 4.2); 2) We compareour multimodal summarization with text summa-rization from both ROUGE score and manual eval-uation (Sec. 4.3); 3) To verify the effectiveness of

our evaluation metrics, we calculate the correla-tion between these metrics and human judgments(Sec. 4.4); 4) We conduct two experiments to showthe effectiveness of our proposed MMAE and thegeneralization of MMAE respectively (Sec. 4.5);5) Finally, we evaluate our multimodal attentionmodel with MMAE (Sec. 4.6).

The hyperparameters in our model are similar toSee et al. (2017), except that we set the maximumnumber of images to 10, 7, and 7 for ATG, ATL,and HAN respectively, because different articleshave the image collection of different sizes. Theimages are sorted in the order of the position inthe article.

4.1 Dataset

There is no large-scale benchmark dataset forMSMO. We follow Hermann et al. (2015) to con-struct a corpus from Daily Mail website5. Similarto Hermann et al. (2015), we use the manually-written highlights offered by Daily Mail as a ref-erence text summary. From Daily Mail, we ran-domly select articles within a week and find that2,917 out of 2,930 articles contain images. Moredetails are illustrated in Table 1.

train valid test

#Documents 293,965 10,355 10,261#ImgCaps 1,928,356 68,520 71,509#AvgTokens(S) 720.87 766.08 730.80#AvgTokens(R) 70.12 70.02 72.16#AvgCapTokens 22.07 22.64 22.34#AvgImgCaps 6.56 6.62 6.97

Table 1: Corpus statistics. Each image on the websiteis paired with a caption. #ImgCaps denotes the numberof image-caption pairs. #AvgTokens(S), #AvgTokens(R)and #AvgCapTokens denote the average number of to-kens in articles, highlights, and captions respectively.

To get the pictorial reference, we employ 10graduate students to select the relevant imagesfrom the article for each reference text summary.We allow annotators to select up to three imagesto reduce the difference between different annota-tors. If the annotators find that there is no relevantimage, they will select none of them. Each arti-cle is annotated by at least two students6. Sincewe use the text reference to guide the generationof the pictorial summary, we do not use the ref-erence image during training. Therefore, we onlyconduct the annotation on the test set.

5http://www.dailymail.co.uk6A third annotator will be asked to decide the final anno-

tation for the case of divergence for the first two annotators.

4159

4.2 User Satisfaction Test

We conduct an experiment to investigate whethera pictorial summary can improve the user satis-faction for the informativeness of the summary.For a fair comparison, we propose a novel strat-egy to compare text summaries and pictorial sum-maries. We take an example to illustrate our strat-egy. Given 100 source news pages, we have theircorresponding reference text summaries and picto-rial summaries. We divide them into two parts ofthe same size, part 1 and part 2. In part 1, humanannotator A evaluates the text summaries accord-ing to the input news, and human annotator B eval-uates the pictorial summaries. In part 2, annotatorA evaluates the pictorial summaries and annota-tor B evaluates the text summaries. All annotatorswill give a score of 1 to 5. The input news is thesame for annotator A and annotator B.

Format AnnotatorA AnnotatorB Overall

Text 3.67 3.75 3.71Pictorial 4.14 4.20 4.17

Table 2: User satisfaction test results. In total, we usethe strategy mentioned in Section 4.2 to evaluate 400randomly selected source news pages. Overall denotesthe average score on these 400 samples.

Table 2 shows our results for user satisfac-tion test. User ratings of pictorial summaries are12.4% higher than text summaries. It shows thatusers prefer this way of presenting information. Italso confirms our motivation for MSMO.

4.3 Comparison with Text Summarization

Our user satisfaction test in Sec. 4.2 is done inan ideal situation, comparing the text referencewith the pictorial reference. To show the effec-tiveness of our model, we also compare our modelwith text summarization from ROUGE and humanjudgment scores. We compare several abstrac-tive summarization methods with our multimodalsummarization methods. PGC7 (See et al., 2017)refers to the pointer-generator network (Sec. 2.2).AED (Nallapati et al., 2016) uses an attentionalencoder-decoder framework and adds some lin-guistic features such as POS, named-entities, andTF-IDF into the encoder. We also implementa seq2seq model with attention (S2S+attn). Tocompare the multimodal output with our multi-modal model, we propose an extractive method

7https://github.com/abisee/pointer-generator

based on GuideRank (GR) (Li et al., 2016a,2018b). GuideRank applies LexRank (Erkan andRadev, 2004) with guidance strategy. In this strat-egy, captions recommend the sentences related tothem. The rankings of sentences and captions areobtained through GR; we extract sentences thatsatisfy the length limit as a text summary accord-ing to the ranking of text. We select an imagewhose caption ranks the first in the captions. Andfinally, the pictorial summary is obtained. Weevaluate different summarization models with thestandard ROUGE metric, reporting the F1 scoresfor ROUGE-1, ROUGE-2, and ROUGE-L. OurROUGE results are given in Table 3, and humanjudgment scores are given in Table 4.

Model ROUGE-1 ROUGE-2 ROUGE-L

S2S+attn 32.32 12.44 29.65Base AED 34.78 13.10 32.24

PGC 41.11 18.31 37.74

ATG 40.63 18.12 37.53

MM ATL 40.86 18.27 37.75HAN 40.82 18.30 37.70GR 37.13 15.03 30.21

Table 3: ROUGE F1 scores on our test set. All ourROUGE scores are reported by official ROUGE script.

Model PGC ATG ATL HAN

HS 3.07 3.30 3.22 3.20

Table 4: Human judgment scores for our multimodalmodel and PGC. We randomly select 400 articles anduse the same strategy as Sec. 4.2. HS denotes the aver-age human judgment scores.

From Table 3, all multimodal models lead toa decrease in ROUGE scores which can attributeto the following reasons. There are 6.56 imageson average in each article and not every image isclosely related to the event of the article. In otherwords, some images are noise. On the other hand,our text input is long text, and it contains enoughinformation for text generation. In Table 4, multi-modal models are better than text model in humanjudgments. It further illustrates our motivation,and also proves the effectiveness of our models.

4.4 Correlation Test

To illustrate the effectiveness of our evaluationmetrics, we conduct an experiment on correla-tions between these metrics and human judgmentscores. Human annotators give a score whichranges from 1 to 5 to a pictorial summary accord-

4160

ing to the reference8. The reference consists of atext summary and up to three relevant images se-lected by humans. We randomly extract the pic-torial summaries from the output of different sys-tems. In response to the three aspects we proposedin Section 3, we propose some related metrics re-spectively. For text salience, we apply ROUGE-1,ROUGE-2, ROUGE-L, and BLEU. For image-text relevance of candidate pictorial summaries,we propose two ways. One is to calculate thesimilarity (Img-Sum) between the image and thewhole text summary. The other is to calculatethe similarities between the image and each sen-tence in the text summary. Then we take the maxi-mum and average values as two metrics: MAXsimand AVGsim. For image salience, in addition tothe IP metric mentioned in Section 3.2, we try tocalculate the similarity between the candidate im-age and each reference image in three ways: 1)I-I: similarities between the global fc7 features,2) Hist: Bhattacharyya distance9 for histogramcomparison (Bhattacharyya, 1943), and 3) Temp:Fourier Analysis template matching (Briechle andHanebeck, 2001).

We employ annotators to evaluate 600 samples(randomly selected from the outputs of each modelon the validation set). Each sample is scored bytwo persons and we take the average score as thefinal score. We use 450 of them as training set totrain the MMAE model in Sec. 4.5, the rest is usedas test set. The scores calculated by each evalua-tion metric are then tested on the training set to seehow well they correlate with human judgments.The correlation is evaluated with three metrics, in-cluding 1) Pearson correlation coefficient (r), 2)Spearman rank coefficient (ρ), and 3) Kendall rankcoefficient (τ ). Our results of correlation test aregiven in Table 5.

As shown in Table 5, IP (Image Precision) cor-relates best with human assessments according tothe three correlation coefficients. It illustrates thatpeople pay more attention to images when assess-ing pictorial summaries. If we choose the rightimage for the summary, people are more likely toassign a high score. We also note that the corre-lation score of IP is significantly higher than four

8Some articles are annotated with no relevant images(about 3.9%), we directly skipped these articles without man-ual scoring

9In statistics, the Bhattacharyya distance measures thesimilarity of two discrete or continuous probability distribu-tions. For a distance d, we take (1− d) as the similarity.

Metric r ρ τ

BLEU .1949 .1542 .1198

Text ROUGE-1 .3006 .2941 .2152ROUGE-2 .2735 .2742 .2002ROUGE-L .3144 .3087 .2272

AVGsim .2662 .2388 .1774Image-Text MAXsim .2849 .2749 .2033

Img-Sum .2380 .2075 .1556

I-I (max) .0169 .0258 .0196I-I (avg) -.0262 -.0140 -.0113Histavg .4688 .5077 .3725

Image Histmax .5974 .6388 .5149Tempavg .4913 .4944 .3631Tempmax .5967 .6435 .5080IP .6407 .6482 .5789

Table 5: Correlation with human judgment scores(training set), measured with Pearson r, Spearman ρ,and Kendall τ coefficients. The max and avg denotethe maximum and average value of the scores.

text metrics. Because it is easy for a person tojudge the importance of images based on refer-ence, such as to see whether the image appears inreference. However, measuring the semantic sim-ilarity of two texts is difficult. The four metrics allmeasure the degree of n-gram overlap which can-not accurately measure semantic similarity.

For the image-text relevance, MAXsim performsbest and is comparable to the several ROUGE met-rics. It shows that in a good pictorial summary,the image and text should be relevant. In somecases, even though the generated text is not so im-portant, the image is closely related to the text.At this time, people can also be satisfied. On theother hand, our VSE0 (Sec. 3.3) model can cap-ture some fluency of sentences by adopting GRU.Compare MAXsim, AVGsim, and Img-Sum, thisis very intuitive. Once people find a sentence (ora part) relevant to the image, they will think theimage is related to the text. Besides, the worstperformance of Img-Sum metric is probably be-cause the average length of captions used to trainVSE0 model is about 22, far less than the lengthof the summary. We find the I-I (max) and I-I(avg) nearly do not correlate with human assess-ments. It shows that the visual features extractedfrom VGG19 are not suitable for calculating thesimilarity between news images. The analysis ofHist (Temp)avg and Hist (Temp)max is similar tothe analysis of MAXsim and AVGsim above.

4.5 Effectiveness and Generalization ofMMAE

We then select the best-performing metrics sep-arately from the three sets of metrics, namely

4161

ROUGE-L, MAXsim, and IP. We apply LR, MLP,and Logis to learn our MMAE model that com-bines the three metrics. We calculate the three co-efficients for the three metrics on the test set as acomparison. The correlation results are given inTable 6.

Metric r ρ τ

ROUGE-L .3488 .3554 .2669MAXsim .2541 .2339 .1773IP .5982 .5966 .5485

MMAELR .6646 .6644 .5265MMAEMLP .6632 .6646 .5265MMAELogis .6630 .6653 .5277

Table 6: Correlation with human judgment scores (testset).

As shown in Table 6, the MMAE learned bythree methods correlates better with human judg-ments. Although MMAELogis gets a slightlyhigher correlation score according to Spear-man and Kendall coefficients, we choose theMMAELR

10 as our final MMAE model due to Oc-cam’s Razor11.

It is crucial that MMAE can generalize for apreviously unseen system. To test the generaliza-tion of MMAE, we use MMAE to evaluate a newsystem and calculate the correlation with humanjudgment scores. The new system is a naive modelwhich applies LexRank to extract sentences andrandomly select an image from source. We can ob-serve that MMAE still correlates well with humanjudgment scores, as shown in Table 7. It illustratesthat MMAE generalize well for a new model. Wegive some examples of MMAE in supplementarymaterial.

Metric r ρ τ

ROUGE-L .3223 .3514 .2615MMAE .6352 .6318 .4728

Table 7: Correlation results for the new model on thesame 150 test samples as in Sec. 4.4.

4.6 Model Performances

According to our analyses above, we have provedMMAE can evaluate multimodal output. In thissection, we report the MMAE scores for our pro-posed multimodal attention model, as shown inTable 8.

10The weight for ROUGE-L, MAXsim, and IP is 1.641,0.854, 0.806 respectively and the intercept is 1.978.

11https://en.wikipedia.org/wiki/Occam%27s_razor

Model ROUGE-L MAXsim IP MMAE

ATG 40.76 25.82 59.28 3.35ATL 40.80 13.26 62.44 3.26HAN 40.82 12.22 61.83 3.25GR 30.20 26.60 61.70 3.20

Table 8: Results evaluated by our MMAE method. Weskipped the articles that are labeled as no relevant im-ages. Finally, only 9,851 of the 10,261 articles are left.

Surprisingly, the model ATG achieves the high-est MMAE score despite the mediocre perfor-mance in three individual metrics. The MAXsimscore of ATG is much higher than ATL and HAN.It shows the global features can help to learn bet-ter image-text alignments. Since GR itself makesuse of the image-caption pairs, it is natural toget a high image-text relevance score. Our pro-posed multimodal attention models all achieveshigher performance than the extractive baselineGR, which further indicate the effectiveness of ourmodels.

5 Related Work

Different from text summarization (Wan andYang, 2006; Rush et al., 2015; Zhu et al., 2017;See et al., 2017; Celikyilmaz et al., 2018; Pauluset al., 2018), Multimodal Summarization is a taskto generate a condensed text summary or a fewkeyframes to help acquire the gist of multimediadata. One of the most significant advantages of thetask is that it does not rely solely on text informa-tion, but it can also utilize the rich visual contentfrom the images.

In recent years, much work has focused on mul-timodal summarization. Evangelopoulos et al.(2013) detect salient events in a movie based onthe saliency of individual features for aural, visual,and linguistic representations. Li et al. (2017)generate the text summary from an asynchronouscollection of text, image, audio, and video. Therehas also been some work (Bian et al., 2013, 2015;Wang et al., 2016; Qian et al., 2016) focused onproducing multimodal output for summarization.Bian et al. (2013, 2015) aim to produce a visual-ized summary for microblogs. Wang et al. (2016)generate a pictorial storyline for summarization.Qian et al. (2016) generate the multimedia topicsfor social events. But these researches all treatimage-text pairs, in which texts and images arealigned, as a basic summarization unit. For exam-ple, the images are aligned with the text in a mi-

4162

croblog post; Wang et al. (2016) obtain the image-text pairs by using image search engine. None ofthe above works focuses on generating multimodaloutput from a collection of texts and images thatare not explicitly aligned. This is one of the goalsin this paper. Another difference is that they sep-arately evaluate texts and images when evaluatingthe final results. In our work, we propose a newautomatic evaluation which jointly considers twoaspects of textual and visual modalities.

6 Conclusion

In this paper, we focus on a novel task which aimsto automatically generate a multimodal summaryfrom multimodal news, where the images and thetexts are not explicitly aligned. We provide a mul-timodal summarization method to jointly gener-ate text and the most relevant image, which canbe referred as the baseline for further study. Ourproposed metrics have been proved to be effectivein evaluating the multimodal output. Moreover,the idea of constructing our MMAE can be eas-ily extended to other modalities. That is, we bothconsider the intramodality salience and the inter-modality relevance.

7 Acknowledgments

The research work described in this paper hasbeen supported by the National Key Research andDevelopment Program of China under Grant No.2016QY02D0303. We thank the three anonymousreviewers for their careful reading of our paper andtheir many insightful comments and suggestions.We also thank Shaonan Wang, Xiaomian Kang,and Yang Zhao for helpful discussions.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In Proceedings ofthe International Conference on Learning Represen-tations (ICLR).

Anil Bhattacharyya. 1943. On a measure of divergencebetween two statistical populations defined by theirprobability distributions. Bull. Calcutta Math. Soc.,35:99–109.

Jingwen Bian, Yang Yang, and Tat-Seng Chua. 2013.Multimedia summarization for trending topics in mi-croblogs. In Proceedings of the 22nd ACM interna-tional conference on Conference on information &knowledge management, pages 1807–1812.

Jingwen Bian, Yang Yang, Hanwang Zhang, and Tat-Seng Chua. 2015. Multimedia summarization forsocial events in microblog stream. IEEE Transac-tions on multimedia, 17(2):216–228.

Kai Briechle and Uwe D Hanebeck. 2001. Templatematching using fast normalized cross correlation.In Optical Pattern Recognition XII, volume 4387,pages 95–103.

Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, andYejin Choi. 2018. Deep communicating agents forabstractive summarization. In Proceedings of the2018 Conference of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies (NAACL-HLT), pages1662–1675.

Ronan Collobert, Jason Weston, Michael Karlen, Ko-ray Kavukcuoglu, and Pavel Kuksa. 2011. Nat-ural language processing (almost) from scratch.Journal of Machine Learning Research (JMLR),12(1):2493–2537.

Gunes Erkan and Dragomir R Radev. 2004. Lexrank:Graph-based lexical centrality as salience in textsummarization. Journal of Artificial IntelligenceResearch, 22:457–479.

Georgios Evangelopoulos, Athanasia Zlatintsi, Alexan-dros Potamianos, Petros Maragos, Konstantinos Ra-pantzikos, Georgios Skoumas, and Yannis Avrithis.2013. Multimodal saliency and fusion for moviesummarization based on aural, visual, and tex-tual attention. IEEE Transactions on Multimedia,15(7):1553–1568.

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, andSanja Fidler. 2018. Vse++: Improving visual-semantic embeddings with hard negatives. In Pro-ceedings of the British Machine Vision Conference(BMVC).

Karl Moritz Hermann, Tomas Kocisky, EdwardGrefenstette, Lasse Espeholt, Will Kay, Mustafa Su-leyman, and Phil Blunsom. 2015. Teaching ma-chines to read and comprehend. In Proceedingsof Neural Information Processing Systems (NIPS),pages 1693–1701.

Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descrip-tions. In Proceedings of the IEEE conference oncomputer vision and pattern recognition (CVPR),pages 3128–3137.

George Karypis. 2001. Evaluation of item-based top-nrecommendation algorithms. In Proceedings of thetenth international conference on Information andknowledge management, pages 247–254.

Ryan Kiros, Ruslan Salakhutdinov, and Richard SZemel. 2014. Unifying visual-semantic embeddingswith multimodal neural language models. arXivpreprint arXiv:1411.2539.

4163

Haoran Li, Jiajun Zhang, Yu Zhou, and ChengqingZong. 2016a. Guiderank: A guided ranking graphmodel for multilingual multi-document summariza-tion. In The Fifth Conference on Natural LanguageProcessing and Chinese Computing & The TwentyFourth International Conference on Computer Pro-cessing of Oriental Languages (NLPCC-ICCPOL),pages 608–620.

Haoran Li, Junnan Zhu, Tianshang Liu, Jiajun Zhang,and Chengqing Zong. 2018a. Multi-modal sentencesummarization with modality attention and imagefiltering. In Proceedings of the Twenty-Fifth Inter-national Joint Conference on Artificial Intelligence(IJCAI), pages 4152–4158.

Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, andChengqing Zong. 2017. Multi-modal summariza-tion for asynchronous collection of text, image, au-dio and video. In Proceedings of the Conference onEmpirical Methods in Natural Language Processing(EMNLP), pages 1092–1102.

Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, andChengqing Zong. 2018b. Read, watch, listen andsummarize: Multi-modal summarization for asyn-chronous text, image, audio and video. IEEETransactions on Knowledge and Data Engineering(TKDE).

Zechao Li, Jinhui Tang, Xueming Wang, Jing Liu, andHanqing Lu. 2016b. Multimedia news summariza-tion in search. ACM Transactions on Intelligent Sys-tems and Technology (TIST), 7(3):33.

Chin-Yew Lin. 2004a. Looking for a few goodmetrics: Automatic summarization evaluation-howmany samples are enough? In Proceedings of NIITestbeds and Community for information access Re-search (NTCIR).

Chin-Yew Lin. 2004b. Rouge: A package for auto-matic evaluation of summaries. Text SummarizationBranches Out.

Thang Luong, Hieu Pham, and Christopher D. Man-ning. 2015. Effective approaches to attention-basedneural machine translation. In Proceedings of theConference on Empirical Methods in Natural Lan-guage Processing (EMNLP), pages 1412–1421.

Ioannis Mademlis, Anastasios Tefas, Nikos Nikolaidis,and Ioannis Pitas. 2016. Multimodal stereoscopicmovie summarization conforming to narrative char-acteristics. IEEE Transactions on Image Process-ing, 25(12):5828–5840.

Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre,Bing Xiang, et al. 2016. Abstractive text summa-rization using sequence-to-sequence rnns and be-yond. In Proceedings of the 20th SIGNLL Confer-ence on Computational Natural Language Learning(CoNLL).

Paul Over and James Yen. 2004. An introduction toduc 2004 intrinsic evaluation of generic new textsummarization systems. In Proceedings of the Doc-ument Understanding Conference (DUC).

Karolina Owczarzak, John M Conroy, Hoa TrangDang, and Ani Nenkova. 2012. An assessment ofthe accuracy of automatic evaluation in summariza-tion. In Proceedings of Workshop on EvaluationMetrics and System Comparison for Automatic Sum-marization, pages 1–9.

Romain Paulus, Caiming Xiong, and Richard Socher.2018. A deep reinforced model for abstractive sum-marization. In Proceedings of the InternationalConference on Learning Representations (ICLR).

Shengsheng Qian, Tianzhu Zhang, and ChangshengXu. 2016. Multi-modal multi-view topic-opinionmining for social event analysis. In Proceedings ofthe 2016 ACM on Multimedia Conference, pages 2–11.

Alexander M. Rush, Sumit Chopra, and Jason Weston.2015. A neural attention model for abstractive sen-tence summarization. In Proceedings of the Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 379–389.

Abigail See, Peter J. Liu, and Christopher D. Manning.2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the AnnualMeeting of the Association for Computational Lin-guistics (ACL), pages 1073–1083.

Rajiv Ratn Shah, Yi Yu, Akshay Verma, Suhua Tang,Anwar Dilawar Shaikh, and Roger Zimmermann.2016. Leveraging multimodal information for eventsummarization and concept-level sentiment analy-sis. Knowledge-Based Systems, 108:102–109.

Karen Simonyan and Andrew Zisserman. 2015. Verydeep convolutional networks for large-scale imagerecognition. In Proceedings of the InternationalConference on Learning Representations (ICLR).

Xiaojun Wan and Jianwu Yang. 2006. Improved affin-ity graph based multi-document summarization. InProceedings of the Human Language TechnologyConference of the NAACL, pages 181–184.

Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazeb-nik. 2018. Learning two-branch neural networks forimage-text matching tasks. IEEE Transactions onPattern Analysis and Machine Intelligence (TPAMI).

William Yang Wang, Yashar Mehdad, Dragomir RRadev, and Amanda Stent. 2016. A low-rank ap-proximation approach to learning joint embeddingsof news stories and images for timeline summariza-tion. In Proceedings of the 2016 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies (NAACL-HLT), pages 58–68.

4164

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,Aaron Courville, Ruslan Salakhudinov, Rich Zemel,and Yoshua Bengio. 2015. Show, attend and tell:Neural image caption generation with visual at-tention. In International Conference on MachineLearning (ICML), pages 2048–2057.

Peter Young, Alice Lai, Micah Hodosh, and JuliaHockenmaier. 2014. From image descriptions tovisual denotations: New similarity metrics for se-mantic inference over event descriptions. Transac-tions of the Association for Computational Linguis-tics (TACL), 2:67–78.

Junnan Zhu, Long Zhou, Haoran Li, Jiajun Zhang,Yu Zhou, and Chengqing Zong. 2017. Augment-ing neural sentence summarization through extrac-tive summarization. In Proceedings of the 6th Con-ference on Natural Language Processing and Chi-nese Computing (NLPCC), pages 16–28.

Keneilwe Zuva and Tranos Zuva. 2012. Evaluation ofinformation retrieval systems. International Jour-nal of Computer Science & Information Technology,4(3):35.