An Empirical Study of Spatial Attention Mechanisms in Deep Networks

Xizhou Zhu1,3†∗ Dazhi Cheng2,3†∗ Zheng Zhang3∗ Stephen Lin3 Jifeng Dai3

1University of Science and Technology of China2Beijing Institute of Technology

3Microsoft Research Asiaezra0408@mail.ustc.edu.cn

v-dachen,zhez,stevelin,jifdai@microsoft.com

Abstract

Attention mechanisms have become a popular compo-nent in deep neural networks, yet there has been little ex-amination of how different influencing factors and meth-ods for computing attention from these factors affect per-formance. Toward a better general understanding of atten-tion mechanisms, we present an empirical study that ab-lates various spatial attention elements within a generalizedattention formulation, encompassing the dominant Trans-former attention as well as the prevalent deformable convo-lution and dynamic convolution modules. Conducted on avariety of applications, the study yields significant findingsabout spatial attention in deep networks, some of which runcounter to conventional understanding. For example, wefind that the comparison of query and key content in Trans-former attention is negligible for self-attention, but vital forencoder-decoder attention. On the other hand, a propercombination of deformable convolution with key contentsaliency achieves the best accuracy-efficiency tradeoff inself-attention. Our results suggest that there exists muchroom for improvement in the design of attention mecha-nisms.

1. Introduction

Attention mechanisms enable a neural network to fo-cus more on relevant elements of the input than on irrel-evant parts. They were first studied in natural languageprocessing (NLP), where encoder-decoder attention mod-ules were developed to facilitate neural machine transla-tion [2, 31, 16]. In computing the output for a given queryelement (e.g., a target word in the output sentence), cer-tain key elements (e.g., source words in the input sentence)

∗Equal contribution. †This work is done when Xizhou Zhu and DazhiCheng are interns at Microsoft Research Asia.

are prioritized according to the query. Later, self-attentionmodules were presented for modeling intra-sentence rela-tions [7, 29, 33, 34, 41], where both the key and queryare from the same set of elements. In a milestone pa-per [41], the Transformer attention module is presented,superseding past works and substantially surpassing theirperformance. The success of attention modeling in NLPhas led to its adoption in computer vision, where differ-ent variants of Transformer attention are applied to recog-nition tasks such as object detection and semantic segmen-tation [22, 43, 19, 24, 51, 15], where the query and key arevisual elements such as image pixels or regions of interest.

In determining the attention weight assigned to a certainkey for a given query, there exist just a few properties of theinput that are commonly considered. One is the content ofthe query. For the case of self-attention, the query contentmay be the features at the query pixel in an image, or of aword in a sentence. Another is the content of the key, wherea key may be a pixel within a local neighborhood of thequery, or another word within the sentence. The third is therelative position of the query and key.

Based on these input properties, there are four possi-ble attention factors from which the attention weight fora key with respect to a query is determined, as these fac-tors must account for information about the key. Specifi-cally, these factors are (1) the query and key content, (2)the query content and relative position, (3) the key con-tent only, and (4) the relative position only. In the latestversion of Transformer attention [11], attention weights areexpressed as a sum of four terms (E1, E2, E3, E4), one foreach of these attention factors as illustrated in Fig. 1. Thenature of the dependencies involved with these terms vary.For example, the first two (E1, E2) are sensitive to the querycontent. While, the latter two (E3, E4) do not account forquery content, but rather they mainly capture salient keyelements and exploit global positional biases, respectively.Although attention weights can be decomposed into terms

1

arX

iv:1

904.

0587

3v1

[cs

.CV

] 1

1 A

pr 2

019

(a) query and key content (b) query content and relative position (c) key content only (d) relative position only

querykey

query key

relative position

query keyrelative position

query key

key

key query keyrelative positionquery key

relative position

Figure 1. Illustration of different attention terms. The color bar above a sampling point denotes its content feature. The existence of contentfeatures and/or relative position indicates that the term uses them for attention weight calculation.

based on these factors, their relative significance in vari-ous inference problems has not been closely examined inthe literature. Moreover, prevalent modules like deformableconvolution [10, 52] and dynamic convolution [44], thoughseemingly orthogonal to Transformer attention, also employmechanisms that focus on certain parts of an input. Whetherthese modules can all be viewed from a unified perspectiveand how their operational mechanisms differ also have notbeen explored.

In this work, we perceive Transformer attention, de-formable convolution, and dynamic convolution modules asvarious instantiations of spatial attention, involving differ-ent subsets of the attention factors and accounting for thesefactors in different ways. Towards disentangling the effectsof different attention factors and mechanisms, we presentan empirical study of spatial attention, in which various el-ements of attention mechanisms are ablated within a gen-eralized attention formulation. This investigation is con-ducted on a variety of applications, namely neural machinetranslation, semantic segmentation, and object detection.From this study, we find that: 1) In the Transformer atten-tion module, the query-sensitive terms, especially the queryand key content term, play a minor role in self-attention.But in encoder-decoder attention, the query and key con-tent term is vital. 2) Though deformable convolution uti-lizes an attention mechanism based only on the query con-tent and relative position term, it operates more effectivelyand efficiently on image recognition than the counterpart inTransformer attention. 3) In self-attention, the factors ofquery content & relative position and key content only arethe most important. A proper combination of deformableconvolution and the key content only term in Transformerattention delivers higher accuracy than that of the Trans-former attention module, with much lower computationaloverhead on image recognition tasks.

The observations made in this paper challenge the con-ventional understanding of current spatial attention mecha-nisms. For example, it is widely believed that their successcan mainly be attributed to query-sensitive attention, espe-cially the query and key content term. This understandingperhaps originates from the initial success of the encoder-decoder attention module in neural machine translation.Thus, in some recent variants [43, 24, 48, 15], like thenon-local block [43] and criss-cross attention module [24],

only the query and key content term is kept, with all theother terms removed. These modules still function wellin self-attention applications, which strengthen this percep-tion. However, our study suggests that this understanding isincorrect. We find that these attention modules with onlyquery-sensitive terms actually perform on par with thosewith only query-irrelevant terms. Our study further sug-gests that this degeneration is likely due to the design ofthe attention modules, rather than an inherent characteris-tic of self-attention, since deformable convolution is foundto exploit query content & relative position effectively andefficiently in image recognition tasks.

This empirical analysis suggests that there is much roomfor improvement in the design of spatial attention mecha-nisms in deep networks. Its findings are used in this pa-per to make some initial headway in this direction, and it ishoped that this study will spur further investigation into theoperational mechanisms used in modeling spatial attention.

2. Related Work

Development and application of attention-based mod-ules. The field of NLP has witnessed steady development ofattention mechanisms in recent years [2, 31, 16, 41, 38, 11].Starting from the introduction of an attention module inneural machine translation [2], various attention factors andweight assignment functions based on these factors havebeen utilized. In [31], the inner product of vectors encod-ing query and key contents is recommended for computingattention weights, and absolute spatial positions are incor-porated as an attention factor. In [16], the weight assign-ment additionally accounts for the inner product of spatialpositions encoded in high-dimensional vectors. The land-mark work of Transformer [41] set a new standard, and itslatest variants use relative positions instead of absolute posi-tions for better generalization ability [38, 11]. In this paper,we conduct the empirical study on the latest instantiation ofTransformer attention [11] from this family of works.

Motivated by their success in NLP tasks [2, 31, 16, 7, 29,33, 34, 41], attention mechanisms have also been employedin computer vision applications such as relational reasoningamong objects [3, 37], image captioning [46], image gener-ation [50, 47], image recognition [22, 43, 19, 24, 51, 15],and video recognition [53, 45]. In vision, the key and

2

query refer to visual elements, but aside from that, mostof these works use a formulation similar to Transformer at-tention. Since the effects of different attention module ele-ments may vary with the target application, we conduct theempirical study on three different tasks that have been in-fluenced greatly by attention modeling, namely neural ma-chine translation in NLP, and object detection and semanticsegmentation in computer vision.

Aside from Transformer attention, there are variants ofconvolution, such as deformable convolution [10, 52] anddynamic convolution [44], that also can be viewed as typesof attention mechanisms which operate on a subset of theattention factors using different attention weight functions.They also are included in the study for examination.

It is worth mentioning a dual form of spatial attention,called channel-wise feature attention [42, 49, 23, 15]. Asdifferent feature channels encode different semantic con-cepts, these works seek to capture the correlations amongthese concepts through activation/deactivation of certainchannels. Meanwhile, in the spatial domain, relationshipsamong elements at different spatial positions are modeled,with the same attention weights on feature channels as-signed to related spatial positions. The development ofchannel-wise feature attention has been focused on cer-tain image recognition tasks, like semantic segmentationand image classification. In this paper, our empiricalstudy specifically examines spatial attention mechanismsdesigned for broad application.

Analysis of spatial attention mechanisms. There ex-ists relatively little analysis of spatial attention mecha-nisms despite their prevalence in deep networks. This re-search has largely been conducted by visualizing or an-alyzing the learned attention weights of a whole atten-tion module on only NLP tasks [17, 40, 18, 25]. Manyworks [17, 40, 18] suggest that attention weight assignmentin encoder-decoder attention plays a role similar to wordalignment in traditional approaches [1, 8, 30, 6]. The im-plicit underlying assumption in these works is that the inputelements accorded high attention weights are responsiblefor the model outputs. However, recent research casts doubton this assumption [25], finding that attention weights donot correlate well with feature importance measures, andthat counterfactual attention weight configurations do notyield corresponding changes in prediction.

In this paper, we conduct the first comprehensive empir-ical study on the elements of spatial attention modules overboth NLP and computer vision tasks. Different attentionfactors and weight assignment functions are carefully dis-entangled, with their effects directly measured by the finalperformance on these tasks.

3. Study of Spatial Attention MechanismsTo facilitate our study, we develop a generalized atten-

tion formulation that is able to represent various moduledesigns. We then show how the dominant attention mecha-nisms can be represented within this formulation, and howablations can be conducted using this formulation with re-spect to different attention module elements.

Generalized attention formulationGiven a query element and a set of key elements, an at-

tention function adaptively aggregates the key contents ac-cording to attention weights that measure the compatibilityof query-key pairs. To allow the model to attend to key con-tents from different representation subspaces and differentpositions, the outputs of multiple attention functions (heads)are linearly aggregated with learnable weights. Let q indexa query element with content zq , and k index a key elementwith content xk. Then the multi-head attention feature yq iscomputed as

yq =

M∑m=1

Wm

[ ∑k∈Ωq

Am(q, k, zq, xk)W ′mxk], (1)

where m indexes the attention head, Ωq specifies thesupporting key region for the query, Am(q, k, zq, xk)denotes the attention weights in the m-th attentionhead, and Wm and W ′m are learnable weights. Usu-ally, the attention weights are normalized within Ωq , as∑

k∈ΩqAm(q, k, zq, xk) = 1.

In encoder-decoder attention, the key and the query arefrom two different sets of elements, where in most applica-tions the two sets of elements need to be properly aligned.For example, in the encoder-decoder attention of neural ma-chine translation, the key and the query elements corre-spond to the words in the input and the output sentences,respectively, where proper alignment is necessary for cor-rect translation. Meanwhile, in self-attention, the key andthe query are from the same set of elements. For exam-ple, both the key and the query are of words in the input oroutput sentence. In such scenarios, the self-attention mech-anism is expected to capture intra-relationships among theelements, and usually the query and the key contents aremodeled by the same set of features, i.e., x = z.

Transformer attentionIn the most recent instantiation of the Transformer at-

tention module [11], the attention weight of each query-keypair is computed as the sum of four terms Ej4j=1 that arebased on different attention factors, as

ATransm (q, k, zq, xk) ∝ exp

( 4∑j=1

Ej), (2)

normalized by∑

k∈ΩqATrans

m (q, k, zq, xk) = 1 where thesupporting key region Ωq spans the key elements (e.g., the

3

attention mechanism spatial properties query content key content relative position complexity

Transformer attention

E1 dense, global X X O(N2sC +NsC

2)E2 dense, global X X O(N2

sC +NsC2)

E3 dense, global X O(NsC2)

E4 dense, global X O(N2sC +NsC

2)

Regular convolution sparse, local X O(NsC2Nk)

Deformable convolution sparse, global X X O(NsC2Nk)

Dynamic convolution sparse, local X X O(NsCNgNk +NsC2)

Table 1. Comparison of different attention mechanisms. Ns denotes number of spatial elements, i.e. width by height for images, andnumber of tokens for text; C denotes representation dimension; Nk denotes kernel size of convolution (Nk = 3 × 3 for images andNk = 3 for text, by default); Ng denotes number of feature groups in dynamic convolution.

whole input sentence). By default, 8 attentional heads areutilized in this paper.

The E1 and E2 terms are sensitive to the query content.The E1 term measures the compatibility of the query andkey content, as E1 = z>q U

>mV

Cmxk, where Um, V C

m arelearnable embedding matrices for the query and key con-tent, respectively. It enables the network to focus moreon the keys compatible with the query in terms of content.A possible outcome is the correspondence between similarquery and key elements, as illustrated in Fig. 1 (a). Forthe E2 term, it is based on the query content and relativeposition, as E2 = z>q U

>mV

RmRk−q , whereRk−q encodes the

relative position k−q by projecting it to a high-dimensionalrepresentation through computing sine and cosine functionsof different wavelengths1 [41]. V R

m is a learnable embed-ding matrix for the encoded relative position Rk−q . Thisterm allows the network to adaptively determine where toassign high attention weights based on the query content. Itmay help to disentangle appearance from spatial transfor-mations in image recognition, as illustrated in Fig. 1 (b).

The E3 and E4 terms are irrelevant to the query content.The E3 term involves key content only, as E3 = u>mV

Cmxk,

where um is a learnable vector. It captures salient key con-tent which should be focused on for the task, and is irrel-evant to the query. An illustration is shown in Fig. 1 (c).As for the E4 term, it involves relative position only, asE4 = v>mV

RmRk−q , where vm is a learnable vector. It cap-

tures global positional bias between the key and query ele-ments, as illustrated in Fig. 1 (d).

It is widely believed that query-sensitive prioritization,especially the query and key content compatibility term E1,is the key to the success of Transformer attention. Thus, insome recent variants [43, 24, 48, 15], only E1 is kept, whilethe other terms are all removed.

In Transformer attention, both Wm and W ′m in Eq. (1)are learnable. W ′m projects the features of xk to a relativelylow dimension for reducing computational overhead, andWm projects the aggregated features back to the same di-mension as yq .

1For 2-d image data, we separately encode the x-axis relative positionRX

k−q and y-axis relative position RYk−q , and concatenate them to be the

final encoding Rk−q = [RXk−q , R

Yk−q ].

Regular and deformable convolutionRegular and deformable convolution can be deemed

as special instantiations of spatial attention mechanisms,where subsets of the attention factors are involved.

In regular convolution, given a query element, a fixednumber of key elements (e.g., 3 × 3) are sampled, accord-ing to predetermined positional offsets with respect to thequery. From the perspective of Eq. (1), the attention weightof regular convolution can be expressed as

Aregularm (q, k) =

1 if k = q + pm

0 else,(3)

where each sampled key element is of a separate attentionhead (e.g., 3×3 regular convolution corresponds to 9 atten-tion heads), and pm denotes the offset for them-th samplingposition. In addition, the weight W ′m in Eq. (1) is fixed asidentity, leaving Wm as learnable. In regular convolution,only relative position is involved, without learnable parame-ters for adapting attention to content. The supporting key re-gion Ωq is restricted to a local window centered at the queryposition and determined by the convolution kernel size.

In deformable convolution [10, 52], learnable offsets areadded to adjust the sampling positions of the key elements,so as to capture spatial transformations. The learnable off-sets are predicted based on the query content, and are thusdynamic to the input. The key and the query elements arefrom the same set. It can also be incorporated into the gen-eralized attention formulation as a special instantiation ofself-attention, where the attention weight is

Adeformm (q, k, xq) = G(k, q + pm + w>mxq), (4)

where pm also denotes a predetermined offset, and w>mxqprojects the query content xq to a deformation offset ac-cording to a learnable vector wm

2. G(a, b) is the bilin-ear interpolation kernel in N -d space, which can be de-composed into 1-d bilinear interpolations as G(a, b) =∏N

n=1 g(an, bn), where an and bn denote the n-th dimen-sion of a and b respectively, and g(an, bn) = max(0, 1 −

2Following [10], the learning rate of wm is set to 0.1 times that of otherparameters to stabilize training.

4

|an − bn|). Similar to regular convolution, the weight W ′min Eq. (1) is fixed as identity.

In deformable convolution, the attention factors arequery content and relative position. The supporting keyregion Ωq can span over all the input elements due to theintroduced learnable offsets, while non-zero weights are as-signed to a sparse set of key elements where bilinear inter-polation is performed.

Dynamic convolutionDynamic convolution [44] is recently proposed to re-

place the Transformer attention module in self-attention,and is claimed to be simpler and more efficient. It is builtupon depth-wise separable convolution [21] with shared dy-namic kernel weights, which are predicted based on thequery content. In depth-wise separable convolution, a stan-dard convolution is factorized into a depth-wise convolutionand a 1×1 convolution called a point-wise convolution, forreducing computation and model size. In depth-wise convo-lution, a single filter is applied to each input channel, whichis fixed for all positions. In dynamic convolution, the kernelweights for the depth-wise convolution are dynamically pre-dicted from the input features, followed by a Softmax nor-malization. For computational savings, the input channelsare divided into several groups, where each group shares thesame dynamic kernel weights. In the system of [44], an or-thogonal module called the gated linear unit (GLU) [12] isapplied before the dynamic convolution module to improveaccuracy. We include the GLU to respect the original de-sign.

Dynamic convolution can also be incorporated into thegeneral attention formulation in Eq. (1) with minor modi-fications, where each input feature channel is of a separateattention head. It can be expressed as

yq =

Cin∑c=1

Wc

[ ∑k∈Ωq

Adynamicc (q, k, xq) · xk,c

], (5)

where c enumerates the channels of the input features (Cinchannels in total), xk,c denotes the feature value at the c-thchannel of xk, and Wc is of the 1 × 1 point-wise convolu-tion. Adynamic

c (q, k, xq) is the attention weight specified bythe dynamic kernel in depth-wise convolution, written as

Adynamicc (q, k, xq) =

Kj,c if k = q + pj

0 else,(6)

where pj denotes the j-th sampling position in the dynamickernel, and Kj,c is the corresponding kernel weight. Zeroattention weight is assigned to keys outside of the kernel.The kernel weight Kj,c is predicted from the input features,and is shared among channels in the same group, as

Kj,c = Ksharej,g ∝ exp

(d>j,gxq

), g = d c

Cin/Nge. (7)

The input features are divided into Ng groups (Ng = 16 bydefault). Kshare

j,g denotes the dynamic kernel weight for theg-th group, and dj,g is the corresponding learnable weightvector. Kshare

j,g is normalized by∑Nk

j=1Ksharej,g = 1, where

Nk denotes the number of elements in the dynamic kernel.In dynamic convolution, attention assignment is based

on the query content and relative position factor. The sup-porting key region Ωq is restricted to a local window aroundthe query position covered by the dynamic kernel.

Comparing attention mechanismsTab. 1 compares the three attention mechanisms dis-

cussed above. Transformer attention exploits comprehen-sive content and position information from both query andkey. The E1, E2 and E4 terms require computation propor-tional to the product of the query and key element num-bers, because they involve a traversal of each query-keypair. The E3 term captures key content only, and thus in-volves computation linear to the key element number. Inneural machine translation, the key and query elements arecommonly dozens of words in a sentence, so the computa-tional overheads of E1, E2 and E4 are comparable to E3. Inimage recognition, the key and query elements consist ofnumerous pixels in an image. The computational overheadsof E1, E2 and E4 are thus much heavier than E3. Note thatwhen the four terms are put together, some computationaloverhead can be shared among them.

Similar to the E2 term, deformable convolution also isbased on query content and relative position. But de-formable convolution samples just a sparse set of key el-ements for each query, and the complexity is linear to thequery element number. Deformable convolution is thusmuch faster to compute than E2 for image recognition, andis comparable in speed to E2 for machine translation.

Dynamic convolution also relies on query content andrelative position. The attention weights of key elements areassigned by the dynamic convolution kernel, based on thequery content. Non-zero attention weights only exist in alocal range covered by the dynamic kernel. The computa-tional overhead is proportional to the product of the kernelsize and query element number. Compared to the E2 term,the computational overhead can be considerably lower if thekernel size is much smaller than the key element number.

We seek to further disentangle the effects of different at-tention factors, and to facilitate comparison to other instan-tiations of spatial attention that use a subset of the factors.Thus, manual switches are introduced into the Transformerattention module, which enable us to manually activate / de-activate particular terms. This is expressed as

ATransm (q, k, zq, xk) ∝ exp

( 4∑j=1

βTransj Ej

), (8)

where βTransj takes values in 0, 1 to control the activa-

5

tion of corresponding terms, and ATransm (q, k, zq, xk) is nor-

malized by∑

k∈ΩqATrans

m (q, k, zq, xk) = 1.

Incorporating attention modules into deep networksWe incorporate various attention mechanisms into deep

networks to study their effects. There are different designchoices in inserting the modules, e.g., whether to connectthem in series or in parallel, and where to place the mod-ules in the backbone network. We empirically observed theresults to be quite similar for different well-considered de-signs. In this paper, we select the design choices in Fig. 2.

For the object detection and semantic segmentationtasks, ResNet-50 [20] is chosen as the backbone and justthe self-attention mechanism is involved. The Transformerattention module is incorporated by applying it on the 3× 3convolution output in the residual block. For insertion intoa pre-trained model without breaking the initial behavior,the Transformer attention module includes a residual con-nection, and its output is multiplied by a learnable scalarinitialized to zero, as in [43]. The manner of incorporatingdynamic convolution is the same. To exploit deformableconvolution, the 3 × 3 regular convolution in the residualblock is replaced by its deformable counterpart. The result-ing architecture is called “Attended Residual Block”, shownin Fig. 2 (a).

In the neuron machine translation (NMT) task, the net-work architecture follows the Transformer base model [41],where both self-attention and encoder-decoder attentionmechanisms are involved. Different from the original paper,we update the absolute position embedding in the Trans-former attention module by the latest relative position ver-sion [11] as in Eq. 2. Because both deformable convolutionand dynamic convolution capture self-attention, they areadded to only the blocks capturing self-attention in Trans-former. For dynamic convolution, we replace the Trans-former attention module by dynamic convolution directly,as in [44]. The architecture is shown in Fig. 2 (b). Forits deformable convolution counterpart, because the Trans-former model does not utilize any spatial convolution (withkernel size larger than 1), we insert the deformable convo-lution unit (with kernel size of 3) prior to the input of theTransformer attention module. The resulting architecture iscalled “Transformer + Deformable”, shown in Fig. 2 (c).

4. Experiments and Analysis4.1. Experimental settings

Image Object DetectionModels are trained on the 118k images of the COCO

2017 [28] train set. Evaluation is done on the 5k imagesof the COCO 2017 validation set. Accuracy is measured bythe standard mean AP scores at different box IoUs (mAP).

Faster R-CNN [36] with Feature Pyramid Networks(FPN) [27] is chosen as the baseline system. ImageNet [13]

pre-trained ResNet-50 is utilized as the backbone. Theattended residual blocks in Fig. 2 (a) are applied in thelast two stages (conv4 and conv5 stages) of ResNet-50.In Transformer attention, the relative position encoding isof the same dimension as the content feature embedding,specifically 256-d and 512-d in the conv4 and conv5 stages,respectively.

Experiments are implemented based on the open sourcemmdetection [5] code base. The hyper-parameter settingstrictly follows FPN [27]. Anchors of 5 scales and 3 as-pect ratios are utilized. 2k and 1k region proposals are gen-erated at a non-maximum suppression threshold of 0.7 attraining and inference respectively. In SGD training, 256anchor boxes (of positive-negative ratio 1:1) and 512 regionproposals (of positive-negative ratio 1:3) are sampled forbackpropagating their gradients. In our experiments, thenetworks are trained on 8 GPUs with 2 images per GPU for12 epochs. The learning rate is initialized to 0.02 and is di-vided by 10 at the 8-th and the 11-th epochs. The weightdecay and the momentum parameters are set to 10−4 and0.9, respectively.

Image Semantic SegmentationModels are trained on the 5,000 finely annotated images

of the Cityscapes [9] train set. Evaluation is done on the 500images of the validation set. The standard mean IoU score(mIoU) is used to measure semantic segmentation accuracy.

The CCNet [24] for semantic segmentation is utilized,with ImageNet pre-trained ResNet-50 and without the criss-cross attention module proposed in [24], which is a variantof Transformer attention. As done for object detection, theattended residual blocks in Fig. 2 (a) are applied in the lasttwo stages. An additional Transformer attention / dynamicconvolution module is placed after the ResNet-50 outputfollowing the practice in [24] for improving performance.

The hyper-parameter setting strictly follows that in theCCNet paper [24]. In SGD training, the training images areaugmented by randomly scaling (from 0.7 to 2.0), randomlycropping (size of 769 × 769 pixels) and random flippinghorizontally. In our experiments, the networks are trainedon 8 GPUs with 1 image per GPU for 60k iterations. The“poly” learning rate policy is employed, where the initiallearning rate is set as 0.005 and multiplied by

(1− iter

itermax

)0.9.

Synchronized Batch Normalization [35] is placed after ev-ery newly added layer with learnable weights. The weightdecay and the momentum parameters are set as 10−4 and0.9, respectively.

Neural Machine Translation (NMT)Model training is conducted on the standard WMT 2014

English-German dataset, consisting of about 4.5 millionsentence pairs. Sentences are encoded using byte-pair en-coding [4], with a shared source-target vocabulary of about37k tokens. Evaluation is on the English-to-German new-

6

(a) Attended Residual Block

3 x 3(Deformable)Convolution

Transformer attention / Dynamic convolution

1 x 1Convolution

1 x 1Convolution

Transformer attention / Dynamic convolution

FeedForward

FeedForward

Attention module

Transformer attention / Dynamic convolution

(b) Transformer / Dynamic convolution

Encoder Decoder

FeedForward

3 x 1Deformableconvolution

Transformer attention

(c) Transformer + Deformable

Encoder Decoder

6 x 6 x

FeedForward

Transformer attention

3 x 1Deformableconvolution

Attention module

6 x 6 x

Figure 2. Illustration of attention module configurations for empirical study. The modules in blue color are newly added to existing blocks.

stest2014 set. Accuracy is measured by the standard bilin-gual evaluation understudy (BLEU) scores [32].

The Transformer base model [41] with relative positionencoding [11] is utilized as the backbone. Experiments areimplemented based on the open source fairseq [14] codebase. The hyper-parameters follows the original settingin [41]. We used the Adam optimizer [26] with β1 = 0.9,β2 = 0.98 and ε = 10−9. In our experiments, the networksare trained on 8 GPUs for 100k iterations. Each trainingbatch contained a set of sentence pairs containing approxi-mately 30k source tokens and 30k target tokens. The initiallearning rate is set as 10−7 and linearly increased to 0.001after iterwarmup = 4000 iterations, and then multiplied by

iteriterwarmup

−0.5. No weight decay is adopted. During training,

label smoothing [39] of value 0.1 is employed.

4.2. Effects of various attention-based modules

Disentanglement in Transformer attentionWe first seek to disentangle the effects of the four terms

in the Transformer attention module. This is achieved bymanually setting the βTrans

j 4j=1 values in Eq. (8) to controlthe activation / deactivation of individual terms. The net-work is trained and tested for all 16 possible configurationsof βTrans

j 4j=1. In this set of experiments, no other attentionmechanisms are involved. Thus, for the object detection andsemantic segmentation tasks, the 3×3 convolution is of reg-ular convolution in the network of Fig. 2 (a). For the NMTtask, the network architecture in Fig. 2 (b) is utilized. Trans-former attention is used in the choices of “Transformer at-tention / Dynamic convolution” in Fig. 2 (a) and (b). Notethat for the NMT task, Transformer attention modules areutilized for both self-attention and encoder-decoder atten-tion. To reduce experimental complexity, the Transformerattention modules in encoder-decoder attention are kept astheir full version (βTrans

j = 1, j = 1, . . . , 4, abbreviated asconfiguration “1111” here) when we study self-attention.

Fig. 3 plots the accuracy-efficiency tradeoffs of differentβTrans

j 4j=1 configurations, where the accuracy-efficiency

envelopes are indicated by connected line segments. Notethat only the computational overheads from the Transformerattention modules under study are counted here, without theoverheads from other parts of the network. From the plot,we draw the following conclusions:

(1) In self-attention, the query-sensitive terms play a mi-nor role compared to the query-irrelevant terms. Especially,the query and key content term have a negligible effecton accuracy, while being computationally heavy in imagerecognition tasks. Overall, the accuracy gain brought bythe Transformer attention module is large (from the con-figuration where the Transformer attention module is re-moved (“w/o”) to that where the full version of Trans-former attention is utilized (“1111”)). It can be seen thatthe gain brought by the query-irrelevant terms (from con-figuration “w/o” to “0011”) is much larger than that broughtby the query-sensitive terms (from configuration “0011” to“1111”). Particularly, the performance gain brought by thequery and key content term (controlled by βTrans

1 ) is negligi-ble. Removing it (from configuration “1111” to “0111”) in-curs only a tiny drop in accuracy, while considerably reduc-ing the computational overhead in image recognition tasks.

(2) In encoder-decoder attention, the query and key con-tent term is vital. Deactivation of it (controlled by βTrans

1 ) in-curs a noticeable drop in accuracy, while only utilizing thequery and key content term (configuration “1000”) deliversaccuracy almost the same as the full version (configuration“1111”). This is because the key step in NMT is to align thewords in the source and the target sentences. A traversal ofthe query and key content is essential for such alignment.

(3) In self-attention, the attention factors of query con-tent & relative position and the key content only are mostimportant. The corresponding configuration “0110” de-livers accuracy very close to the full version (configura-tion “1111”), while saving a considerable amount of com-putational overhead in image recognition tasks. It is alsoworth noting that the key content only term, which cap-tures saliency information, can effectively improve the per-

7

210 220 230 240 250 260 270 280 290

36.5

37

37.5

38

38.5

39

39.5

40

40.5

GFLOPs

mA

P

w/o

001000110001

01100111

01000101 11111011

11011110

1100

1001

1000 1010

0010 + deformable

(a) Image Object detection on COCO

400 500 600 700 800 900 1000 1100 1200

72

73

74

75

76

77

GFLOPs

mIo

U

0010

w/o

01000101

00110001

01100111

1000 10111010 1001

11001101

111011110010 + deformable

(b) Image Semantic Segmentation on Cityscapes

1.7 1.9 2.1 2.3 2.5 2.7 2.9

21

22

23

24

25

26

27

28

GFLOPs

BL

EU

1010

0001

0101

10111010

11011110

1100

00 + deformable

(c) Translation on newstest2014 (self-attention)

2.45 2.5 2.55 2.6 2.65 2.7 2.75

20

21

22

23

24

25

26

27

28

GFLOPs

BL

EU

0010

0001

1000 1010

01000101

0011

01100111

11111011

1010

11011110

1100

(d) Translation on newstest2014 (encoder-decoder attention)

Figure 3. Accuracy-efficiency tradeoffs of the four terms in Transformer attention (E1 for query and key content, E2 for query contentand relative position, E3 for key content only, and E4 for relative position only). The activation and deactivation of particular terms isset by configuration βTrans

j 4j=1 (e.g., “0011” denotes the activation of E3 and E4). Because the encoder-decoder attention mechanism isindispensable for NMT, there is no “w/o” setting in (d). The results of some configurations overlap in the plots because they are of thesame accuracy and computational overhead. The key configurations under study are highlighted in red. The recommended configurationof “0010 + deformable” for self-attention in Tab. 2 is also plotted here.

βTrans1,2,3,4→ βTrans

1,2,3,4 + deformableObject Detection (self-attention) Semantic Segmentation (self-attention) Neural Machine Translation (self-attention)

mAP ∆ mAP GFLOPs ∆% FLOPs mIoU ∆ mIoU GFLOPs ∆% FLOPs BLEU ∆ BLEU GFLOPs ∆% FLOPsw/o→ 1111 + deformable 36.4→ 41.0 +4.6 213.7→ 281.4 +31.7% 71.9→ 77.8 +5.9 449.5→ 1112.1 +147.4% 20.9→ 28.0 +7.1 1.7→ 3.2 +88.2%

1111→ 1011 + deformable 38.8→ 41.0 +2.2 281.4→ 281.4 -0.0% 76.7→ 77.8 +1.1 1112.1→ 1112.1 -0.0% 27.7→ 28.0 +0.3 2.7→ 3.2 +17.3%1110→ 1010 + deformable 38.8→ 40.9 +2.1 281.4→ 281.2 -0.1% 76.7→ 77.7 +1.0 1112.1→ 1111.2 -0.1% 27.7→ 28.0 +0.3 2.7→ 2.9 +5.8%1101→ 1001 + deformable 38.8→ 41.0 +2.2 281.4→ 281.4 -0.0% 76.7→ 77.8 +1.1 1112.1→ 1112.1 -0.0% 27.7→ 28.0 +0.3 2.7→ 3.2 +17.3%1100→ 1000 + deformable 38.8→ 40.9 +2.1 281.4→ 281.2 -0.1% 76.7→ 77.7 +1.0 1112.1→ 1111.2 -0.1% 27.7→ 28.0 +0.3 2.7→ 2.9 +5.8%0111→ 0011 + deformable 38.8→ 41.0 +2.2 253.6→ 250.1 -1.4% 76.6→ 77.5 +0.9 814.0→ 794.4 -2.4% 27.6→ 27.7 +0.1 2.7→ 3.0 +10.9%0110→ 0010 + deformable 38.8→ 40.8 +2.0 253.6→ 221.1 -12.8% 76.6→ 77.3 +0.7 814.0→ 489.5 -39.9% 27.6→ 27.7 +0.1 2.7→ 2.7 -1.1%0101→ 0001 + deformable 38.6→ 40.7 +2.1 251.1→ 247.6 -1.4% 76.3→ 77.3 +1.0 800.7→ 781.1 -2.5% 27.4→ 27.6 +0.2 2.6→ 2.9 +11.6%0100→ w/o + deformable 38.6→ 39.9 +1.3 251.1→ 213.7 -14.9% 76.3→ 77.2 +0.9 800.7→ 449.5 -43.9% 27.4→ 27.3 -0.1 2.6→ 2.2 -13.5%

Table 2. Deformable convolution vs. E2 in Transformer attention, where both exploit query content and relative position information. Theunderlined configuration of “0010 + deformable” is recommended for an optimal accuracy-efficiency tradeoff.

formance with little additional overhead.Our findings contradict the widespread belief that query-

sensitive terms, especially the query and key content term,are crucial for the success of Transformer attention. Theexperimental results suggest that this is only true for theencoder-decoder attention scenario. In self-attention sce-narios, the query and key content term is even removable.

Deformable convolution vs. E2 in Transformer attentionHere, we compare deformable convolution and the E2

term from Transformer attention in Eq. (2). Becausedeformable convolution is designed for capturing self-attention, we restrict the experiments to self-attention sce-narios only. Note that when deformable convolution is uti-lized in the NMT task, the network architecture is of “Trans-former + Deformable” in Fig. 2 (c).

Tab. 2 compares deformable convolution and the E2 termin a variety of settings. We find that:

(1) For object detection and semantic segmentation, de-

8

βTrans1,2,3,4→ dynamic

Object Detection (self-attention) Semantic Segmentation (self-attention) Neural Machine Translation (self-attention)mAP ∆ mAP GFLOPs ∆% FLOPs mIoU ∆ mIoU GFLOPs ∆% FLOPs BLEU ∆ BLEU GFLOPs ∆% FLOPs

0100 38.6 - 251.1 - 76.3 - 800.7 - 27.4 - 2.6 -0100 (nk = 31)→ dynamic (nk = 31) 38.6→ 37.9 -0.7 229.4→ 352.9 +53.8% 75.5→ 74.2 -1.3 523.3→ 1029.0 +96.6% 27.4→ 27.6 +0.2 2.4→ 2.4 +1.8%0100 (nk = 25)→ dynamic (nk = 25) 38.6→ 37.8 -0.8 226.6→ 306.8 +35.4% 75.5→ 74.2 -1.3 511.8→ 840.4 +64.2% 27.4→ 27.6 +0.2 2.3→ 2.3 +1.4%0100 (nk = 19)→ dynamic (nk = 19) 38.6→ 37.6 -1.0 224.4→ 270.6 +20.6% 75.4→ 73.7 -1.7 502.6→ 692.1 +37.7% 27.4→ 27.5 +0.1 2.3→ 2.3 +1.1%0100 (nk = 13)→ dynamic (nk = 13) 38.5→ 37.5 -1.0 222.7→ 244.3 +9.7% 74.4→ 71.9 -2.5 495.9→ 584.3 +17.8% 27.3→ 27.4 +0.1 2.3→ 2.3 +0.7%

Table 3. Dynamic convolution vs. E2 in Transformer attention, where both exploit query content and relative position information. Thekernel size of dynamic convolution Nk is n2

k for image recognition and nk for NMT. The spatial range of Transformer attention is alsoconstrained to be the kernel size of dynamic convolution for ablation.

formable convolution considerably surpasses the E2 term inboth accuracy and efficiency. While for NMT, deformableconvolution is on par with the E2 term in both accuracy andefficiency. In terms of efficiency, deformable convolutiondoes not need to traverse all the key elements. This ad-vantage is obvious on images, where numerous pixels areinvolved. In terms of accuracy, the bilinear sampling in de-formable convolution is based on the hypothesis of locallinearity of feature maps. This hypothesis holds better onimages where local image content changes gradually, thanon languages where words change abruptly.

(2) The combination of deformable convolution and thekey content only term (“0010 + deformable”) delivers thebest accuracy-efficiency tradeoff. The accuracy is on parwith using deformable convolution and the whole atten-tion module (“1111 + deformable”), while the overheadis slightly higher than that of deformable convolution only(“w/o + deformable”). This finding is in line with finding(3) of “Disentanglement in Transformer attention”. It fur-ther suggests the importance of the query content & relativeposition and key content only factors in self-attention. Theconfiguration “0010 + deformable” is also plotted in Fig. 3.

Dynamic convolution vs. E2 in Transformer attentionWe compare these two instantiations in self-attention

scenarios. The network architectures are of Fig. 2 (a) forimage recognition tasks, and of Fig. 2 (b) for NMT, whereeither the Transformer attention with E2 only (configuration“0100”) or dynamic convolution is utilized.

Tab. 3 presents the results. We can find that for NMT,dynamic convolution achieves accuracy on par with the E2term at reduced computational cost. However, dynamicconvolution is not effective for object detection and seman-tic segmentation, delivering considerably lower accuracy.To further study the influence of kernel size in dynamic con-volution, we also constrain the spatial range of the E2 termto be the same as that in dynamic convolution. The accuracydrops as the spatial range shrinks for both dynamic convo-lution and the E2 term. But it is worth noting that the E2term still surpasses dynamic convolution at the same spatialrange in image recognition tasks, with even smaller compu-tational overhead. The inferior accuracy of dynamic convo-lution in image recognition tasks might be because dynamicconvolution is originally designed for NMT, and some de-sign choices may not be suitable for image recognition.

References[1] T. Alkhouli, G. Bretschner, J.-T. Peter, M. Hethnawi,

A. Guta, and H. Ney. Alignment-based neural machine trans-lation. In Proceedings of the First Conference on MachineTranslation, 2016. 3

[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine trans-lation by jointly learning to align and translate. In ICLR,2015. 1, 2

[3] P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, et al. In-teraction networks for learning about objects, relations andphysics. In NIPS, 2016. 2

[4] D. Britz, A. Goldie, M.-T. Luong, and Q. Le. Massive ex-ploration of neural machine translation architectures. arXivpreprint arXiv:1703.03906, 2017. 6

[5] K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun,W. Feng, Z. Liu, J. Shi, W. Ouyang, C. C. Loy, and D. Lin.mmdetection. https://github.com/open-mmlab/mmdetection, 2018. 6

[6] W. Chen, E. Matusov, S. Khadivi, and J.-T. Peter. Guidedalignment training for topic-aware neural machine transla-tion. arXiv preprint arXiv:1607.01628, 2016. 3

[7] J. Cheng, L. Dong, and M. Lapata. Long short-termmemory-networks for machine reading. arXiv preprintarXiv:1601.06733, 2016. 1, 2

[8] T. Cohn, C. D. V. Hoang, E. Vymolova, K. Yao, C. Dyer,and G. Haffari. Incorporating structural alignment biasesinto an attentional neural translation model. arXiv preprintarXiv:1601.01085, 2016. 3

[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,R. Benenson, U. Franke, S. Roth, and B. Schiele. Thecityscapes dataset for semantic urban scene understanding.In CVPR, 2016. 6

[10] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei.Deformable convolutional networks. In ICCV, 2017. 2, 3, 4

[11] Z. Dai, Z. Yang, Y. Yang, W. W. Cohen, J. Carbonell, Q. V.Le, and R. Salakhutdinov. Transformer-xl: Attentive lan-guage models beyond a fixed-length context. arXiv preprintarXiv:1901.02860, 2019. 1, 2, 3, 6, 7

[12] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Languagemodeling with gated convolutional networks. In ICML,2017. 5

[13] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, 2009. 6

[14] S. Edunov, M. Ott, and S. Gross. fairseq. https://github.com/pytorch/fairseq, 2017. 7

9

[15] J. Fu, J. Liu, H. Tian, Z. Fang, and H. Lu. Dual at-tention network for scene segmentation. arXiv preprintarXiv:1809.02983, 2018. 1, 2, 3, 4

[16] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N.Dauphin. Convolutional sequence to sequence learning. InICML, 2017. 1, 2

[17] H. Ghader and C. Monz. What does attention in neu-ral machine translation pay attention to? arXiv preprintarXiv:1710.03348, 2017. 3

[18] R. Ghaeini, X. Z. Fern, and P. Tadepalli. Interpreting recur-rent and attention-based neural models: a case study on nat-ural language inference. arXiv preprint arXiv:1808.03894,2018. 3

[19] J. Gu, H. Hu, L. Wang, Y. Wei, and J. Dai. Learning regionfeatures for object detection. In ECCV, 2018. 1, 2

[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In CVPR, 2016. 6

[21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Effi-cient convolutional neural networks for mobile vision appli-cations. arXiv preprint arXiv:1704.04861, 2017. 5

[22] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei. Relation networksfor object detection. In CVPR, 2018. 1, 2

[23] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation net-works. In CVPR, 2018. 3

[24] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, andW. Liu. Ccnet: Criss-cross attention for semantic segmen-tation. arXiv preprint arXiv:1811.11721, 2018. 1, 2, 4, 6

[25] S. Jain and B. Wallace. Attention is not explanation. arXivpreprint arXiv:1902.10186, 2019. 3

[26] D. P. Kingma and J. Ba. Adam: A method for stochasticoptimization. arXiv preprint arXiv:1412.6980, 2014. 7

[27] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, andS. Belongie. Feature pyramid networks for object detection.In CVPR, 2017. 6

[28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-manan, P. Dollar, and C. L. Zitnick. Microsoft coco: Com-mon objects in context. In ECCV, 2014. 6

[29] Z. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou,and Y. Bengio. A structured self-attentive sentence embed-ding. arXiv preprint arXiv:1703.03130, 2017. 1, 2

[30] L. Liu, M. Utiyama, A. Finch, and E. Sumita. Neural ma-chine translation with supervised attention. arXiv preprintarXiv:1609.04186, 2016. 3

[31] M.-T. Luong, H. Pham, and C. D. Manning. Effective ap-proaches to attention-based neural machine translation. InEMNLP, 2015. 1, 2

[32] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: amethod for automatic evaluation of machine translation. InProceedings of the 40th annual meeting on association forcomputational linguistics, pages 311–318. Association forComputational Linguistics, 2002. 7

[33] A. P. Parikh, O. Tackstrom, D. Das, and J. Uszkoreit. A de-composable attention model for natural language inference.arXiv preprint arXiv:1606.01933, 2016. 1, 2

[34] R. Paulus, C. Xiong, and R. Socher. A deep rein-forced model for abstractive summarization. arXiv preprintarXiv:1705.04304, 2017. 1, 2

[35] C. Peng, T. Xiao, Z. Li, Y. Jiang, X. Zhang, K. Jia, G. Yu,and J. Sun. Megdet: A large mini-batch object detector. InCVPR, 2018. 6

[36] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towardsreal-time object detection with region proposal networks. InNIPS, 2015. 6

[37] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski,R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neuralnetwork module for relational reasoning. In NIPS, 2017. 2

[38] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attentionwith relative position representations. arXiv preprintarXiv:1803.02155, 2018. 2

[39] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.Rethinking the inception architecture for computer vision. InProceedings of the IEEE conference on computer vision andpattern recognition, pages 2818–2826, 2016. 7

[40] G. Tang, R. Sennrich, and J. Nivre. An analysis of attentionmechanisms: The case of word sense disambiguation in neu-ral machine translation. arXiv preprint arXiv:1810.07595,2018. 3

[41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is allyou need. In NIPS, 2017. 1, 2, 4, 6, 7

[42] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang,X. Wang, and X. Tang. Residual attention network for imageclassification. In CVPR, 2017. 3

[43] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neuralnetworks. In CVPR, 2018. 1, 2, 4, 6

[44] F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Payless attention with lightweight and dynamic convolutions.arXiv preprint arXiv:1901.10430, 2019. 2, 3, 5, 6

[45] F. Xiao and Y. Jae Lee. Video object detection with analigned spatial-temporal memory. In ECCV, 2018. 2

[46] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudi-nov, R. Zemel, and Y. Bengio. Show, attend and tell: Neuralimage caption generation with visual attention. In ICML,2015. 2

[47] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, andX. He. Attngan: Fine-grained text to image generation withattentional generative adversarial networks. In CVPR, 2018.2

[48] Y. Yuan and J. Wang. Ocnet: Object context network forscene parsing. arXiv preprint arXiv:1809.00916, 2018. 2, 4

[49] H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, andA. Agrawal. Context encoding for semantic segmentation.In CVPR, 2018. 3

[50] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attention generative adversarial networks. arXiv preprintarXiv:1805.08318, 2018. 2

[51] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. Change Loy, D. Lin,and J. Jia. Psanet: Point-wise spatial attention network forscene parsing. In ECCV, 2018. 1, 2

[52] X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable con-vnets v2: More deformable, better results. arXiv preprintarXiv:1811.11168, 2018. 2, 3, 4

[53] X. Zhu, Y. Wang, J. Dai, L. Yuan, and Y. Wei. Flow-guidedfeature aggregation for video object detection. In ICCV,2017. 2

10