Context Analysis for Pre-trained Masked Language Models · 2020. 11. 12. · Findings of the...

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3789–3804November 16 - 20, 2020. c©2020 Association for Computational Linguistics

3789

Context Analysis for Pre-trained Masked Language Models

Yi-An Lai Garima Lalwani Yi ZhangAWS AI HLT

{yianl,glalwani,yizhngn}@amazon.com

Abstract

Pre-trained language models that learn contex-tualized word representations from a large un-annotated corpus have become a standard com-ponent for many state-of-the-art NLP systems.Despite their successful applications in vari-ous downstream NLP tasks, the extent of con-textual impact on the word representation hasnot been explored. In this paper, we presenta detailed analysis of contextual impact inTransformer- and BiLSTM-based masked lan-guage models. We follow two different ap-proaches to evaluate the impact of context: amasking based approach that is architectureagnostic, and a gradient based approach thatrequires back-propagation through networks.The findings suggest significant differences onthe contextual impact between the two modelarchitectures. Through further breakdown ofanalysis by syntactic categories, we find thecontextual impact in Transformer-based MLMaligns well with linguistic intuition. We fur-ther explore the Transformer attention pruningbased on our findings in contextual analysis.

1 Introduction

Pre-trained masked language models (MLM) suchas BERT (Devlin et al., 2019) and ALBERT (Lanet al., 2019) have set state-of-the-art performanceon a broad range of NLP tasks. The success is oftenattributed to their ability to capture complex syntac-tic and semantic characteristics of word use acrossdiverse linguistic contexts (Peters et al., 2018). Yet,how these pre-trained MLMs make use of the con-text remains largely unanswered.

Recent studies have started to inspect the linguis-tic knowledge learned by pre-trained LMs such asword sense (Liu et al., 2019a) , syntactic parse trees(Hewitt and Manning, 2019), and semantic rela-tions (Tenney et al., 2019). Others directly analyzemodel’s intermediate representations and attention

weights to understand how they work (Kovalevaet al., 2019; Voita et al., 2019).

While previous works either assume access tomodel’s internal states or take advantage of model’sspecial structures such as self-attention maps, theseanalysis are difficult to generalize as the architec-tures evolve. In this paper, our work complementsthese previous efforts and provides a richer under-standing of how pre-trained MLMs leverage con-text without assumptions on architectures. We aimto answer following questions: (i) How much con-text is relevant to and used by pre-trained MLMswhen composing representations? (ii) How fardo MLMs look when leveraging context? Thatis, what are their effective context window sizes?We further define a target word’s essential contextas the set of context words whose absence willmake the MLM indiscriminate of its prediction.We analyze linguistic characteristics of these essen-tial context words to better understand how MLMsmanage context.

We investigate the contextual impacts in MLMsvia two approaches. First, we propose the contextperturbation analysis methodology that graduallymasks out context words following a predeterminedprocedure and measures the change in the targetword probability. For example, we iteratively maskwords that have the least change to the target wordprobability until the probability deviates too muchfrom the start. At this point, the remaining wordsare relevant to and used by the MLM to representthe target word, since further perturbation causes anotable prediction change. Being model agnostic,our approach looks into the contextualization in theMLM task itself, and quantify them only on theoutput layer. We refrain from inspecting internalrepresentations since new architectures might nothave a clear notion of ”layer” with inter-leavingjump connections such as those in Guo et al. (2019)and Yao et al. (2020).

3790

The second approach is adapted from Falenskaand Kuhn (2019) and estimates the impact of aninput subword to the target word probability via thenorm of the gradients. We study pre-trained MLMsbased on two different architectures: Transformerand BiLSTM. The former is essentially BERT andthe latter resembles ELMo (Peters et al., 2018).Although the scope in this work is limited to thecomparison between two popular architectures, thesame novel methodology can be readily applied tomultilingual models as well as other Transformer-based models pre-trained with MLM.

From our analysis, when encoding words usingsentence-level inputs, we find that BERT is ableto leverage 75% of context on average in termsof the sentence length, while BiLSTM has the ef-fective context size of around 30%. The gap iscompelling for long-range context more than 20words away, wherein, BERT still has a 65% chanceto leverage the words in comparison to BiLSTMthat only has 10% or less to do so. In addition,when restricted to a local context window aroundthe target word, we find that the effective contextwindow size of BERT is around 78% of the sen-tence length, whereas BiLSTM has a much smallerwindow size of around 50%. With our extensivestudy on how different pre-trained MLMs operatewhen producing contextualized representations andwhat detailed linguistic behaviors can be observed,we exploited these insights to devise a pilot appli-cation. We apply attention pruning that restrictsthe attention window of BERT based on our find-ings. Results show that the performance remainsthe same with its efficiency improved. Our maincontributions can be briefly summarized as:

• Standardize the pre-training setup (model size,corpus, objective, etc.) for a fair comparisonbetween different underlying architectures.

• Novel design of a straight-forward and intu-itive perturbation-based analysis procedure toquantify impact of context words.

• Gain insights about how different architec-tures behave differently when encoding con-texts, in terms of number of relevant contextwords, effective context window sizes, andmore fine-grained break-down with respect toPOS and dependency structures.

• Leverage insights from our analysis to con-duct a pilot application of attention pruningon a sequence tagging task.

2 Related Work

Pre-training language models (LM) to learn contex-tualized word representations from a large amountof unlabeled text has been shown to benefit down-stream tasks (Howard and Ruder, 2018; Peters et al.,2018; Radford et al., 2019). Masked language mod-eling (MLM) introduced in BERT (Devlin et al.,2019) has been widely used as the pre-training taskin works including RoBERTa (Liu et al., 2019b),SpanBERT (Joshi et al., 2020), and ALBERT (Lanet al., 2019). Many of them employ the Trans-former architecture (Vaswani et al., 2017) that usesmulti-head self-attention to capture context.

To assess the linguistic knowledge learned bypre-trained LMs, probing task methodology sug-gest training supervised models on top of the wordrepresentations (Ettinger et al., 2016; Hupkes et al.,2018; Belinkov and Glass, 2019; Hewitt and Liang,2019). Investigated linguistic aspects span acrossmorphology (Shi et al., 2016; Belinkov et al., 2017;Liu et al., 2019a), syntax (Tenney et al., 2019; He-witt and Manning, 2019), and semantics (Conneauet al., 2018; Liu et al., 2019a).

Another line of research inspects internal statesof pre-trained LMs such as attention weights (Ko-valeva et al., 2019; Clark et al., 2019) or interme-diate word representations (Coenen et al., 2019;Ethayarajh, 2019) to facilitate our understandingof how pre-trained LMs work. In particular, Voitaet al. (2019) studies the evolution of representa-tions from the bottom to top layers and finds that,for MLM, the token identity tends to be recreatedat the top layer. A close work to us is Khandel-wal et al. (2018), they conduct context analysis onLSTM language models to learn how much contextis used and how nearby and long-range context isrepresented differently.

Our work complements prior efforts by analyz-ing how models pre-trained by MLM make useof context and provides insights that different ar-chitectures can have different patterns to capturecontext. Distinct from previous works, we leverageno specific model architecture nor intermediate rep-resentations while performing the context analysis.

Another related topic is generic model inter-pretations including LIME (Ribeiro et al., 2016),SHAP (Lundberg and Lee, 2017), and Ancona et al.(2017). Despite the procedural similarity, our workfocuses on analyzing how pre-trained MLMs be-have when encoding contexts and our methodologyis both model-agnostic and training-free.

3791

Model MNLI-(m/mm) QQP QNLI SST-2 CoLA STS-B MRPC RTE Avg

BERT (Devlin et al., 2019) 84.6/83.4 71.2 90.5 93.5 52.1 85.8 88.9 66.4 79.6BiLSTM + ELMo 72.9/73.4 65.6 71.7 90.2 35.0 64.0 80.8 50.1 67.1

BERT (ours) 84.6/84.0 71.0 91.5 93.6 55.7 86.2 88.6 67.4 80.3BiLSTM (ours) 70.9/70.2 63.0 73.7 90.6 30.5 67.6 81.2 54.6 66.9

Table 1: GLUE benchmark test results. BiLSTM+ELMo numbers are cited from (Wang et al., 2018). The compa-rable performance to previous works validates our pre-training process.

3 Masked Language Modeling

Given a sentence X = (w1, w2, ..., wL) whereeach word wi is tokenized into li subwords(si1, ..., sili), a portion of tokens are randomlymasked with the [MASK] token. MLMs are trainedto recover the original identity of masked tokensby minimizing the negative log likelihood (NLL).In practice, BERT (Devlin et al., 2019) randomlyreplaces 15% tokens by [MASK] for 80% of thecases, keep the original token for 10% of the time,and replace with a random token for the remaining10% of the cases.

For context analysis, we perform the maskingand predictions at the word level. Given a targetword wt, all its subwords are masked X\t =(...s(t−1)lt−1

,[MASK], ...,[MASK], s(t+1)1...).Following Devlin et al. (2019), the conditionalprobability of wt can be computed from outputs ofMLMs with the independence assumption betweensubwords:

P (wt|X\t) = P (st1 . . . stlt |X\t)

=

lt∏i=1

P (sti|X\t).(1)

To investigate how MLMs use context, we pro-pose procedures to perturb the input sentence fromX\t to X\t and monitor the change in the targetword probability P (wt|X\t).

4 Approach

Our goal is to analyze the behaviors of pre-trainedMLMs when leveraging context to recover identityof the masked target word wt, e.g. to answer ques-tions such as how many context words are consid-ered and how large the context window is. To thisend, we apply two analysis approaches. The firstone is based on the masking or perturbation of inputcontext which is architecture agnostic. The secondgradient-based approach requires back-propagationthrough networks.

Our first approach performs context perturba-tion analysis on pre-trained LMs at inference timeand measures the change in masked target wordprobabilities. To answer each question, we startfrom X\t and design a procedure Ψ that itera-tively processes the sentence from last perturbationXk+1\t = Ψ(Xk

\t). The patterns of P (wt|Xk\t) of-

fer insights to our question. An example of Ψ is tomask out a context word that causes the least or neg-ligible change in P (wt|Xk

\t). It’s worth mentioningthat as pre-trained LMs are often used off-the-shelfas a general language encoder, we do not furtherfinetune the model on the analysis dataset but di-rectly analyze how they make use of context. Inpractice, we loop over a sentence word-by-word toset the word as the target first and use rest of wordsas the context for our masking process. Since wedo the context analysis only with model inference,the whole process is fast - around half day on a4-GPU machine to process 12k sentences.

Our second approach estimates the impact of aninput subword sij to P (wt|X\t) by using deriva-tives. Specifically, we adapt the IMPACT scoreproposed in Falenska and Kuhn (2019) to our ques-tions. The score IMPACT(sij , wt) can be computedwith the gradients of the negative log likelihood(NLL) with respect to the subword embedding:

IMPACT(sij , wt) =‖∂(logP (wt|X\t))

∂sij‖∑L

m

∑lmn ‖

∂−logP (wt|X\t)∂smn

‖. (2)

The l2-norm of the gradient is used as the impactmeasure and normalized over all the subwords ina sentence. In practice, we report the impact of acontext word wi by adding up the scores from itssubwords

∑lij IMPACT(sij , wt).

We investigate two different encoder architec-tures of pre-trained MLMs. The first one is BERTthat employs 12 Transformer encoder layers, 768dimension, 3072 feed-forward hidden size, and 110million parameters. The other uses a standard bi-directional LSTM (Hochreiter and Schmidhuber,

3792

60 50 40 30 20 10 0 10 20 30 40 50 60Relative Position

0

20

40

60

80

100Pr

obab

ility

of b

eing

use

d (%

) BERT on EWTBERT on GUMBiLSTM on EWTBiLSTM on GUM

(a) Masking-based context impacts

60 50 40 30 20 10 0 10 20 30 40 50 60Relative Position

0

2

4

6

8

10

12

Mea

n Im

pact

(%)

BERT on EWTBERT on GUMBiLSTM on EWTBiLSTM on GUM

(b) Gradient-based context impacts

Figure 1: Analysis of how much context is used by MLMs. (a) Context words at all relative positions havesignificantly higher probabilities to be considered by BERT, compared with BiLSTM. (b) Gradient-based IMPACTscore also shows that BERT considers more distant context than BiLSTM, impact scores are normalized to 100%.

EWT GUM

Sentences 9,673 3,197Words 195,093 67,585Mean Length 20.17 21.14Median Length 17 19Max Length 159 98

Table 2: Statistics of datasets used for analysis

1997) that has 3 layers, 768 embedding dimension,1200 hidden size, and around 115 million param-eters. The BiLSTM model parameters are chosenso that they resemble ELMo while being close toBERT in model size. To have a fair comparison, wepre-train both encoders from scratch on the uncasedWikipedia-book corpus (wikibook) with the samepre-training setup as in Devlin et al. (2019). ForBiLSTM, we add a linear layer and a LayerNorm(Ba et al., 2016) on top, to project outputs into 768dimension. We validate our pre-trained models byfine-tuning them on GLUE benchmark (Wang et al.,2018) in single-task manner and report test perfor-mance comparable to previous works in Table 1.Our pre-trained BiLSTM-based MLM also getscomparable results to ELMo (Peters et al., 2018).

We perform MLM context analysis on two En-glish datasets from the Universal Dependencies(UD) project, English Web Treebank (EWT) (Sil-veira et al., 2014) and Georgetown University Mul-tilayer corpus (GUM) (Zeldes, 2017). Datasetsfrom the UD project provide consistent and rich lin-guistic annotations across diverse genres, enablingus to gain insights towards the contexts in MLMs.We use the training set of each dataset for analy-sis. EWT consists of 9, 673 sentences from web

blogs, emails, reviews, and social media with themedian length being 17 and maximum length be-ing 159 words. GUM comprises 3, 197 sentencesfrom Wikipedia, news articles, academic writing,fictions, and how-to guides with the median lengthbeing 19 and maximum length being 98 words. Thestatistics of datasets are summarized in Table 2.

5 How much context is used?

Self-attention is designed to encode informationfrom any position in a sequence, whereas BiL-STMs model context through the combination oflong- and short-term memories in both left-to-rightand right-to-left directions. For MLMs, the entiresequence is provided to produce contextualized rep-resentations, it is unclear how much context in thesequence is used by different MLMs.

In this section, we first propose a perturbationprocedure Ψ that iteratively masks out a contextword contributing to the least absolute change ofthe target word probability P (wt|Xk

\t). That is, weincrementally eliminate words that do not penalizeMLMs predictions one by one, until further mask-ing cause P (wt|Xk

\t) to deviate too much from theoriginal probability P (wt|X\t). At this point, theremaining unmasked words are considered beingused by the MLM since corrupting any of themcauses a notable change in target word prediction.

In practice, we identify deviations using thenegative log likelihood (NLL) that correspondsto the loss of MLMs. Assuming NLL has a vari-ance of ε at the start of masking, we stop the per-turbation procedure when the increase on NLLlogP (wt|X\t) − logP (wt|Xk

\t) exceeds 2ε. Weobserve that NLLs fluctuate around [−0.1, 0.1] at

3793

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95<100100Masked context (% of length)

0

2

4

6

8In

crea

se in

NLL


Figure 2: Context usage analysis for MLMs via elimi-nation of irrelevant context. BERT uses about 75% ofcontext while BiLSTM uses around 30%.

the start of masking, hence we terminate our proce-dure when the NLL increase reaches 0.2. We reportthe effective context size in terms of percentage oflength to normalize the length impact. The analysisprocess is repeated using each word in a sentenceas the target word for all sentences in the dataset.

For our second approach, we follow equation 2to calculate the normalized impact of each subwordto the target word and aggregate them for each con-text word to get IMPACT(wi, wt). We group the IM-PACT scores by relative position of a word wi to thetarget word wt and plot the average. To comparewith our first approach, we also use masking-basedmethod to analyze that for a word with a specificrelative position, what would be its probability ofbeing used by a MLM.BERT uses distant context more than BiLSTM.After our masking process, a subset of contextwords are tagged as ”being used” by the pre-trainedLM. In Figure 1a, we aggregated results in termsof relative positions (context-word-to-target-word)for all targets and sentences. ”Probability of beingused %” denotes when a context word appears ata relative position to target, how likely is it to berelevant to the pre-trained LM.

Figure 1a shows that context words at all relativepositions have substantially higher probabilities tobe considered by BERT than BiLSTM. And BiL-STM focuses sharply on local context words, whileBERT leverages words at almost all the positions.A notable observation is that both models considera lot more often, words within distance around[−10, 10] and BERT has as high as 90% probabil-ity to use the words just before and after the targetword. Using gradient-based analysis, Figure 1bshows similar results that BERT considers moredistant context than BiLSTM and local words have


0

2

4

6

8

10

Incr

ease

in N

LL

NOUN - BERTADJ - BERTVERB - BERTDET - BERTADP - BERTNOUN - BiLSTMADJ - BiLSTMVERB - BiLSTMDET - BiLSTMADP - BiLSTM

(a) Masking-based: Different syntactic categories


0

2

4

6

8

10

Incr

ease

in N

LL

Short - BERTMedium - BERTLong - BERTShort - BiLSTMMedium - BiLSTMLong - BiLSTM

(b) Masking-based: Different length buckets

Figure 3: Context usage analysis for MLMs, instancesbucketed by syntactic categories of target words or in-put lengths. (a) More context is used to model con-text words than function words. (b) BERT uses fixedamounts of context while BiLSTM’s context usage per-centage varies by input length.

more impact to both models than distant words.There are notable differences between two anal-

ysis approaches. Since the gradient-based IM-PACT score is normalized into a distribution acrossall positions, it does not show the magnitude ofthe context impact on the two different models.On the other hand, the masking-based analysisshows that BERT uses words at each position morethan BiLSTM based on absolute probability values.Another important difference is that the gradient-based approach is a glass-box method and requiresback-propagation through networks, assuming themodels to be differentiable. On the other hand,the masking-based approach treats the model as ablack-box and has no differentiability assumptionon models. In the following sections, we will con-tinue analysis with the masking-based approach.BERT uses 75% of words in a sentence as con-text while BiLSTM considers 30%. Figure 2shows the increase in NLL when gradually mask-ing out the least relevant words. BERT’s NLLincreases considerably when 25% of context are

3794

masked, suggesting that BERT uses around 75% ofcontext. For BiLSTM, its NLL goes up remarkablyafter 70% of context words are masked, meaningthat it considers around 30% of context. Albeithaving the same capacity, we observe that BERTuses more than two times of context words intoaccount than BiLSTM. This could explain the su-perior fine-tuning performance of BERT on tasksdemanding more context to solve. We observe thatpre-trained MLMs have consistent behaviors acrosstwo datasets that have different genres. For the fol-lowing analysis, we report results combining EWTand GUM datasets.Content words needs more context than func-tion words. We bucket instances based on thepart-of-speech (POS) annotation of the target word.Our analysis covers content words including nouns,verbs and adjectives, and function words includ-ing adpositions and determiners. Figure 3a showsthat both models use significantly more contextto represent content words than function words,which is aligned with linguistic intuitions (Boyd-Graber and Blei, 2009). The findings also showthat MLMs handle content and function words ina similar manner as regular language models do,which are previously analyzed by Wang and Cho(2016); Khandelwal et al. (2018).BiLSTM context usage percentage varies byinput sentence length, whereas for BERT, itdoesn’t. We categorize sentences with lengthshorter than 25 as short, between 25 and 50 asmedium, and more than 50 as long. Figure 3bshows that BiLSTM uses 35% of context for shortsentences, 20% for medium, and only 10% for longsentences. On the other hand, BERT leveragesfixed 75% of context words regardless of the sen-tence length.

6 How far do MLMs look?

In the previous section, we looked at how muchcontext is relevant to the two MLMs via an elimi-nation procedure. From Figure 1a and 1b, we alsoobserve that local context is more impactful thanlong-range context for MLMs. In this section, weinvestigate this notion of locality of context evenfurther and try to answer the question of how faraway do MLMs actually look at in practice, i.e.,what is the effective context window size (cws) ofeach MLM.

For context perturbation analysis, we introducea locality constraint to the perturbation procedure

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Access to combined context window size on both sides (% of available context)

0

2

4

6

8

10

Incr

ease

in N

LL

BERT on EWTBiLSTM on EWTBERT on GUMBiLSTM on GUM

Figure 4: Change in NLL as the context window sizearound target word (left and right combined) changes

while masking words. We aim to identify how localversus distant context impacts the target word prob-ability differently. We start with masking all thewords around the target, i.e., the model only relieson its priors learned during pre-training (cws∼ 0%1). We iteratively increase the cws on both sidesuntil all the surrounding context is available (cws∼ 100%). Details of the masking procedure can befound in Appendix. We report the increase in NLLcompared to when the entire context is availablelogP (wt|X\t) − logP (wt|Xk

\t), with respect tothe increasing cws. This process is repeated usingeach word as the target word, for all the sentencesin the dataset. We aggregate and visualize the re-sults similar to section 5 and use the same threshold(0.2) as before to mark the turning point.

As shown in Figure 4, increasing the cws aroundtarget word reduces the change of NLL until a pointwhere the gap is closed. The plot clearly highlightsthe differences in the behavior of two models -for BERT, words within cws of 78% impact themodel’s ability to make target word predictions,whereas, for BiLSTM, only words within cws of50% affect the target word probability. This showsthat BERT, leveraging entire sequence by self-attention, looks at a much wider context windowsize (effective cws ∼ 78%) in comparison to therecurrent architecture BiLSTM (effective cws∼ 50%). Besides, BiLSTM shows a clear notionof contextual locality that it tends to consider verylocal context for target word prediction.

Furthermore, we investigate the symmetricity ofcws on either side by following the same procedurebut now separately on each side of the target word.We iteratively increase cws either on left side orright side while keeping the rest of the words un-masked. More details of the analysis procedure can

1% here denotes the percent of available context w.r.t.(sentence-length - 1) context words, excluding target word.

3795

25 20 15 10 5 00 5 10 15 20 25Access to |x| context window (in #words) on corresponding side

0

2

4

6

8

10

Incr

ease

in N

LLNOUN - BERTNOUN - BiLSTM

(a) Target word belonging to POS - NOUN


0

2

4

6

8

10

Incr

ease

in N

LL

DET - BERTDET - BiLSTM

(b) Target word belonging to POS - DET

Figure 5: Symmetricity analysis of context windowsize for two target word syntactic categories from shortsentences l ≤ 25 (a) For NOUN as target, BERT looksat words within the window [-16, 16], while BiLSTMhas the context window [-7, 7]. (b) When target wordis DET, BERT looks at words within the window [-14,18], while BiLSTM has the context window [-1, 3].

be found in the Appendix. The analysis results arefurther bucketed by the POS categories of targetwords as well as input sentence lengths, similar toSection 5, to gain more fine-grained insights. InFigure 5, we show the symmetricity analysis ofcws for short length sentences and target word withPOS tags - NOUN and DET. The remaining plotsfor medium and long length sentences with targetword from other POS tags are shown in Appendixdue to the lack of space.

From Figure 5, both models show similar behav-iors across different POS tags when leveraging sym-metric/asymmetric context. The cws attended to oneither side is rather similar when target words areNOUN, whereas for DET, we observe both mod-els paying more attention to right context wordsthan the left. This observation aligns well withlinguistic intuitions for English language. We canalso observe the striking difference between twomodels in effective cws, with BERT attending to amuch larger cws than BiLSTM. The difference in


0

2

4

6

8

10

Incr

ease

in N

LL


Figure 6: Identifying essential context by maskingmost important words. 35-40% of context is criticalto BERT while BiLSTM sees about 20% as essential.

the left and right cws for DET appears to be morepronounced for BiLSTM in comparison to BERT.We hypothesize that this is due to BiLSTM’s over-all smaller cws (left + right) which makes it onlyattend to the most important words that happen tobe mostly in the right context.

7 What kind of context is essential?

There is often a core set of context words that isessential to capture the meaning of target word.For example, “Many people think cotton is themost comfortable to wear in hot weather.”Although most context is helpful to understand themasked word fabric, cotton and wear are essentialas it would be almost impossible to make a guesswithout them.

In this section, we define essential context aswords such that when they are absent, MLMswould have no clue about the target word identity,i.e., the target word probability becomes close tomasking out the entire sequence P (wt|Xmask all).To identify essential context, we design the pertur-bation Ψ to iteratively mask words bringing largestdrop in P (wt|Xk

\t) until we reach a point, wherethe increase in NLL just exceeds the 100% masksetting (logP (wt|X\t) − logP (wt|Xmask all)).The words masked using above procedure are la-belled as essential context words. We further ana-lyze linguistic characteristics of the identified es-sential context words.BERT sees 35% of context as essential, whereasBiLSTM perceives around 20%. Figure 6 showsthat on average, BERT recognizes around 35% ofcontext as essential when making predictions, i.e.,when the increase in NLL is on par with mask-ing all context. On the other hand, BiLSTM seesonly 20% of context as essential. This implies thatBERT would be more robust than the BiLSTM-

3796

Context Distance All targets NOUN ADJ VERB DET ADP

Full-context Linear 9.37 9.33 9.23 8.97 9.47 9.47BERT-essential Linear 6.25 6.42 5.89 5.87 5.65 6.11BiLSTM-essential Linear 5.49 6.43 6.03 6.32 4.20 3.77

Full-context Tree 3.63 3.37 3.73 2.83 4.13 4.31BERT-essential Tree 2.91 2.66 2.88 2.20 3.18 3.46BiLSTM-essential Tree 2.74 2.66 2.90 2.28 2.74 2.73

Table 3: Mean distances from essential context words to target words. Linear means linear positional distance andTree denotes the dependency tree walk distance. Results are bucketed by part-of-speech tags of target words.

who is responsible for [completing] all paperwork for entering a new market ?PRON AUX ADJ SCONJ VERB

nsubj

cop mark

DET NOUN SCONJ VERB DET ADJ NOUN PUNCT

advcl

detobj

markacl det

amod

objpunctroot

i love how it really depends on how a really is , hePRON VERB ADV ADJ

conj

DET NOUN PUNCT

[good] horse your horse not how talented is .PRON ADV VERB ADP ADV PRON NOUN ADV AUX ADV ADV ADJ PRON AUX PUNCT

nsubjadvmod

advmodnsubj

ccomp

advmodmark

advcl

detobl:npmod

nmod:poss

nsubjadvmod

coppunct

advmodadvmod

nsubjcop

root

punct

the service is great and during weekends it to but the [wait]DET VERBAUX NOUNNOUN PUNCT

tends get busy , is worthwhile .PARTADP ADJPRONNOUN DET AUX ADJ PUNCTADJ CCONJ VERB CCONJ

detnsubj

cop nsubjobl

case

ccconj

auxmark

xcomp

det copnsubj

ccpunct

conjpunct

root

Figure 7: Essential context identified by BERT along with POS tags and dependency trees. Words in brackets aretargets. Words underlined are essential.

based encoder in the presence of noisy input, afinding also supported by Yin et al. (2020); Jin et al.(2019), as it will be harder to confuse the modelcompletely given larger size of essential contextwords set in comparison to BiLSTM.

Essential words are close to target words inboth linear position and on dependency tree.Table 3 calculates the mean distances fromidentified essential words to the target wordson combined EWT and GUM datasets. Boththe models tend to identify words much closerto the target as essential, whether we considerlinear positional distance or node distance independency trees. We use annotated dependencyrelations to extract the traversal paths from eachessential word to the target word in dependencytree. We find that the top 10 most frequentdependency paths often correspond with thecommon syntactic structures in natural language.For example, when target words are NOUN,the top 3 paths are DET(up:det)⇒NOUN,ADP(up:case)⇒NOUN, ADJ(up:amod)⇒NOUN for both models. Further, we also look atthe dependency paths of essential words whichare unique to each model. The comparison shows

that words of common dependency paths aresometimes identified as essential by BERT butnot by BiLSTM and vice versa. This suggeststhat there is room to improve MLMs by makingthem consistently more aware of input’s syntacticstructures, possibly by incorporating dependencyrelations into pre-training. The full lists of topdependency paths are presented in the Appendix.

Figure 7 shows examples of essential words fromBERT with POS tags and dependency relations.Words in square brackets are target words and theunderlined words are essential words. We observethat words close to the target in the sentence as wellas in the dependency tree are often seen as essential.We can also see that BERT often includes the rootof the dependency tree as an essential word.

8 Application: Attention Pruning forTransformer

As a pilot application, we leverage insights fromanalysis in previous sections to perform attentionpruning for Transformer. Transformer has achievedimpressive results in NLP and has been used forlong sequences with more than 10 thousand tokens(Liu et al., 2018). Self-attention for a sequence of

3797

Model Dev F1 Test F1

BERT - Full 94.9(0.2) 90.8(0.1)BERT - Dynamic Pruning 94.7(0.2) 90.6(0.2)BERT - Static Pruning 94.5(0.2) 90.3(0.1)

Table 4: CoNLL-2003 Named Entity Recognition re-sults (5 seeds). The attention pruning based on our find-ings gives comparable results to the original BERT.

length L is of O(L2) complexity in computationand memory. Many works attempt to improve theefficiency of self-attention by restricting the num-ber of tokens that each input query can attend to(Child et al., 2019; Kitaev et al., 2020).

Our analysis in Section 6 shows that BERT haseffective cws of around 78%. We perform a dy-namic attention pruning by making self-attentionneglect the furthest 22% of tokens. Due to theO(L2) complexity, this could save around 39% ofcomputation in self-attention. We apply this lo-cality constraint to self-attention when fine-tuningBERT on a downstream task. Specifically, weuse the CoNLL-2003 Named Entity Recognition(NER) dataset (Sang and Meulder, 2003) with 200kwords for training. We fine-tune BERT for NERin the same way as in Devlin et al. (2019). Wealso explore a static attention pruning that restrictsthe attention span to be within [−5,+5]2. Resultsin Table 4 show that BERT with attention prun-ing has comparable performance to the originalBERT, implying successful application of our anal-ysis findings. Note that we use an uncased vocab-ulary, which could explain the gap compared toDevlin et al. (2019).

9 Conclusion

In our context analysis, we have shown that BERThas an effective context size of around 75% of inputlength, while BiLSTM has about 30%. The differ-ence in context usage is striking for long-range con-text beyond 20 words. Our extensive analysis ofcontext window size demonstrate that BERT usesmuch larger context window size than BiLSTM.Besides, both models often identify words withcommon syntactic structures as essential context.These findings not only help to better understandcontextual impact in masked language models, butalso encourage model improvements in efficiencyand effectiveness in future works. On top of that,diving deep into the connection between our con-

2 With average training set sentence length of 14, this spanequates to cws of 78%.

text analysis and a model’s robustness to noisy textsis also an interesting topic to explore.

Acknowledgments

The authors would like to acknowledge the entireAWS Lex Science team for thoughtful discussions,honest feedback, and full support. We are also verygrateful to the reviewers for insightful commentsand helpful suggestions.

ReferencesMarco Ancona, Enea Ceolini, Cengiz Oztireli, and

Markus Gross. 2017. Towards better understandingof gradient-based attribution methods for deep neu-ral networks. arXiv preprint arXiv:1711.06104.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin-ton. 2016. Layer normalization. arXiv preprintarXiv:1607.06450.

Yonatan Belinkov and James Glass. 2019. Analysismethods in neural language processing: A survey.Transactions of the Association for ComputationalLinguistics, 7:49–72.

Yonatan Belinkov, Lluıs Marquez i Villodre, HassanSajjad, Nadir Durrani, Fahim Dalvi, and James R.Glass. 2017. Evaluating layers of representation inneural machine translation on part-of-speech and se-mantic tagging tasks. In IJCNLP.

Jordan L Boyd-Graber and David M Blei. 2009. Syn-tactic topic models. In Advances in neural informa-tion processing systems, pages 185–192.

Rewon Child, Scott Gray, Alec Radford, and IlyaSutskever. 2019. Generating long sequences withsparse transformers. ArXiv, abs/1904.10509.

Kevin Clark, Urvashi Khandelwal, Omer Levy, andChristopher D. Manning. 2019. What does bertlook at? an analysis of bert’s attention. ArXiv,abs/1906.04341.

Andy Coenen, Emily Reif, Ann Yuan, Been Kim,Adam Pearce, Fernanda B. Viegas, and Martin Wat-tenberg. 2019. Visualizing and measuring the geom-etry of bert. In NeurIPS.

Alexis Conneau, German Kruszewski, Guillaume Lam-ple, Loıc Barrault, and Marco Baroni. 2018. Whatyou can cram into a single vector: Probing sentenceembeddings for linguistic properties. In ACL.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training of deepbidirectional transformers for language understand-ing. ArXiv, abs/1810.04805.

Kawin Ethayarajh. 2019. How contextual are contex-tualized word representations? comparing the geom-etry of bert, elmo, and gpt-2 embeddings. ArXiv,abs/1909.00512.

3798

Allyson Ettinger, Ahmed Elgohary, and Philip Resnik.2016. Probing for semantic evidence of compositionby means of simple classification tasks. In RepE-val@ACL.

Agnieszka Falenska and Jonas Kuhn. 2019. The (non-)utility of structural features in bilstm-based depen-dency parsers. In ACL.

Zhijiang Guo, Yan Zhang, Zhiyang Teng, and WeiLu. 2019. Densely connected graph convolutionalnetworks for graph-to-sequence learning. Transac-tions of the Association for Computational Linguis-tics, 7:297–312.

John Hewitt and Percy Liang. 2019. Designingand interpreting probes with control tasks. InEMNLP/IJCNLP.

John Hewitt and Christopher D. Manning. 2019. Astructural probe for finding syntax in word represen-tations. In NAACL-HLT.

Sepp Hochreiter and Jurgen Schmidhuber. 1997.Long short-term memory. Neural computation,9(8):1735–1780.

Jeremy Howard and Sebastian Ruder. 2018. Universallanguage model fine-tuning for text classification. InACL.

Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema.2018. Visualisation and’diagnostic classifiers’ re-veal how recurrent and recursive neural networksprocess hierarchical structure. Journal of ArtificialIntelligence Research, 61:907–926.

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and PeterSzolovits. 2019. Is bert really robust? a strong base-line for natural language attack on text classificationand entailment. arXiv: Computation and Language.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld,Luke Zettlemoyer, and Omer Levy. 2020. Spanbert:Improving pre-training by representing and predict-ing spans. Transactions of the Association for Com-putational Linguistics, 8:64–77.

Urvashi Khandelwal, He He, Peng Qi, and Dan Juraf-sky. 2018. Sharp nearby, fuzzy far away: How neu-ral language models use context. In ACL.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.2020. Reformer: The efficient transformer. ArXiv,abs/2001.04451.

Olga V. Kovaleva, Alexey Romanov, Anna Rogers, andAnna Rumshisky. 2019. Revealing the dark secretsof bert. In EMNLP/IJCNLP.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman,Kevin Gimpel, Piyush Sharma, and Radu Soricut.2019. Albert: A lite bert for self-supervised learn-ing of language representations. arXiv preprintarXiv:1909.11942.

Nelson F. Liu, Matt Gardner, Yonatan Belinkov,Matthew E. Peters, and Noah A. Smith. 2019a. Lin-guistic knowledge and transferability of contextualrepresentations. ArXiv, abs/1903.08855.

Peter J. Liu, Mohammad Saleh, Etienne Pot, BenGoodrich, Ryan Sepassi, Lukasz Kaiser, and NoamShazeer. 2018. Generating wikipedia by summariz-ing long sequences. ArXiv, abs/1801.10198.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019b.Roberta: A robustly optimized bert pretraining ap-proach. arXiv preprint arXiv:1907.11692.

Scott M Lundberg and Su-In Lee. 2017. A unifiedapproach to interpreting model predictions. In Ad-vances in neural information processing systems,pages 4765–4774.

Matthew E. Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word repre-sentations. ArXiv, abs/1802.05365.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Languagemodels are unsupervised multitask learners. OpenAIBlog, 1(8):9.

Marco Tulio Ribeiro, Sameer Singh, and CarlosGuestrin. 2016. ” why should i trust you?” explain-ing the predictions of any classifier. In Proceed-ings of the 22nd ACM SIGKDD international con-ference on knowledge discovery and data mining,pages 1135–1144.

Erik F. Tjong Kim Sang and Fien De Meulder.2003. Introduction to the conll-2003 shared task:Language-independent named entity recognition.ArXiv, cs.CL/0306050.

Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Doesstring-based neural mt learn source syntax? InEMNLP.

Natalia Silveira, Timothy Dozat, Marie-Catherinede Marneffe, Samuel Bowman, Miriam Connor,John Bauer, and Christopher D. Manning. 2014. Agold standard dependency corpus for English. InProceedings of the Ninth International Conferenceon Language Resources and Evaluation (LREC-2014).

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,Adam Poliak, R. Thomas McCoy, Najoung Kim,Benjamin Van Durme, Samuel R. Bowman, Dipan-jan Das, and Ellie Pavlick. 2019. What do youlearn from context? probing for sentence struc-ture in contextualized word representations. ArXiv,abs/1905.06316.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is all

3799

you need. In Advances in neural information pro-cessing systems, pages 5998–6008.

Elena Voita, Rico Sennrich, and Ivan Titov. 2019. Thebottom-up evolution of representations in the trans-former: A study with machine translation and lan-guage modeling objectives. In EMNLP/IJCNLP.

Alex Wang, Amanpreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R Bowman. 2018.Glue: A multi-task benchmark and analysis platformfor natural language understanding. arXiv preprintarXiv:1804.07461.

Tian Wang and Kyunghyun Cho. 2016. Larger-contextlanguage modelling with recurrent neural network.In ACL.

Shaowei Yao, Tianming Wang, and Xiaojun Wan.2020. Heterogeneous graph transformer for graph-to-sequence learning. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 7145–7154.

Fan Yin, Quanyu Long, Tao Meng, and Kai-WeiChang. 2020. On the robustness of language en-coders against grammatical errors. arXiv preprintarXiv:2005.05683.

Amir Zeldes. 2017. The GUM corpus: Creating mul-tilayer resources in the classroom. Language Re-sources and Evaluation, 51(3):581–612.

https://doi.org/http://dx.doi.org/10.1007/s10579-016-9343-x

https://doi.org/http://dx.doi.org/10.1007/s10579-016-9343-x

3800

A Appendix

B Context Window Size Analysis

B.1 Masking Strategies for Context WindowSize Analysis

As mentioned in Section 6, for analyzing how farmasked LMs look at within the available context,we follow a masking strategy with locality con-straints applied. The masking strategy is as follows- we start from no context available, i.e., all thecontext words masked and iteratively increase theavailable context window size (cws) on both sidessimultaneously, till the entire context is available.This procedure is also depicted in Figure 8. Forsymmetricity analysis of cws, we follow similarprocess as above but considering each side of thetarget word separately. Hence, when consideringcontext words to the left, we iteratively increase thecws on the left of target word, keeping the rest ofthe context words on the right unmasked as shownin Figure 9.

Iteration 1 [ MASK] [ MASK ] [ MASK ] potent [ MASK ] [ MASK ]Target Word

[ MASK ]

Sentence It is a very potent psychological weapon

Iteration 2 [ MASK] [ MASK ] [ MASK ] potent psychological [ MASK ]Target Word

very

Iteration 3 [ MASK] [ MASK ] a potent psychological weaponTarget Word

very

Iteration 4 [ MASK] is a potent psychological weaponTarget Word

very

Iteration 5 It is a potent psychological weaponTarget Word

very

Figure 8: Masking strategy for context window sizeanalysis

B.2 Additional Plots for SymmetricityAnalysis of Context Window Size

In Figure 10, we show various plots investigatinghow context around the target word impact’s modelperformance as we look at left and right contextseparately. Figures 10a, 10d, 10g, 10j, 10m showleft and right cws for sentences belonging to shortlength category (l ≤ 25). The trends show that,where NOUN, ADJ, VERB leverage somewhatsymmetric context windows, DET and ADP showasymmetric behavior relying more heavily on rightcontext words for both the models - BERT andBiLSTM. Similar observations can be made forsentences belonging to medium length bucket (l >25 and l ≤ 50) with ADP being an exceptionwhere BiLSTM shows more symmetric contextdifferent than BERT, as shown in Figures 10b, 10e,10h, 10k, 10n. However, for sentences belonging to

Iteration 1 [ MASK] [ MASK ] [ MASK ] potentTarget Word

[ MASK ]

Sentence It is a very potent psychological weapon

Iteration 2 [ MASK] [ MASK ] [ MASK ] potentTarget Word

very

Iteration 3 [ MASK] [ MASK ] a potent psychological weaponTarget Word

very

Iteration 4 [ MASK] is a potent psychological weaponTarget Word

very

Iteration 5 It is a potent psychological weaponTarget Word

very

psychological weapon

psychological weapon

Figure 9: Masking strategy for symmetricity analysisof cws on the left

long length bucket (l > 50), left and right contextwindow sizes are leveraged quite differently.

We can also see that BiLSTM leverages almostsimilar number of context words as we moved on tobuckets of longer sentence lengths in comparisonto BERT which can leverage more context whenits available. This is aligned with our observationfrom Section 5.

C Dependency Paths from EssentialWords to Target Words

Given a target word, BERT or BiLSTM identifiesa subset of context words as essential. Based onthe dependency relations provided in the datasets,we extract the dependency paths starting from eachessential word to the target words, i.e., the path totraverse from an essential word to the given targetword in the dependency tree. We summarize the top10 most frequent dependency paths recognized byBERT or BiLSTM given the target words being aspecific part-of-speech category. Table 5, 6, 7, 8, 9show the results for NOUN, ADJ, VERB, DET, andADP, respectively. The up and down denote the di-rection of traversal, followed by the correspondingrelations in the dependency tree. We can see thatthe top dependency paths for BERT and BiLSTMare largely overlapped with each other. We alsoobserve that these most frequent dependency pathsare often aligned with common syntactic patterns.For example, the top 3 paths for NOUN are DET=(up:det)⇒ NOUN that could be “the” cat,ADP =(up:case)⇒ NOUN that could be “at”home, and ADJ =(up:amod)⇒ NOUN whichcould be “white” car. This implies that both mod-els could be aware of the common syntactic struc-tures in the natural language.

To further compare the behaviors of BERT andBiLSTM when identifying essential context, wecount the occurrence of dependency paths based onthe disjoint essential words. That is, given an inputsentence, we only count the dependency paths of

3801


0

2

4

6

8

10In

crea

se in

NLL

NOUN - BERTNOUN - BiLSTM

(a) BERT looking at context window size[-16, 16]; biLSTM looking at contextwindow size [-7, 7]

50 45 40 35 30 25 20 15 10 5 00 5 10 15 20 25 30 35 40 45Access to |x| context window (in #words) on corresponding side

0

2

4

6

8

10

Incr

ease

in N

LL


(b) BERT looking at context window size[-29, 32]; biLSTM looking at contextwindow size [-12, 12]

160 140 120 100 80 60 40 20 00 20 40 60 80 100 120 140 160Access to |x| context window (in #words) on corresponding side

0

2

4

6

8

10

Incr

ease

in N

LL


(c) BERT looking at context window size[-135, 72]; biLSTM looking at contextwindow size [-19, 5]


0

2

4

6

8

10

Incr

ease

in N

LL

ADJ - BERTADJ - BiLSTM

(d) BERT looking at context window size[-15, 16]; biLSTM looking at contextwindow size [-5, 5]


0

2

4

6

8

10

Incr

ease

in N

LL


(e) BERT looking at context window size[-28, 30]; biLSTM looking at contextwindow size [-7, 6]


0

2

4

6

8

10

Incr

ease

in N

LL


(f) BERT looking at context window size[-54, 77]; biLSTM looking at contextwindow size [-6, 4]


0

2

4

6

8

10

Incr

ease

in N

LL

VERB - BERTVERB - BiLSTM

(g) BERT looking at context window size[-14, 16]; biLSTM looking at contextwindow size [-7, 6]


0

2

4

6

8

10

Incr

ease

in N

LL


(h) BERT looking at context window size[-28, 30]; biLSTM looking at contextwindow size [-9, 8]


0

2

4

6

8

10

Incr

ease

in N

LL


(i) BERT looking at context window size[-102, 148]; biLSTM looking at contextwindow size [-10, 9]


0

2

4

6

8

10

Incr

ease

in N

LL


(j) BERT looking at context window size[-14, 18]; biLSTM looking at contextwindow size [-1, 3]


0

2

4

6

8

10

Incr

ease

in N

LL


(k) BERT looking at context window size[-25, 31]; biLSTM looking at contextwindow size [-1, 2]


0

2

4

6

8

10

Incr

ease

in N

LL


(l) BERT looking at context window size[-50, 75]; biLSTM looking at contextwindow size [-2, 2]


0

2

4

6

8

10

Incr

ease

in N

LL

ADP - BERTADP - BiLSTM

(m) BERT looking at context windowsize [-13, 16]; biLSTM looking at contextwindow size [-2, 3]


0

2

4

6

8

10

Incr

ease

in N

LL


(n) BERT looking at context window size[-25, 30]; biLSTM looking at contextwindow size [-3, 3]


0

2

4

6

8

10

Incr

ease

in N

LL


(o) BERT looking at context window size[-99, 113]; biLSTM looking at contextwindow size [-3, 3]

Figure 10: Symmetricity analysis of context window size for different syntactic categories of target word belongingto sentences from buckets of different lengths; along the rows, we consider sentences of different lengths for a givensyntactic category: (a) - (c) analysis for NOUN; (d) - (f) analysis for ADJ; (g) - (i) analysis for VERB; (j) - (l)analysis for DET; (m) - (o) analysis for ADP; along the columns, we consider different syntactic categories forgiven bucket ranging from short (first column), medium (second column) to long (third column)

3802

BERT BiLSTMDET =(up:det)⇒ NOUN DET =(up:det)⇒ NOUNADP =(up:case)⇒ NOUN ADP =(up:case)⇒ NOUNADJ =(up:amod)⇒ NOUN ADJ =(up:amod)⇒ NOUNVERB =(down:obj)⇒ NOUN VERB =(down:obj)⇒ NOUNADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN VERB =(down:obl)⇒ NOUNNOUN =(down:compound)⇒ NOUN ADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUNNOUN =(up:compound)⇒ NOUN NOUN =(up:nmod)⇒ NOUNNOUN =(up:nmod)⇒ NOUN NOUN =(down:nmod)⇒ NOUNNOUN =(down:nmod)⇒ NOUN NOUN =(down:compound)⇒ NOUNVERB =(down:obl)⇒ NOUN NOUN =(up:compound)⇒ NOUN

Table 5: Top 10 most frequent dependency paths when the target words are NOUN.

BERT BiLSTMNOUN =(down:amod)⇒ ADJ NOUN =(down:amod)⇒ ADJDET =(up:det)⇒ NOUN =(down:amod)⇒ ADJ DET =(up:det)⇒ NOUN =(down:amod)⇒ ADJADP =(up:case)⇒ NOUN =(down:amod)⇒ ADJ ADP =(up:case)⇒ NOUN =(down:amod)⇒ ADJAUX =(up:cop)⇒ ADJ AUX =(up:cop)⇒ ADJVERB =(down:obj)⇒ NOUN =(down:amod)⇒ ADJ ADV =(up:advmod)⇒ ADJADV =(up:advmod)⇒ ADJ VERB =(down:obj)⇒ NOUN =(down:amod)⇒ ADJADJ =(up:amod)⇒ NOUN =(down:amod)⇒ ADJ ADJ =(up:amod)⇒ NOUN =(down:amod)⇒ ADJADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:amod)⇒ ADJ PUNCT =(up:punct)⇒ ADJPUNCT =(up:punct)⇒ NOUN =(down:amod)⇒ ADJ VERB =(down:obl)⇒ NOUN =(down:amod)⇒ ADJPUNCT =(up:punct)⇒ ADJ PUNCT =(up:punct)⇒ NOUN =(down:amod)⇒ ADJ

Table 6: Top 10 most frequent dependency paths when the target words are ADJ.

BERT BiLSTMPRON =(up:nsubj)⇒ VERB PRON =(up:nsubj)⇒ VERBNOUN =(up:obj)⇒ VERB NOUN =(up:obj)⇒ VERBPUNCT =(up:punct)⇒ VERB PUNCT =(up:punct)⇒ VERBAUX =(up:aux)⇒ VERB AUX =(up:aux)⇒ VERBADV =(up:advmod)⇒ VERB ADV =(up:advmod)⇒ VERBADP =(up:case)⇒ NOUN =(up:obl)⇒ VERB ADP =(up:case)⇒ NOUN =(up:obl)⇒ VERBNOUN =(up:obl)⇒ VERB NOUN =(up:obl)⇒ VERBPART =(up:mark)⇒ VERB PART =(up:mark)⇒ VERBDET =(up:det)⇒ NOUN =(up:obj)⇒ VERB DET =(up:det)⇒ NOUN =(up:obj)⇒ VERBSCONJ =(up:mark)⇒ VERB SCONJ =(up:mark)⇒ VERB

Table 7: Top 10 most frequent dependency paths when the target words are VERB.

BERT BiLSTMNOUN =(down:det)⇒ DET NOUN =(down:det)⇒ DETADP =(up:case)⇒ NOUN =(down:det)⇒ DET ADP =(up:case)⇒ NOUN =(down:det)⇒ DETADJ =(up:amod)⇒ NOUN =(down:det)⇒ DET ADJ =(up:amod)⇒ NOUN =(down:det)⇒ DETVERB =(down:obj)⇒ NOUN =(down:det)⇒ DET VERB =(down:obj)⇒ NOUN =(down:det)⇒ DETADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DET ADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DETVERB =(down:obl)⇒ NOUN =(down:det)⇒ DET VERB =(down:obl)⇒ NOUN =(down:det)⇒ DETNOUN =(up:compound)⇒ NOUN =(down:det)⇒ DET NOUN =(down:nmod)⇒ NOUN =(down:det)⇒ DETPROPN =(down:det)⇒ DET NOUN =(up:compound)⇒ NOUN =(down:det)⇒ DETNOUN =(down:nmod)⇒ NOUN =(down:det)⇒ DET PROPN =(down:det)⇒ DETNOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DET NOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DET

Table 8: Top 10 most frequent dependency paths when the target words are DET.

BERT BiLSTMNOUN =(down:case)⇒ ADP NOUN =(down:case)⇒ ADPDET =(up:det)⇒ NOUN =(down:case)⇒ ADP DET =(up:det)⇒ NOUN =(down:case)⇒ ADPVERB =(down:obl)⇒ NOUN =(down:case)⇒ ADP VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADPNOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADP NOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADPPROPN =(down:case)⇒ ADP PROPN =(down:case)⇒ ADPADJ =(up:amod)⇒ NOUN =(down:case)⇒ ADP ADJ =(up:amod)⇒ NOUN =(down:case)⇒ ADPDET =(up:det)⇒ NOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADP DET =(up:det)⇒ NOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADPPUNCT =(up:punct)⇒ VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADP PRON =(up:nmod:poss)⇒ NOUN =(down:case)⇒ ADPNOUN =(down:nmod)⇒ PROPN =(down:case)⇒ ADP NOUN =(down:nmod)⇒ PROPN =(down:case)⇒ ADPPRON =(up:nmod:poss)⇒ NOUN =(down:case)⇒ ADP PRON =(down:case)⇒ ADP

Table 9: Top 10 most frequent dependency paths when the target words are ADP.

3803

essential words which are unique to each model,e.g., words essential to BERT but not essential toBiLSTM. Our goal is to see for these essentialwords unique to a model, whether some specialdependency paths are captured by the model. Ta-ble 10, 11, 12, 13, 14 show the results for NOUN,ADJ, VERB, DET, and ADP, respectively. Weobserve that around top 5 dependency paths foressential words unique to BERT or BiLSTM aremostly overlapping with each other as well as theresults in Table 5, 6, 7, 8, 9. This implies thatsometimes words of common dependency pathscan be identified by BERT as essential while BiL-STM fails to do so and sometimes it’s another wayaround. In other words, there is a room to makemodels to be more consistently aware of syntacticstructures of an input. The observation suggeststhat explicitly incorporating dependency relationsinto pre-training could potentially benefit maskedlanguage models.

3804

BERT BiLSTMDET =(up:det)⇒ NOUN ADP =(up:case)⇒ NOUNADP =(up:case)⇒ NOUN DET =(up:det)⇒ NOUNADJ =(up:amod)⇒ NOUN VERB =(down:obl)⇒ NOUNPUNCT =(up:punct)⇒ NOUN VERB =(down:obj)⇒ NOUNVERB =(down:obl)⇒ NOUN PUNCT =(up:punct)⇒ NOUNVERB =(down:obj)⇒ NOUN NOUN =(up:nmod)⇒ NOUNADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN PUNCT =(up:punct)⇒ VERB =(down:obj)⇒ NOUNNOUN =(up:nmod)⇒ NOUN NOUN =(down:nmod)⇒ NOUNNOUN =(down:nmod)⇒ NOUN PUNCT =(up:punct)⇒ VERB =(down:obl)⇒ NOUNNOUN =(up:compound)⇒ NOUN PRON =(up:nsubj)⇒ VERB =(down:obj)⇒ NOUN

Table 10: Top 10 dependency paths from essential words unique to each model to the target words that are NOUN.

BERT BiLSTMNOUN =(down:amod)⇒ ADJ NOUN =(down:amod)⇒ ADJDET =(up:det)⇒ NOUN =(down:amod)⇒ ADJ PUNCT =(up:punct)⇒ ADJADP =(up:case)⇒ NOUN =(down:amod)⇒ ADJ ADP =(up:case)⇒ NOUN =(down:amod)⇒ ADJVERB =(down:obj)⇒ NOUN =(down:amod)⇒ ADJ VERB =(down:obl)⇒ NOUN =(down:amod)⇒ ADJPUNCT =(up:punct)⇒ NOUN =(down:amod)⇒ ADJ PUNCT =(up:punct)⇒ NOUN =(down:amod)⇒ ADJAUX =(up:cop)⇒ ADJ NOUN =(up:nmod)⇒ NOUN =(down:amod)⇒ ADJADJ =(up:amod)⇒ NOUN =(down:amod)⇒ ADJ DET =(up:det)⇒ NOUN =(down:amod)⇒ ADJPUNCT =(up:punct)⇒ ADJ VERB =(down:obj)⇒ NOUN =(down:amod)⇒ ADJADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:amod)⇒ ADJ PUNCT =(up:punct)⇒ VERB =(down:obj)⇒ NOUN =(down:amod)⇒ ADJVERB =(down:obl)⇒ NOUN =(down:amod)⇒ ADJ PRON =(up:nsubj)⇒ ADJ

Table 11: Top 10 dependency paths from essential words unique to each model to the target words that are ADJ.

BERT BiLSTMPUNCT =(up:punct)⇒ VERB PUNCT =(up:punct)⇒ VERBADP =(up:case)⇒ NOUN =(up:obl)⇒ VERB NOUN =(up:obl)⇒ VERBNOUN =(up:obj)⇒ VERB NOUN =(up:obj)⇒ VERBNOUN =(up:obl)⇒ VERB PRON =(up:nsubj)⇒ VERBDET =(up:det)⇒ NOUN =(up:obj)⇒ VERB DET =(up:det)⇒ NOUN =(up:obl)⇒ VERBPRON =(up:nsubj)⇒ VERB VERB =(up:advcl)⇒ VERBADV =(up:advmod)⇒ VERB ADP =(up:case)⇒ NOUN =(up:obl)⇒ VERBNOUN =(up:nsubj)⇒ VERB VERB =(down:advcl)⇒ VERBCCONJ =(up:cc)⇒ VERB VERB =(up:conj)⇒ VERBSCONJ =(up:mark)⇒ VERB VERB =(down:conj)⇒ VERB

Table 12: Top 10 dependency paths from essential words unique to each model to the target words that are VERB.

BERT BiLSTMNOUN =(down:det)⇒ DET NOUN =(down:det)⇒ DETADP =(up:case)⇒ NOUN =(down:det)⇒ DET VERB =(down:obl)⇒ NOUN =(down:det)⇒ DETVERB =(down:obj)⇒ NOUN =(down:det)⇒ DET NOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DETNOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DET NOUN =(down:nmod)⇒ NOUN =(down:det)⇒ DETVERB =(down:obl)⇒ NOUN =(down:det)⇒ DET PUNCT =(up:punct)⇒ VERB =(down:obl)⇒ NOUN =(down:det)⇒ DETADJ =(up:amod)⇒ NOUN =(down:det)⇒ DET PUNCT =(up:punct)⇒ NOUN =(down:det)⇒ DETADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DET ADP =(up:case)⇒ NOUN =(down:det)⇒ DETPRON =(up:nsubj)⇒ VERB =(down:obj)⇒ NOUN =(down:det)⇒ DET ADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:det)⇒ DETNOUN =(up:compound)⇒ NOUN =(down:det)⇒ DET VERB =(down:obj)⇒ NOUN =(down:det)⇒ DETDET =(up:det)⇒ NOUN =(down:nmod)⇒ NOUN =(down:det)⇒ DET PUNCT =(up:punct)⇒ VERB =(down:obj)⇒ NOUN =(down:det)⇒ DET

Table 13: Top 10 dependency paths from essential words unique to each model to the target words that are DET.

BERT BiLSTMNOUN =(down:case)⇒ ADP NOUN =(down:case)⇒ ADPDET =(up:det)⇒ NOUN =(down:case)⇒ ADP VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADPVERB =(down:obl)⇒ NOUN =(down:case)⇒ ADP PUNCT =(up:punct)⇒ VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADPADJ =(up:amod)⇒ NOUN =(down:case)⇒ ADP DET =(up:det)⇒ NOUN =(down:case)⇒ ADPPUNCT =(up:punct)⇒ VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADP ADJ =(up:amod)⇒ NOUN =(down:case)⇒ ADPPROPN =(down:case)⇒ ADP AUX =(up:aux)⇒ VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADPNOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADP PROPN =(down:case)⇒ ADPDET =(up:det)⇒ NOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADP PRON =(up:nsubj)⇒ VERB =(down:obl)⇒ NOUN =(down:case)⇒ ADPADP =(up:case)⇒ NOUN =(down:nmod)⇒ NOUN =(down:case)⇒ ADP NOUN =(up:nmod)⇒ NOUN =(down:case)⇒ ADPNOUN =(up:compound)⇒ NOUN =(down:case)⇒ ADP ADP =(up:case)⇒ NOUN =(up:nmod)⇒ NOUN =(down:case)⇒ ADP

Table 14: Top 10 dependency paths from essential words unique to each model to the target words that are ADP.

Context Analysis for Pre-trained Masked Language Models · 2020. 11. 12. · Findings of the...

Documents

Transcript of Context Analysis for Pre-trained Masked Language Models · 2020. 11. 12. · Findings of the...