Getting the Most out of AMR Parsing

Chuan Wang and Nianwen XueBrandeis University

{cwang24,xuen}@brandeis.edu

Abstract

This paper proposes to tackle the AMRparsing bottleneck by improving two com-ponents of an AMR parser: concept iden-tification and alignment. We first build aBidirectional LSTM based concept iden-tifier that is able to incorporate richercontextual information to learn sparseAMR concept labels. We then extend anHMM-based word-to-concept alignmentmodel with graph distance distortion anda rescoring method during decoding to in-corporate the structural information in theAMR graph. We show integrating the twocomponents into an existing AMR parserresults in consistently better performanceover the state of the art on various datasets.

1 Introduction

Abstract Meaning Representation (AMR) (Ba-narescu et al., 2013) is a semantic representationwhere the meaning of a sentence is encoded as arooted, directed graph. A number of AMR parsershave been developed in recent years (Flaniganet al., 2014; Wang et al., 2015b; Artzi et al.,2015; Pust et al., 2015; Peng et al., 2015; Zhouet al., 2016; Goodman et al., 2016a), and theinitial benefit of AMR parsing has been demon-strated in various downstream applications such asInformation Extraction (Pan et al., 2015; Huanget al., 2016), Machine Comprehension (Sachanand Xing, 2016), and Language Generation (Flani-gan et al., 2016b; Butler, 2016). However, AMRparsing parsing accuracy is still in the high 60%,as measured by the SMatch score (Cai and Knight,2013), and a significant improvement is needed inorder for it to positively impact a larger number ofapplications.

Previous research has shown that concept iden-tification is the bottleneck to further improvementof AMR parsing. For example, JAMR (Flani-gan et al., 2014), the first AMR parser, is ableto achieve an F-score of 80% (close to the inter-annotator agreement of 83) if gold concepts areprovided. Its parsing accuracy drops sharply to62.3% when the concepts are identified automat-ically.

One of the challenges in AMR concept iden-tification is data sparsity. A large portion ofAMR’s concepts are either word lemmas orsense-disambiguated lemmas drawn from Prop-bank (Palmer et al., 2005). Since the AMR Bankis relatively small, many of the concept labels inthe development or test set only occur a few timesor never appear in the training set. Werling etal. (2015) addresses this problem by defining aset of generative actions that maps words in thesentence to their AMR concepts and use a localclassifier to learn these actions. Given such sparsedata, making full use of contextual information iscrucial to accurate concept labeling. BidirectionalLSTM has shown its success on many sequence la-beling tasks since it is able to combine contextualinformation from both directions and avoid man-ual feature engineering. However, it is non-trivialto formalize concept identification as a sequencelabeling problem because of the large concept la-bel set. Inspired by Foland and Martin (2016;2017), who first apply the Bidirectional LSTM toAMR concept identification by categorizing thelarge labels into a finite set of predefined types,we propose to address concept identification us-ing Bidirectional LSTM with Factored ConceptLabels (FCL), where we re-group the concept la-bel set based on their shared graph structure. Thismakes it possible for different concepts to be rep-resented by one common label that captures theshared semantics of these concepts.

Accurate concept identification also cruciallydepends on the word-to-AMR-concept alignment.Since there is no manual alignment in the AMRannotation, typically either a rule-based or un-supervised aligner is applied to the training datato extract the mapping between words and con-cepts. This mapping will then be used as ref-erence data to train concept identification mod-els. The JAMR aligner (Flanigan et al., 2014)greedily aligns a span of words to graph frag-ments using a set of heuristics. While it can easilyincorporate information from additional linguis-tic sources such as WordNet, it is not adaptableto other domains. Unsupervised aligners borrowtechniques from Machine Translation and treatsentence-to-AMR alignment as a word alignmentproblem between a source sentence and its lin-earized AMR graph (Pourdamghani et al., 2014)and solve it with IBM word alignment models(Brown et al., 1993). However, the distortionmodel in the IBM models is based on the lin-ear distance between source side words while thelinear order of the AMR concepts has no lin-guistic significance, unlike word order in naturallanguage. A more appropriate sentence-to-AMRalignment model should be one that takes the hi-erarchical structure of the AMR into account. Wedevelop a Hidden Markov Model (HMM)-basedsentence-to-AMR alignment method with a novelGraph Distance distortion model to take advan-tage of the structural information in AMR, and ap-ply a structural constraint to re-score the posteriorduring decoding time.

We present experimental results that showincorporating these two improvements toCAMR (Wang et al., 2016), a state-of-the-arttransition-based AMR parser, results in consis-tently better Smatch scores over the state of the arton various datasets. The rest of paper is organizedas follows. Section 2 describes related work onAMR parsing. Section 3 describes our improvedLSTM based concept identification model, andSection 4 describes our alignment method. Wepresent experimental results in Section 5, andconclude in Section 6.

2 Related Work

Existing AMR parsers are either transition-basedor graph-based. Transition-based AMR parsers(Wang et al., 2015b,a; Goodman et al., 2016a,b),focus on modeling the correspondence between

the dependency tree and the AMR graph of a sen-tence by designing a small set of actions that trans-form the dependency tree into the AMR graph.Pust et al. (2015) formulates AMR parsing as amachine translation problem in which the sen-tence is the source language input and the AMRis the target language output. AMR parsing sys-tems that focus on modeling the graph aspect ofthe AMR includes JAMR (Flanigan et al., 2014,2016a; Zhou et al., 2016), which treats AMR pars-ing as a procedure for searching for the Maxi-mum Spanning Connected Subgraphs (MSCGs)from an edge-labeled, directed graph of all pos-sible relations. Parsers based on Hyperedge Re-placement Grammars (HRG) (Chiang et al., 2013;Bjorklund et al., 2016; Groschwitz et al., 2015) putmore emphasis on modeling the formal propertiesof the AMR graph. One practical implementationof HRG-based parsing is that of (Peng et al., 2015;Peng and Gildea, 2016). The adoption of Combi-natory Categorical Grammar (CCG) in AMR pars-ing has also been explored in (Artzi et al., 2015;Misra and Artzi, 2016), where a number of exten-sions have been proposed to enable CCG to workon the broad-coverage AMR corpus.

More recently, Foland and Martin (2016; 2017)describe a neural network based model that de-composes the AMR parsing task into a seriesof subproblems. Their system first identifiesthe concepts using a Bidirectional LSTM Recur-rent Neural Network (Hochreiter and Schmidhu-ber, 1997), and then locates and labels the argu-ments and attributes for each predicate, and fi-nally constructs the AMR using the concepts andrelations identified in previous steps. (Barzdinsand Gosko, 2016) first applies the sequence-to-sequence model (Sutskever et al., 2014) typicallyused in neural machine translation to AMR pars-ing by simply treating the pre-order traversal ofAMR as foreign language strings. (Peng et al.,2017) also adopts the sequence-to-sequence modelfor neural AMR parsing and focuses on reducingdata sparsity in neural AMR parsing with cate-gorization of the concept and relation labels. Incontrast, (Konstas et al., 2017) adopts a differ-ent approach and tackles the data sparsity prob-lem with a self-training procedure that can utilizea large set of unannotated external corpus. (Buysand Blunsom, 2017) design a generic transition-based system for semantic graph parsing and ap-ply sequence-to-sequence framework to learn the

transformation from natural language sequences toaction sequences.

3 Concept Identification withBidirectional LSTM

In this section, we first introduce how we catego-rize AMR concepts using Factored Concept La-bels. We then integrate character-level informa-tion into a Bidirectional LSTM through Convolu-tional Neural Network (CNN)-based embeddings.

3.1 Background and NotationGiven a pair of AMR graph G and English sen-tence S, a look-up tableM is first generated whichmaps a span of tokens to concepts using an aligner.Although there are differences among results gen-erated by different aligners, in general, the alignedAMR concepts can be classified into the followingtypes:

• PREDICATE. Concepts with sense tags,which are frames borrowed from Propbank,belong to this case. Most of the tokensaligned to this type are verbs and nouns thathave their own argument structures.• NON-PREDICATE. This type of concepts

are mostly lemmatized word tokens from theoriginal English sentences.• CONST. Most of the numerical expressions

in English sentences are aligned to this type,where AMR concepts are normalized numer-ical expressions.• MULTICONCEPT. In this type, one or more

word tokens in an English sentence arealigned to multiple concepts that form asub-structure in an AMR graph. The mostfrequent case is named entity subgraphs.For example, in Figure 1, “Mr. Vinken”is aligned to subgraph (p / person:name (m / name :op1 "Mr.":op2 "Vinken").

3.2 Factored Concept LabelsTo be able to fit AMR’s large concept label spaceinto a sequence labeling framework, redefining thelabel set is necessary in order to make the learningprocess feasible. While it is trivial to categorizethe PREDICATE, NON-PREDICATE, CONST cases,there is no straightforward way to deal with theMULTICONCEPT type. Foland and Martin (2016)only handle named entities, which constitute the

Figure 1: An example AMR graph for sentence:“Mr. Vinken is chairman of Elsevier N.V., theDutch publishing group.”.

majority of the MULTICONCEPT cases, where theyadopt BIOES tags to detect the boundary and usean additional Bidirectional LSTM to learn the fine-grained named entity concept types. For otherMULTICONCEPT cases, they only use the leaf con-cepts and ignore other parts of the subgraphs. Fig-ure 2 shows the concept label distribution on de-velopment set of LDC2015E86, where we can seenearly half of the MULTICONCEPTcases are notnamed entities.

6267!

9100!

596!

5278!

0!

2000!

4000!

6000!

8000!

10000!

Preica

te!

Non-pred

icate!

Const!

Multico

ncep

t!

Non-Named Entity !

Figure 2: AMR concept label distribution for de-velopment set of LDC2015E86

Based on the observation that many of theMULTICONCEPT cases are actually similarlystructured subgraphs that only differ in the lexicalitems, we choose to factor the lexical items out ofthe subgraph fragments and use the skeletal struc-ture as the fine-grained labels, which we refer asFactored Concept Label (FCL).

Figure 4 shows that although English words“visitor” and “worker” have been aligned to differ-ent subgraph fragments, after replacing the lexicalitems, in this case the leaf concepts visit-01and work-01 with a placeholder “x”, we are ableto arrive at the same FCL. The strategy for deter-mining the FCL for a word is simple: for each En-glish word w and the subgraph s it aligns to, ifthe length of the longest overlapping substring be-

Figure 3: One example of generating FCL for sentence “NATO allies said the cyber attack was unprece-dented.”

Figure 4: One example of generating FCL

tween w and the leaf concept c of s is greater than4, we replace c with a placeholder.

Despite this simple strategy, our results showthat it can capture a wide range of MULTICON-CEPT cases while keeping the new label spacemanageable. While the named entity can beeasily categorized using FCL, it can also coversome other common cases such as morphologi-cal expressions of negation (e.g., “inadequate”)and comparatives (e.g., “happier”). Setting a fre-quency threshold to prune out the noisy labels,we are able to extract 91 canonical FCL labelson the training set. Our empirical results showthat this canonical FCL label set can cover 96%of the MULTICONCEPT cases on the developmentset. Figure 3 gives one full example of FCLs gen-erated for one sentence. For the PREDICATE cases,following (Foland and Martin, 2016), we only usethe sense tag as its label.

We use label 〈other〉 to label stop words thatdo not map to AMR concepts. The MULTICON-CEPT cases are handled by FCL. The FCL labelset generated by this procedure can be treated asan abstraction of the original AMR concept labelspace, where it groups concepts that have similarAMR subgraphs into the same category.

3.3 CNN-based Character-level Embedding

After constructing the new label set with FCL, weset up a baseline Bidirectional LSTM using theconcatenation of word and NER embeddings asthe input. For each input word w and its NER tagt, their embeddings ew and et are extracted from aword embedding matrix Wwd ∈ Rdwd×|Vwd| and aNER tag embedding matrix Wt ∈ Rdt×|Vt| respec-tively, where dwd and dt are the dimensions of theword and NER tag embedding matricies, |Vwd| and|Vt| are the sizes of the word and NER tag vocab-ulary.

Although this architecture is able to capturelong-range contextual information, it fails to ex-tract information originating from the word formitself. As we have discussed above, in some ofthe MULTICONCEPT cases the concepts are asso-ciated with the word forms themselves and won’tbenefit from its contextual information. For ex-ample, in “unprecedented”, the prefix “un” itselfalready gives enough information to predict theFCL label 〈x〉 : polarity -, which indicates neg-ative polarity. In order to incorporate such mor-phological and shape information, we choose toadd a convolutional layer to extract character-levelrepresentations. A similar technique has been ap-plied to Named Entity Recognition (Santos andGuimaraes, 2015; Chiu and Nichols, 2015) andwe only provide a brief description of the archi-tecture here. For a word w composed of charac-ters {c1, c2, . . . , cl}, where l is the length of wordw, we learn a character embedding matrix Wc ∈Rdc×|Vc|, where dc is the character embedding di-mension defined by the user and Vc is charactervocabulary size. After retrieving the character em-bedding chi for each character ci in word w, we

obtain a sequence of vectors {ch1, ch2, . . . , chl}.This serves as the input to convolutional layer.

Figure 5: The architecture of the CNN-basedcharacter-level embedding.

The convolutional layer applies a linear trans-formation to the local context of a character in theinput sequence, where the local context is parame-terized by window size k. Here we define the localcontext of the character embedding chi to be:

fi = (chi−(k−1)/2, . . . , chi+(k−1)/2)> (1)

The j-th element of the convolutional layer out-put vector ewch is computed by element-wise max-pooling (Ranzato et al., 2007):

[ewch]j = max1≤i≤l

[W 0fi + b0]j (2)

W 0 and b0 are the parameters of the convolutionallayer. And the output vector ewch is the characterlevel representation of the word w. The architec-ture of the model is shown in Figure 5.

The final input to the Bidirectional LSTM is theconcatenation of three embeddings [ew, et, ewch]for each word position.

4 Aligning an English Sentence to itsAMR graph

Given an AMR graph G and English sentencee = {e1, e2, . . . , ei, . . . , eI}, in order to fit theminto the traditional word alignment framework, theAMR graph G is normally linearized using depthfirst search by printing each node as soon as itit visited. The re-entrance node is printed butnot expanded to preserve the multiple mentions of

concept. The relation (also called AMR role to-ken) between concepts are preserved in the unsu-pervised aligner (Pourdamghani et al., 2014) be-cause they also try to align relations to Englishwords. We ignore the relations here since wefocus on aligning concepts. Therefore the lin-earized concept sequences can be represented asg = {g1, g2, . . . , gj , . . . , gJ}. However, althoughthis configuration makes it easy to adopt existingword alignment models, it also ignores the struc-tural information in the AMR graph.

In this section, we proposes a method that in-corporates the structural information in the AMRgraph through a distortion model inside an HMM-based word aligner. We then further improve themodel with a re-scoring method during decodingtime.

4.1 HMM-based Aligner with GraphDistance Distortion

Given a sequence pair (e, g), the HMM-basedword alignment model assumes that each sourceword is assigned to exactly one target word, anddefines an asymmetric alignment for the sentencepair as a = {a1, a2, . . . , ai, . . . , aI}, where eachai ∈ [0, J ] is an alignment from source position ito target position ai, ai = 0 means that ei is notaligned to any target words. Note that in the AMRto English alignment context, both the alignmentand the graph structure is asymmetric, since weonly have AMR graph annotation on in linearizedAMR sequence g. Unlike the traditional wordalignment for machine translation, here we willhave different formulas for each translation direc-tion. In this section, we only discuss the transla-tion from English (source) to linearized AMR con-cepts (target) and we will discuss the AMR to En-glish direction in the following section.

The HMM-based model breaks the generativealignment process into two factors:

P (e,a | g)

=I∏

i=1

Pd(ai | ai−1, J)Pt(ei|gai)(3)

where Pd is the distortion model and Pt is thetranslation model. Traditionally, the distortionprobability Pd(j | j′, J) is modeled to dependonly on the jump width (j−j′) (Vogel et al., 1996)and is defined as:

Pd(j | j′, J) =ct(j − j′)∑J

j′′=1 ct(j′′ − j′)

(4)

where ct(j − j′) is the count of jump width. Thisformula simultaneously satisfies the normalizationconstraint and captures the locality assumptionthat words that are adjacent in the source sentencetend to align to words that are closer in the targetsentence.

As the linear locality assumption does not holdamong linearized AMR concepts, we choose in-stead to encode the distortion probability throughgraph distance, which is given by:

Pgd(j | j′, G) =ct(d(j, j′))∑j′′ ct(d(j′′, j′))

(5)

The graph distance d(j, j′) is the length of short-est path on AMR graph G from concept j to con-cept j′. Note that we have to artificially normal-ize Pgd(j | j′, G), because unlike the linear dis-tance between word tokens in a sentence, there canbe multiple concepts that can have the same dis-tance from the j′-th concept in the AMR graph, aspointed out in (Kondo et al., 2013).

During training, just like the original HMM-based aligner, an EM algorithm can be applied toupdate the parameters of the model.

4.2 Improved Decoding with PosteriorRescoring

So far, we have integrated the graph structureinformation into the forward direction (Englishto AMR). To also improve the reverse directionmodel (AMR to English), we choose to use thegraph structure to rescore the posterior during de-coding time.

Compared with Viterbi decoding, posteriorthresholding has shown better results in wordalignment tasks (Liang et al., 2006). Given thresh-old γ, for all possible alignments, we select thefinal alignment based on the following criteria:

a = {(i, j) : p(aj = i | g, e) > γ} (6)

where the state probability p(aj = i | g, e) iscomputed using the forward-backward algorithm.The forward algorithm is defined as:

αj,i

=∑i′

αj−1,i′p(aj = i | aj−1 = i′)p(gj | eaj )

(7)

To incorporate the graph structure, we rescale thedistortion probability in reverse direction model

as:

pnew(aj = i |aj−1 = i′)

= p(aj = i | aj−1 = i′)e∆d (8)

where the scaling factor ∆d = dj − dj−1 is thegraph depth difference between the adjacent AMRconcepts gj and gj−1. We also apply the same pro-cedure for the backward computation. Note thatsince the model is in reverse direction, the distor-tion p(aj = i | aj−1 = i′) here is still based onEnglish word distance, jump width.

This rescaling procedure is based on the intu-ition that after we have processed the last con-cept gj−1 in some subgraph, the next concept gj’saligned English position i is not necessarily re-lated to the last aligned English position i′. Fig-ure 6 illustrates this phenomenon: Although weand current are adjacent concepts in linearizedAMR sequence, they are actually far away fromeach other in the graph (with a graph depth differ-ence of -2). However, the distortion based on theEnglish word distance mostly tends to choose thecloser word, which may yield a very low proba-bility for our correct answer here (the jump widthbetween “Currently” and “our” is -6). By apply-ing the exponential scaling factor, we are able toreduce the differences between different distortionprobabilities. On the contrary, when the distor-tion probability is reliable (the absolute value ofthe graph depth difference is small), the modelchooses to trust the distortion and picks the closerEnglish word.

The rescaling factor can be viewed as a selec-tion filter for decoding, where it relies on the graphdepth difference ∆d to control the effect of learneddistortion probability. Note that after the rescaling,the resulting distortion probability no longer sat-isfies the normalization constraint. However, weonly apply this during decoding time and experi-ments show that the typical threshold γ = 0.5 stillworks well for our case.

4.3 Combining Both Directions

Empirical results show that combining alignmentsfrom both directions improve the alignment qual-ity (DeNero and Klein, 2007; Och and Ney, 2003;Liang et al., 2006). To combine the alignments,we adopt a slightly modified version of posteriorthresholding, competitive thresholding, as pro-posed in (DeNero and Klein, 2007), which tendsto select alignments that form a contiguous span.

Figure 6: AMR graph annotation, linearized con-cepts for sentence “Currently, there is no asbestosin our products”. The concept we in solid lineis the (j − 1)-th token in linearized AMR. It isaligned to English word “our” and its depth ingraph dj−1 is 3. While the word distance-baseddistortion prefers an alignment near “our”, the cor-rect alignment needs a longer distortion.

5 Experiments

We first test the performance of our BidirectionalLSTM concept identifier and HMM-based aligneras standalone tasks, where we investigate the ef-fectiveness of each component in AMR parsing.Then we report the final results by incorporatingboth components to CAMR (Wang et al., 2016).At the model development stage, we mainly usethe dataset LDC2015E86 used in the SemEvalShared Task (May, 2016). Note that this datasetincludes :wiki relations where every named en-tity concept is linked to its wikipedia entry. We re-move this information in the training data through-out the development of our models. At the finaltesting stage, we add wikification using an off-the-shelf AMR wikifier (Pan et al., 2015) as a post-processing step. All AMR parsing results are eval-uated using the Smatch (Cai and Knight, 2013)scorer.

5.1 Bidirectional LSTM ConceptIdentification Evaluation

In order to isolate the effects of our concept iden-tifier, we first use the official alignments providedby SemEval. The alignment is generated by theunsupervised aligner described in (Pourdamghaniet al., 2014). After getting the alignment table, wegenerate our FCL label set by filtering out noisyFCL labels that occur fewer than 30 times in thetraining data. The remaining FCL labels accountfor 96% of the MULTICONCEPT cases in the de-

velopment set. Adding other labels that includePREDICATE, NON-PREDICATE and CONST givesus 116 canonical labels. UNK label is added tohandle the unseen concepts.

In the Bidirectional LSTM, the hyperparametersettings are as follows: word embedding dimen-sion dwd = 128, NER tag embedding dimensiondt = 8, character embedding dimension dc = 50,character level embedding dimension dwch = 50,convolutional layer window size k = 2.

Input P R F1 Accword,NER 81.2 80.6 80.9 85.4word,NER,CNN 83.3 82.7 83.0 87.0

Table 1: Performance of Bidirectional LSTM withdifferent input.

Table 1 shows the performance on the develop-ment set of LDC2015E86, where the precision, re-call and F-score are computed by treating 〈other〉as the negative label and accuracy is calculatedusing all labels. We include accuracy here sincecorrectly predicting words that don’t invoke con-cepts is also important. We can see that utilizingCNN-based character level embedding yields animprovement of around 2 percentage points abso-lute for both F-score and accuracy, which indicatesthat morphological and word shape information isimportant for concept identification.

Impact on AMR Parsing In order to test theimpact of our concept identification component onAMR parsing, we add the predicted concept labelsas features to CAMR. Here is the detailed featureset we add to CAMR’s feature templates. To clar-ify the notation, we refer the concept labels pre-dicted by our concept identifier as cpred and thecandidate concept labels in CAMR as ccand:

• pred label. cpred used directly as a fea-ture.

• is eq sense. A binary feature of whethera cpred and ccand have the same sense (if ap-plicable).

One reason why we choose to add the concept la-bel and sense as features to predict the conceptrather than using the predicted label to recover theconcept directly is that the latter is not a straight-forward process. For example, since we generalizeall the predicates to a compact form <pred-xx>,for irregular verbs like “became”⇒ become-01,simply stemming the inflected verb form will not

give us the correct concept even if the sense ispredicted correctly. However, since CAMR usesthe alignment table to store all possible conceptcandidates for a word, adding our predicated la-bel as a feature could potentially help the parserto choose the correct concept. In order to takefull advantage of the predicted concept labels, wealso extend CAMR so that it can discover candi-date concepts outside of the alignment table. Toachieve this, during the FCL label generation pro-cess, we first store the string-to-concept mappingas a template. For example, when we generatethe FCL label (person :ARG0-of <x>-01)from “worker”, we also store the template <x>er-> (person :ARG0-of <x>-01). Thenduring decoding time, we would enumerate everytemplate and try to use the left hand side of thetemplate (which is <x>er) as a regular expres-sion to match current word. Once we find a matchin all the template entries, we would substitute theplaceholder in right hand side with the matchedsubstring to get the candidate concept label. As aresult, even we haven’t seen “teacher”, by match-ing teacher with the regular expression (.?)er,we could generate the correct answer (person:ARG0-of teach-01). We refer this processas unknown concept generation. Table 2 summa-rizes the impact of our proposed methods on de-velopment set of LDC2015E86. We can see thatby utilizing the unknown concept generation andfeatures derived from cpred , both precision and re-call improve by about 1 percentage point, whichindicates that the new feature brings richer infor-mation to the concept prediction model to helpcorrectly score candidate concepts from the align-ment table.

Parsers P R F1

CAMR (Wang et al., 2016) 72.3 61.4 66.5CAMR-gen 72.1 62.0 66.6CAMR-gen-cpred 73.6 62.6 67.6

Table 2: Performance of AMR parsing withcpred features without wikification on dev set ofLDC2015E86. The first row is performance of thebaseline parser. The second row adds unknownconcept generation and the last row additionallyextends the baseline parser with cpred features.

5.2 HMM-based AMR-to-English AlignerEvaluation

To validate the effectiveness of our proposedalignment methods, we first evaluate our for-

91.6!

79.9!

85.3!

94.3!

81.9!

87.7!

70!

75!

80!

85!

90!

95!

100!

Precision! Recall! F-1!

HMM baseline(f)!HMM graph!

(a) HMM forward

92.3!

84.7!

88.3!

96.1!

83.6!

89.4!

75!

80!

85!

90!

95!

100!

Precision ! Recall! F-1!

HMM baseline(r)!HMM rescale!

(b) HMM reverse

Figure 7: Our improved forward (graph) and re-verse(rescale) model compared with HMM base-line on hand aligned development set.

ward (English-to-AMR) and reverse (AMR-to-English) aligners against the baseline HMM wordalignment model, which is the Berkeley alignertoolkit (DeNero and Klein, 2007). Then we com-bine the forward and reverse alignment results us-ing competitive thresholding. We set the thresholdγ to be 0.5 in the following experiments. To evalu-ate the alignment quality, we use 200 hand-alignedsentences from (Pourdamghani et al., 2014) splitequally as the development and test sets. Weprocess the English sentences by removing stopwords, following similar procedure as in (Pour-damghani et al., 2014). When linearizing AMRgraphs, we instead remove all the relations andonly keep the concepts. For all models, we run5 iterations of IBM Model 1 and 2 iterations ofHMM on the whole dataset.

From Figure 7a, we can see that our graph-distance based model improves both the precisionand recall by a large margin, which indicates thegraph distance distortion better fits the English-to-AMR alignment task. For the reverse model,although our HMM rescaling model loses accu-racy in recall, it is able to improve the precisionby around 4 percentage points, which confirmsour intuition that the rescoring factor is able tokeep reliable alignments and penalize unreliableones. We then combine our forward and reversealignment result using competitive thresholding.Table 3 shows the combined result against hand-aligned dev and test sets.

Datasets P R F1

dev 97.7 84.3 90.5test 96.9 84.6 90.3

Table 3: Combined HMM alignment result evalu-ation.

Dataset Aligner P R F1

DevJAMR 73.6 62.6 67.6ISI 72.8 64.6 68.4Our HMM 73.6 63.6 68.2

TestJAMR 71.6 61.3 66.0ISI 70.6 62.7 66.4Our HMM 72.1 62.5 67.0

Table 4: AMR parsing result (without wikifica-tion) with different aligner on development andtest of LDC2015E86, where JAMR is the rule-based aligner, ISI is the modified IBM Model 4aligner

Impact on AMR Parsing To investigate ouraligner’s contribution to AMR parsing, we replacethe alignment table generated by the best perform-ing aligner (the forward and reverse combined) inthe previous section and re-train CAMR with thepredicted concept label features included.

From Table 4, we can see that the unsuper-vised aligner (ISI and HMM) generally outper-forms the JAMR rule-based aligner, and our im-proved HMM aligner is more consistent than theISI aligner (Pourdamghani et al., 2014), which isa modified version of IBM Model 4.

5.3 Comparison with other Parsers

We first add the wikification information to theparser output using the off-the-shelf AMR wiki-fier (Pan et al., 2015) and compare results with thestate-of-the-art parsers in 2016 SemEval SharedTask. We also report our result on the previousrelease (LDC2014T12), AMR annotation Release1.0, which is another popular dataset that most ofthe existing parsers report results on. Note that theRelease 1.0 annotation doesn’t include wiki infor-mation.

Dataset Parsers P R F1

SemEvalTest

CAMR 70.3 63.1 66.5RIGA - - 67.2JAMR(2016a) 70 65 67Our parser 71.7 64.9 68.1

SemEvalBlindTest

CAMR 67.4 57.3 62.0RIGA 68.0 57.0 62.0JAMR(2016a) - - 56Our parser 68.2 59.5 63.6

Table 5: Comparison with the winning systems inSemEval (with wikification) on test and blind testsets

CAMR and RIGA (Barzdins and Gosko, 2016)are the two best performing parsers that partici-pated in SemEval 2016 shared task. While we useCAMR as our baseline system, the parser fromRIGA is also based on a version of CAMR ex-tended with a error-correction wrapper and an en-semble with a character-level neural sequence-to-sequence model. Our parser outperforms bothsystems by around 1.5 percentage points, wherethe improvement in recall is more significant, ataround 2 percentage points.

Parsers P R F1

CAMR 71.3 62.2 66.5(Zhou et al., 2016) 70 62 66(Pust et al., 2015) - - 65.8Our parser 72.7 64.0 68.07

Table 6: Comparison with the existing parsers onfull test set of LDC2014T12

Table 6 shows the performance of our parser onthe full test set of LDC2014T12. We include theprevious best results on this dataset. The parserproposed in (Zhou et al., 2016) jointly learns theconcept and relation through an incremental jointmodel. We also include the AMR parser by(Pust et al., 2015) that models AMR parsing asa machine translation task and incorporates vari-ous external resources. Our parser still achievesthe best result without incorporating external re-sources other than the NER information.

6 Conclusion

In this paper, we presents work that improvesAMR parsing performance by focusing on twocomponents of the parser: concept identificationand alignment. We first build a BidirectionalLSTM based concept identifier which is able toincorporate richer context and learn sparse con-cept labels. Then we extend the HMM-based wordalignment model with a graph distance distortionand a rescoring method during decoding to incor-porate the graph structure information. By inte-grating the two components into an existing AMRparser, our parser is able to outperform state-of-the-art AMR parsers and establish a new state ofthe art.

Acknowledgments

We want to thank the anonymous reviewers fortheir suggestions and detailed error checking.

ReferencesYoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015.

Broad-coverage CCG semantic parsing with AMR.In Proceedings of the 2015 Conference on Empiri-cal Methods in Natural Language Processing, pages1699–1710, Lisbon, Portugal. Association for Com-putational Linguistics.

Laura Banarescu, Claire Bonial, Shu Cai, MadalinaGeorgescu, Kira Griffitt, Ulf Hermjakob, KevinKnight, Philipp Koehn, Martha Palmer, and NathanSchneider. 2013. Abstract meaning representationfor sembanking. In Proceedings of the 7th Linguis-tic Annotation Workshop and Interoperability withDiscourse.

Guntis Barzdins and Didzis Gosko. 2016. Riga atsemeval-2016 task 8: Impact of smatch extensionsand character-level neural translation on amr pars-ing accuracy. arXiv preprint arXiv:1604.01278.

Henrik Bjorklund, Frank Drewes, and Petter Ericson.2016. Between a rock and a hard place–uniformparsing for hyperedge replacement dag grammars.In International Conference on Language and Au-tomata Theory and Applications, pages 521–532.Springer.

Peter F Brown, Vincent J Della Pietra, Stephen A DellaPietra, and Robert L Mercer. 1993. The mathemat-ics of statistical machine translation: Parameter esti-mation. Computational linguistics, 19(2):263–311.

Alastair Butler. 2016. Deterministic natural lan-guage generation from meaning representations formachine translation. In Proceedings of the 2ndWorkshop on Semantics-Driven Machine Transla-tion (SedMT 2016), pages 1–9. Association forComputational Linguistics.

Jan Buys and Phil Blunsom. 2017. Robust incre-mental neural semantic graph parsing. CoRR,abs/1704.07092.

Shu Cai and Kevin Knight. 2013. Smatch: an evalua-tion metric for semantic feature structures. In Pro-ceedings of the 51st Annual Meeting of the Associa-tion for Computational Linguistics (Volume 2: ShortPapers), pages 748–752. Association for Computa-tional Linguistics.

David Chiang, Jacob Andreas, Daniel Bauer,Karl Moritz Hermann, Bevan Jones, and KevinKnight. 2013. Parsing graphs with hyperedgereplacement grammars. In Proceedings of the51st Meeting of the Association of ComputationalLinguistics, Sofia, Bulgaria.

Jason P. C. Chiu and Eric Nichols. 2015. Named en-tity recognition with bidirectional lstm-cnns. CoRR,abs/1511.08308.

John DeNero and Dan Klein. 2007. Tailoring WordAlignments to Syntactic Machine Translation. Pro-ceedings of the 45th Annual Meeting of the Associa-tion of Computational Linguistics, pages 17–24.

Jeffrey Flanigan, Chris Dyer, A. Noah Smith, andJaime Carbonell. 2016a. CMU at SemEval-2016task 8: Graph-based AMR Parsing with InfiniteRamp Loss. In Proceedings of the 10th Interna-tional Workshop on Semantic Evaluation (SemEval-2016), pages 1202–1206. Association for Computa-tional Linguistics.

Jeffrey Flanigan, Chris Dyer, A. Noah Smith, andJaime Carbonell. 2016b. Generation from AbstractMeaning Representation using tree transducers. InProceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 731–739. Association for Computational Lin-guistics.

Jeffrey Flanigan, Sam Thomson, Jaime Carbonell,Chris Dyer, and Noah A. Smith. 2014. A Discrim-inative Graph-Based Parser for the Abstract Mean-ing Representation. In Proceedings of the 52nd An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1426–1436, Baltimore, Maryland. Association for Com-putational Linguistics.

William Foland and H. James Martin. 2016. CU-NLP at SemEval-2016 Task 8: AMR parsing us-ing LSTM-based recurrent neural networks. In Pro-ceedings of the 10th International Workshop on Se-mantic Evaluation (SemEval-2016), pages 1197–1201. Association for Computational Linguistics.

William Foland and James Martin. 2017. Abstractmeaning representation parsing using lstm recurrentneural networks. In Proceedings of the 55th An-nual Meeting of the Association of the Computa-tional Linguistics, Vancouver, Canada. Associationfor Computational Linguistics.

James Goodman, Andreas Vlachos, and Jason Narad-owsky. 2016a. Noise reduction and targeted explo-ration in imitation learning for abstract meaning rep-resentation parsing. In Proceedings of the 54th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1–11.Association for Computational Linguistics.

James Goodman, Andreas Vlachos, and Jason Narad-owsky. 2016b. UCL+Sheffield at SemEval-2016task 8: Imitation learning for amr parsing with analpha-bound. In Proceedings of the 10th Interna-tional Workshop on Semantic Evaluation (SemEval-2016), pages 1167–1172. Association for Computa-tional Linguistics.

Jonas Groschwitz, Alexander Koller, Christoph Teich-mann, et al. 2015. Graph parsing with s-graph gram-mars. In ACL (1), pages 1481–1490.

Sepp Hochreiter and Jurgen Schmidhuber. 1997.Long short-term memory. Neural computation,9(8):1735–1780.

Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji,R. Clare Voss, Jiawei Han, and Avirup Sil. 2016.Liberal event extraction and event schema induction.In Proceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 258–268. Association for Com-putational Linguistics.

Shuhei Kondo, Kevin Duh, and Yuji Matsumoto. 2013.Hidden markov tree model for word alignment. InProceedings of the Eighth Workshop on StatisticalMachine Translation, pages 503–511.

Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, YejinChoi, and Luke Zettlemoyer. 2017. Neural AMR:sequence-to-sequence models for parsing and gen-eration. CoRR, abs/1704.08381.

Percy Liang, Ben Taskar, and Dan Klein. 2006. Align-ment by agreement. In Proceedings of the mainconference on Human Language Technology Con-ference of the North American Chapter of the As-sociation of Computational Linguistics, pages 104–111. Association for Computational Linguistics.

Jonathan May. 2016. Semeval-2016 task 8: Mean-ing representation parsing. Proceedings of SemEval,pages 1063–1073.

Dipendra Kumar Misra and Yoav Artzi. 2016. Neuralshift-reduce ccg semantic parsing. In Proceedings ofthe 2016 Conference on Empirical Methods in Natu-ral Language Processing, pages 1775–1786, Austin,Texas. Association for Computational Linguistics.

Franz Josef Och and Hermann Ney. 2003. A systematiccomparison of various statistical alignment models.Computational linguistics, 29(1):19–51.

Martha Palmer, Daniel Gildea, and Paul Kingsbury.2005. The proposition bank: An annotated cor-pus of semantic roles. Computational Linguistics,31(1):71–106.

Xiaoman Pan, Taylor Cassidy, Ulf Hermjakob, Heng Ji,and Kevin Knight. 2015. Unsupervised entity link-ing with abstract meaning representation. In Pro-ceedings of the 2015 Conference of the North Amer-ican Chapter of the Association for ComputationalLinguistics: Human Language Technologies, pages1130–1139.

Xiaochang Peng and Daniel Gildea. 2016. UofR atSemEval-2016 Task 8: Learning synchronous hy-peredge replacement grammar for AMR parsing. InProceedings of the 10th International Workshop onSemantic Evaluation (SemEval-2016), pages 1185–1189. Association for Computational Linguistics.

Xiaochang Peng, Linfeng Song, and Daniel Gildea.2015. A synchronous hyperedge replacement gram-mar based approach for AMR parsing. In Proceed-ings of the Nineteenth Conference on ComputationalNatural Language Learning.

Xiaochang Peng, Chuan Wang, Daniel Gildea, and Ni-anwen Xue. 2017. Addressing the data sparsity is-sue in neural amr parsing. In Proceedings of the15th Conference of the European Chapter of the As-sociation for Computational Linguistics: Volume 1,Long Papers, pages 366–375, Valencia, Spain. As-sociation for Computational Linguistics.

Nima Pourdamghani, Yang Gao, Ulf Hermjakob, andKevin Knight. 2014. Aligning English strings withabstract meaning representation graphs. In Proceed-ings of the 2014 Conference on Empirical Methodsin Natural Language Processing (EMNLP), pages425–429.

Michael Pust, Ulf Hermjakob, Kevin Knight, DanielMarcu, and Jonathan May. 2015. Parsing Englishinto abstract meaning representation using syntax-based machine translation. In Proceedings of the2015 Conference on Empirical Methods in Natu-ral Language Processing, pages 1143–1154, Lis-bon, Portugal. Association for Computational Lin-guistics.

M. Ranzato, F. J. Huang, Y. L. Boureau, and Y. LeCun.2007. Unsupervised learning of invariant feature hi-erarchies with applications to object recognition. In2007 IEEE Conference on Computer Vision and Pat-tern Recognition, pages 1–8.

Mrinmaya Sachan and Eric Xing. 2016. Machinecomprehension using rich semantic representations.In Proceedings of the 54th Annual Meeting of theAssociation for Computational Linguistics (Volume2: Short Papers), pages 486–492. Association forComputational Linguistics.

Cicero Nogueira dos Santos and Victor Guimaraes.2015. Boosting named entity recognition withneural character embeddings. arXiv preprintarXiv:1505.05008.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.Sequence to sequence learning with neural net-works. CoRR, abs/1409.3215.

Stephan Vogel, Hermann Ney, and Christoph Tillmann.1996. HMM-based word alignment in statisticaltranslation. In Proceedings of the 16th conferenceon Computational linguistics-Volume 2, pages 836–841. Association for Computational Linguistics.

Chuan Wang, Sameer Pradhan, Xiaoman Pan, HengJi, and Nianwen Xue. 2016. CAMR at semeval-2016 task 8: An extended transition-based AMRparser. In Proceedings of the 10th InternationalWorkshop on Semantic Evaluation (SemEval-2016),pages 1173–1178, San Diego, California. Associa-tion for Computational Linguistics.

Chuan Wang, Nianwen Xue, and Sameer Pradhan.2015a. Boosting transition-based AMR parsingwith refined actions and auxiliary analyzers. In Pro-ceedings of the 53rd Annual Meeting of the Associ-ation for Computational Linguistics and the 7th In-ternational Joint Conference on Natural LanguageProcessing (Short Papers), pages 857–862.

Chuan Wang, Nianwen Xue, and Sameer Pradhan.2015b. A transition-based algorithm for AMR pars-ing. In Proceedings of the 2015 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, pages 366–375, Denver, Colorado. Asso-ciation for Computational Linguistics.

Keenon Werling, Gabor Angeli, and Christopher Man-ning. 2015. Robust subgraph generation improvesabstract meaning representation parsing. arXivpreprint arXiv:1506.03139.

Junsheng Zhou, Feiyu Xu, Hans Uszkoreit, WeiguangQU, Ran Li, and Yanhui Gu. 2016. AMR Parsingwith an Incremental Joint Model. In Proceedings ofthe 2016 Conference on Empirical Methods in Nat-ural Language Processing, pages 680–689, Austin,Texas. Association for Computational Linguistics.