Cross-lingual Semantic Parsingesslli2018.folli.info/wp-content/uploads/01_dimensions.pdf · Weakly...

Input Output Supervision LexStruct Parsing Features Model Learning LexLearn Setup Evaluation References

Cross-lingual Semantic ParsingPart I: 11 Dimensions of Semantic Parsing

Kilian Evang

University of Dusseldorf

1 / 94


Abbreviations

NL natural languagee.g., English, Bulgarian

NLU natural language utterancee.g., a sentence, a question, a command

MRL meaning representation language (a formal language)e.g., first-order logic, SQL

MR meaning representatione.g., a first-order logical formula, an SQL query

2 / 94


What Is a Semantic Parser?

A program that translates NLUs to MRs, which a computer canexecute.

Example

(1) a. Give me the cities in Virginia.b. answer(A, (city(A), loc(A, B), const(B, stateid(virginia))))

3 / 94


Semantic Parsing—Why?

• Virtual assistants“Wake me up at seven.”

• Natural-language database interfaces“List all 1990s French comedy movies.”

• Machine readingExtract structured information from text

4 / 94


Canonicalization

• many ways to say the same thing in NL

• semantic parsers need to canonicalize

Example

(2) a. What’s the population of Alabama?b. How many people live in Alabama?c. How many citizens in Alabama?d. answer(A, (population(B, A), const(B, stateid(alabama)))

5 / 94


Subproblems of Semantic Parsing

1. Predict the meaning of each word

2. Predict how to assemble word meanings into NLU meanings

6 / 94


Rule-based vs. Learning-based Methods

• Rule-based semantic parsing• high precision, low recall• requires expert effort for each new construction• suitable for small domains, controlled NL

• Learning-based semantic parsing• learns from examples• important for scaling to large domains, variable NL• focus of this course

7 / 94


Subproblems of Semantic Parser Learning

1. lexicon learning: which words mean what?

2. parser learning: how to assemble word meanings into sentencemeanings?

8 / 94


11 Questions for Learning a Semantic Parser

1. What is the input? (NL)

2. What is the output? (MRL)

3. What kind of supervision is available?

4. How do we structure the lexicon?

5. What parsing algorithm do we use?

6. What features do we use?

7. What kind of model do we use?

8. What learning algorithm do we use?

9. How do we learn the lexicon?

10. What experimental setup do we use?

11. How do we evaluate the semantic parser?

9 / 94


Example Semantic Parser

• GeoPar

• simple semantic parser for illustration

• parses questions about United States geography intoGeoQuery MRs

• inspired by Zelle and Mooney (1996)

• optional exercise: implement GeoPar• starter code:https://github.com/texttheater/geopar

• solution for peeking: https://github.com/texttheater/geopar/tree/solution

10 / 94

https://github.com/texttheater/geopar

https://github.com/texttheater/geopar/tree/solution

https://github.com/texttheater/geopar/tree/solution


What Is the Input?

Some Possible Answers

• List of words (GeoPar)e.g., [’what’, ’is’, ’the’, ’capital’, ’of’,’the’, ’state’, ’with’, ’the’, ’largest’,’population’, ’?’]

• List of words + part-of-speech tags

• List of words + word vectors

11 / 94


What Is the Output?

• depends on purpose


• MRs in a special-purpose MRLe.g., GeoQuery (GeoPar)

• MRs in a query language for databases/knowlege basese.g., SQL, SPARQL, description logics

• MRs in a wide-coverage MRLe.g., DRT, AMR

12 / 94


GeoQuery

Example

(3) a. What is the capital of the state with the largestpopulation?

b. answer(C, (capital(C), loc(C, S), largest(P, (state(S),population(S, P)))))

Zelle and Mooney (1996)

13 / 94


GeoQuery (cont.)

• supports queries about U.S. states, cities, rivers, etc.

• Prolog subset

• datasets• Geo880: 880 NLU-MRL pairs, English, widely used as a

benchmark in early work on learning-based semantic parsing(c. 1996–2011), http://www.cs.utexas.edu/users/ml/nldata/geoquery.html

• Geo250: subset with three additional NLs (Turkish, Spanish,Japanese), https://github.com/jimwhite/UBL

14 / 94

http://www.cs.utexas.edu/users/ml/nldata/geoquery.html

http://www.cs.utexas.edu/users/ml/nldata/geoquery.html

https://github.com/jimwhite/UBL


GeoQuery Predicates

Concepts

capital( ) city( ) lake( ) major( ) mountain( ) place( ) state( )river( )

Properties

area( , ) density( , ) elevation( , ) high point( , ) len( , )population( , ) size( , )

Relationscapital( , ) loc( , ) next to( , ) traverse( , )

15 / 94


GeoQuery Predicates (cont.)

Constantsconst( , countryid(usa))const( , stateid(mississippi))const( , cityid(’los angeles’, ca))const( , riverid(mississippi))const( , placeid(’mount whitney’))...

Aggregation

count( , , ) fewest( , , ) higher( , ) highest( , ) largest( , )longer( , ) lowest( , ) most( , , ) shortest( , ) smallest( , )sum( , , )

Negation

\+

16 / 94


Description Logics (DL)

Example

(4) a. What college did Obama go to?b. Type.University u Education.BarackObama

Berant et al. (2013)

17 / 94


Description Logics (cont.)

• DL expressions describe concepts

• Application-dependent set of concept symbols and relationsymbols

• .: join a relation with a concept

• u: intersection of concepts

• translated to SQL or SPARQL for querying databases such asFreebase

• datasets• Free917 (917 NLU-MR pairs)• WebQuestions (5810 NLU-answer pairs)• https://nlp.stanford.edu/software/sempre/

18 / 94

https://nlp.stanford.edu/software/sempre/


Discourse Representation Structures (DRS)

Example

(5) a. All equipment will be completely manufactured.

b. x1equipment.n.01(x1)

⇒

e1 s1manufacture.v.01(e1)Manner(e1, s1)Result(e1, x1)complete.r.01(s1)

Bjerva et al. (2016)

19 / 94


Discourse Representation Structures (DRS) (cont.)

• Discourse Representation Theory (DRT) (Kamp, 1984)

• general-purpose meaning representation language

• uses discourse referents, event semantics, logical connectives

• WordNet sense inventory, VerbNet frame inventory

• datasets• Groningen Meaning Bank (http://gmb.let.rug.nl),

English• Parallel Meaning Bank (http://pmb.let.rug.nl),

English, Dutch, Italian, German

20 / 94

http://gmb.let.rug.nl

http://pmb.let.rug.nl


Abstract Meaning Representations (AMR)

Example

(6) a. All equipment will be completely manufactured.b.

Bjerva et al. (2016)

21 / 94


Abstract Meaning Representations (AMR) (cont.)

• graph-based general-purpose meaning representation(Banarescu et al., 2013)

• somewhat limited expressive power compared to DRS (Bos,2016)

• no word sense annotation, PropBank frame inventory

• datasets• https://amr.isi.edu/download.html

22 / 94

https://amr.isi.edu/download.html


What Kind of Supervision is Available?


• NLU-MR pairs (GeoPar)

• question-answer pairs + database

• text + database

• NLU-MR pairs in another language (cross-lingually supervised)

23 / 94


Supervised Learning from NLU-MR Pairs

Example


b. answer(C, (capital(C), loc(C, S), largest(P, (state(S),population(S, P)))))

Zelle and Mooney (1996)

24 / 94


Supervised Learning from NLU-MR Pairs (cont.)

• parser trained to produce gold-standard MRs

• requires expert annotation

25 / 94


Supervised Learning from Question-Answer Pairs +Database

Liang et al. (2011)

26 / 94


Supervised Learning from Question-Answer Pairs +Database (cont.)

• parser trained to produce any MR, as long as executing it onDB gives gold-standard answer

• enables annotation by non-experts

27 / 94


Weakly Supervised Learning from Text + Database

Reddy et al. (2014)

28 / 94


Weakly Supervised Learning from Text + Database (cont.)

• convert output of syntactic parser to ungrounded MR

• learn to map ungrounded MRs to grounded MRs (databasesubgraphs) so that facts found in Web text correspond tofacts found in database

29 / 94


Cross-lingually Supervised Learning from NLU-MR Pairs inanother Language

Evang and Bos (2016)

30 / 94


Cross-lingually Supervised Learning from NLU-MR Pairs inanother Language (cont.)

• have: semantic parser for language A

• want: semantic parser for language B

• use parallel data to project parses from A to B, train semanticparser on projected parses

31 / 94


How do We Structure the Lexicon?


• Lexicon = a mapping from (multi)words to predicates(GeoPar)

• Lexicon = a mapping from (multi)words to lambdaexpressions

• often with CCG categories (adding syntactic type and wordorder information)

32 / 94


The GeoPar Lexicon Format

Example NLU-MR Pair

(8) a. What states border Texas?b. answer(S, (state(S), next to(S, T), const(T,

stateid(texas))))

Corresponding Lexicon Entries

states ` state( , )border ` next to( , )Texas ` const( , stateid(texas))

33 / 94


A λ-based Lexicon Format

Example NLU-MR Pair

(9) a. What states border Texas?b. λs.state(s) ∧ next to(s, texas)


What ` λf .λg .λx .(f (x) ∧ g(x))states ` λx .state(x)border ` λx .λy .next to(y , x)Texas ` texasZettlemoyer and Collins (2005)

34 / 94


A λ-based Lexicon Format with CCG Categories

Example NLU-MR Pair



What ` λf .λg .λx .(f (x) ∧ g(x))states ` λx .state(x)border ` λx .λy .next to(y , x)Texas ` texasZettlemoyer and Collins (2005)

34 / 94



Example NLU-MR Pair



What ` λf .λg .λx .(f (x) ∧ g(x))states ` N : λx .state(x)border ` λx .λy .next to(y , x)Texas ` texasZettlemoyer and Collins (2005)

34 / 94



Example NLU-MR Pair



What ` λf .λg .λx .(f (x) ∧ g(x))states ` N : λx .state(x)border ` λx .λy .next to(y , x)Texas ` NP : texasZettlemoyer and Collins (2005)

34 / 94



Example NLU-MR Pair



What ` λf .λg .λx .(f (x) ∧ g(x))states ` N : λx .state(x)border ` (S \NP)/NP : λx .λy .next to(y , x)Texas ` NP : texasZettlemoyer and Collins (2005)

34 / 94



Example NLU-MR Pair



What ` (S /(S \NP))/N : λf .λg .λx .(f (x) ∧ g(x))states ` N : λx .state(x)border ` (S \NP)/NP : λx .λy .next to(y , x)Texas ` NP : texasZettlemoyer and Collins (2005)

34 / 94


What Parsing Algorithm Do We Use?


• CYK variant

• Shift-reduce variant (GeoPar)

35 / 94


CYK Parsing

• for a long time the mostpopular algorithm for syntacticand semantic parsing

• each cell stands for one subspanof the input

• cells filled with all possibleconstituents/arcs/MRs overthat span

• shorter spans first, longer spanslater

• quadratic running time evenwith beam search

36 / 94


Shift-reduce Parsing

• a transition-based family of algorithms

• more popular for syntactic and semantic parsing recently

• processes input from left to right

• builds constituents/arcs/MRs from left to right

• advantage: linear running time (when greedy or using beamsearch)

• advantage: psychological plausbility

37 / 94


A Shift-reduce Algorithm for GeoQuery

• a parser state (aka parse item, configuration) consists of astack and a queue

• in the initial state, the stack contains 1 term: answer( , ),and the queue contains the whole input NLU

• in a final state, the stack contains 1 term (the output MR)and the queue is empty

• actions (aka transitions) take the parser from one state (thepredecessor) to the next state (its successor)

38 / 94


We Need Actions to...

Example


b. answer(C, (capital(S, C), largest(P, (state(S),population(S, P)))))

• ...introduce predicate instances

• ...coreference variables

• ...embed one predicate instance under another

• ...conjoin two predicate instances

39 / 94



Example


b. answer(C, ( capital (S, C), largest(P, (state(S),population(S, P)))))





39 / 94



Example


b. answer(C, (capital( S , C), largest(P, (state( S ),population(S, P)))))





39 / 94



Example


b. answer(C, (capital(S, C), largest (P, (state(S),

population (S, P)))))





39 / 94



Example


b. answer(C, (capital(S, C), largest(P, ( state (S),

population (S, P)))))





39 / 94


Notation

• q[0]: leftmost element of queue

• q[:n]: first n elements of queue

• s[0]: topmost element of stack

• s[1]: second-topmost element of stack

• Each stack element s[i] has a secondary stack s[i].sec which isinitially empty but may later contain subterms of s[i]. Whens[i].sec is empty, then s[i].sec[0] refers to s[i].

• j marks the subterm found at s[i].sec[j]

• t.args: the arguments of the term t

40 / 94


Action Types

• shift-n-P: remove q[:n] from queue and append P to stacksuch that q[:n]` P in lexicon

• skip: remove q[0]

• coref- i -j- k -l: unify the variables at s[1].sec[i].args[j - 1] ands[0].sec[k].args[l - 1]

• drop-i : move s[0] into s[1].sec[0].args[i - 1], push it ontos[1].sec

• precondition: s[0].sec is empty

• lift-i : move s[1] into s[0].sec[0].args[i - 1], push it ontos[0].sec

• precondition: s[1].sec is empty

• conj: move s[0] to form a conjunction with s[1].sec[0], push itonto s[1].sec

• pop: pop 1 element from s[0].sec

• idle: no-op for final items

41 / 94


Example Parseanswer(C, (capital(S, C), largest(P, (state(S), population(S, P)))))

action stack queue0 answer( , ) what is the capital of the state with the largest population ?

1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?

5 coref- 0 -1- 0 -2 answer(C, ) capital( , C) of the state with the largest population ?

6 drop-2 answer(C, 0 capital( , C)) of the state with the largest population ?

7 skip answer(C, 0 capital( , C)) the state with the largest population ?

8 skip answer(C, 0 capital( , C)) state with the largest population ?

9 shift-1-state( ) answer(C, 0 capital( , C)) state( ) with the largest population ?

10 coref- 0 -1- 0 -1 answer(C, 0 capital(S, C)) state(S) with the largest population ?

11 skip answer(C, 0 capital(S, C)) state(S) the largest population ?

12 skip answer(C, 0 capital(S, C)) state(S) largest population ?

13 shift-1-largest( , ) answer(C, 0 capital(S, C)) state(S) largest( , ) population ?

14 lift-2 answer(C, 0 capital(S, C)) largest( , 0 state(S)) population ?

15 shift-1-population( , ) answer(C, 0 capital(S, C)) largest( , 0 state(S)) population( , ) ?

16 coref- 0 -1- 0 -1 answer(C, 0 capital(S, C)) largest( , 0 state(S)) population(S, ) ?

17 coref- 1 -1- 0 -2 answer(C, 0 capital(S, C)) largest(P, 0 state(S)) population(S, P) ?

18 conj answer(C, 0 capital(S, C)) largest(P, 1 (state(S), 0 population(S, P))) ?

19 pop answer(C, 0 capital(S, C)) largest(P, 0 (state(S), population(S, P))) ?

20 pop answer(C, 0 capital(S, C)) largest(P, (state(S), population(S, P))) ?

21 conj answer(C, 1 (capital(S, C), 0 largest(P, (state(S), population(S, P))))) ?

22 pop answer(C, 0 (capital(S, C), largest(P, (state(S), population(S, P))))) ?23 pop answer(C, (capital(S, C), largest(P, (state(S), population(S, P))))) ?24 skip answer(C, (capital(S, C), largest(P, (state(S), population(S, P)))))

42 / 94



action stack queue0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?

2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?



















42 / 94



action stack queue0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?

3 skip answer( , ) capital of the state with the largest population ?4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?



















42 / 94



action stack queue0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?

4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?



















42 / 94



action stack queue0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?



















42 / 94





















22 pop answer(C, 0 (capital(S, C), largest(P, (state(S), population(S, P))))) ?

23 pop answer(C, (capital(S, C), largest(P, (state(S), population(S, P))))) ?24 skip answer(C, (capital(S, C), largest(P, (state(S), population(S, P)))))

42 / 94





















22 pop answer(C, 0 (capital(S, C), largest(P, (state(S), population(S, P))))) ?23 pop answer(C, (capital(S, C), largest(P, (state(S), population(S, P))))) ?

24 skip answer(C, (capital(S, C), largest(P, (state(S), population(S, P)))))

42 / 94






















42 / 94


Parser States as Python Objects

class ParserState:def __init__(self, stack, queue, pred, action):

self.stack = stackself.queue = queueself.pred = pred # predecessor parser stateself.action = action # action leading from pred

# to selfdef shift(self, n, term):

queue = self.queuefor i in range(n):

queue = queue.pop()stack = self.stack.push(term)return ParserState(stack, queue, self,

(’shift’, n, term.to_string()))def skip(self):

return ParserState(self.stack, self.queue.pop(),self, (’skip’,))

# ...

43 / 94


What Features Do We Use?

• parser must be decide what is the next action for each state

• represent state as feature vectors to make this easier


• Discrete representations of parser states (do featureengineering; GeoPar)

• Continuous representations of parser states (use neuralnetwork)

44 / 94


Predicting Actions using Featuresaction stack queue

0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?



















45 / 94



0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?




















45 / 94



0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?

3 skip answer( , ) capital of the state with the largest population ?




















45 / 94



0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?

2 skip answer( , ) the capital of the state with the largest population ?





















45 / 94



0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?4 shift-1-capital( , ) answer( , ) capital( , ) of the state with the largest population ?



















46 / 94



0 answer( , ) what is the capital of the state with the largest population ?1 skip answer( , ) is the capital of the state with the largest population ?2 skip answer( , ) the capital of the state with the largest population ?3 skip answer( , ) capital of the state with the largest population ?




















46 / 94



0 answer( , ) what is the capital of the state with the largest population ?

1 skip answer( , ) is the capital of the state with the largest population ?

2 skip answer( , ) the capital of the state with the largest population ?





















46 / 94


Feature Extraction Code

class ParserState:# ...def local_features(self):

def get_stack_terms():for sp, ssp in ((0, 0), (0, 1),

(1, 0), (1, 1), (1, 2)):try:

yield self.stack[sp].sec[ssp]except IndexError:

yield None(s00, s01, s10, s11, s12) = get_stack_terms()# ...

47 / 94


Feature Extraction Code (cont.)


# ...def get_words():

for i in (-4, -3, -2, -1, 0, 1, 2, 3):try:

yield self.words[self.offset + i]except IndexError:

yield None(W4, W3, W2, W1, w0, w1, w2, w3) = get_words()# ...

48 / 94




# ...tf = {} # map from templates to features# feature group 1: stack predicatestf[’s00p’] = s00.pred if s00 is not None else Nonetf[’s01p’] = s00.pred if s01 is not None else Nonetf[’s10p’] = s10.pred if s10 is not None else Nonetf[’s11p’] = s11.pred if s10 is not None else Nonetf[’s12p’] = s12.pred if s12 is not None else None# ...

49 / 94




# ...# feature group 2: combinations of stack predicatesfor s0ip in (’s00p’, ’s01p’):

for s1jp in (’s10p’, ’s11p’, ’s12p’):tf[(s0ip, s1jp)] = (tf[s0ip], tf[s1jp])

# ...

50 / 94




# ...# feature group 3: unigramstf[’W4’] = W4tf[’W3’] = W3tf[’W2’] = W2tf[’W1’] = W1tf[’w0’] = w0tf[’w1’] = w1tf[’w2’] = w2tf[’w3’] = w3# ...

51 / 94




# ...# feature group 4: bigramsunigram_templates = (’W4’, ’W3’, ’W2’, ’W1’,

’w0’, ’w1’, ’w2’, ’w3’)for (wi, wj) in zip(unigram_templates,

unigram_templates[1:]):tf[(wi, wj)] = (tf[wi], tf[wj])

# feature group 5: trigramsfor (wi, wj, wj) in zip(unigram.templates,

unigram_templates[1:],unigram.templates[2:]):

tf[(wi, wj, wk)] = (tf[wi], tf[wj], tf[wk])# ...

52 / 94




# ...# Yield all template features as strings:for (template, value) in tf.items():

if isinstance(template, tuple):template = ’ ’.join(template)

if isinstance(value, tuple):value = ’ ’.join(str(x) for x in value)

yield template + ’ = ’ + value# Also yield a bias feature:yield ’bias’

53 / 94



class ParserState:# ...def features(self):

if self._features is None: # memoizeif self.pred is None:

self._features = collections.Counter()else:

self._features = collections.Counter(self.pred.features())

action = ’ ’.join(str(x) for x in self.action)

for f in self.pred.local_features():f = f + ’ : ’ + actionself._features[f] += 1

return self._features

54 / 94


Some Caveats

• in practice, use numpy arrays rather thancollections.Counter for efficiency

• use a vector compression method such as hash kernels(Bohnet, 2010) to limit vector size

• need more complex features for good performance, e.g.,• combinations of queue words and stack predicates• stack predicate classes, e.g. aggregate vs. concept• previous actions• various combinations thereof

55 / 94


Example Features

Features Usually Indicative of Good Parsing Decisions

’w0 = capital : shift 1 capital’’s10p W3 W2 W1 = answer what is the : coref 0 1 0 2’

Features Usually Indicative of Bad Parsing Decisions

’w0 = capital : skip’’W1 = the : skip’

56 / 94


What Kind of Model Do We Use?


• Linear model (GeoPar)

• Log-linear model

• Neural network

57 / 94


Linear Model

• associates every feature with a real-valued weight• positive weights for features indicative of good parsing

decisions• negative weights for features indicative of bad parsing decisions• weights close to 0 for features not very indicative

• the score of a parser state is the sum of the counts of each ofits features multiplied by its weight

• parser will choose actions that lead to parser states withhigher scores

58 / 94


Model Code

class LinearModel:

def __init__(self):self.weights = collections.Counter()

def score(self, features):result = 0for feature in features:

result += self.weights[feature]return result

59 / 94


Parsing with Beam Search

def parse(words, model, beam_size=10):agenda = [initial_state(words)]while any(not st.is_final() for st in agenda):

agenda = [su for st in agendafor su in st.successors()]

agenda.sort(key=lambda st: model.score(st.features()),reverse=True)

beam = agenda[:min(beam_size, len(agenda))]agenda = beam

if not agenda:raise ValueError(’no parse found’)

return agenda[0].stack[0]

60 / 94


What Learning Algorithm Do We Use?

• How do we set the weights?

• Need to train our model on the training data


• Perceptron (GeoPar)

• Some variant of stochastic gradient descent

61 / 94


Perceptron Update

class LinearModel:

# ...

def update(self, gold_features, predicted_features):update = collections.Counter()for gf in gold_features:

update[gf] += 1for pf in predicted_features:

update[pf] -= 1for feature, delta in update.items():

if delta != 0:self.weights[feature] += delta

62 / 94


Perceptron Training

def train_one_epoch(train_data, model, beam_size=10):for words, gold_action_sequence in train_data:

def is_correct(state):# a state is considered correct if the action sequence leading up# to it is a prefix of the complete gold action sequence, or vice# versa:return all(g == i for g, i in zip(gold_action_sequence,

state.action_sequence()))agenda = [initial_state(words)]while any(not st.is_final() for st in agenda):

agenda = [su for st in agendafor su in st.successors()]

agenda.sort(key=lambda st: model.score(st.features()),reverse=True)

beam = agenda[:min(beam_size, len(agenda))]if not any(is_correct(st) for st in beam):

break # early updateagenda = beam

best = agenda[0]best_correct = next(st for st in agenda if is_correct(st))if best != best_correct:

model.update(best_correct.features(),best.features())

63 / 94


Perceptron Training for Multiple Epochs

def train(train_data, num_epochs):train_data = list(train_data)model = LinearModel()for t in range(num_epochs):

random.shuffle(train_data)train_one_epoch(train_data, model)

return model

64 / 94


Some Caveats

• again, model.weights should be a numpy array in practice

• use averaged perceptron: keep track of weights after eachupdate, average them at the end

• rather than fixing number of epochs, use validation data tostop training once further epochs do not improve accuracy

65 / 94


How Do We Learn the Lexicon?


• specify lexicon manually (GeoPar)

• use some combination of• unsupervised word-predicate alignment• extension-based lexicon induction• lexical templates• splitting via higher-order unification• iterative lexicon refinement• cross-lingual lexicon induction

66 / 94


Unsupervised Word-predicate Alignment

What states border Texas

answer(S, ( state(S), next to(S, T) const(T, stateid(texas))))

67 / 94


Unsupervised Word-predicate Alignment (cont.)

• widely used technique in machine translation

• use, e.g., GIZA++ or fast align

• alignments are uncertain

• possible solution: generate multiple lexical entries; let thelearning algorithm figure out which ones are good/bad

68 / 94


Extension-based Lexicon Induction

Berant et al. (2013)

69 / 94


Extension-based Lexicon Induction (cont.)

• use predicates in a DB, e.g., PlaceOfBirth

• the extension of a predicate is the set of pairs of entities it istrue of, e.g., (BarackObama, Honolulu), (MichelleObama,Chicago)

• find phrases in raw text, e.g., “lived in”

• the extension of a phrase is the set of pairs of entitiesmentioned in the same sentence, e.g. (BarackObama,Honolulu), (BarackObama, Chicago)

• map entity mentions to entities based on string similarityheuristics

• learn to map phrases to predicates based on the similarity oftheir extensions

70 / 94


Lexical Templates

Zettlemoyer and Collins (2005)

71 / 94


Lexical Templates (cont.)

• use templates to generate CCG lexical entries for words basedon co-occurring predicates and semantic types

• e.g., templates for nouns, names, verbs

• disadvantage: templates must be hand-written and depend onNL and MRL

72 / 94


Splitting Entries via Higher-order Unification

L1New York borders Vermont ` S : next to(ny , vt)

L2New York borders ` S /NP : λx .next to(ny , x)Vermont ` NP : vt

L3New York ` NP : nyborders ` (S \NP)/NP : λx .λy .next to(y , x)Kwiatkowksi et al. (2010); Kwiatkowski et al. (2011)

73 / 94


Splitting Entries via Higher-order Unification (cont.)

• initial lexical entries = NLU-MR pairs

• repeatedly split entries in two to arrive at a lexicon thatgeneralizes well

• disadvantage: exponentially many possibilities, need heuristicsand word alignments to guide splitting

74 / 94


Iterative Lexicon Refinement

Artzi et al. (2013)

75 / 94


Iterative Lexicon Refinement (cont.)

Idea: alternate between updating model and lexicon

1. generate candidate lexicon based on templates or splitting

2. parse training sentences with candidate lexicon

3. take lexicon entries used in highest-scoring parses and addthem to main lexicon

4. update the model (as with perceptron training)

5. repeat

76 / 94


What Experimental Setup Do We Use?

data

test data training data

validation data training data proper

77 / 94


k-fold Cross Validation

https://tex.stackexchange.com/a/429464/61093

78 / 94

https://tex.stackexchange.com/a/429464/61093


How Do We Evaluate the Semantic Parser?


• exact match of MR (GeoPar)

• exact match of answer

• partial match of MR (e.g., Smatch)

79 / 94


Exact Match Evaluation

• coverage: parses returned / sentences in test set

• precision: MR = gold MR / parses returned

• recall = accuracy: MR = gold MR / sentences in test set

80 / 94


Exact Match of MR: Some Accuracies

Zettlemoyer and Collins (2005) 79.3Zettlemoyer and Collins (2007) 86.1Kwiatkowksi et al. (2010) 87.9Kwiatkowski et al. (2011) 88.6

81 / 94


Exact Match of Answer: Some Accuracies

Kwiatkowksi et al. (2010) 88.9Liang et al. (2011) 91.1

82 / 94


Partial Match of MR

• For wide-coverage MRs like DRT and AMR, exact matchesare still rare

• Semantic parsers are benchmarked using overlap of gold MRand predicted MR

• Problem: maximum common subgraph problem is NP-hard

• Solution: Smatch approximation (Cai and Knight, 2013)

83 / 94


AMR Partial Match Example

instance(a, want-01) ∧instance(b, boy) ∧instance(c, go-01) ∧ARG0(a, b) ∧ARG1(a, c) ∧ARG0(c, b)

instance(x, want-01) ∧instance(y, boy) ∧instance(z, football) ∧ARG0(x, y) ∧ARG1(x, z)

Cai and Knight (2013)

84 / 94


AMR Partial Match Example

instance(a, want-01) ∧instance(b, boy) ∧

instance(c, go-01) ∧ARG0(a, b) ∧ARG1(a, c) ∧

ARG0(c, b)

instance(x, want-01) ∧instance(y, boy) ∧

instance(z, football) ∧ARG0(x, y) ∧ARG1(x, z)

a=x, b=y, c=zPrecision: 4/5; Recall: 4/6; F1: 0.73

Cai and Knight (2013)

85 / 94


AMR (LDC2014T12): Some Smatch Scores

Flanigan et al. (2016) 0.59Damonte et al. (2016) 0.64Zhou et al. (2016) 0.66Pust et al. (2015) 0.66Wang et al. (2015) 0.66Ballesteros and Al-Onaizan (2017) 0.64

86 / 94


Bibliography I

Artzi, Y., FitzGerald, N., and Zettlemoyer, L. (2013). Semanticparsing with combinatory categorial grammars. ACL 2013tutorial.

Ballesteros, M. and Al-Onaizan, Y. (2017). Amr parsing usingstack-lstms. In Proceedings of the 2017 Conference on EmpiricalMethods in Natural Language Processing, pages 1269–1275.Association for Computational Linguistics.

Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K.,Hermjakob, U., Knight, K., Koehn, P., Palmer, M., andSchneider, N. (2013). Abstract meaning representation forsembanking. In Proceedings of the 7th Linguistic AnnotationWorkshop and Interoperability with Discourse, pages 178–186.Association for Computational Linguistics.

87 / 94


Bibliography II

Berant, J., Chou, A., Frostig, R., and Liang, P. (2013). Semanticparsing on freebase from question-answer pairs. In Proceedingsof the 2013 Conference on Empirical Methods in NaturalLanguage Processing, pages 1533–1544. Association forComputational Linguistics.

Bjerva, J., Bos, J., and Haagsma, H. (2016). The meaning factoryat semeval-2016 task 8: Producing amrs with boxer. InProceedings of the 10th International Workshop on SemanticEvaluation (SemEval-2016), pages 1179–1184. Association forComputational Linguistics.

Bohnet, B. (2010). Top accuracy and fast dependency parsing isnot a contradiction. In Proceedings of the 23rd InternationalConference on Computational Linguistics (Coling 2010), pages89–97. Coling 2010 Organizing Committee.

88 / 94


Bibliography III

Bos, J. (2016). Squib: Expressive power of abstract meaningrepresentations. Computational Linguistics, Volume 42, Issue 3 -September 2016, pages 527–535.

Cai, S. and Knight, K. (2013). Smatch: an evaluation metric forsemantic feature structures. In Proceedings of the 51st AnnualMeeting of the Association for Computational Linguistics(Volume 2: Short Papers), pages 748–752. Association forComputational Linguistics.

Damonte, M., Cohen, S. B., and Satta, G. (2016). An incrementalparser for abstract meaning representation. arXiv preprintarXiv:1608.06111.

89 / 94


Bibliography IV

Evang, K. and Bos, J. (2016). Cross-lingual learning of anopen-domain semantic parser. In Proceedings of COLING 2016,the 26th International Conference on Computational Linguistics:Technical Papers, pages 579–588. The COLING 2016 OrganizingCommittee.

Flanigan, J., Dyer, C., Smith, N. A., and Carbonell, J. (2016).Cmu at semeval-2016 task 8: Graph-based amr parsing withinfinite ramp loss. In Proceedings of the 10th InternationalWorkshop on Semantic Evaluation (SemEval-2016), pages1202–1206. Association for Computational Linguistics.

Kamp, H. (1984). A Theory of Truth and SemanticRepresentation. In Groenendijk, J., Janssen, T. M., and Stokhof,M., editors, Truth, Interpretation and Information, pages 1–41.FORIS, Dordrecht, Holland/Cinnaminson, U.S.A.

90 / 94


Bibliography V

Kwiatkowksi, T., Zettlemoyer, L., Goldwater, S., and Steedman,M. (2010). Inducing probabilistic CCG grammars from logicalform with higher-order unification. In Proceedings of the 2010Conference on Empirical Methods in Natural LanguageProcessing, pages 1223–1233.

Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., and Steedman,M. (2011). Lexical generalization in ccg grammar induction forsemantic parsing. In Proceedings of the 2011 Conference onEmpirical Methods in Natural Language Processing, pages1512–1523. Association for Computational Linguistics.

Liang, P., Jordan, M., and Klein, D. (2011). Learningdependency-based compositional semantics. In Proceedings ofthe 49th Annual Meeting of the Association for ComputationalLinguistics: Human Language Technologies, pages 590–599.Association for Computational Linguistics.

91 / 94


Bibliography VI

Pust, M., Hermjakob, U., Knight, K., Marcu, D., and May, J.(2015). Parsing english into abstract meaning representationusing syntax-based machine translation. In Proceedings of the2015 Conference on Empirical Methods in Natural LanguageProcessing, pages 1143–1154. Association for ComputationalLinguistics.

Reddy, S., Lapata, M., and Steedman, M. (2014). Large-scalesemantic parsing without question-answer pairs. Transactions ofthe Association for Computational Linguistics, 2:377–392.

Wang, C., Xue, N., and Pradhan, S. (2015). Boostingtransition-based amr parsing with refined actions and auxiliaryanalyzers. In Proceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the 7thInternational Joint Conference on Natural Language Processing

92 / 94


Bibliography VII

(Volume 2: Short Papers), pages 857–862. Association forComputational Linguistics.

Zelle, J. M. and Mooney, R. J. (1996). Learning semanticgrammars with constructive inductive logic programming. InProceedings of the Eleventh National Conference of theAmerican Association for Artificial Intelligence (AAAI-93), pages817–822.

Zettlemoyer, L. and Collins, M. (2005). Learning to map sentencesto logical form: Structured classification with probabilisticcategorial grammars. In Proceedings of the Twenty-FirstConference Annual Conference on Uncertainty in ArtificialIntelligence (UAI-05), pages 658–666. AUAI Press.

93 / 94


Bibliography VIII

Zettlemoyer, L. and Collins, M. (2007). Online learning of relaxedccg grammars for parsing to logical form. In Proceedings of the2007 Joint Conference on Empirical Methods in NaturalLanguage Processing and Computational Natural LanguageLearning (EMNLP-CoNLL).

Zhou, J., Xu, F., Uszkoreit, H., QU, W., Li, R., and Gu, Y. (2016).Amr parsing with an incremental joint model. In Proceedings ofthe 2016 Conference on Empirical Methods in Natural LanguageProcessing, pages 680–689. Association for ComputationalLinguistics.

94 / 94

Cross-lingual Semantic Parsingesslli2018.folli.info/wp-content/uploads/01_dimensions.pdf · Weakly...

Documents

Transcript of Cross-lingual Semantic Parsingesslli2018.folli.info/wp-content/uploads/01_dimensions.pdf · Weakly...