Introduction Derivation Projection Experiments Conclusions References

Cross-lingual CCG Induction

Kilian Evang@texttheater

University of Dusseldorf

2019-06-04NAACL-HLT

1 / 23

Introduction Derivation Projection Experiments Conclusions References

Outline

Introduction

Derivation Projection

Experiments

Conclusions

2 / 23

Introduction Derivation Projection Experiments Conclusions References

Outline

Introduction

Derivation Projection

Experiments

Conclusions

3 / 23

Introduction Derivation Projection Experiments Conclusions References

Combinatory Categorial Grammar

We

NP

sang

S \NP

S<0

3 / 23

Introduction Derivation Projection Experiments Conclusions References

Combinatory Categorial Grammar

We

NP

saw

(S \NP)/NP

the

NP /N

car

N

that

(N \N)/(S /NP)

John

N

bought

(S \NP)/NP

NP∗

S /(S \NP)T>

S /NP>1

N \N>0

N<0

NP>0

NP>0

S \NP>0

S<0

4 / 23

Introduction Derivation Projection Experiments Conclusions References

Appeal

• coordination

• universal rules

• syntax-semantics interface

5 / 23

Introduction Derivation Projection Experiments Conclusions References

Most CCG Parsers

• trained on large treebanks, or

• hand-crafted

6 / 23

Introduction Derivation Projection Experiments Conclusions References

David Blackwell, CC-BY-NC

What about low-resource languages?

7 / 23

Introduction Derivation Projection Experiments Conclusions References

Unsupervised CCG Induction?

target-language text+

magic=

target-language CCG parser

(Bisk and Hockenmaier, 2013; Bisk et al., 2015)

8 / 23

Introduction Derivation Projection Experiments Conclusions References

Cross-lingual CCG Induction?

English CCG parser+

parallel corpus+

magic=

target-language CCG parser

9 / 23

Introduction Derivation Projection Experiments Conclusions References

Cross-lingual CCG Induction via Derivation Projection

parallel corpus+

English CCG derivations+

word alignments+

derivation projection=

target-language CCG derivations=

target-language training data

10 / 23

Introduction Derivation Projection Experiments Conclusions References

Outline

Introduction

Derivation Projection

Experiments

Conclusions

11 / 23

Introduction Derivation Projection Experiments Conclusions References

Derivation Projection

• project lexical categories along word alignments

• n:1 alignment → merge

• word order difference → flip slash

11 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva tre figli

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva tre

N4 /N5

figli

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva tre

N4 /N5

figli

N5

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva

S2 /NP3

tre

N4 /N5

figli

N5

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva

S2 /NP3

tre

N4 /N5

figli

N5

N4

>0

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva

S2 /NP3

tre

N4 /N5

figli

N5

N4

>0

NP3

∗

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 1/3

He

NP1

had

(S2 \NP1)/NP3

three

N4 /N5

sons

N5

N4>0

NP3 ∗

S2 \NP1

>0

S2<0

Aveva

S2 /NP3

tre

N4 /N5

figli

N5

N4

>0

NP3

∗

S2

>0

12 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una pianta molto decorativa

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta molto decorativa

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta

N3

molto decorativa

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta

N3

molto

(N2 \N3)/(N4 \N5)

decorativa

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta

N3

molto

(N2 \N3)/(N4 \N5)

decorativa

N4 \N5

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta

N3

molto

(N2 \N3)/(N4 \N5)

decorativa

N4 \N5

N2 \N3

>0

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta

N3

molto

(N2 \N3)/(N4 \N5)

decorativa

N4 \N5

N2 \N3

>0

N2

<0

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 2/3

a

NP1 /N2

very

(N2 /N3)/(N4 /N5)

decorative

N4 /N5

plant

N3

N2 /N3

>0

N2>0

NP1>0

una

NP1 /N2

pianta

N3

molto

(N2 \N3)/(N4 \N5)

decorativa

N4 \N5

N2 \N3

>0

N2

<0

NP1

>0

13 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 3/3

Do

(S1 \NP2)/(S3 \NP4)

n’t

(S5 \NP6)\(S1 \NP2)

mess

((S3 \NP4)/PR7)/NP8

it

NP8

up

PR7

(S5 \NP6)/(S3 \NP4)<1

×(S3 \NP4)/PR7

>0

S3 \NP4

>0

S5 \NP6

>0

Versau es nicht

14 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 3/3

Do

(S1 \NP2)/(S3 \NP4)

n’t

(S5 \NP6)\(S1 \NP2)

mess

((S3 \NP4)/PR7)/NP8

it

NP8

up

PR7

(S5 \NP6)/(S3 \NP4)<1

×(S3 \NP4)/PR7

>0

S3 \NP4

>0

S5 \NP6

>0

Versau es

NP8

nicht

14 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 3/3

Do

(S1 \NP2)/(S3 \NP4)

n’t

(S5 \NP6)\(S1 \NP2)

mess

((S3 \NP4)/PR7)/NP8

it

NP8

up

PR7

(S5 \NP6)/(S3 \NP4)<1

×(S3 \NP4)/PR7

>0

S3 \NP4

>0

S5 \NP6

>0

Versau es

NP8

nicht

(S5 \NP6)\(S3 \NP4)

14 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 3/3

Do

(S1 \NP2)/(S3 \NP4)

n’t

(S5 \NP6)\(S1 \NP2)

mess

((S3 \NP4)/PR7)/NP8

it

NP8

up

PR7

(S5 \NP6)/(S3 \NP4)<1

×(S3 \NP4)/PR7

>0

S3 \NP4

>0

S5 \NP6

>0

Versau

(S3 \NP4)/NP8

es

NP8

nicht

(S5 \NP6)\(S3 \NP4)

14 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 3/3

Do

(S1 \NP2)/(S3 \NP4)

n’t

(S5 \NP6)\(S1 \NP2)

mess

((S3 \NP4)/PR7)/NP8

it

NP8

up

PR7

(S5 \NP6)/(S3 \NP4)<1

×(S3 \NP4)/PR7

>0

S3 \NP4

>0

S5 \NP6

>0

Versau

(S3 \NP4)/NP8

es

NP8

nicht

(S5 \NP6)\(S3 \NP4)

S3 \NP4

>0

14 / 23

Introduction Derivation Projection Experiments Conclusions References

Example 3/3

Do

(S1 \NP2)/(S3 \NP4)

n’t

(S5 \NP6)\(S1 \NP2)

mess

((S3 \NP4)/PR7)/NP8

it

NP8

up

PR7

(S5 \NP6)/(S3 \NP4)<1

×(S3 \NP4)/PR7

>0

S3 \NP4

>0

S5 \NP6

>0

Versau

(S3 \NP4)/NP8

es

NP8

nicht

(S5 \NP6)\(S3 \NP4)

S3 \NP4

>0

S5 \NP6

<0

14 / 23

Introduction Derivation Projection Experiments Conclusions References

Outline

Introduction

Derivation Projection

Experiments

Conclusions

15 / 23

Introduction Derivation Projection Experiments Conclusions References

Training

• Parallel corpus: tatoeba.org

• English parser: EasyCCG trained on CCGrebank

• Word aligments: GIZA++

• Target-language parser: EasyCCG trained on projectedderivations

15 / 23

Introduction Derivation Projection Experiments Conclusions References

Evaluation

• PASCAL challenge on unsupervised grammar induction:Arabic, Czech, Danish, Basque, Dutch, Portuguese, Slovenian,Swedish

• unlabeled dependency f-score

16 / 23

Introduction Derivation Projection Experiments Conclusions References

Training (cont.)

ara ces dan eus nld por slv swe

sentence pairs 20K 11K 21K 2K 44K 161K 835 24Kprojected 7K 4K 11K 590 18K 50K 364 12K

17 / 23

Introduction Derivation Projection Experiments Conclusions References

Baselines

• BH13: CCG induction from raw text + POS tags(Bisk and Hockenmaier, 2013)

• BCH15: CCG induction from raw text(Bisk et al., 2015)

18 / 23

Introduction Derivation Projection Experiments Conclusions References

Results

Language ara ces dan eus nld por slv swe

Monolingual training on PASCALTrain tokens 5K 436K 25K 81K 79K 159K 54K 62KBH13 .651 .507 .585 .450 .544 .629 .464 .669BCH15 .437 .324 .377 .352 .438 .516 .236 .529

Cross-lingual training on TatoebaTrain tokens 20K 11K 21K 2K 44K 161K 835 24Kthis work .468 .449 .630 .290 .614 .678 .350 .637

19 / 23

Introduction Derivation Projection Experiments Conclusions References

Induced Lexicons

eng deu ita nld

SOV (S \NP)\NP - + - +right adj N \N, (N \N)/(N \N) - - + -pro-drop S, S /NP - - + -

20 / 23

Introduction Derivation Projection Experiments Conclusions References

Outline

Introduction

Derivation Projection

Experiments

Conclusions

21 / 23

Introduction Derivation Projection Experiments Conclusions References

Take-home Message

• CCG derivations can be automatically projected along wordalignments

• Cross-lingual supervision helps CCG induction

• Induces linguistically plausible lexicons

• Used for bootstrapping the Parallel Meaning Bank:https://pmb.let.rug.nl

21 / 23

Introduction Derivation Projection Experiments Conclusions References

Bibliography I

Bisk, Y., Christodoulopoulos, C., and Hockenmaier, J. (2015).Labeled grammar induction with minimal supervision. InProceedings of the 53rd Annual Meeting of the Association forComputational Linguistics and the 7th International JointConference on Natural Language Processing (Volume 2: ShortPapers), pages 870–876. Association for ComputationalLinguistics.

Bisk, Y. and Hockenmaier, J. (2013). An HDP model for inducingcombinatory categorial grammars. Transactions of theAssociation for Computational Linguistics, 1:75–88.

22 / 23

Introduction Derivation Projection Experiments Conclusions References

Bibliography II

Honnibal, M., Curran, J. R., and Bos, J. (2010). RebankingCCGbank for improved NP interpretation. In Proceedings of the48th Annual Meeting of the Association for ComputationalLinguistics, pages 207–215. Association for ComputationalLinguistics.

Lewis, M. and Steedman, M. (2014). A* CCG parsing with asupertag-factored model. In Proceedings of the 2014 Conferenceon Empirical Methods in Natural Language Processing(EMNLP), pages 990–1000, Doha, Qatar.

Steedman, M. (2001). The Syntactic Process. The MIT Press.

23 / 23