A Syntax-basedStatistical MachineTranslation Model

Alexander Friedl, Georg Teichtmeister 4.12.2006

A Syntax-based Statistical Translation Model

� Introduction

� The model

� Experiment

� Conclusion

A Syntax-based Statistical Translation Model

� Statistical Translation Model (STM):

- mathematical model

- statistical modelling of human-languagetranslation

- parameters estimated with training corpus

A Syntax-based Statistical Translation Model

� First steps

- IBM 1998: string-to-string word-basedtranslation

- completely independent process

A Syntax-based Statistical Translation Model

� Question: Why not word-based translation?

- no structural or synactic aspects

- how to handle different word order?

A Syntax-based Statistical Translation Model

� Solution:

A Syntax-based Statistical Translation Model

A Syntax-based Statistical Translation Model

� Input: parse tree(syntactic parser)

� Output: string

A Syntax-based Statistical Translation Model

� Three operations on each node of the tree:

- reorder

- inserting

- translating

A Syntax-based Statistical Translation Model

� Reorder

- different word order

- English vs. Japanese

A Syntax-based Statistical Translation Model

� Word-insertion

- e.g. capture linguistic differnces in specifyingsyntactic cases

A Syntax-based Statistical Translation Model

� Translation

- Translate leaf words into the destination language

A Syntax-based Statistical Translation Model

� Output

A Syntax-based Statistical Translation Model

� Introduction

� The model

� Experiment

� Conclusion

A Syntax-based Statistical Translation Model

� English parse tree into a noisy channel

� Output should be a Japanese sentence

� Stochastical operations an each nodes

A Syntax-based Statistical Translation Model

� Reorder

- N! possiblereorderings for N children

- probability given bythe r-table

A Syntax-based Statistical Translation Model

� Word-insertion

- left, right oder nowhere

- probability given by the n-table

A Syntax-based Statistical Translation Model

� Translation

- dependent only on the word itself

- Probability given by the t-table

A Syntax-based Statistical Translation Model

� Total probability

- product of the single operation probabilities

� Tables

- English-Japanese training corpus

A Syntax-based Statistical Translation Model

� Formal Transcription

Input:

English parse tree ε (in nodes ε1, ε2, … εn)

Output:

French sentence f (in words f1, f2, … fm )

A Syntax-based Statistical Translation Model

� Probability getting f for ε

Str(Θ(ε )) is a sequence of leaf words

A Syntax-based Statistical Translation Model

� P(Θ|ε ): Probability of a particular set of RVsin a parse tree

A Syntax-based Statistical Translation Model

� RVs θi = <νi,ρi,τi> independent on each other, but dependent on features of εi

A Syntax-based Statistical Translation Model

� Probability of getting a French sentence fgiven an English parse tree ε

A Syntax-based Statistical Translation Model

� Automatic Estimation of model parameters

- update parameter to maximize the likelihood of the training corpus

A Syntax-based Statistical Translation Model

� Algorithm

1. Initialize probability tables2. Reset counters3. For each iteration the number of events

are counted and weighted by theprobability of events

4. Parameter re-estimated by the counts

A Syntax-based Statistical Translation Model

� Introduction

� The model

� Experiment

� Conclusion

A Syntax-based Statistical Translation Model

� Experiment with small English-Japanesecorpus

- 2121 translation sentence pairs

- taggers build the English parse trees

� Comparison with IBM 5

A Syntax-based Statistical Translation Model

� Evaluation

- generate the most probable alignment ofthe training corpus (Viterbi)

- average score of the first 50 sentences

A Syntax-based Statistical Translation Model

0 pointsAlignment wrong

0,5 pointNot sure

1 pointAlignment okay

00,431IBM Model 5

100,582Our Model

Perfect sentencesAverage score

A Syntax-based Statistical Translation Model

A Syntax-based Statistical Translation Model

� Perplexity

- Our Model: 15,79

- IBM Model 5: 9,84

A Syntax-based Statistical Translation Model

� Introduction

� The model

� Experiment

� Conclusion

A Syntax-based Statistical Translation Model

� Syntax-based translation model

� Statistical modelling the translation process

� Syntactic information for languages withdifferent word order

� Better alignment results in an experiment

A Syntax-based Statistical Translation Model

Outline – 2nd Part

� Problems of Tree to String� Clone Operation� Tree to Tree

� Phrasal Translation� Translation System

A Syntax-based Statistical Translation Model

Problem of Tree to String Model

� Not all re-orderings of terminal nodes are possible

� Constrains syntactic correspondence between native and foreign language

A Syntax-based Statistical Translation Model

The Clone operation

� Insert a copy of a subtree at any point in the tree

� Delete original node

Clone Delete

A Syntax-based Statistical Translation Model

When to clone?

� Should a clone be inserted as child of

� Decide which node should be cloned

� Probability of cloning is independent of previous cloning operations

)|( jins cloneP ε

∑==

kkmakeclone

imakecloneiclone P

PcloneP

)(

)()1|(

εεε

jε

A Syntax-based Statistical Translation Model

Example

A Syntax-based Statistical Translation Model

Tree – to - Tree

� Syntactic trees for foreign and native language

� Output tree instead of output string� Additional tree transformations

� Single source node → Two target nodes� Two source nodes → Single target node

� New model: )|( ab TTP

A Syntax-based Statistical Translation Model

Building the output tree

� At each level at the output tree:�Choose a elementary tree

�Align the children of the the elementary tree

� Translate the leaves

A Syntax-based Statistical Translation Model

Elementary tree

�

� Means that two nodes can grouped together� Example:

� Nodes A and B are considered a elementary tree

))(|( aaaelem childrentP εε ⇒

A Syntax-based Statistical Translation Model

Alignment of children� All children of an elementary tree are aligned at

once according to:

� Insertions and Deletions are also done in this step

))(|( aaalign tchildrenP ⇒εα

A Syntax-based Statistical Translation Model

Tree – to – Tree Clone

� Same reordering problems as Tree – to –String

� Cloning will now added to alignment step

))(|( aains tchildrencloneP ⇒∈ εα

∑=∈

kkmakeclone

imakecloneiclone P

PcloneP

)(

)()|(

εεαε

A Syntax-based Statistical Translation Model

Parameter comparison

A Syntax-based Statistical Translation Model

Experiments

� Data from Korean – English corpus (Military domain)

� Korean suffixes often carry meaning�This suffixes became leaves in the syntax tree

�Vocabulary was reduced from 10059 to 3279

� Average Korean: 13 words and 21 tokens� Average English: 16 words

A Syntax-based Statistical Translation Model

Results

Word pairings by System (A) and Gold Standard(G)

||||

||21

GA

GAAER

+∩−=

A Syntax-based Statistical Translation Model

Phrasal Translation

� 1 to 1 word translation not perfect� Compound nouns

�German vs English

� Idiomatic phrases� “To kick the bucket” vs “Den Löffel abgeben”

A Syntax-based Statistical Translation Model

Model

� 1 to 1 Model:

� 1 to N with fertility µ:

� N to N:

)|()|( eftt =Ττ

∏=

==Τl

iil efteleffftt

121 )|()|()|..()|( µτ

∏=

==Φl

imim

ml

eeefteeel

eeeffftph

12121

2121

)..|()..|(

)..|..()|(

µ

φ

A Syntax-based Statistical Translation Model

Incorporation into TM�

� R: feature forreordering

� N: feature for insertion� ρ: Reorder operation

� υ: Insert operation

)|()|()1()|()|( iiiiiiii NnRrphPii

υρλφλεθ ΦΦ −+Φ=

A Syntax-based Statistical Translation Model

From Models to the Translation

� Task: Translate from Foreign to English

� Reformulation

)|( FEP

)()|()|( EPEFPFEP =

A Syntax-based Statistical Translation Model

Language Model

� Probability of an English Sentence P(E)� N-gram LM in original Implementation� No syntactic Information used� Improvement: Immediate-head parsing for

language models

A Syntax-based Statistical Translation Model

Immediate-head parsing

� English Sentence� Non-lexical PCFG → Large Parse Forest� Pruning of the Large Parse Forest

�Which edges have high probability of beingcorrect?

� Evaluation of pruned Parse with a lexicalPCFG

A Syntax-based Statistical Translation Model

Decoder

� Decoder works in reverse direction to TM

� Find most probable syntactic tree E from a Sentence F

� Basic idea: “Translate“ Parse tree using TM to the foreign language

� Parse the foreign sentence

� “Translate” back to English� And check LM

A Syntax-based Statistical Translation Model

Conclusion� Improvements to Syntactic Translation Model

� Tree to String Clone� Tree to Tree� Tree to Tree Clone� Phrasal Translation

� Brief overview over Translation Process

A Syntax-based Statistical Translation Model

References

� A Syntax based Statistical Translation ModelKenji Yamada and Kevin Knight, Proceedings of the Conference of the Association for Computational Linguistics 2001

� Loosely Tree-based Alignment for Machine TranslationDaniel Gildea, In Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics (ACL-03), Supporo, Japan, 2003.

� Syntax baed Language Models for Statistical Machine TranslationEugene Charniak, Kevin Knight and Kenji Yamada, In MT Summit IX. Intl. Assoc. for Machine Translation, 2003

� A Decoder for Syntax-Based Statistical MT Kenji Ymada and Kevin Knight, Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics (ACL-02) , 2002

A Syntax-based Statistical Translation Model

Thank you for your attention!