Towards Syntactically Constrained Statistical Word Alignment
description
Transcript of Towards Syntactically Constrained Statistical Word Alignment
Towards Syntactically Towards Syntactically Constrained Statistical Constrained Statistical
Word AlignmentWord Alignment
Greg HannemanGreg Hanneman
11-734: Advanced Machine Translation SeminarApril 30, 2008
Outline
• The word alignment problem
• Base approaches
• Syntax-based approaches
– Distortion models
– Tree-to-string models
– Tree-to-tree models
• Discussion
Word Alignment
• Parallel sentence pair: F and E
• Most general: map a subset of F to a subset of E
Word Alignment
• Very large alignment spaces!
– An n-word parallel sentence has n2 possible links and 2n2 possible alignments
– Restrict to one-to-one alignments: n! possible alignments
• Alignment models try to restrict or learn a probability distribution over this space to get the “best” alignment of a sentence
Outline
• The word alignment problem
• Base approaches
• Syntax-based approaches
– Distortion models
– Tree-to-string models
– Tree-to-tree models
• Discussion
A Generative Story[Brown et al. 1990]
The proposal will not be implemented
English sentence
Fertility
Les propositions neseront pas applicationmises en
Lexical generation
Les propositions ne seront pas applicationmises en
Distortion
The Framework
• F: words f1 … fj … fn
• E: words e1 … ei … em
• Compute P(F, A | E) for hidden alignment variable A: a1 … aj … an
– The major step: decomposition, model parameters, EM algorithm, etc.
• aj = i: word fj is aligned to word ei
The IBM Models[Brown et al. 1993; Och and Ney 2003]
• Model 1: “Bag of words” — word order doesn’t affect alignment
• Model 2: Position of words being aligned does matter
The IBM Models[Brown et al. 1993; Och and Ney 2003]
• Later models use more implicit structural or linguistic information, but not really syntax, and not really overtly
– Fertility: P(φ | ei) of ei producing φ words in F
– Distortion: P(τ, π | E) for a set of F words τ in a permutation π
– Previous alignments: Probs. for positions in F of the different words of a fertile ei
The HMM Model[Vogel et al. 1996; Och and Ney 2003]
• Linguistic intuition: words, and their alignments, tend to clump together in clusters
• aj depends on absolute size of “jump” between it and aj–1
Discriminative Training
• Consider all possible alignments, score them, and pick the best ones under some set of constraints
• Can incorporate arbitrary features; generative models more fixed
• Generative models’ EM requires lots of unlabeled training data; discriminative requires some labeled data
Discriminative Alignment[Taskar et al. 2005]
•
– Co-occurrence
– Position difference
– Co-occurrence of following words
– Word-frequency rank
– Model 4 prediction
– …
The
proposal
will
not
be
implemented
Les
propositions
ne
seront
pas
application
mises
en
),(),( jiT
ji fefev fw
Outline
• The word alignment problem
• Base approaches
• Syntax-based approaches
– Distortion models
– Tree-to-string models
– Tree-to-tree models
• Discussion
Syntax-Based Approaches
• Constrain alignment space by looking beyond flat text stream: take higher-level sentence structure into account
• Representations
– Constituency structure
– Inversion Transduction Grammar
– Dependency structure
An MT Motivation
Syntax-Based Distortion[DeNero and Klein 2007]
• Syntax-based MT should start from syntax-aware word alignments
• HMM model + target-language parse trees: prefer alignments that respect tree
• Handled in distortion model: jumps should reflect tree structure
Syntax-Based Distortion[DeNero and Klein 2007]
• HMM distortion: size of jump between aj–1 and aj
• Syntactic distortion: tree path between aj–
1 and aj
Syntax-Based Distortion[DeNero and Klein 2007]
• Training:100,000 parallel French–English and Chinese–English sentences with English parse trees
• Both E→F and F → E; combined with different unions and intersections, plus thresholds
• Test: Hand-aligned Hansards and NIST MT 2002 data
Syntax-Based Distortion[DeNero and Klein 2007]
• HMMs roughly equal, better than GIZA++
• Soft union for French; hard union for Chinese; competitive thresholding
Tree-to-String Models
Tree-to-String Models
• New generative story
• Word-level fertility and distortion replaced with node insertion and sibling reordering
• Lexical translation still the same
• Word alignment produced as a side effect from lexical translations
Tree-to-String Alignment[Yamada and Knight 2001]
• Discussed in other sessions this semester
• Training: 2121 short Japanese–English sentences, modified Collins parser output for English
• Test: First 50 sentences of training corpus
• Beat IBM Model 5 on human judgements; perplexity between Model 1 and Model 5
Subtree Cloning[Gildea 2003]
• Original tree-to-string model is too strict
– Syntactic divergences, reordering
• Soft constraint: allow alignments that violate tree structure, but at a cost
– Tweak the tree side of the alignment to contain things needed for the string side
– Ex.: SVO to OSV
Subtree Cloning[Gildea 2003]
S
VP
AUX VP
do ADVP VB
RB
entirely
understand
NP
I
PRP
NP
PRP$ NN
your language
NP
I
PRP
Subtree Cloning[Gildea 2003]
S
VP
AUX VP
do
NP
I
PRP
ADVP VB
RB
entirely
understand
NP
PRP$ NN
your language
NP
I
PRP
Subtree Cloning[Gildea 2003]
S
VP
AUX VP
do
NP
I
PRP
ADVP VB
RB
entirely
understand
NP
PRP$ NN
your language
NP
I
PRP
men ti
NULL NULL
ni hua wo tu
tung
Subtree Cloning[Gildea 2003]
• For a node np:
– Probability of cloning something as a new child of np: single EM-learned constant for all np
– Probability of making that clone a node nc: uniform over all nc
• Surprising that this works…
Subtree Cloning[Gildea 2003]
• Compared with IBM 1–3, basic tree-to-string, basic tree-to-tree models
• Training: 4982 Korean–English sentence pairs, with manual Korean parse trees
• Test: 101 hand-aligned held-out sentences
Subtree Cloning[Gildea 2003]
• Cloning helps: as good or better than IBM
• Tree-to-tree model runs faster
Tree-to-Tree Models
• Alignment must conform to tree structure on both sides — space is more constrained
• Requires more transformation operations to handle divergent structures [Gildea 2003]
• Or we could be more permissive…
Inversion Transduction Grammar
[Wu 1997]
• For bilingual parsing; get one-to-one word alignment as a side effect
• Parallelbinary-branchingtrees with reordering
ITG Operations
• A → [A A]
– Produce “A1 A2” in source and target streams
• A → <A A>
– Produce “A1 A2” in source stream, “A2 A1” in target stream
• A → e / f
– Produce “e” in source stream, “f” in target stream
ITG Operations
• “Canonical form” ITG produces only one derivation for a given alignment
– S → A | B | C
– A → [A B] | [B B] | [C B] | [A C] | [B C] | [C C]
– B → <A A> | <B A> | <C A> | <A C> | <B C> | <C C>
– C → e / f
Alignment with ITG[Zhang and Gildea 2004]
• Compared IBM 1, IBM 4, ITG, and tree-to-string (with and without cloning)
• Training: Chinese–English (18,773) and French–English (20,000) sentences less than 25 words long
• Test: Hand-aligned Chinese–English (48) and French–English (447)
Alignment with ITG[Zhang and Gildea 2004]
• ITG best, or at least as good as IBM or tree-to-string plus cloning
• ITG has no linguistic syntax…
Dependency Parsing
• Discussed in other sessions this semester
• Notion of violating “phrasal cohesion”
– Usually bad, but not always
Dependencies + ITG[Cherry and Lin 2006]
• Find invalid dependency spans; assign score of –∞ if used by the ITG parser
• Simple model: maximize co-occurrence score with penalty for distant words
• ITG reduces AER by 13% relative; dependencies + ITG reduce by 34%
nj
mi
jiji feφfev 52 10),(),(
Dependencies + ITG[Cherry and Lin 2006]
• Discriminative training with an SVM
• Feature vector for each ITG rule instance
– Features from Taskar et al. [2005]
– Feature marking ITG inversion rules
– Feature (penalty) marking invalid spans based on dependency tree
Dependencies + ITG[Cherry and Lin 2006]
• Compared Taskar et al. to D-ITG with hard and soft constraints
• Training: 50,000 French–English sentence pairs for counts and probabilities; 100 hand-annotated pairs with derived ITG trees for discriminative training
• Test: 347 hand-annotated sentences from 2003 parallel text workshop
Dependencies + ITG[Cherry and Lin 2006]
• Relative improvement smaller in discriminative training scenario with stronger objective function
• Hard constraint starts to hurt recall
Outline
• The word alignment problem
• Base approaches
• Syntax-based approaches
– Distortion models
– Tree-to-string models
– Tree-to-tree models
• Discussion
All These Tradeoffs…
• Mathematical and statistical correctness vs. computability
• Simple model vs. capturing linguistic phenomena
• Not enough syntactic information vs. too much syntactic information
• Ruling out bad alignments vs. keeping good alignments around
• Completely unconstrained: every alignment link (ei, fj) either “on” or “off”
• Permutation space: one-to-one alignment with reordering [Taskar et al. 2005]
• ITG space: permutation space satisfying binary tree constraint [Wu 1997]
• Dependency space: permutation space maintaining phrasal cohesion
Alignment Spaces
Alignment Spaces
• D-ITG space: Dependency ∩ ITG space [Cherry and Lin 2006]
• HD-ITG space: D-ITG space where each span must contain a head [Cherry and Lin 2006a]
Examining Alignment Spaces
[Cherry and Lin 2006a]• Alignment score
– Learned co-occurrence score
– Gold-standard oracle score
Examining Alignment Spaces
[Cherry and Lin 2006a]• Learned co-occurrence score
– More restricted spaces give better results
Examining Alignment Spaces
[Cherry and Lin 2006a]• Oracle score: subsets of permutation
space
– ITG rules out almost nothing correct
– Beam search in dependency space does worst
Conclusions
• Base alignment models are mathematical, limited notions of sentence structure
• Syntax-aware alignment helpful for syntax-aware MT [DeNero and Klein 2007]
• Using structure as a hard constraint is harmful for divergent sentences; tweaking trees [Gildea 2003] or using soft constraints [Cherry and Lin 2006] helps fix this
Conclusions
• Surprise winner: ITG
– Computationally straightforward
– Permissive, simple grammar that mostly only rules out bad alignments [Cherry and Lin 2006a]
– Does a lot, even when it’s not the best
• Discriminative framework looks promising and flexible — can incorporate generative models as features [Taskar et al. 2005]
Towards the Future
• Easy-to-run GIZA++ made complicated IBM models the norm — promising discriminative or syntax-based models currently lack such a toolkit
• Syntax-based discriminative techniques — morphology, POS, semantic information…
• Any other ideas?
References• Brown, P., J. Cocke, S. Della Pietra, V. Della Pietra, F. Jelinek, J. Lafferty,
R. Mercer, and P. Roossin, “A statistical approach to machine translation,” Computational Linguistics, 16(2):79-85, 1990.
• Brown, P., S. Della Pietra, V. Della Pietra, and R. Mercer, “The mathematics of statistical machine translation: Parameter estimation,” Computational Linguistics, 19(2):263-311.
• Cherry, Colin and Dekang Lin, “Soft syntactic constraints for word alignment through discriminative training,” Proceedings of the COLING/ACL Poster Session, 105-112, 2006.
• Cherry, Colin and Dekang Lin, “A comparison of syntactically motivated alignment spaces,” Proceedings of EACL, 145-152, 2006a.
• DeNero, John and Dan Klein, “Tailoring word alignments to syntactic machine translation,” Proceedings of ACL, 17-24, 2007.
• Gildea, Daniel, “Loosely tree-based alignment for machine translation,” Proceedings of ACL, 80-87, 2003.
References• Och, Franz and Hermann Ney, “A systematic comparison of various
statistical alignment models,” Computational Linguistics, 29(1):19-51, 2003.
• Taskar, B., S. Lacoste-Julien, and D. Klein, “A discriminative matching approach to word alignment,” Proceedings of HLT/EMNLP, 73-80, 2005.
• Vogel, S., H. Ney, and C. Tillmann, “HMM-based word alignment in statistical translation,” Proceedings of COLING, 836-841, 1996.
• Wu, Dekai, “Stochastic inversion transduction grammars and bilingual parsing of parallel corpora,” Computational Linguistics, 23(3):377-403.
• Yamada, Kenji and Kevin Knight, “A syntax-based statistical translation model,” Proceedings of ACL, 523-530, 2001.
• Zhang, Hao and Daniel Gildea, “Syntax-based alignment: Supervised or unsupervised?” Proceedings of COLING, 418-424, 2004.