Large-Context Models for Large-Scale Machine...

Large-Context Models forLarge-Scale Machine Translation

John DeNeroDissertation Talk

Statistical Language Systems are Working

1


1

Google spent $5.6 billion on infrastructure

in the last 3 years

How?

Data!

Google.com annual report of capital expenditure, “the majority of which was related to IT infrastructure investments.”1


1

People use [Google Translate] hundreds of millions of times a week.“ ” 2

“Google’s Computing Power Refines Translation Tool,” New York Times, 9 March 2010, Technology Section.2

Google spent $5.6 billion on infrastructure

in the last 3 years

How?

Data!

Google.com annual report of capital expenditure, “the majority of which was related to IT infrastructure investments.”1

The Many Use(r)s of Machine Translation

Assimilation Dissemination Communication



• Document translation

• Broadcast monitoring

• Intelligence gathering



73%Most Internet users can’t read the English Web










machine translated







machine translated

John I’m in Berkeley







machine translated


Juan Estoy en Berkeley







machine translated



• Emergency room triage

• Military deployments

• Multilingual education

• 9-1-1 Response

• Commerce with tourists







machine translated



• Emergency room triage

• Military deployments

• Multilingual education

• 9-1-1 Response

• Commerce with tourists

婆婆

John

[Grandma-in-law]

Data-Driven Machine Translation

Parallel corpus gives translation examples

Yo lo haré de muy buen grado

I will do it gladly

Después lo veras

You will see later

Machine translation system:

Target language corpus gives examples of well-formed sentences

I will get to it later See you later He will do it




I will do it gladly

Después lo veras

You will see later


Model of translation



I will do it later

Target language




I will do it gladly

Después lo veras

You will see later


Model of translation



Yo lo haré despuésNOVEL SENTENCE

Source language

VB

MD VP

VPNP

S

PRP ADV

Stitching Together Fragments


I will do it gladly

Después lo veras

You will see later

PRPVB

MD VP

VPNP

S

PRP ADV

I will do it laterModel of translation

Yo lo haré después



VB

MD VP

VPNP

S

PRP ADV



I will do it gladly

Después lo veras

You will see later

PRPVB

MD VP

VPNP

S

PRP ADV




ADV ADV


VB

MD VP

VPNP

S

PRP ADV



I will do it gladly

Después lo veras

You will see later

PRPVB

MD VP

VPNP

S

PRP ADV



Machine translation system:S S

ADV ADV


An Example Syntax-Based TranslationNew Arabic v5.1 base system - sentence 211 Generated by Jens-S. Vöckler 2008-04-10 21:29 2

New Arabic v5.1 base system - sentence 211foreign:tac-lang: urfD albaz aladla’ baá tSryHat fur uSulh alá almqaT‘e .bckwltr: wrfD AlbAz AlAdlA’ bAY tSryHAt fwr wSwlh AlY AlmqATEp .

Tune.nw.0: al @-@ baz declined to make any statements upon his arrival in the province .Tune.nw.1: al @-@ baz refused to give any statements on arriving at al @-@ muqataah .Tune.nw.2: immediately upon his arrival in the area , al @-@ baz declined to give any statements .Tune.nw.3: al @-@ baz refused to make any statement upon his arrival at the moqata’ah .1-best: al @-@ baz declined to give any statements upon his arrival in the province .

[ara-tune4600:211] 1-best Dot Productfeature weight value product

derivation-size 0.41 8 3.30glue-rule 3.89 2 7.78green -0.08 0 0gt_prob 0.40 36.18 14.43identity -9.97 0 0

is_lexicalized -0.65 6 -3.91lex_pef 1.02 5.47 5.60lex_pfe 0.31 4.44 1.39lm1 1 22.76 22.76

lm1-unk 30.08 0 0lm2 0.74 26.66 19.79

lm2-unk -39.18 0 0missingWord -1.29 0 0model1inv 1.02 10.60 10.81model1nrm 1.35 11.29 15.22

nonmonotone 4.17 0 0olive 1.95 0 0psm1n 0.50 24.65 12.30

text-length -3.87 15 -58.05trivial_cond_prob 0.41 3.34 1.38

unk-rule 19.28 0 0reported totalcost 52.82 v · w 52.82

Arabic source sentence:







lm1-unk 30.08 0 0lm2 0.74 26.66 19.79






Al-baz declined to make any statements upon his arrival in the province

Reference translation from a human translator:


[ara-tune4600:211] 1-best PoS-Tree

al

NNP

@-@

HYPH

baz

NNP

NML

NPB

NP-C63425995

declined

VBD

to

TO

give

VB

any

DT

statements

NNS30229081

NPB

upon

IN

his

PRP$

arrival

NN

NPB

in

IN

the

DT

province

NN

NPB

NP-C59736686

PP

NP-C

PP114470921

NP-C

VP-C

VP220719583

SG-C

VP

S-BAR

.

.

S151963398

GLUE265961794

TOP265961890

64

2190

13

3

7

26

New Arabic v5.1 base system - sentence 211 Generated by Jens-S. Vöckler 2008-04-10 21:29 2






lm1-unk 30.08 0 0lm2 0.74 26.66 19.79









[ara-tune4600:211] 1-best PoS-Tree

al

NNP

@-@

HYPH

baz

NNP

NML

NPB

NP-C63425995

declined

VBD

to

TO

give

VB

any

DT

statements

NNS30229081

NPB

upon

IN

his

PRP$

arrival

NN

NPB

in

IN

the

DT

province

NN

NPB

NP-C59736686

PP

NP-C

PP114470921

NP-C

VP-C

VP220719583

SG-C

VP

S-BAR

.

.

S151963398

GLUE265961794

TOP265961890

64

2190

13

3

7

26







lm1-unk 30.08 0 0lm2 0.74 26.66 19.79














lm1-unk 30.08 0 0lm2 0.74 26.66 19.79





Features:

The Steps in a Modern Translation System

Choose a translation

Learn a model

Apply the model



Learn a model

Apply the model

Proceedings

United Nations

will do it ADVVP

lo haré ADV



Learn a model

Apply the model


Later do it I will

Proceedings

United Nations

will do it ADVVP

lo haré ADV



Learn a model

Apply the model


Later do it I willI will later do it

That I’ll do laterLater that I’ll do

I will do it later...

...

Proceedings

United Nations

will do it ADVVP

lo haré ADV


Learn a model

Apply the model


The Alignment Problem in Translation

Thank you , I will do it gladly .

Gracias,loharédemuybuengrado.




Thank you

Gracias




Thank you

muy buen

Thank you

Gracias




About the task:• A lot can be inferred

from lexical statistics

• Correct alignments are not one-to-one

• Some cases are tricky, even for people

Thank you

muy buen

Thank you

Gracias








About solutions:

• Word-to-word links

• Learning driven by conditional word distributions

Thank you

muy buen

Thank you

Gracias








About solutions:

• Word-to-word links

• Learning driven by conditional word distributions

P(gracias|you)

Thank you

muy buen

Thank you

Gracias


Large-Context Alignment Challenges


Goal: Model multi-word structures during alignment





1

P(gracias|you)

Challenge• Jointly infer phrase

boundaries and alignments

• Boundaries depend on both languages

P(gracias,Thank you)





Challenge• Capture context

• Compose phrases

1

2

P(gracias|you)

Challenge• Jointly infer phrase

boundaries and alignments

• Boundaries depend on both languages

P(gracias,Thank you)�(lo hare, I will do it)

Modeling Phrasal Correspondence1

Train a generative model that explains observed translations via latent structure

Paradigm:



Phrase pairs are generated independentlyProcess:

Paradigm:

Thank you , I will do it gladly

Gracias , lo haré de muy buen grado




Paradigm:

Thank you , I will do it gladly

Gracias , lo haré de muy buen grado




Paradigm:

Optimization: Explain all translations with shared parameters

Modeling Phrasal Correspondence

P(A = a) = �(Thank you,Gracias) · �(I will do,hare) · �(it, lo) · · ·


Gracias

,

lo

haré

de

muy

buen

grado

.

1

We learn , a multinomial distribution over phrase pairs �




Gracias

,

lo

haré

de

muy

buen

grado

.

1


P (A = a) =�

(e,s)�a

�(e, s)

* Terms omitted: Phrase pair count and phrase permutation

*




Gracias

,

lo

haré

de

muy

buen

grado

.

1


L(�) =�

d�D

�

��

a�A(d)

P (A = a)

�

�

P (A = a) =�

(e,s)�a

�(e, s)


*




Gracias

,

lo

haré

de

muy

buen

grado

.

For each sentence pair:

1


L(�) =�

d�D

�

��

a�A(d)

P (A = a)

�

�

P (A = a) =�

(e,s)�a

�(e, s)


*




Gracias

,

lo

haré

de

muy

buen

grado

.

For each sentence pair:For each alignment:

1


L(�) =�

d�D

�

��

a�A(d)

P (A = a)

�

�

P (A = a) =�

(e,s)�a

�(e, s)


*




Gracias

,

lo

haré

de

muy

buen

grado

.

For each sentence pair:For each alignment:

Maximizing likelihood gives a degenerate solution: huge phrases!

1


L(�) =�

d�D

�

��

a�A(d)

P (A = a)

�

�

P (A = a) =�

(e,s)�a

�(e, s)


*

.


Gracias,loharédemuybuengrado

Guiding Phrasal Correspondence Models

� � DP(�0,�)

Prefers short phrases�0Base distribution:

DP( · ,�) Non-parametric cache modelDirichlet process:

1

(Thank you, Gracias)

(Thanks, Gracias)

(Thank you, Muchas gracias)

...

...English-Spanish phrase pair Count

Phrase Pair Cache (c):

.




� � DP(�0,�)



1


(Thanks, Gracias)


...



P(z|c) =c(z) + � · �0(z)|c| + �

.




� � DP(�0,�)



1


(Thanks, Gracias)


...



P(z|c) =c(z) + � · �0(z)|c| + �

grow fixed

.




� � DP(�0,�)



Iterative realignment of all the data by sampling Consistent, efficient estimation

1


(Thanks, Gracias)


...



P(z|c) =c(z) + � · �0(z)|c| + �

grow fixed

Thank you , I shall do so gladly .

What Happens in Practice

Gracias

,

lo

haré

de

muy

buen

grado

.

Thank you , I shall do so gladly .

A state-of-the-art word-level alignment

A sampled phrase alignmentfrom our system

1

Performance Results

Translation performance in a phrase-based system (Moses)for Spanish-to-English parliamentary proceedings (Europarl)

Translation quality (BLEU)

28293031

30.129.8

Word-level baselinePhrase-level model [DeNero et al. EMNLP ’08]*

0246

3.14.4Millions of

phrase pairs

Higher is better

Lower is cheaper

* John DeNero, Alex Bouchard-Côté, and Dan Klein. Sampling Alignment Structure under a Bayesian Translation Model, EMNLP 2008.

1

Subsequent Work

We described a non-parametric Bayesian prior and a consistent sampling procedure (EMNLP 2008)

• Phil Blunsom, Trevor Cohn, Chris Dyer, and Miles Osborne. A Gibbs sampler for phrasal synchronous grammar induction, ACL 2009.

• Abhishek Arun, Chris Dyer, Barry Haddow, Phil Blunsom, Adam Lopez, and Philipp Koehn. Monte Carlo inference and maximization for phrase-based translation, CoNLL 2009.

• Ding Liu and Daniel Gildea. Bayesian Learning of Phrasal Tree-to-String Templates, EMNLP 2009.

• Matt Post and Daniel Gildea. Bayesian Learning of a Tree Substitution Grammars, ACL 2009.

• Phil Blunsom and Trevor Cohn. Inducing Synchronous Grammars with Slice Sampling, NAACL 2010.

• Trevor Cohn and Phil Blunsom. A Bayesian Model of Syntax-Directed Tree to String Grammar Induction, EMNLP 2009.

1

A Model of Composed Phrases

过去

In the past two years

[past]

两 [two]

年 [year]

中 [in]

2


过去 [past]

两 [two]

年 [year]

中 [in]

A model can predict the whole analysis above, including minimal links & composed phrase pairs .


2


过去 [past]

两 [two]

年 [year]

中 [in]



< P(in|中) = 0.8 , InDictionary = 1.0, ... >

Features on

word links:

2


过去 [past]

两 [two]

年 [year]

中 [in]



< Count(the past two, 过去两 ) = 7, Size(3,2) = 1, ... > phrase pairs:

< P(in|中) = 0.8 , InDictionary = 1.0, ... >

Features on

word links:

2

Learning from Supervised Data

过去 [past]

两 [two]

年 [year]

中 [in]

Guess: Model Prediction


2


过去 [past]

两 [two]

年 [year]

中 [in]



Gold: Human Annotation


2


过去 [past]

两 [two]

年 [year]

中 [in]





Loss function: Number of differing rounded rectangles

2


过去 [past]

两 [two]

年 [year]

中 [in]





Online learning (MIRA) adjusts model parameters to prefer the gold over the guess by a margin of the loss

Loss function: Number of differing rounded rectangles

2

过去 [past]

两 [two]

年 [year]

中 [in]


Finding the Optimal Correspondence2

arg maxy�ITG(x)

� · [ �word(x, y) + �phrase(x, y) ]

过去 [past]

两 [two]

年 [year]

中 [in]



Hierarchical decomposition

arg maxy�ITG(x)


过去 [past]

两 [two]

年 [year]

中 [in]



Hierarchical decomposition

ITG parser with a state space that tracks peripheral alignments for each region

arg maxy�ITG(x)


Experimental Results

Unsupervised word model baselineSupervised word model [Haghighi, Blitzer, DeNero, and Klein. ACL ’09]*Composed Phrase Pair Model [DeNero and Klein. In submission]**

** John DeNero and Dan Klein. Supervised Modeling of Extraction Sets for Machine Translation, in submission.

* Aria Haghighi, John Blitzer, John DeNero, and Dan Klein. Better Word Alignments with Supervised ITG Models, ACL 2009.

55

65

75

71.668.464.1

Phrase Pair F1

Alignment quality relative to human-annotated data

33343536

35.934.734.5

Translation quality for Chinese-to-English

BLEU

2


Learn a model

Apply the model



Learn a model

Apply the model


Large data sets provide statistics for larger structures


Learn a model

Apply the model


Non-parametric models scale with the data



Learn a model

Apply the model


Non-parametric models scale with the data


The more context we incorporate, the better we do


Learn a model

Apply the model


Extracting Translation Rules






VP

PRPVB

MD VP

NP

.S

PRP ADV

S

S

VB NP

PRP

VP

,

VP




VP

PRPVB

MD VP

NP

.S

PRP ADV

S

S

VB NP

PRP

VP

,

VP




ADV

AD

V

VP

PRPVB

MD VP

NP

.S

PRP ADV

S

S

VB NP

PRP

VP

,

VP




ADV

AD

V

VP

PRPVB

MD VP

NP

.S

PRP ADV

S

S

VB NP

PRP

VP

,

will do it ADVVP

lo haré ADV

VP




ADV

AD

V

VP

PRPVB

MD VP

NP

.S

PRP ADV

S

S

VB NP

PRP

VP

,

will do it ADVVP

lo haré ADV

Frequency statistics on these rules guide

translation

Synchronous Context-Free Grammars

Mi dormitorio nuevo no es ni grande ni pequeño

GrammarDerivation

Translation:

Source:



GrammarDerivation

dormitorioNN bedroom

Translation:

Source:



GrammarDerivation


NN

Translation:

Source:



GrammarDerivation


NN

bedroomNN

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

NN

bedroomNN

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

NN JJ JJ JJ

newJJ

bigJJ

smallJJ

bedroomNN

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

Mi NN JJNP

My JJ NN

NN JJ JJ JJ

newJJ

bigJJ

smallJJ

bedroomNN

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

Mi NN JJNP

My JJ NN

NN JJ JJ JJ

NP

newJJ

bigJJ

smallJJ

bedroomNN

My

NP

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

Mi NN JJNP

My JJ NN

NP no es ni JJ ni JJSNP is neither JJ nor JJ

NN JJ JJ JJ

NP

newJJ

bigJJ

smallJJ

bedroomNN

My

NP

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

Mi NN JJNP

My JJ NN


NN JJ JJ JJ

NP

S

NP no es ni JJ ni JJ

newJJ

bigJJ

smallJJ

bedroomNN

My

NP

Translation:

Source:



GrammarDerivation


nuevoJJ new

grandeJJ big

pequeñoJJ small

Mi NN JJNP

My JJ NN


NN JJ JJ JJ

NP

S


newJJ

bigJJ

smallJJ

bedroomNN

S

is neither norMy

NP

Translation:

Source:

The Size of the Grammar

0

22,500

45,000

67,500

90,000

1 2 3 4 5 6 7 8 9 10+

332,000 rules match a 30-word sentence to be translated

Size of the source-side yield (terminals + non-terminals)

Rule

Cou

nt

A grammar learned from 220 million words of Arabic-to-English example translations:

John DeNero, Adam Pauls, Mohit Bansal, and Dan Klein. Efficient Parsing for Transducer Grammars, NAACL 2009.

The Size of the Grammar

0

22,500

45,000

67,500

90,000

1 2 3 4 5 6 7 8 9 10+

332,000 rules match a 30-word sentence to be translated

Size of the source-side yield (terminals + non-terminals)

Rule

Cou

nt

A grammar learned from 220 million words of Arabic-to-English example translations:

NP no es ni JJ ni JJ . NP is neither JJ nor JJ .

S

John DeNero, Adam Pauls, Mohit Bansal, and Dan Klein. Efficient Parsing for Transducer Grammars, NAACL 2009.

The Structure of the Grammar


S


NP JJ JJ

S → NP no es ni JJ ni JJ




NP JJ JJ

ni JJ ni JJ

ADJP

ADJP → ni JJ ni JJ




NP JJ JJ

es ADJP

VP

VP→ es ADJP

ni JJ ni JJ

ADJP





NP JJ JJ

es ADJP

VP

VP→ es ADJP

ni JJ ni JJ

ADJP





NP JJ JJ

S

NP no VP

S → NP no VP

Coarse-to-Fine Translation



1 Apply a subset of the grammar with only small rules





ADJP

VP

S

NP

JJ JJJJNN



2 Prune away unlikely portions of the search space


ADJP

VP

S

NP

JJ JJJJNN




Apply the full translation grammar to the pruned space

3


Experimental Results

* John DeNero, Adam Pauls, Mohit Bansal, and Dan Klein. Efficient Parsing for Transducer Grammars, NAACL 2009.

Baseline

+Optimization

+Transformation

+Coarse-to-Fine

0 75 150 225 300

50

104

181

264

5x speed-up with the largest translation grammars in use today (ISI Syntax-Based MT System) [DeNero et al. NAACL ’09]*

Minutes required to analyze a 300 sentence test set


Learn a model

Apply the model



Learn a model

Apply the model


Fully exploiting large data sets requires searching over very large spaces


Learn a model

Apply the model


Coarse-to-fine inference is a powerful technique for doing so

Fully exploiting large data sets requires searching over very large spaces


Learn a model

Apply the model


Even the Best Models are Wrong

Model

NOVEL SENTENCE


I will later do itLater do it I will


...


Model

NOVEL SENTENCE



Total model score for 1000 sentences

Tran

slat

ion

Qua

lity

(BLE

U)

Samples from output spaceSamples near maximumHighest scoring translation

Model



...

NOVEL SENTENCE



511660 513245 51483050.0

50.2

50.4

50.6

50.8

Total model score for 1000 sentences

Tran

slat

ion

Qua

lity

(BLE

U)

Samples from output spaceSamples near maximumHighest scoring translation

Model



...

NOVEL SENTENCE


Consensus by Averaging Over Sentences

Model



...

NOVEL SENTENCE



Intuition: “Happy families are all alike; every unhappy family is unhappy in its own way.” [Tolstoy. 1877]*

* Leo Tolstoy. Анна Каренина. 1877.

Model



...

NOVEL SENTENCE





Idea: Average over sentences to find the phrases that are alike. [DeNero et al. ACL ’09]**Model



...

NOVEL SENTENCE

** John DeNero, David Chiang, and Kevin Knight. Fast Consensus Decoding over Translation Forests, ACL 2009.








...

NOVEL SENTENCE


“Later” “do” “do it” “I’ll” “do later” “do it I will”... ...








...

NOVEL SENTENCE



1 1 1 0 0 1

1 1 1 0 0 0

1 1 0 1 1 0

1 1 0 1 0 0








...

NOVEL SENTENCE



1 1 1 0 0 1

1 1 1 0 0 0

1 1 0 1 1 0

1 1 0 1 0 0

0.12

0.10

0.07

0.07...








...

NOVEL SENTENCE



0.97 0.98 0.54 0.41 0.34 0.121.00

1 1 1 0 0 1

1 1 1 0 0 0

1 1 0 1 1 0

1 1 0 1 0 0

0.12

0.10

0.07

0.07...

Expected output


Phrase Expectations from Forests




YoNP

I

I



YoNP

I

I

lo haréVP

shall do it

shall do it



YoNP

I

I

lo haréVP

shall do it

shall do it

loPRP it

it



YoNP

I

I

lo haréVP

shall do it

shall do it

loPRP it

it

PRP haréVP

will do PRP

will do it

“do it”



YoNP

I

I

lo haréVP

shall do it

shall do it

loPRP it

it

PRP haréVP

will do PRP

will do it

NP VPS

NP VP“I will”

“do it”

NP VPS

NP VP



YoNP

I

I

lo haréVP

shall do it

shall do it

loPRP it

it

PRP haréVP

will do PRP

will do it

NP VPS

NP VP“I will”

“I shall”

“do it”

NP VPS

NP VP



YoNP

I

I

lo haréVP

shall do it

shall do it

loPRP it

it

PRP haréVP

will do PRP

will do it

NP VPS

NP VP

I {will, shall} do it

“I will”

“I shall”

“do it”

NP VPS

NP VP



YoNP

I

I

lo haréVP

shall do it

shall do it

loPRP it

it

PRP haréVP

will do PRP

will do it

NP VPS

NP VP

I {will, shall} do it

I {will, shall} do it later

S despuésS

S later

“it later”“I will”

“I shall”

“do it”

Single System Translation Results

Translation quality in ISI’s Full-Scale Arabic-to-English Hierarchical Translation System

Translation Quality (BLEU)

50

51

52

5353.0

52.252.0

Model-Only BaselineConsensus from a List [DeNero et al. ACL ’09]*Consensus from a Forest [DeNero et al. ACL ’09]*

* John DeNero, David Chiang, and Kevin Knight. Fast Consensus Decoding over Translation Forests, ACL 2009.

Model 1 Model 2 Model 3

Translating Using Multiple Systems

NOVEL SENTENCE




NOVEL SENTENCE


0.96 0.99 0.57 0.44 0.30 0.00

0.94 0.92 0.81 0.67 0.11 0.09

0.97 0.98 0.54 0.41 0.34 0.12



NOVEL SENTENCE


I will do it later

0.96 0.99 0.57 0.44 0.30 0.00

0.94 0.92 0.81 0.67 0.11 0.09

0.97 0.98 0.54 0.41 0.34 0.12

Consensus Modeling FAQ


Q: How do we combine different models?



Models Which model?

Phrase lengths

Expected counts

Model score

Length

A: Train a linear consensus model scoring a derivation d:

I�

i=1

�w(�)

i �i(d) +4�

n=1

w(n)i v(n)

i (d)

�+ w(b) · b(d) + w(�) · �(d)



Models Which model?

Phrase lengths

Expected counts

Model score

Length


I�

i=1

�w(�)

i �i(d) +4�

n=1

w(n)i v(n)

i (d)

�+ w(b) · b(d) + w(�) · �(d)

Q: What output sentences are considered?



Models Which model?

Phrase lengths

Expected counts

Model score

Length


I�

i=1

�w(�)

i �i(d) +4�

n=1

w(n)i v(n)

i (d)

�+ w(b) · b(d) + w(�) · �(d)

Q: What output sentences are considered?

A: The union of output spaces of models:R

Rpb Rh

Multi-System Translation Results

Google’s Full-Scale Research Translation System for Arabic-to-English

Translation quality (BLEU)

42

43

44

45

46

45.3

43.9

Best Single-System Model-Only BaselineMulti-System Forest-Based Consensus [DeNero et al. NAACL ’10]*

* John DeNero, Shankar Kumar, Ciprian Chelba, and Franz Och. Model Combination for Machine Translation, NAACL 2010.


Learn a model

Apply the model



Learn a model

Apply the model


Statistical models provide distributions over outputs


Learn a model

Apply the model



Leveraging those distributions improves performance


Learn a model

Apply the model



Leveraging those distributions improves performance

Compact representations can enable large-scale computation

Summary of Translation Research

Learn a model

Apply the model



Learn a model

Apply the model


Non-parametric models

Large-context models

[DeNero et al. EMNLP ’08]

[DeNero & Klein. ACL ’10]


Learn a model

Apply the model



Large-context models Coarse-to-fine

[DeNero et al. NAACL ’09][DeNero et al. EMNLP ’08]



Learn a model

Apply the model




Compact encodings

Full distributions

[DeNero et al. NAACL ’09] [DeNero et al. ACL ’09]

[DeNero et al. NAACL ’10]




Learn a model

Apply the model




Compact encodings

Full distributions





Are we done yet?


Learn a model

Apply the model




Compact encodings

Full distributions





Are we done yet?

• Morphology in alignment modeling


Learn a model

Apply the model




Compact encodings

Full distributions





Are we done yet?


• Unsupervised composed phrase learning


Learn a model

Apply the model




Compact encodings

Full distributions





Are we done yet?


• Unsupervised composed phrase learning

• Adding information to consensus models

Acknowledgements

Thank you!

and many thanks to my excellent coauthors on this work:

Berkeley: Mohit Bansal, John Blitzer, Alex Bouchard-Côté, Aria Haghighi, Dan Klein, and Adam Pauls

Information Sciences Institute: David Chiang and Kevin Knight

Google: Ciprian Chelba, Shankar Kumar, and Franz Och

John

Juan Gracias!

Large-Context Models for Large-Scale Machine...

Documents

Transcript of Large-Context Models for Large-Scale Machine...