Meta-level Statistical Machine Translation System

Human Language Technology LabAmirkabir University of Technology

IJCNLP 2013, Nagoya, Japan

Amirkabir University

of Technology

Sajad Ebrahimi, Kourosh Meshgi, Shahram Khadivi

and Mohammad Ebrahim Shiri Ahmad Abady

Amirkabir University

of Technology

Outli

neIntroductionBackgroundStacking for classificationAdapting Stacking to SMT Experiments and ResultsRelated WorkConclusion and Future Work

1 .Introduction Traditional approaches to System Combination need multiple structurally different

SMT systems. In this research, we focus on a single SMT system. We try to introduce a meta-level SMT which can learn how to decrease or modify

translation errors. To do this, we utilize an Ensemble Learning algorithm, called Stacking. The basic idea :

a collection of base-level SMTs is generated for obtaining a meta-level corpus Then a meta-level SMT is trained on this corpus

We address the issue of how to adapt Stacking to SMT.

Amirkabir University

of Technology

2 .Background2.1 Log-linear model and statistical machine translation : Given a source string S , the goal of SMT is to find a target string from all

possible translations :

In meta-SMT, given a machinery output , the goal is to find a target sentence :

Amirkabir University

of Technology

�̂�=𝑎𝑟𝑔𝑚𝑎𝑥𝑡 2{𝑝𝑟 ( 𝑡2|𝑡 ) }

𝑡=𝑎𝑟𝑔𝑚𝑎𝑥𝑡 1{𝑝𝑟 (𝑡 1|𝑠 ) }

disjoint almost equal parts : set of different Learning algorithms

2.1 Stacking for Classification

Overview Proposed by Wolpert(1992) learn a meta-level (or level-1) classifier based on the output of base-level (or level-0)

classifiers, estimated via cross-validation as follows: Define

Amirkabir University

of Technology

Class ValueFeature Vector

J-fold cross-validation

2.1 Stacking for Classification

Overview Define ,

Amirkabir University

of Technology

Training setTest set

At each j-th step, given the learning algorithms, we invoke each of them on to induce and apply to the test part .

The concatenated predictions + the original class value => At the end of the entire cross-validation :

full meta-level data set

2.1 Stacking for Classification

3.1. Overview is applied to a learning algorithm to induce meta-level classifier . Finally ,

All the learning algorithms () are applied to the entire data set inducing the final base-level classifiers to be used at runtime.

Amirkabir University

of Technology

to classify a new instance : the concatenated predictions of all base-level classifiers form a meta-level vector that is assigned a class value by the meta-level classifier

We adopt stacking to SMT in a principled way…

3 .Adapting Stacking to SMT

we adapt it to SMT as follows:

𝑆𝑀𝑇 𝑃𝑎𝑟𝑎𝑑𝑖𝑔𝑚𝑆𝑀𝑇 𝑗

𝑆𝑀𝑇 𝑃𝑎𝑟𝑎𝑑𝑖𝑔𝑚

𝑆𝑀𝑇 𝑃𝑎𝑟𝑎𝑑𝑖𝑔𝑚

𝑆𝑀𝑇

𝑚𝑒𝑡𝑎−𝑆𝑀𝑇

New Source Sentences

Amirkabir University

of Technology

Target Sentences

3.1 Training base-level SMTs we train 5 phrase-based SMT systems on the training part and obtain

the result of these systems on the corresponding test sets. We need these results for the next step.

3.2 Training meta-level SMTs We gathered the n-best outputs of base-level SMTs on the corresponding

test sets to : build a meta-level corpus using these outputs along with correct human

translations Then, train a meta-SMT on this new corpus. We train our meta-SMT on 10 meta-level corpus which is progressively created

from n-best outputs of base-level systems, . we call these systems as meta-SMT (1-best) and meta-SMT (2-best) and so on.

Amirkabir University

of Technology

3.3 Tuning meta-level SMTs To build a meta-level development set, we tune 5 base-level SMT

systems on the tuning part and obtain the result of these systems on the corresponding test sets.

Finally a meta-level development set is created by gathering these outputs paired with correct human translations to tune meta-level SMTs.

Amirkabir University

of Technology

4 .Experiments

The corpus that is used for training and cross-validation process is Verbmobil project corpus

Amirkabir University

of Technology

4.1 Data

#of words #of sentences

249K 23K English216K 23K Persian

4 .Experiments

Giza++ => bi-directional word alignment SRILM => language model training case-insensitive BLEU => quality measuring Moses decoder => a phrase-based SMT (both base-level and meta-

level) MERT => tune the feature weights on the development data

Amirkabir University

of Technology

4.2 Experimental setup

4 .Experiments Amirkabir University

of Technology

4.3 Evaluation

Type of SMT Test setbaseline SMT 30.47meta-SMT (1-best) 31.20meta-SMT (2-best) 31.00meta-SMT (3-best) 31.37meta-SMT (4-best) 31.49meta-SMT (5-best) 31.41meta-SMT (6-best) 31.05meta-SMT (7-best) 31.19meta-SMT (8-best) 31.40meta-SMT (9-best) 31.30meta-SMT (10-best) 31.54

BLEU (%) scores of baseline SMT and meta-SMTs on the Verbmobil test set that has 250 sentences with four reference translations.

4 .Experiments Amirkabir University

of Technology

4.3 Evaluation• Some examples:

• Delete a wrong word: • EN : that is perfect . then we have talked about everything . goodbye .

•FA (main) . : ردیمMک صحبت اش درباره چیز همه ما پس است عالی خداحافظ. . میبینمآن•FA (meta) . . . : خداحافظ کردیم صحبت دیروز چیز همه ما پس است عالی آن

• Translate an untranslated word:• EN : I think we will take the Metropol hotel . could you reserve two single

rooms ?•FA (main) : هتل را ما میکنم فکر رزرو . Metropolمن دو شما ؟ roomsمیتوانیم مجزا• FA (meta) :هتل را ما میکنم فکر رزرو . Metropolمن دو شما ؟ اتاقهامیتوانیم تک بیندازم

• EN : yes , I would suggest the flight at a quarter past seven .•FA (main) : میکنم پیشنهاد را من ، هفت . flightبله ساعت از بعد ربع یک•FA (meta) : میکنم پیشنهاد را من ، هفت . پروازبله ساعMت از بعد ربع یک

4 .Experiments Amirkabir University

of Technology

4.3 Evaluation• Some examples:

• Rephrase and reordering : • EN : the best thing would be for us to take the subway from our hotel to the

station.•FA (main) : تا بود خواهد ما برای چیز ما مترواز را بهترین . ایستگاهتا هتل

•FA (meta) : از تا بود خواهد ما برای چیز ما بهترین . مترو ایستگاهتا هتل

4 .Experiments Amirkabir University

of Technology

4.3 Evaluation two factors possibly contribute to these results :

performing cross-validation on the training set the re-optimization on the system

we perform two experiments to investigate the effect of each factor : (Straight1) => test the approach without any cross-validation process, but

with the development set obtained from stacking. (Straight2) => to build meta-level SMTs tuned with a development set which

is obtained directly from baseline SMT (i.e., without performing cross-validation on it).

4 .Experiments Amirkabir University

of Technology

4.3 Evaluation

29

29.5

30

30.5

31

31.5

32

n-best list

% B

LEU

StackingStraight1Straight2

Comparison of Stacking, Straight1 and Straight2

4 .Experiments Amirkabir University

of Technology

4.3 EvaluationAfter analyzing the results:

it can be concluded that both factors, i.e., cross-validation and re-optimizing the system with the stacking-based development set, are important to outperform the baseline SMT system. Since use of both factors, consistently lead to the best results.

We conducted statistical significance tests using paired bootstrap resampling proposed by Koehn (2004) to measure the reliability of the conclusion that meta-SMTs are really better than baseline SMT. It is observed that all stacking-based meta-SMTs are really better than the baseline SMT in 99% of the times.

5 .Related Work Amirkabir University

of Technology

Xiao et al. (2010) presented a general solution for adaption of bagging and boosting to SMT. Their results showed that ensemble learning algorithms are promising in SMT.

Simard et al. (2007a), trained a “mono-lingual” Phrase-based SMT system on the output of an RBMT system for the source side of the training set of the Phrase-based SMT system and the corresponding human translated (manually post-edited) reference.

Béchara et al. (2011) designed a full phrase-based SMT pipeline that included a translation step and a post-editing step. They use a novel context aware approach.

5 .Conclusion and future work Amirkabir University

of Technology

We have presented a simple and effective approach to translation error modification by building a meta-level SMT using a meta-level corpus that is created form original corpus by cross validation.

Experimental results showed that such a meta-SMT can fix many translation errors that occur in the baseline translations.

As a future work, we have planned to develop a technique for combining multiple SMT systems using stacked generalization algorithm

5 .Conclusion and future work Amirkabir University

of Technology

Moreover, we are running more tests with different language-pairs and larger corpora.

As another future work, we will apply our framework under different SMT paradigms such as hierarchical phrase-based SMT and syntax-based SMT.

6 .References Amirkabir University

of Technology

1. Almut Silja Hildebrand and Stephan Vogel. 2008. Combination of machine translation systems via hypothesis selection from combined n-best lists. In Proc. of the 8th AMTA conference, pages 254-261.

2. Evegeny Matusov, Nicola Ueffing and Hermann Ney. 2006. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment. In Proc. of EACL 2006, pages 33-40.

3. Antti-Veikko Rosti, Spyros Matsoukas and Richard Schwartz. 2007. Improved word-level system combination for machine translation. In Proc. of the 45th Annual Meeting of the Association for Computational Linguistics, pages. 312-319.

4. Michel Simard, Cyril Goutte, and Pierre Isabelle. 2007a. Statistical phrase-based post-editing. In NAACL-HLT, pages 508-515

5. Béchara, H., Y. Ma, and J. van Genabith. 2011. Post-editing for a statistical MT system. In MT Summit XIII, pages 308-315

6. David H. Wolpert. 1992. Stacked generalization. Neural Networks, 5(2): 241-259.7. Leo Breiman. 1996b. Bagging predictors. Machine Learning, 24(2):123-140.

THANK YOU

Amirkabir University

of Technology