Post on 04-Jan-2016
Data Collection and Language Technologies for Mapudungun
Lori Levin, Rodolfo Vega,
Jaime Carbonell, Ralf Brown,
Alon LavieLanguage Technologies Institute
Carnegie Mellon University
Eliseo CañulefInstituto de Estudias Indígenas
Universidad de la Frontera
Carolina HuenchullanMinisterio de Educación
Chile
Presented by Ariadna Font-LlitjosLanguage Technologies Institute
Carnegie Mellon University
Overview
• Chile’s programs in bilingual and multicultural education
• The AVENUE project at Carnegie Mellon University
• The Mapudungun corpus• Plans for Example-Based Machine
Translation• Plans for Rule-Based Machine Translation
Bilingual and Multicultural Education in Chile
• Eight ethnic groups: Mapuche, Aymara, Rapa Nui (Pascuense), Likay Antai, Quechua, Colla, Kawashkar (Alacalufe), Yamana (Yagan).
• Make education culturally and linguistically relevant.
• Languages of instruction are native language and second language (Spanish).
• Community involvement in curriculum design.
AVENUE: Automatic Voice Enabled Natural language Understanding Environment
• Affordable machine translation for languages with scarce resources.– No large corpus in electronic form– Few or no native speakers trained in
computational linguistics
AVENUE: Omnivorous MT
• AVENUE can consume whatever resources are available– EBMT: if a parallel corpus is available– Human-Engineered MT: if a human
computational linguist is available– Seeded Version Space Learning for automatic
acquisition of transfer rules: if no corpus or computational linguist is available
Mapudungun
• Language of the Mapuche– Over 900,000 Mapuche in Chile and Argentina
• Words contain several morphemes including multiple open class items.
• Still spoken by a majority of Mapuche• Still spoken as a first language• Competing orthographies• Some vocabulary loss• Some written literature and newsletters
The Mapudungun Corpora
• First step toward:– Corpus-based machine translation– Authentic corpus for instructional purposes
• Written corpus
• Spoken corpus
The Written Mapudungun Corpus
• Existing texts were entered in electronic form and translated into Spanish:– Memorias de Pascual Coña: the life story of a
Mapuche leader written by Ernesto Wilhelm de Moessbach.
– Las Ultimas Familias by Tomás Guevara.
– Nuestros Pueblos newspaper published by Corporación Nacional de Desarrollo Indígena (CONADI).
• Total of around 200,000 words
The Spoken Mapudungun Corpus
• Recorded with Sony DAT recorder and digital stereo microphone.
• Downloaded with CoolEdit
• Transcribed with TransEdit– Alignment of audio and transcript for speech
recognition
The Spoken Mapudungun Corpus
• All sessions were scheduled and recorded by a native speaker interviewer
• Subject matter: primary and preventive health– Limited domain for higher quality machine
translation– People were asked to describe their experiences
with an illness and how it was treated by modern or traditional medicine
The Spoken Mapudungun Corpus
• Speakers: – 21-75 years old; most 40-65– Fully native speakers– Some auxiliary nurses for rural areas in Chilean
Public health system– Some machi:
• Did not reveal specialized knowledge
The Mapudungun Spoken Corpus
• Dialects:– Lafkenche, Nguluche, Pewenche – Williche will be recorded at a later stage of the
project• more morpho-syntactic differences from the other
dialects
The Mapudungun Spoken Corpus
• Orthography:– Pan-dialectal:
• 32 phones
• Some are dialectal variants of each other
– Supra-dialectal• 28 letters covering the 32 phones
– Typable on Spanish keyboard with some diacritics such as apostrophes
– Use Spanish letters for phonemes that sound like Spanish phonemes
Plans for Machine Translation
• Example-Based MT
• Seeded Version Space Learning for automated acquisition of transfer rules
Example-Based MT
• Insert one of Ralf’s slides
Automated Acquisition of Transfer Rules
• Elicitation Tool
• Seeded Version Space Learning
• Run-time transfer system for MT
Chinese-English Transfer Rule for Yes-No Questions
S::S : [NP VP MA] -> [AUX NP VP]((x1::y2) ; set alignments (x2::y3)
((x0 subj) = x1) ; create Chinese f-structure ((x0 subj case) = nom) ; Chinese has no case, so add it ((x0 act) = quest) ; set speech act to question (x0 = x2) ; create Chinese f-structure
((y1 form) = do) ; set base form of AUX to "do" ; proper form will be selected based on subj-verb agreement
((y3 vform) =c inf) ; verb must be infinitive ((y1 agr) = (y2 agr)) ; subject and "do" must agree)
Example of Seed Rule and Generalization
• Pair 1: the man::der mann• Pair 2: the woman::die frau
Seed Rule 1 Seed Rule 2 Generalization
Det N Det N Det N Det N Det N Det N
X1::Y1 X1::Y1 X1::Y1
X2::Y2 X2::Y2 X2::Y2((X1 AGR) = *3-SING) ((X1 AGR) = *3-SING) ((X1 AGR) = *3-SING)
((X1 DEF) = *DEF) ((X1 DEF) = *DEF) ((X1 DEF) = *DEF)
((X2 AGR) = *3-SING) ((X2 AGR) = *3-SING) ((X2 AGR) = *3-SING)
((X2 COUNT) = +) ((X2 COUNT) = +) ((X2 COUNT) = +)
((Y1 AGR) = *3-SING) ((Y1 AGR) = *3-SING) ((Y1 AGR) = *3-SING)
((Y1 CASE) = *NOM) ((Y1 CASE) =
(*NOT* *GEN *DAT))
((Y1 DEF) = *DEF) ((Y1 DEF) = *DEF) ((Y1 DEF) = *DEF)
((Y2 GENDER) = *M) ((Y2 GENDER) = *F) ((Y2 GENDER) = *F)
((Y2 AGR) = *3-SING) ((Y2 AGR) = *3-SING) ((Y2 AGR) = *3-SING)
((Y2 CASE) = *NOM)
((Y2 GENDER) = *M) ((Y2 GENDER) = *F) ((Y2 GENDER) = (Y1 GENDER))
Elicitation Tool
Elicitation Process
• Bilingual informant
• Literate in the elicitation language and the elicited language
• Translate sentences
• Align words
Elicitation Corpus: ExcerptHe has sold both of his cars. English promptEl ha vendido sus dos automóviles Spanish promptfey weluiñi epu awtu Mapudungun provided by informant He can move both of his thumbs. El puede mover sus dos pulgaresfey pepi newüleliñi epu fütrarumechangüll He loves both of his sisters. El ama a sus dos hermanas fey poyeyñi epu deya He loves both of his brothers. El ama a sus dos hermanos fey poyeyñi epu peñi
Elicitation Corpus• Compositional:
– Small phrases are elicited first and then are combined into larger phrases
– For learnability
• Minimal Pairs:– Sentences that differ in only one feature (e.g.,
number of the subject)– For automatic feature detection
• If the minimal pair differs only in the number of the subject, and the verbs are different in the two sentences, the language may have agreement in number between subjects and verbs.
Elicitation Corpus: Current Coverage
• 864 Sentences (pilot corpus)• Transitive and intransitive sentences• Animate and inanimate subjects and objects• Definite and indefinite subjects and objects• Present/ongoing and past/completed• Singular, plural, and dual nouns• Simple noun phrases with definiteness, modifiers• Possessive noun phrases
Elicitation Corpus: Future Work
• Probst and Levin (2002) – Pitfalls of automated elicitation
• Automatic Branching and skipping:– Automatically skip parts of the corpus
depending on what features have been detected
Status of automated rule learning
• Preliminary results – Learned some compositional rules for German
• Current work:– Interaction of compositional rules– Seed rule generation– Generalization and verification of seed rule
hypothesis
Status of Transfer Rule System
• Preliminary experiments on Chinese-English MT
• Integrated into a multi-engine system with Example-Based MT
Tools for Field Linguists?
• Can feature detection and automatically learned rules be useful to alert a field worker to possible interesting data?
• Can automated elicitation with branching and skipping be helpful?