AUTOMATIC IDENTIFICATION OF CLOSE LANGUAGES:case of Malay and Indonesian

Bali Ranaivo - ranaivo@cs.usm.mySiti Khaotijah Mohamad - sitijah@cs.usm.my

Computer Aided Translation UnitSchool o/Computer Sciences

Universiti Sains Malaysia11800, Penang, Malaysia

Abstract - Resume

The Malay used in Malaysia and the oneused in Indonesia are very closed to eachother since they both have the samecommon origin, the bahasa Melayu. Despitethis fact there exist differences which are tobe found in the spelling, lexical andmorphological levels. Any and allcorrection, translation, understanding of aMalay text without knowing whether it isMalaysian or Indonesian may lead to seriouserrors or to the use of irrelevant program.For these reasons we intend to create aprogram able to determine whether a text isMalay or Indonesian using the saiddifferences according to the digressive orderof their reliability. This research being at itsvery beginning, we shall, in this article, only'present the descriptive part of thedifferences existing between Malay andIndonesian.

Le malais utilise en Malaisie et celui utilise enlndonesie sont tres proches puisqu 'ils sont issusde la meme langue, Ie Bahasa Melayu. Malgrecette parente, il existe des differences qui sesituent aux niveaux orthographique, lexicale etmorphologique. Toutes correction, traduction,comprehension d'un texte malais sans savoir s 'ilest malaisien ou indonesien peut conduire il deserreurs importantes ou a l'application deprogramme inadequat. Pour toutes ces raisons,nous projetons de creer un systeme capable dedeterminer si un texte est malais ou indonesienen utilisant ces differences selon leur ordre defiabilite: du plus sur au moins sur. Cetterecherche n 'etant qu'il son debut, nous nepresentons dans cet article que la partiedescriptive des differences qui existent entre Iemalais et l'indonesien.

Key words - Mots clefs

Automatic Language identification, Malay,Indonesian.

Identification automatique des langues,malais, indonesien.

1. Introduction

For several centuries, the Malay language orBahasa Melayu, has been used as the linguafranca of various ethnic groups in the Malayarchipelago. Today it is known as BahasaMelayu (BM) in Malaysia (after 1963) andas Bahasa Indonesia (BI) in Indonesia (after1928) and it is still known as Bahasa Melayuin Singapore and Brunei Darussalam. BMand BI originate from the same languageand remain mutually intelligible. However,because of different socio-politicaldevelopments in Malaysia and Indonesiaduring the last one hundred years or so, thetwo varieties of Malay spoken in the twocountries have developed differently, eachinfluenced by different factors resulting inmany noticeable differences.

For example, because of different colonizer,the Malay language of Malaysia containsmore English loan words and theIndonesian, Dutch loan words. Besides, theimpact of Islam among the Malays inMalaysia is much more entrenched in BMthan in B1. In Indonesia the use of Arabic ismore limited. Instead, western and localvocabularies have constantly enriched the

Pre-COLING 2002 Seminar - Penang, Malaysia - 19 August, 2002 - 1

Indonesian language because of theinfluence of dialects.

With the rapid development of Malaysianand Indonesian web sites and the expansionof multilingual text corpus, the number ofdocuments writen in Malay (BM and BI)increase every day. Then a question ariseson how to identify the closed languages likeBM and BI and how to process thosedocuments automatically.

Therefore our aim is to build one rapid andefficiency program that able to determinewhether one written text said Malay text isMalaysian or Indonesian. We could reachour object if we classify and order thosedifferences.

Before enumarating the differences betweenBM and BI, we'll cite the works that havealready done to identify Malay texts.

2. Language identifiers for Malay texts

We query on the Web by using the searchbrowser Google "Malay languageidentification" and "Indonesian languageidentification". For the first query we got4,480 answers and for the second 10,900answers. As usual, we got a lot of answersrelated or not to our queries. But among allthose miscellaneous answers we find fewlanguage identifiers for Malay and forIndonesian. One of them is the Xerox'slanguage identifier' that recognise 47languages of which Malay and Indonesian.The demo is available on line. We havetested it by using some of the criteria usedfor us to identify Malay from Indonesian.For example, we submit the sentence Sayasukar bepergian. The system recognised itas a Malay text and not Indonesian. InMalay, the equivalent word of bepergian isberpergian. Without testing all systems thatexist to identify one of the two languages,BM or BI, we can say that most of them

I http://www.xrce.xerox.com/cgi­bin/rnlttlLanguageGuesser

don't really make the difference betweenBM and BI. The criteria they use canrecognise Malay language meaning BM andBI.

3. Spelling system

In 1972, Malaysia and Indonesia announcedofficially the implementation of the NewRomanized Spelling System. Before thisdate, the two countries had their ownspelling system influenced by English inMalaysia and by Dutch in Indonesia.

BM and BI have respectively 9 vowels and26 consonants. The grapheme < e >represents two phonemes: lal (schwa) andleI.

VowelsOne letter a, e, i, 0, u

Two letters ai, au, oiOne letter b, c, d, f, g, h,j, k,

ConsonantsI, rn, n, p, q, r, s, t,

v, w, x, Y, zTwo letters gh, kh, ng, ny, sy

Figure 1: Graphemes system in BM and BI

If we can say today that this reform spellingis a success, few parts of those Dutch andEnglish spelling systems still remain.

3.1. Combined letters

In BI, the spelling of some proper namesused under Dutch colonization is still on.

This concerns the sounds lUI, Idzl and Itf/.Before the reform spelling, they werewritten differently in Malaysia and inIndonesia.

8M BINew Reforrn

Spelling

lui u oe uSoekarno Sukarno

Idzl j dj jDardjowidjojo Darjowijojo

Itfl ch tj cDarmasoetjipta DarmasuciJJta

Pre-COLING 2002 Seminar - Penang, Malaysia - 19 August, 2002 - 2

Some of borrowed words from Dutch showspecific ending when they are retranscribedinto BI. For example, all words ending withthe trigrams < oar > find in Malay textscome from Dutch: supletoar 'supplement',repertoar 'repertoire', trotoar 'pavement'.BM use other words for the same meaning:penambah, himpunan, laluan jalan kaki.

between the two. Sometimes those wholereduplicated words are written differently.The word is written once, the hyphendisappears, and the number '2' follows theword. The difference between BM and BI isthe position of this numeral. In BM, '2'occupies the same baseline as the wordreduplicated. In BI, it is raised as the squaresymbol '2'.

3.4. Spelling of numerals: the use of fullstop and comma'

In BM and BI, the use of full stop andcomma are inverted. If in BM, the comma isused in digits to indicate numbers inthousand, million and so on, in BI, it is thefull stop that plays this kind of role.

3.2. Apostrophe' and inverted comma'

Apostrophe and inverted comma are usedequally in BM and BI. The rule doesn'tprecise when those marks could be used. Itsays only that "an apostrophe is thepunctuation mark used to show the omissionof a letter or letters". So in BM and BI wecan find those kinds of reductions in onetext:

I'gris (= Inggeris) EnglishP 'cis (= Perancis) FrenchAli 'kan (= akan) kusurati.I'll write to Ali.

In Indonesia, the apostrophe can be alsoused to show the omission of numerals:

1januari '88 (= 1988)

In Malaysia, there are two words in whichthe apostrophe or the inverted comma isused: Za'ba or Za'ba2 and Dato,3. They arethe vestiges of former spelling system: theapostrophe is used to mark the glottal stop.

BMbuku-bukubuku2

BMtwo thousandtwo thousandthstwo thousand ringgit

BItwo thousandtwo thousandthstwo thousand rupiah

4. Morphology

books

2,000ke-2,000 = kedua ribu$2,000.00

2.000ke-2,000 = kedua ribuRp2.000,00

3.3. Reduplicating symbol '2'

The Malay language has different forms forreduplicating words (cf. Asmah Haj i Omar,1975: 185-223, "Reduplication in Malay").One of those forms is the wholereduplication, where the whole word isrepeated twice by inserting a hyphen

2 It's the name of a Malaysian grammarian; wecan find also his name written: Zaaba.3 It's the synonym of Datuk, a honorificMalaysian title.

Because of their relationship, the differencebetween BM and BI at morphology level isvery slight. We have found only twodifferences. The first one is about the ruleswith the prefix {beR-} and the circumfix{beR-an}. The second is aboutabbreviations.

4.1. Affixation with {beR-}, {beR-an}

The affixation with {beR-} and {beR-an}depends on the initial phoneme type of theroot. If it is a Irl, the Irl of the prefix and the

Pre-COLING 2002 Seminar - Penang, Malaysia - 19 August, 2002 - 3

Irl of the root fuse and become one Ir/. Inother contexts, the prefix is just put beforethe root without any changing. But in BIthere is another context for those twoaffixes. With some roots in which the firstsyllable is closed with Irl, the prefix has theform "be-".

1) {beR-} ~ be I _/r/{beR-}+ RUMAH => berumah

2) {beR-} ~ ber I _other context{beR-}+IDU => beribu{beR-}+SURAt => bersurat

Those two rules are common for BM andBI. The third rule is specific to BI.3) {beR-} ~ be I --fCVrl

{beR-}+SERTA => beserta

4.2. Abbreviations

5. Lexicon

Data >< 10 000 wordsRedundant >< 7694 wordsWordlist >< 2306 wordsSame BM/BI words >< 1 605BI words only >< 701

Figure 2: Data statistic

In our data that we have retrieved fromMedia Indonesia Online, dated on August 8,2002, they used 2 306 words to write thenews with total of words is about 10 000words. From 2 306 words, we got about 701words that used only in BI. This mean 1605words are the same words used in BM.Therefore we use this 701 words as our mainguide to identify the difference of lexiconbetween these two languanges.

{

The b lexicon differences can be groupedunder five main aspects:

1) Words with different spellings

BI used the Dutch spelling while BM usedEnglish.

Malay people seem enjoying to shortenwords. They use them very often so forsomeone who learns Malay it becomes veryfuzzy to understand the meaning of thesentence. For our research, we will use thosewords to disguish BM and BI.

Examples of abbreviations in BMKDN = Kementerian Dalam Negeri

The Ministry of Domestic AffairsPernas = Perbadanan Nasional

National Consortiumtadika = taman didikan kanak-kanak

Kindergarten

BMtelevisyenstesenkuizkeranabahawaOgosakhbarbahagian

HItelevisestasiunkuiskarenabahwaAgustusakbarbagian

2) Words with different meanings

It is true that a much larger number of wordsin BM and BI have the same meanings.However, there are also many words with

Examples of abbreviations in BI

ABRI = Angkatan Bersenjata RepublikIndonesia

Army of Republic ofIndonesiaAkabri = Akademi Angkatan BersenjataRepublik Indonesia

Academy of Indonesian Armypemilu = pemilihan umun

general election

pejabatsulitbolehpolisibisa

BMofficesecretcanpolicyvenom

BIgovernment officerdifficultpermisablepolicecan

Pre-COLING 2002 Seminar - Penang, Malaysia - 19 August, 2002 - 4

3) Different words for one thing /concept

Most of the 701 words that have beenmentioned before are included in this type ofdifferences.

various meanings and the first or moreimportant meaning in BM often differs fromthat in BI.

BMikan bilistualabilikdibenarkanrancanganmaklumatpontengitik

BIikan teri

handukkamardiizinkanacarainJormasiabsenbebek

Meaninganchoviestowelroompermittedprogrammeinformationabsentduck

lexicons by mapping the Kamus Dewan andthe Kamus Besar Bahasa Indonesia. We'llextract from this comparison three lists ofwords: 1) list of words belonging to BM andBI, 2) list of BM words and 3) list of BIwords. Our work will make easy if we canuse the electronical form of the Indonesiandictionary.

If we want to know exactly the language ofone text, the best way for it is to check ifmost of the words of this text belong to thedictionary of one language. Instead of usingthe whole dictionary, it's more reliable touse only part of this dictionary meaning thewords that belong only to this language. It'sin this point of view that we want to createthe proper lexicon of BM and BI.

4) Different loan words

BM: abdomen, abstrak, absurd, birokrasi

BI: adhesi, administrasi, adopsi, mobil,direktur

BM has adopted many Briitish words whileBI has incorporated many Dutch words.

5) Different title as an indicating rank,status

BM: En. Zakaria Idris, Pengurus Besar

BI: Sdr. Zakaria Idris, Direktur Utama

In Malaysia there are many titles toaddressing someone depend on their rank,status and age but may be not so many inIndonesia.

Building proper lexicon

We can see through those lexicondifferences that each language hasdeveloped its own lexicon by borrowing,shorting words or derivating. Let's call thiskind of lexicon, "proper lexicon of BM (orBI)". We are planning to get those proper

6. Conclusion

From what we have said, we can divide thecriteria identifications into three groups:criteria at the characters level, criteria at themorphology level and criteria at the lexiconlevel. Between the three, it's of course thelast one that can be really reliable forlanguage identification. But we think thatbefore looking up in the dictionary, theresult get from the two other levels maydirect the program to open the appropriatelexicon that can reduce the time foridentification.

In our article, we raise the problem of theidentification of closed languages. We hopethat through our research, the differencesbetween Malay and Indonesian will appearclearly and that our language identifier willbe an unavoidable tool for any systemworking on Malay texts.

REFERENCES

Asmah Haji Omar, Essays on MalaysianLinguistics, Dewan Bahasa dan Pustaka,1975.

Pre-COLING 2002 Seminar - Penang, Malaysia - 19 August, 2002 - 5

Asmah Haji Gmar, The Linguistic Sceneryin Malaysia, Dewan Bahasa dan Pustaka,1992.

General guidelines for Malay spelling,Dewan Bahasa dan Pustaka, 1992.

General guidelines for the formation ofterms in Malay, Dewan Bahasa dan Pustaka,1992.

Ismail Bin Dahaman, Pedoman Ejaan danSebutan Bahasa Melayu, Dewan Bahasa danPustaka, 2000.

Ismail Bin Dahaman, Pedoman Ejaan RumiBahasa Melayu, Dewan Bahasa danPustaka, 2000.

Kamus Besar Bahasa Indonesia,Departemen Pendidikan dan Kebudayaan,1990.

Pedoman Umum Ejaan Bahasa IndonesiaYang Disempurnakan, Edisi kedua,berdasarkan Keputusan Menteri Pendidikandan Kebudayan Republik Indonesia, 1987.

Samsuri Ikip Malang, Tata Kalimat BahasaIndonesia, Sastra Hudaya, 1985.

Van Dyck, A.-M., Malherbe, V., ParlonsIndomfsien, L'Harmattan, 1997.

Vinsensius Berlian Vega SN, Bresnan S.,"Indexing the Indonesian Web: LanguageIdentification and Miscellaneous Issues",http:///www.comp.nus.edu.sgl-vinsensi/pubs/wwwl O-IndexingIndonesianWeb.pdf.

/

Pre-COLING 2002 Seminar - Penang, Malaysia - 19 August, 2002 - 6