1

A RULE BASED APPROACH ON STEMMING OF BENGALI

VERBS

Presented by:Joyeeta Bagchi

Sneha SarkarAnasuya Paul

Koushik Dutta

Under the guidance of-

Mr. Alok Ranjan Pal

2

CONTENTS Introduction Difficulty of the language Overview of stemming Related Work Pictorial Representation of proposed approach Example of each step Module 1(Suffix Stripping) Module 2(Applying Rules) Explanation of module 2 with example Algorithm Partial view of input file List of suffixes Partial view of output file Efficiency and time complexity Conclusion and future work

3

INTRODUCTIONStemming is an operation that splits a word into the constituent root part and affix without doing complete morphological analysis.For example, Eating = Eat + -ing

Worked = Work + -ed

The main purpose of stemming is to reduce different grammatical forms / word forms of a word like its noun, adjective, verb, adverb etc. to its root form.

We can say that the goal of stemming is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form.

4

DIFFICULTY OF THE LANGUAGE

Bengali is one of the most morphologically rich language and stemming of Bengali verb is the most problematic area for Stemming.

Sometimes, nearly 10x5 forms for a certain verb in Bengali may appear in different contexts.

5

OVERVIEW OF STEMMING

A typical simple stemmer algorithm involves removing suffixes using a list of frequent suffixes, while a more complex one would use morphological knowledge to derive a stem from the words.

6

RELATED WORK In 1980 Martin Porter developed the “Porter

Stemmer”.It uses the fact that English language suffixes are mostly a combination of smaller and simpler suffixes.

Ramanathan and Rao (2003) proposed a lightweight stemmer for Hindi which has used a hand crafted suffix list and has performed longest match stripping

Dasgupta and Ng (2006) proposed unsupervised morphological parsing of Bengali. When evaluated on a set of 4,110 human-segmented Bengali words, the algorithm achieves 83% success.

Majgaonker and Siddiqui (2010) developed an unsupervised approach for Marathi stemmer. The maximum accuracy observed is 82.5% for the statistical suffix stripping approach.

Suba et al. (2011) proposed two stemmers for Gujarati, with an average accuracy of about 90.7%.

7

PICTORIAL REPRESENTATION OF PROPOSED APPROACH

8

EXPLANATION OF EACH STEP

9

MODULE 1 (SUFFIX STRPPING)

10

MODULE 2 (APPLYING RULES)

11

EXPLANATION OF MODULE 2 USING EXAMPLE

12

ALGORITHMSTEP1: Start of algorithm.

STEP 2: Create 4 new string[] namely splits1[], splits2[ ] and splits3[ ].

STEP 3: Read the contents of the doc files and split the words by space (‘ ’) separator.

3.1. Store the words of each sentence in splits1[ ].

3.2. Store the inflexions in splits2[ ].

3.3. Store the desired root words in splits3[ ].

STEP 4: Declare and initialize variables l1=length of splits1[ ] , 12=length of splits2[ ] .

STEP 5: Fetch the inflected verb forms in input1[] from splits1[i] if ‘/verb’ is contained by the currently fetched word. This step is repeated 11 times.

5.1. Determine the subroot from input1[i] by repeating the steps 12 times.

5.1.1. if splits2[j] in contained in input1[i] then,

5.1.1.a. Declare variable index which stores the index of last occurrence of splits2[j] in input1[i].

13

5.1.1.b. If index is greater than equal to 2 then,5.1.1.b.i. Store the substring of input1[i]

from begindex=0 to endindex=index in input1[i].

5.1.1.b.ii. Break the loop.5.2. Determine the actual root input1[i] by repeating the steps l1 times. 5.2.1. Check the ending kar of input1[i].

5.2.1.a. I f input1[i] ends with e-kar(‘ি�’), o-kar(‘ো��’), a-kar(‘ো�’) or aa-kar(‘��’) then, replace it with aa-kar(‘��’).

5.2.1.b. if length of input1[i] is less than 3, concate it with aa-kar(‘��’).

5.2.2. Check the starting kar of input1[i].5.2.2.a. if input1[i] starts with e-kar

(‘ি�’), then replace it with a-kar(‘ো�’).

5.2.2.b. if input1[i] starts with u-kar (‘��’), then replace it with o-kar(‘ো��’).

5.2.2.c. if input1[i] starts with a-kar(‘ো�’), then replace it with aa-kar(‘��’).

14

STEP 6: Generate the output doc file by copying the contents of splits1[] and concatenating it with their obtained root words from input1[] wherever the word contains ‘/verb’.

STEP 7: Compare the obtained sentences in splits1[ ] with the desired sentences in splits3[ ] and calculate the efficiency.

STEP 8: End of algorithm.

 

15

PARTIAL VIEW OF INPUT FILE

16

LIST OF SUFFIXES

17

PARTIAL VIEW OF OUTPUT FILE

18

EFFICIENCY:

Dealing with 450 inflections of 14 selected root verbs, the proposed approach gives an efficiency of 99.36%.

TIME COMPLEXITY:

The time complexity of the proposed algorithm in worst case is O(n2).

EFFICIENCY & TIME COMPLEXITY

19

CONCLUSION AND FUTURE WORK

In this project, we present a lightweight stemmer for 14 selected Bengali Verbs that strips the suffixes using a predefined suffix list, on a “longest match” basis, and then finds root on basis of some rules.

Except a few cases, the result obtained from our algorithm is quite satisfactory according to our expectation.

We argue that a stronger and populated learning set would invariably yield better result. In future , we plan to test our algorithm with more sets of Bengali verbs.

 

20

THANK YOU