lems. In the case of grammar checking, two specific

A utomated Diagnosis of N on- N ative English

Speaker's Natural Language

Mari BowdenDepartment of Computer Science

The University of Texas -

Pan AmericanEdinburg, TX 78539Bowdenma@aol. com

Richard FoxDepartment of Mathematics

and Computer ScienceNorthern Kentucky UniversityHighland Heights, KY 41099

[email protected]

Abstract

Typical grammar checking software use some formof natural language parsing to determine if errors ex-ist in the text. If a sentence is found ungrammatical,

the grammar checker usually seeks a single grammat-

ical error as an explanation. For non-native speakers

of English, it is possible that a given sentence contain

multiple errors and grammar checkers may not ade-

quately explain these mistakes. This paper presents

GRADES, a diagnostic program that detects and ex-

plains grammatical mistakes made by non-native En-

glish speakers. GRADES performs its diagnostic task,not through parsing, but through the application of

classification and pattern matching rules. This makesthe diagnostic process more efficient than other gram-

mar checkers. GRADES is envisioned as a tool to help

non-native English speakers learn to correct their En-

glish mistakes, but is also a demonstration that gram-

mar checking need not rely on parsing techniques.

1. Introduction

or a true grammar diagnostic system. Most revolvearound the concept of bottom-up unification-basedparsing. This process requires that the system matchthe words of a sentence to grammatical roles, or con-stituents. Constituents of identified grammatical rolesare then merged into larger constituents. For instance,a determiner, adjective and noun are merged into anoun phrase, an auxiliary verb, verb, and preposi-tional phrase are merged into a verb phrase, and thenoun phrase and verb phrase are merged into a le-gal sentence. Unfortunately, the English language isrife with ambiguity since so many words can take onmultiple grammatical roles and there are a large num-ber of ways that constituents can be legally combined.Therefore, bottom-up parsing is a computationallyintractable process. Some shortcuts have been ap-plied to parsing to reduce the effort, such as usinglists of common well-formed phrases, and partial pars-ing. Other shortcut processes include combining top-down and bottom-up parsing where top-down pars-ing is used to disambiguate partially constructed con-stituents, and ill-formed phrase lists to further reducethe amount of processing. But even with these short-cut methods, the parsing process remains intractable.A variety of systems have been developed which trythese variations [6,7,9, 10, 11]. Interested readersmight also see [1, 8, 12] for useful overviews of Natu-ral Language Understanding research and commonlyused parsing algorithms.

Here, a different approach is taken. Grammarchecking is in fact a diagnostic process. It is one ofidentifying the cause of a malformed sentence. Artifi-cial Intelligence (AI) has pioneered many approachesto automated diagnosis. One successful approach hasbeen the Generic Tasks paradigm [4], where severalinformation-processing strategies have been identifiedthat can be united to solve a wide variety of prob-

People who learn English as a second languagemight produce erroneous sentences in spite of know-ing the proper grammatical rules of English. Teachingaids are sought to improve their performance. On~ so-lution is the grammar checker [13] .Grammar check-ers are standard in today's word processor softwareand are also available as tutorial systems. However ,the typical grammar checker does not expect mistakes,and when found, anticipates a single error, or an er-ror common to native English speakers. Therefpre,it takes more effort to diagnose the problems of rton-native English speakers.

There are many approaches taken to perform gram-mar checking, whether as part of a word processor

Proceedings of the 14th IEEE International Conference on Tools with Artificial Intelligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE

lems. In the case of grammar checking, two specifictasks, Hierarchical Classification [2] and HypothesisMatching [3], can be applied to solve the problem.This has been the approach taken by the GRADESsystem. GRADES has successfully and efficiently di-agnosed a variety of non-native English speaker sen-tences for their grammatical mistakes. This paperdescribes the GRADES system, the Generic Task-based approach using Hierarchical Classification andHypothesis Matching, and offers some examples of thesystem in action.

function in the hierarchy. In this way, only high-Ievelmalfunctions in the hierarchy are considered, unlessthey are found plausible, and then they are refined byconsidering their children in the hierarchy.

Applying this AI approach to grammar checking isstraightforward. First, a hierarchy of grammatical er-rors is required. This hierarchy will consist of, at ahigh level, the general types of errors, such as "verb-based errors" and "noun-based errors." Each of thesecategories can be broken into more specific types oferrors. For instance, a verb-based error might be a sit-uation where the auxiliary verb "to be" was misused,another might be that a modal verb was misused whileanother might be a disagreement between the subjectand the verb, or a lack of an auxiliary verb. Many ofthese errors can be further decomposed into more spe-cific categories such as incorrect form of the auxiliaryverb "to be," or incorrect order of auxiliary verbs.

To identify if a given error category is relevant, ahypothesis matcher is called upon. There are multiplehypothesis matchers, at least one per error category.It is the job of the hypothesis matcher to examinethe given sentence for the features it expects to findin order to establish that a given grammatical erroris present. These features are words with a specificgrammatical role, the proper modality, number, tenseand word order. A hypothesis matcher that finds fea-tures as expected then alerts the classifier that thereis an error of the type associated with that matcher .If there are child errors underneath that error, thoseare examined in turn. If the hypothesis matcher doesnot find the features it seeks, the error is assumed tonot appear and the next error type is examined.

By applying the Generic Tasks of Hierarchical Clas-sification and Hypothesis Matching to the diagnosis ofgrammatical errors, three things are gained. First, thehypothesis matcher tests usually entail a single passthrough the sentence or a subset of the sentence sothat each test can be performed without a computa-tionally difficult parse of the sentence. Second, theexplanation of the error is easy to present because itis essentially a description of the features found by thehypothesis matcher. Finally, by viewing the problemas one of diagnosis instead of parsing, the knowledgeneeded by the system is easy to identify because itmore closely matches the goal of the grammar checker-to identify errors. The next section presents a systembuilt using this Generic Task-based approach.

2. Using generic tasksgrammatical mistakes

to diagnose

Most grammar checking systems perform a fullor partial parse of the sentence using bottom-upunification-based parsing. Another approach is toview grammar checking as a diagnostic problemwhereby a sentence has a malfunction that is i~en-tified through some AI approach. The malfunctiqn issome improperly applied grammatical rule, an incor-rectly used word (or words), or an incorrect orderingof the words in some phrase. Viewed this way, gram-mar checking is the identification of incorrect rule(s)or word(s),and the generation of solutions to correctthose rule(s) or word(s). Identification of the causeof the error (the word or words making the sentenceungrammatical) can be accomplished through classifi-cation in order to reduce the amount of searching, Inparticular , Hierarchical Classification can be appl.ed.

Hierarchical Classification is a Generic Task.Generic Tasks are domain and problem-independentinformation-processing strategies. Generic Tasks havebeen combined to solve a great variety of problems,including diagnosis. A basic approach to solving di-agnostic problems is to classify the malfunction [5]. InHierarchical Classification, the diagnostic problem issolved through searching some taxonomy of malfunc-tions for the specific cause of the error. A malfunctionin the hierarchy is identified as a possible cause byevaluating specialized knowledge of features that areassociated with that malfunction. This knowledge iscaptured in the form of a hypothesis matcher, a pat-tern matcher that examines the data of the currentcase seeking those features associated with the mal-function. If suitable features are found, the hypoth-esis matcher responds that the malfunction is plau-sible. Classification resumes by cO-nsidering the mal-function's children in the hierarchy, that is, more spe-cific malfunctions. If a hypothesis matcher does !notfind evidence of a malfunction, then the malfunction isdismissed and the search continues with the next rnal-

3. GRADES

GRADES is the GRAmmar Diagnostic Expert Sys-

tem' a program that can detect and explain gram-


.Lack of main verb

.Incorrect choice of auxiliary verb "be" in nega-tion*

.Lack of auxiliary verb "be" for progressive tense*

.Lack of auxiliary verb "have" for present perfect*

Some of the above errors have more specific errors.For instance, there are ten common legal verb elementorderings, so "verb elements out of order" has its ownsub-hierarchy, illustrated in figure 1. In this figure, itcan be seen that if category 2 and category 3 auxil-iary verbs appear in a sentence without a category 1auxiliary verb, then the order must be the category2 auxiliary verb followed by the category 3 auxiliaryverb followed by the main verb. Category 1 verbs aremodal verbs {e.g., "can," "will"), category 2 verbs aresome form of "have," category 3 verbs are some formof "to be," and category 4 verbs are some form of"do." If a different order appears, then there is anerror with the verbal ordering.

matical errors of non-native English speakers. ! Itwas specifically designed to diagnose errors commpnlyfound by native Japanese adults who are learningi En-glish as a second language. GRADES contains a ftirlysmall lexicon of words (approximately 220) but! caneasily be expanded. Each word in the lexicon ~on-tains a variety of grammatical information about ~hatword. For instance, an auxiliary verb will includ~ its

Icategory (have, be, do, modal), form (present, ~ast,

etc) and number (singular, plural). Other gramt at- ical categories contain different attributes. Wor s in

GRADES are categorized in one or more of ten r les,I

where any given word might take on more than tone grammatical category. GRADES is able to diag ose

two general classes of grammatical errors, verb-rel tederrors ( of which there are thirteen subtypes of er~ors )and noun-related errors (of which there are four ~ub-types of errors) .

GRADES performs its diagnosis by applying pat-tern matching rules to the sentence or portions o~ thesentence. First, a verb element list is created by span-ning the sentence for those words that could be ~ux-iliary verbs or the main verb. Words that take onmultiple grammatical roles, where on role is a verbform, are further analyzed to ensure that they sh~uldbe part of the verb element list, otherwise theyi areomitted from this list. Non-verbs found between vprbsare also omitted from this list. Once the verb ele~entlist is available, processing begins as a search thr~ughthe taxonomy of verb-based error types. The se~rchis a goal-driven approach, unlike typical botto~-upparsers, which use a data-driven approach. The!fol-lowing are the specific verb-related list of errors that

Iare detected by GRADES (* denotes errors with morethan a single cause) :

Category 1: modal verbs (e.g.. "can. " "willJ

Category 2: f9rm of "have"Category 3: f9rm of "to be"Category 4: form of "do" Category 2 ,

v

Category 3

Aux v

Category 3Category 3

, Category 3

Category 2

Category 2

-v

Category

v

v

Category 3- V

Category 4

v (no Aux)

Figure Legal verb and auxiliary verb orderings.Subject-verb disagreement

Figure 2 illustrates how GRADES performs verb-based error detection. First, the verb element list isgenerated. Next, errors based on a misuse of "have"are sought. If a version of "have" does not appearin the sentence, this entire portion of the hierarchy isignored. Next, errors based on a misuse of "to be"are sought, followed by errors based on the misuse ofmodal verbs and "do" are sought. The last categorytests various errors that arise if no auxiliary verbs arepresent. Within each of these error causes, there aresub-causes so that if a given error is found plausible,its subtypes are examined for more specific errors. Ifany such error is found, GRADES generates an expla-nation of the error and continues searching (becausesentences might contain more than a single error) .Inthe case of a few errors Oisted in the verb list above

.Subject-auxiliary verb disagreement

.Verb elements (auxiliary verbs, main verb) o~t oforder

.Incorrect form of a verb in the sentence

.Incorrect sub-structure of a verb element

.Lack of auxiliary verb "do" for negation

.Incorrect choice of auxiliary verb "be" wit~ anintransitive verb

.Incorrect use of passive voice

.Lack of auxiliary verb "be" for passive voice


with an *) there is more than one possible cause ofthe error and so multiple explanations are provided.

Neither of these would be conceptually difficult. En-larging the system will not cause its accuracy to de-crease since grammatical errors do not generally in-teract in ways that make the detection of errors moredifficult in multiple-error cases, unlike many diagnos-tic domains where interacting errors can mask or com-plicate symptoms. Errors not currently implementedbut identified as future work are two further forms ofdisagreement: incorrect choice of pronoun and tensemismatch; incorrect modification: wrong selection ofadverb or adjective or wrong form of adjective; incor-rect use of a pronoun or preposition; lack of or extraarticle, preposition or subject; and incorrect word or-der of pronoun or adverb. Most of these errors aresimilar to those already implemented and the effortto include them is merely adding the portions of theclassification hierarchy and hypothesis matchers.

GRADES places three restrictions on its usage.First, and most importantly, GRADES is not ableto accept sentences that have more than one subject-predicate structure. Second, similar to the first re-striction, inserted words and clauses surrounded bycommas or hyphens, and parenthesized words andclauses are not allowed. Third, in order to simplifythe process of compiling the verb list, a shortcut wasinserted into the system whereby the subject of thesentence is specifically annotated as the subject. Allthree of these restrictions limit the scope of GRADESin that more complex forms of sentences cannot behandled (currently). The reason for these restrictionsis to simplify the matching mechanisms of GRADESand to remove any need for bottom-up parsing. How-ever, these are reasonable restrictions in that the sys-tem is intended for non-native English speakers whoare not yet very advanced in their use of English, andso are not yet expected to use such sentences.

I h1pul selnece I

..JJ- .,1iOO~ele" ent ist

first Velement have be modals do nan-oux V

(-verb)-E-wad order test

Type (1) (2) (3) ( ) (5) (6) (7) (8) (9) (1D)-E-verbformtest

I~ "'"'~ I .ubject-verb ogreement test

obsence of mBin \lertJ test

subject-oux v ogreement test negsti\le formotkJn test

pe..ive voice structure

test

Figure 2: Partial hierarchy of verb-related errors

After searching for verb errors, GRADES next eon-centrates on noun-related errors. The list of p<>ssi-ble noun-related errors is given below, a substant~allyshorter list.

.Noun number disagreement in NP

.Lack of a determiner

.Extra determiner

.Modifiers out of order

Figure 3 illustrates how GRADES searches for thenoun-related errors. The noun phrase is examined forone of four errors. The noun is compared to the adliec-tives for number agreement. A determiner is sought tosee if there is either a lack of determiner when one isneeded, or extra determiners are present. And finally,the determiner and adjectives are compared to makesure that they are in a proper order. Again, if anyhypothesis matcher detects an error, an explanationis generated and the search continues. Examples4.

iextra determiner test

modifier word order test

What follows are three examples of the GRADESsystem. The examples illustrate the processes appliedby GRADES. They also present a varied spectrumof error types. Each error is accompanied by bothan explanation of the reasoning process applied byGRADES and GRADES' own run-trace.

The first example is of the sentence "The boy planbuy a car." GRADES' run-trace for this sentence isshown in figure 4. GRADES begins by compiling theverb element list, in this case "plan buy." GRADESthen attempts to identify any or all errors. First,GRADES considers verb-based errors. The first ofthe hypothesis matchers examines the verb list to de-termine if any verbs appear out of le.gal order. The

t

lack of determiner test

Figure 3: Noun-related errors

Increasing the scope of GRADES is a matteX" ofadding error categories and increasing the lexicon size.


legal order of verbs is based on the category of eachverb (refer back to figure 1). Here, "plan" and "buy"are found to be in a legal order. Next, GRADES ton-tinues and another hypothesis matcher examines theverbs to determine if any are in an incorrect verb flt>rm.It finds two main verbs ( "plan," "buy" ) which is, notlegal, therefore one of these verbs has an imprbperform. GRADES determines that "buy" should be averb object (gerund or infinitive) and not a main verb.GRADES recommends that "buy" be changed to theform "to buy." There may be multiple errors in thesentence, so GRADES continues by checking the ~extpossible error cause, a subject-verb disagreement. IThesubject-verb disagreement hypothesis matcher findssuch a disagreement in that "the boy" is singular and"plan" is a plural form of the verb. Either the noun orthe verb is incorrect in that both should be sin~ular("the boy plans") or plural ("the boys plan"). iTheremaining verb and noun tests are negative (no el1rorsfound), and so GRADES has completed its diagnosisof this sentence.

Figure 5: GRADES run-trace for example 2

GRADES' run-trace is shown in figure 6. The verb el-ement list is compiled as "may have be enjoy to read"and processing continues with GRADES searching forboth verb-related and noun-related errors. GRADESdetermines five errors with the sentence, some fromeach general category. The first error is that two verbsare of an incorrect form. Both the auxiliary verb "be"is the wrong form of "to be" and the main verb "en-joy" is the wrong form. Next, the noun "girl" is foundto disagree with the determiner "many" as one de-notes singular and the other plural. A similar error isfound in the object phrase at the end of the sentencewhere "these" and "book" do not agree. GRADES de-tects a final error in that both "my" and "these" aredeterminers for "book" and the noun can only haveone determiner .

5. Conclusion

GRADES is a diagnostic expert system built toidentify the cause of grammatical errors. It is envi-sioned as a tool for people who learn English as a sec-ond language. Unlike grammar checkers that rely onbottom-up parsing and employ shortcut mechanisms,GRADES performs diagnosis through a classificationprocess whereby an error category is considered andpattern matching rules are used to determine if it isplausible or not. If found plausible, the error categoryis refined into more detail by considering that error'ssub-causes. Categories found implausible by their pat-tern matching rules are discarded. If an error is de-tect~d, an explanation is automatically generated tohelp the user learn why the sentence was ungrammati-cal. GRADES has a small lexicon but is able to detectmany of the various grammatical errors in English aslong as the sentence contains no complex, compoundor inserted clauses or words.


The second example is of a very simple sen-tence with incorrect grammar , "The book written."GRADES' run-trace is given in figure 5. The verb el-ement list is merely "written" so most of the possibleerrors are skipped. However, the main verb form hy-pothesis matcher determines that the form of the mainverb is incorrect. Since there are no auxiliary verbs,GRADES further determines that the main verb mustbe in a passive form. The solution to this second erroris in fact the solution to both detected errors as theyare related errors.

The final example for this paper is the sentence"Many girl may have be enjoy to read my these book."


blocks for expert system design. IEEE Expert,1.3, p. 23-30.

[5] W. J. Clancey. Heuristic classification. ArtificialIntelligence, 27(3), p. 289-350, 1985.

[6] W. H. DeSmelt. Ker Kommissar: an ICALLconversation simulator for intermediate German,in Intelligent Language Tutors, Theory ShapingTechnology, ed. V. Mellisa Holland, Jonathan DKaplan and Michelle R. Sams, p. 153-174, 1995.Laurence Erlbaum Associates, Inc., Publishers.

[7] D. Loriz. GPARS: A suite of grammar assess-ment systems. In Intelligent Language Tutors,Theory Shaping Technology, ed. V. Mellisa Hol-land, Jonathan D Kaplan and Michelle R. Sams,p. 121-133,1995. New Jersey: Laurence ErlbaumAssociates, Inc., Publishers.


GRADES has been successfully tested on a vari-ety of sentences ranging from no errors to as many asfive errors in a sentence. These sentences are basedon data accumulated from non-native English speak-ers learning the English language, although limitedto reflect the implemented rules and the small ]exi-con. GRADES has not yet been used in practice asa teaching tool. However, the test results are veryencouraging, and the hope is to use it as a teachingtool once the system has been expanded (in terms oflexicon and completing other error rules).

[8) M. Perlin. LR recursive transition networks forEarley and Tomita parsing. In Proceedings of the29th Annual Meeting of the Association for Com-putational Linguistics, p. 98-105, 1991.

[9) M. R. Sams. Advanced technologies for languagelearning: The BRIDGE project within the ARIlanguage tutor program, in Intelligent LanguageTutors, Theory Shaping Technology, ed. V. M.Holland, J. D. Kaplan and M. R. Sams, p. 7-21,1995. New Jersey: Laurence Erlbaum Associates,Inc, Publishers.

[10) D. Schneider and K. F. McCoy. Recognizing syn-tactic errors in the writing of second languagelearners, in Proceedings of the 36th Annual Meet-ing of the Association for Computational Linguis-tics and 17th International Conference on Com-putational Linguistics, p. 1198-1204, 1998.

[11] G. W. Smith. Computers and Human Language,

1991. New York; Oxford: Oxford University

Press.

References

[1] J. Allen. Natural Language Understanding. 1987,Menlo Park, CA: Benjamin/Cummings.

[2] T. Bylander, T. Johnson, and A. Goel. St~ uc- tured matching: A task-specific technique for

making decisions. Knowledge Acquisition, 3( ):1-

20,1991.

[12] H. Tanaka, T. Tokunaga, K. G. Suresh and K.Inui. Natural language analysis and generationtechnologies. Technical report, Department ofComputer Science, Tokyo Institute of Technol-ogy, 1993.

[13] C. J. Wong. Computer grammar ,cQecker andteaching ESL writing, the 9th Annu.al MidlandsConference on Language and Literature, 1996,

http:/ /www .coe.missouri.edu/ cjw /portofolio/grammar-checker.htm

[3] T. Bylander ~d S. Mittal. CSRL: A languagefor classificatory problem solving and uncertaintyhandling. AI Magazine, 7(3):66- 77, August 1$86.

[4] B. Chandrasekaran. 1988. Generic tasks in

knowledge-based reasoning: High-Ievel building


lems. In the case of grammar checking, two specific

Documents

Transcript of lems. In the case of grammar checking, two specific