ASSESSMENT OF DIRECTED WRITINGBY A GROUP OF TESL STUDENTS IN UPSI
NORMAH BINTI OTHMAN
FACULTY OF LANGUAGES
UPSI RESEARCH CODE: 01-03-41-02UPSI RESEARCH ACCOUNT: 050554
2003
ASSESSMENT OF DIRECTED WRITINGBY A GROUP OF TESL STUDENTS IN UPSI
NORMAH BINTI OTHMAN
FACULTY OF LANGUAGES
UPSI RESEARCH CODE: 01-03-41-02UPSI RESEARCH ACCOUNT: 050554
2003
No DescriptionTable of contentList of figuresList of tablesAbstractAbstrak
TABLE OF CONTENT
Page23456
1.1 Introduction 71.2 Statement of problem 101.3 Objectives 121.4 Research questions 141.5 Literature review 141.6 Research design 231.7 Instrumentation 251.8 Data collection 251.9 Data analysis 261.10 Result 26
1.10.1 The scores obtained from the assessment of dir.ected writing 261.10.2 Salient features of directed writing 'assessment 44
1.11 Discussion and recommendation 521.12 References 531.13 Appendix 57
2
LIST OF FIGURES
No. Description Pagel.1 Sommer's and Cohen's criteria for assessment ofwriting 181.2 Categories for evaluating writing by Brown (2001) 20
3
LIST OF TABLES
No. Description Page1.1A The scores given by twenty-eight TESL students and one expert rater to 27
directed writing DW51 to DW751.1B The scores given by twenty-eight TESL students and one expert rater to 28
directed writing DW51 to DW75 (contd)1.1C The scores given by twenty-eight TESL students and one expert rater to 29
directed writing DW51 to DW75 (contd)1.2A Categories of scores by Teacher A and Expert Rater 1 301.2B Categories of scores by twelve raters using analytic scoring method 321.2C Categories of scores by seven raters using primary trait scoring method .... "
.).)
I.2D Categories of scores by eight raters using holistic scoring method 351.3 Descriptive statistics of scores given by twenty-eight TESL students and 39
one expert rater to twenty-five directed writing samples (DW51 to DW75)1.4 Non-parametric correlation coefficient (Spearman rho) of scores obtained 41
from twenty-eight TESL students and one expert rater1.5 Parametric correlation coefficient (pearson r) of scores obtained from 43
twenty-eight TESL students and one expert rater
4
Abstract
Twenty-eight TESL students in UPSI and one experienced English SPM Examinationexaminer (Expert Rater 1) assessed twenty-five samples ofForm Four ESL secondaryschool students' writing. Twelve of the TESL students used the analytic scoring method;seven used the primary trait scoring method; and eight used the holistic scoring method.The expert rater and one of the TESL students (Teacher A) used the English SPMExamination scoring method. Teacher A and Expert Rater 1 assessed the writing samplesindividually, but the other twenty-eight TESL students were gathered during three
separate seminars and workshops. The scores that twenty-eight TESL students, TeacherA and Expert Rater 1 gave to the writing samples were correlated using descriptivestatistics and parametric (Pearson R) and non-parametric (Spearman Rho) calculations.The analysis of these scores showed that there was significant correlation coefficientamong the scores obtained from all the subjects, even though they used three differentscoring methods to assess the writing samples. The TESL students who took part in theassessment during the seminars and workshops agreed that the three scoring methodswere suitable for classroom assessment as compared to the English SPM Examination
scoring method, which was more suitable for standardized assessment. A few strengthsand weaknesses of each scoring method were identified and solutions for use in theclassroom assessment were recorded in the salient features of assessment.
5
Abstrak
Dua puluh lapan pelajar TESL dari UPSI dan seorang pemeriksa kertas peperiksaanBahasa Inggeris SPM (Expert Rater 1) telah memeriksa dua puluh lima sampel penulisanpelajar Tingkatan Empat dari sekolah menengah. Dua belas orang dari pelajar tersebuttelah menggunakan skema permakahan analytic; tujuh orang menggunakan skema
permakahan primary trait; dan lapan orang menggunakan skema permakahan holistic.
Expert rater dan seorang dari pelajar tersebut (Teacher A) telah menggunakan skema
permakahan Bahasa Inggeris SPM. Teacher A dan Expert Rater 1 memeriksa sampelpenulisan secara individu, tetapi dua puluh lapan pelajar TESL yang lain telah
dikumpulkan dalam tiga seminar dan bengkel yang berlainan. Skor yang diberi oleh dua
puluh lapan pelajar TESL, Teacher a dan Expert Rater 1 telah dikorelasi menggunakanstatistik deskriptif dan parametric (Pearson R) dan non-parametric (Spearman Rho).Analisis skor ini menunjukkan terdapat korelasi yang signifikan, walau pun subjek telahmenggunakan skema permarkahan yang berbeza untuk memeriksa sampel penulisanpelajar sekolah. Pelajar TESL yangmengambil bahagian dalam seminar dan bengkelbersetuju bahawa ketiga-tiga skema permarkahan sesuai digunakan untuk penilaiandalam kelas jika dibandingkan dengan skema permarkahan Bahasa Inggeris SPM yanglebih sesuai untuk penilaian piawai. Beberapa kekuatan dan kelemahan setiap skema
permarkahan telah dikenalpasti dan resolusi untuk penggunaan dalam kelas telahdirekodkan oleh pelajar TESL semasa seminar dan bengkel.
6
1.1 INTRODUCTION
TESL teachers' assessment ofESL students' writing plays an important role in the
process of teaching. It is also important that their assessment provides confidence and
motivation for students to excel in their school-based, district-based, state-based and
national-based ESL examinations in Malaysian secondary schools. The TESL teachers in
schools normally handle the school-based ESL examinations, and a selected group of
TESL teachers who work together in a committee handles the district-based and the state
based ESL examinations. The Malaysian Examination Board (Lembaga Peperiksaan
Malaysia) handles the national-based ESL examination at the Sijil Peperiksaan Malaysia
(SPM) level.
The function of the school-based, district-based and state-based examinations in
Malaysian secondary schools is to give some insight into the progress of students'
learning and achievement while still in school, whereas the national-based examinations
are to give final grades that determine the students' future undertaking in their studies.
Nevertheless all levels of examinations are important to the students' learning process.
As Rabinowitz (2001) says, statewide, or nation-wide tests cannot yield the detailed
information necessary to target instruction for individual students. So this leaves a clear
and essential role for local assessments or school-based assessments to develop
diagnostic information about what students do well, where they are having difficulty and
how the instructional program might be adjusted to address their specific needs.
7
For school-based, district-based and state-based examinations, teachers normally adopt
certain sets of scoring methods to assess their students' writing depending on the
objectives of assessments. And there are many types of scoring methods available for
teachers to refer to when assessing their students' writing tasks. Each scoring method is
different from another in the sense that each has different criteria oflooking at students'
writing product. For example, one scoring method looks at a student's writing product
generallyand does not go into detail about analyzing the student's grammar performance,
whereas another scoring method looks into detail the grammar performance. No matter
what kind of criteria each scoring method has, the ultimate aim is the same, that is, to
grade students' writing.
It is important to make sure that the interpretation of students' writing performance,
regardless ofwhat kind of scoring methods used, helps the students in their process of
learning. It was discovered that a few scoring schemes in several countries/school
systems employ similar aspects. This means that these school systems are interested in
evaluating the same elements in students' writing. That is why the kinds ofwriting
offered to students in almost all schools are more or less in the same form and serves the
same function (Takala, 1988).
Some of the kinds ofwriting that teachers normally give to their students in schools all
over the world are essays, summaries, note taking, letter writing, paraphrasing, report
writing and directed writing. Teachers in assessing their students' performance in
language normally use these kinds ofwriting to evaluate students' language performance.
8
"Written language has always played a dominant role in formal education. Typically the
acquisition of literacy is considered to be one of the most important tasks of the school,
not only as a vehicle of learning, but as a means of achieving other goals as well"
(Takala, 1988). Written examination in the SPM examination is a very important type of
assessment that determines the students' performance in the language use. And the result
.
of the assessment is important because a good grade for English examination is a
prerequisite for the students to further their studies in certain fields such as medicine,
pure science and mathematics, especially at a foreign institution.
The International Association for the Evaluation ofEducational Achievement (IEA) has
carried out several studies on writing tasks and scoring scale. The IEA, which was
founded in 1959, had conducted much research to compare the educational performance
of school students in various countries and systems of education around the world
(Gorman, 1988: vii). The IEA's study ofwritten composition began in 1980 and the
findings were published in several volumes. The writing tasks studied were pragmatic
writing, letter writing, summary writing, descriptive writing, narrative writing, open
writing, argumentative/persuasive writing and reflective writing. There are also studies
conducted to investigate the effectiveness of some scoring methods used to assess
students' writing tasks.
9
1.2 Statement ofProblem
TESL teachers who conduct the school-based, district-based and state-based assessments
ofwriting provide immediate feedback for the students, and thus enable the students to
progress in their learning process, and in their preparation for the national-based
examination. The nation-based assessment ofwriting provides grades that determine the
students' future undertakings in their further studies. Expert raters conduct the national
based assessment and the Malaysian Examination Board (Lembaga Peperiksaan
Malaysia) trains the expert raters to assess the national examination. It is important then
for TESL teachers to balance their school-based, district-based and state-based
assessments with the national-based assessment ofwriting for the sake of students'
learning progress and school service improvement.
The issue about balancing school-based assessment and the nation-based assessment does
not only apply to Malaysian schools. In America for instance, the issue about balancing
state and local assessment has been raised by school administrators. State assessment in
America refers to their national level assessment, and local assessment is their school
based or district-based assessment. Rabinowitz (2001) in his article about balancing state
and local assessments in American schools finds that local assessment programs are still
relevant because effective local assessment is essential to improved student learning, and
that locally developed and administered assessment programs have unique capacity to
provide diagnostic information that, when understood and used effectively, has
immediate impact on classroom practice.
10
Another researcher who shares the same view with Rabinowitz about balancing state and
local assessment in America is Stiggins (2002), who believes that the current assessment
systems in American education are harming huge numbers of students and that harm
arises directly from the failure to balance the use of standardized tests and classroom
assessments in the service of school improvement. He also believes that student
achievement suffers because the once-a-year tests are incapable of providing teachers
with the moment-to-moment and day-to-day information about student achievement that
they need to make crucial instructional decisions. Stiggins (2002) suggests that teachers
rely on classroom assessment to make crucial instructional decisions. However the
problem is that teachers are unable to gather or effectively use dependable information on
student achievement each day because of the drain of resources for excessive
standardized testing. There are no resources left to train teachers to create and conduct
appropriate classroom assessments. For the same reason, Stiggins (2002) states that
district and building administrators have not been trained to build assessment systems
that balance standardized tests and classroom assessments. As a result of these chronic
and long-standing problems, classroom; building; district; state; and national assessment
systems remain in constant crisis, and students suffer the consequences.
The issue about balancing local and national assessment involves students' performance
in writing, because locally and nationally standardized examinations normally require
students to write continuously in a given period. For example, the P:MR, the SPM, the
STPM and the MUET nationally standardized examinations in Malaysia involve writing
components. So writing is commonly used to assess students' language skills and their
11
learning in many academic content-areas. Thus the need to provide students with fair and
supportable assessment approaches is very important because many decisions rest on
writing assessment. It is imperative that decision makers, national examiners, national
raters and schoolteachers who provide language performance report based on assessment
of students' writing give a fair report that really depicts the students' actual performance
because the report given determine the students' future undertakings and even future
career. So it is necessary to study the needs for validated school-based assessment
methods that TESL teachers can make used of to balance school-based and nation-based
assessment.
1.3 Objectives
Teachers assign writing tasks for different instructional purposes: to have learners imitate
some model ofwriting; to train learners in the use and manipulation of linguistic and
rhetorical forms; to reinforce material that students have already learned; to improve
learners' writing fluency; to encourage authentic communication whereby the writer
really wants to impart the information and the reader is genuinely interested in receiving
it; and to learn how to integrate all the purposes mentioned above, with the emphasis on
improving the whole performance, not just one of its aspects (Raimes, 1987 as quoted by
Cohen, 1994). Taking into consideration all these purposes as to why teachers assign
writing tasks to their students, the present study developed two main objectives that cover
TESL teachers' role in assessing their students' writing tasks; and the effectiveness of the
holistic scoring method, the analytic scoring method and the primary trait scoring method
to assess directed writing.
12
The two main objectives are:
1. To investigate TESL teachers' assessment of directed writing product written by
Form Four ESL secondary students. This writing product was given in Paper Two
of the English SPM Examination in Malaysia.
2. To establish the construct validity of the holistic scoring method, analytic scoring
method and primary trait scoring method to assess directed writing in ESL
classrooms in Malaysia.
The specific objectives ofthis study were:
1. To record and analyze the salient features of assessment verbalized by the TESL
teachers as they assessed directed writing.
2. To design and establish the holistic, analytic and primary trait scoring methods to
assess directed writing.
3. To test the validity and reliability of the holistic, analytic and primary trait scoring
methods designed to assess the directed writing.
4. To identify the strengths and weaknesses of the holistic scoring method, analytic
scoring method and primary trait scoring method used in this study to assess
directed writing.
5. To establish the concurrent validity of the scores obtained from the TESL teachers
after assessing directed writing using the holistic scoring method, analytic scoring
method and primary trait scoring method, as compared to the scores given by the
13
expert raters who trained by the Malaysian Examination Board (Lembaga
Peperiksaan Malaysia).
1.4 Research Questions
1. How did TESL teachers assess directed writing, and what were the salient
features of assessment that they verbalized as they reacted to the writing products
using the holistic scoring method, the analytic scoring method and the primary
trait scoring method?
2. To what extent were the holistic scoring method, the analytic scoring method and
the primary trait scoring method valid and reliable for assessment of directed
writing?
3. What was the relationship between the scores given by TESL teachers using the
holistic scoring method, the analytic scoring method and the primary trait scoring
method with the scores given by the expert rater, using the SPM examination
scoring method?
105 Literature Review
In Malaysia, three major national examinations are given to secondary school students.
These examinations are standardized nationally in terms of the questions given and the
assessment of the examinations. The examinations are known as the Penilaian Menengah
Rendah (P:MR) for the Form Three students (aged fifteen to sixteen years old); Sijil
PelajaranMalaysia (SPM) for the Form Five students (aged seventeen to eighteen years
old); and the Sijil Tinggi Pe!ajaranMalaysia (STPM) for the Form Six students (aged
14
nineteen to twenty years old). These examinations that are in the form ofwritten tests,
oral tests and practical tests serve to evaluate the school students' performance in several
subjects. The result of the examinations determines the students' chance for further
studies. They are graded according to their performance in the examination. For example,
students who get good grades in the science subjects will be placed in the science stream
classes; and students who get good grades in the arts subjects will be placed in the arts
stream.
The English Language is offered as a subject in all the national examinations at
secondary school level. At the PN1R and the SPM level, English is compulsory for all
students. At the STPM level, the subject is offered as an elective where the literature
elements are also included. However, since 1999, a new subject known as MUET
(Malaysian University English Test) is offered as a compulsory subject to all Form Six
students. In this subject, four skills are tested in separate examinations: listening skills in
Paper One, speaking skills in Paper Two, reading comprehension skills in Paper Three,
and writing skills in Paper Four. Starting from the year 2001 onwards, students who wish
to continue their studies at the tertiary level will have to take MUET as a prerequisite.
English as a Second Language (ESL) is a compulsory subject in the SPM examination. It
is crucial for ESL students to do well in the subject due to the importance of the
language. There are three major components tested in the SPM English Examination:
Oral English, Paper One and Paper Two. In Paper Two of the SPM examination, the
15
students are tested on three types ofwriting: directed writing, summary writing and essay
writing. The three types ofwriting tasks given in Paper Two of the SPM examination are
very important for secondary school ESL students in Malaysia. As much as it is
important for the students to do well in the writing tasks, it is also equally so for TESL
teachers to assess their students' writing well enough to ensure that the grades given
really depict the students' actual performance in writing.
Teachers' assessment of students' writing can greatly influence students' attitudes for
future learning because they can be easily confused by unclear, vague or ambiguous
responses and can become frustrated with their writing progress and hopes for the results
in their examination. Alternatively, students can be positively motivated if the assessment
given to their written work reflects their actual performance in the national level
examination. Unfortunately, there is no clear set of universal guidelines that will
guarantee such a supportive and positive experience for all students. In a given context
for writing instruction, students will differ, and tasks, topics, and responses will differ
(Grabe and Kaplan, 1996: 377).
Schoolteachers might be using different ways and methods to assess their students'
writing tasks, depending on the school authorities' instruction to them. Students will not
be able to predict their actual performance in the national examination if the performance
assessment method adopted by their ESL teachers may not be the same as the national
raters' performance assessment method. Normally, TESL teachers in secondary schools
invite the English SPM national raters who are considered as the expert raters to come to
16
their schools to conduct seminars and workshops for their students before they sit for the
national examination. This is to ensure that their students get some exposure about the
national raters' expectation when assessing their writing product in the national level
examination. The national raters are trained by the Malaysian Examination Board
(Lembaga Peperiksaan Malaysia) on how to assess the English SPM examination
according to their standards. The Malaysian Examination Board has its own scoring
method that national raters or the expert raters have to refer to when assessing the English
SPM examination papers.
Each scoring method used to assess students' writing tasks has its own unique ways of
looking into the details of the writing pieces. However the ultimate aim is the same that is
to give grades to the writing pieces. The problem is whether the scores given by one
examiner or rater differ from the scores given by another examiner or rater who uses a
different scoring method, ifboth of them have been assigned to rate the same writing
task. Ifwe look at the elements suggested by two different researchers, we will see the
differences in each focus. For example, Sommer (1989) and Cohen (1994) have
suggested someguidelines about the elements to look at when assessing students' writing
(see Table 1.1). There are some similarities and differences in their suggestions.
17
Figure 1.1 Sommer's and Cohen's criteria for assessment ofwriting
Sommer (1989) Cohen (1994)
Can the students write in grammatical Appropriateness of languagesentences? conventions - grammar, spelling,
punctuation;
Accuracy ofmeaning - selection anduse of vocabulary;
How well do students manage the basic Organization - sense of pattern for themechanical elements ofwriting (such as development of ideas;paragraph indentation, punctuation, and
Rhetorical structure - clarity and unityspelling?of the thesis;
How well do they organize information? Style - sense of control and grace;
Can students find a starting place to write?
Can they formulate opinions or arguments inwriting?
Do they stay on the assigned topic?
Can they describe a process?
How well do they understand the formats inwhich they must write?
How well do they understand their Reader's acceptance - efforts made in
audience(s)? the text to solicit the reader's
agreement, if so desired.
Reader's understanding - inclusion ofsufficient information to allow meaningto be conveyed;
Are any students experiencing dialect orsecond-language interference?
Register - appropriateness of level offormality;
Content - depth and breadth of
coverage;
Economy - efficiency of language use;
18
The first four columns in Sommer's and Cohen's show some similarities in the elements
that both look into when assessing students' writing pieces. For example in the first
columns of Sommer's and Cohen's, we can see that grammar is taken into consideration.
However Cohen's description is more detailed because he has included vocabulary as one
element of grammar.
In column five of Sommer's we can. see that this element does not appear in Cohen's.
Sommer looks into whether there is any interference from the dialect and second
language, and Cohen does not take that element into consideration. Instead Cohen takes
into consideration register of the writing piece, which Sommer does not.
In a more recent study, Brown (2001: 357) has come out with six categories that raters or
teachers should consider when assessing students' writing. He has adapted these
categories from J.D. Brown, 1991. Table 1.2 shows the categories that he has listed down
for assessing students' writing. He has listed the categories according to its importance.
19
Figure 1.2 Categories for evaluating writing by Brown (2001)
Content
Thesis statement
• Related ideas
• Development of ideas through personal experience, illustration, facts, opinions
Use of description, cause/effect, comparison/contrast
• Consistent focus
Organization
• Effectiveness of introduction
• Logical sequence of ideas
• Conclusion
• Appropriate length
Discourse
• Topic sentences
• Paragraph unity
• Transition
• Discourse markers
Cohesion
• Rhetorical conventions
• Reference
• Fluency
• Economy
• Variation
Syntax
Vocabulary
Mechanics
Spelling
• Punctuation
• Citation of references (if applicable)
• Neatness and appearance
20
As we can see, there are some differences in the categories given by Sommer (1989),
Cohen (1994) and Brown (2001). Ifone group ofTESL teachers make use of one of these
suggestions, and another group makes use of the other suggestion, there mayor may not
be some differences in their scoring system. This is because some of the elements stated
in each of the scoring methods do not exist in another's.
There are also similarities and differences in the scoring methods available for examiners
and raters to refer to when assessing students' writing. The similarities will not cause
problems for examiners or raters to give grades if they refer to different scoring methods.
The problem lies in the differences because it might cause differences in scores given to
the students' writing. Ifwe look into two scoring methods and compare the elements that
each look into, we will see some differences in their focus. For example holistic scoring
method looks into one single integrated score ofwriting behavior. It is interested in
responding to the writing as a whole and respondents are unlikely to be penalized for
poor performance on one lesser aspect, for example grammatical ability (Cohen,
1994:314). On the other hand, primary trait scoring method narrows its focus on a
specific aspect of the writing piece. So if two TESL teachers assess the same writing
piece, one with the holistic scoring method and the other with primary trait scoring
method, their focus while assessing the writing piece is definitely not the same. The
problem is whether the scores that they give to the same writing piece differ or not.
21
Cohen (1994:312) states that writers and teachers or raters differ in so many aspects
related to the assessment ofwriting. He quotes Ruth and Murphy as saying that:
1. Writers will differ in their notions about the significance of particular features of
the topic.
2. Students and their teachers (raters) differ in their recognition and interpretation of
salient points in a writing topic (with teachers having a wealth of professional
experience in the evaluation ofwriting while students have only their own
experience as test takers).
3. Student writers may construct different writing tasks for themselves at different
stages in their development.
Ruth and Murphy's findings, as quoted by Cohen (1994) above tell us that it is
universally accepted that writers and their raters differ in some ways or other. Even if two
raters are given the same scoring method to assess the same writing piece, there are
bound to be differences in their judgment.
There are a few studies that look into assessment ofwriting performance and their score
relationship. For example Johnson, Penny and Gordon (2001) studied score resolution
and the inter-rater reliability of holistic scores in rating essays; Hayes (2000) studied the
consistency of student performance on holistically scored writing assignments; and
Swartz, Hooper, Montgomery, Wakely, et al (1999) studied on using generalizability
22
theory to estimate the reliability ofwriting scores derived from holistic and analytical
scoring methods. Despite ofmany studies conducted that were related to writing
assessment and scoring relationship, Crehan and Hudson (2001) stated that unresolved
concerns remain for the more basic issues of objective and reliable scoring of
performance assessments, especially for writing products. Both of them conducted a
study on a comparison of two scoring strategies for performance assessments.
1.6 Research Design
This research involves a case study with twenty-eight TESL students from Universiti
Pendidikan Sultan Idris and one experienced TESL teacher. One of the TESL students
was involved in teaching a class ofForm Four ESL students, during which the directed
writing samples were taken as instrument for this research. Three separate seminars and
workshops were conducted to gather twenty-seven TESL students, during which they
were required to assess the directed writing samples. While assessing the directed writing
samples, their salient features of assessment were recorded. The experienced TESL
teacher was an expert rater who had twelve years experience in assessing the English
SPM Examination papers.
During the seminars and the workshops the TESL students were trained to assess the
directed writing samples using the three scoring methods devised for this research.
McNamara (2000:44) believed that initial and on-going rater training was an important
way to improve the quality of rater-mediated assessment schemes. The training normally
took the form of a moderation meeting, and this moderation meeting had the function of
23
Top Related