DOCUMENT RESUME - ERICDOCUMENT RESUME ED 213 035 CS 206 751 AMOR Spandel, Vicki; Stiggins, Richard...

DOCUMENT RESUME

ED 213 035 CS 206 751

AMOR Spandel, Vicki; Stiggins, Richard J.TITLE Direct Measures of Writing Skill: Issues and

Applications. Revised Edition.INSTITUTION Northwest Regional Educational Laboratory, Portland,

OR. Clearinghouse for Applied Performance Testing.SPONS AGENCY National Inst. of Education (DHEW), Washington,

D.C.PUB DATE Aug 81GRANT OB-NIE-G-78-0206NOTE 62p.; For related document see ED 196 038.

EDRS PRICE MF01/PC03 Plus Postage.DESCRIPTORS *Educational Assessment; Educational Planning;

Elementary Secondary Education; *Evaluation Methods;*National Surveys; Testing; Test Results; *Test Use;*Writing (Composition); *Writing Evaluation

ABSTRACTIntended for educators seeking information on direct

writing assessments, this monograph describes general procedures forplanning and conducting a writing assessment and strategies fortailoring that assessment to local needs. The introductory chapteroffers a brief comparison of direct and indirect writing assessmentmethods, highlighting those features of direct assessment that makeit the most popular approach. The status of writing assessment inAmerican education is then summarized with emphasis on cur- ntpatterns and developmental trends. The second chapter presents anoverview of direct writing assessment procedures, touching onconsiderations in maximizing test quality, strategies for exercisedevelopment and alternative scoring approaches. The third chapterdiscusses selection of a writing assessment approach to suit aspecific educational context such as program evaluation or studentscreening. Appended is a profile of statewide writing assessment,which provides in ration on the use of objective tests and orwriting exercises, the developers of such exercises, kinds of writingassessed, kind of scoring method used, those using the results, andcontact persons. (HOD)

************************************************************************ Reproductions supplied by EDRS are the best that can be made ** from the original document. *

***********************************************************************

U.S. DEPANTNIENT OS EDUCATIONNATIONAL INSTITUTE OF EDUCATION

EDUCATIONAL RESOURCES INFORMATION

'IICENTER (ERIC)

his document has bean reproduced asreceived from the person or organizationonginatmg ir

Minor changes have been made to Improvereproduction quality

a Points of view or opinions stated in this documerit do not necessanty represent official NIEpolabon or policy

Direct Measuresof Writing Skill:Issues and Applications

REVISED EDITION

Vicki SpandelRichard J. Stiggins

L..CAPT

tir

"PERMISSION 70 REPRODUCE THISMATERIAL HAS BEEN GRANTED BY

NREL

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)

Clearinghouse for Applied Performance Testing

/cNORTHWEST REGIONAL EDUCATIONAL LABORATORY

300 S.W. Sixth AvenuePortland, Oregon 97204

I

4,

January 1980August 1981 (Revised Edition)

This work is published by the Clearinghouse for Applied PerformanceTesting (C A PT) of the Northwest Regional Educational Laboratory, a pri-vate nonprofit corporation. The work contained herein has been devel-oped under grant OB-NIE-G-78-0206 with the National Institute of Edu-cation (NIE), Department of Health, Education and Welfare. Theopinions expressed in this publication do not necessarily reflect the posi-tion of the National Institute of Education, and no official endorsement bythe institute should be inferred.

TABLE OF CONTENTSPage

PREFACE v

ACKNOWLEDGEMENTS vii

CHAPTER I: Introduction to Writing Assessment . . 1

Comparison of Direct and IndirectWriting Assessment 2

Assessment Focus 3Practical Testing Considerations 3Characteristics of Test Exercises 4Judging Test Quality 5Comparing Assessment Options 5

A Status Report on WritingAssessment Programs 6

StillNo "Best" Answer 11

CHAPTER II: An Overview of Direct WritingAssessment Procedures 13

Ensuring High Quality Assessment 13Reliability 13Validity 15

Developing Exercises 17Assessment planning 17Exercise development 18Review of specifications and exercises . 21Exercise pretesting 22Final exercise revision 23

Procedures for Scoring WritingSamples 23

Holistic scoring 23Analytical scoring 26Primary trait scoring 27

iii

Scoring language usage andmechanics 29

T-unit analysis 31

A Comparison of Scoring Methods 34

CHAPTER III: Adapting Writing Assessment toSpeciU Purposes 35

Using Tests to Manage Instruction 36

Using Tests to Select Students 36

Using Tests to Evaluate Programs 37

Selecting Examinees as a Functionof Purpose 38

De N eloping Exercises as a Functionof Purpose 38

Selecting Scoring Procedures as aFunction of Purpose 39

Ensuring Efficient, Effective, andHigh Quality Assessment 42

REFERENCES 43

APPENDIX: Profiles of Statewide WritingAssessment 45

5

iv

PREFACE

The Clearinghouse for Applied Performance Testing(CAPT) has published a series of monographs on the assess-ment of writing proficiency. Direct Measures of Writing Skill:Issues and Applications, published in the first edition in Janu-ary 1980, was the initial volume in that series. It presentedperspectives on writing assessment as they had developedthrough the 1970s and included results of a 1979 nationalsurvey of statewide writing assessment programs compiledby Vicki Frederick of the Wisconsin State Department ofEducation.

This revised edition contains much of the same informa-tion f')und in the original. However, the views presented havebeen updated to reflect two additional years of writing as-sessment research and development Included in this editionare summarized results of a 1981 national survey of state-wide and large-city school district writing assessment pro-grams compiled by Michael McCready and Virginia Meltonof Louisiana Technological University.

This monograph is written for educators interested inlearning about procedures for the direct measurement ofwriting skills: that is, testing through the use of student writ-ing samples. Minimum attention is given in this volume to theindirect assessment of writing skills via objective languageusage tests. Material presented herein is directly useable byeducators at all levels, from elementary, junior high and highschool to postsecondary and state department levels.

Those interested in additional information on writing as-sessment are directed to three recent CAPT publications: Us-ing Writing Assessment in the Classroom: A Teacizer's Hand-book, A Directory of Writing Assessment Consultants and AGuide to Published Tests of Writing Proficiency. The formerprovides teachers with strategies for using writing assess-ment methods to teach writing skills. The latter provides con-sumer information on available published tests of writingskill, and sources of technical assistance in developing andimplementing writing assessment programs. Anyone in-terested in obtaining these publications is urged to contactCAPT for further details.

CAPT intends to continue its role in collecting, synthesiz-ing and disseminating information on writing assessment.Readers are encouraged to submit comments and sugges-tions regarding this and oth( r CAPT writing assessmentpublications.

Richard J. StigginsCAPT Coordinator

kJ

7

ACKNOWLEDGMENTS

The authors wish to thank CAPT staff members NancyBridgeford and Carol De Witte for their valuable assistancein the production of this revised edition of Direct Measures.Nancy played a key editorial and production coordinationrole, and Carol provided excellent secretarial assistance.

Thanks arc also due the NWREL Media Center for theirproduction assistance and to the many people whose timeand ideas contributed to development of the first edition.

Special thanks to Mike McCready and Virginia Melton ofthe Teacher Education Department, Louisiana TechnologicalUniversity, Ruston, Louisiana, for their willingness to sharethe results of their valuable research on state and local writ-ing assessment programs.

Vicki SpandelRichard J. Etiggins

VII

CHAPTER I: Introduction toWriting Assessment

Until recently, those concerned with the large-scale assess-ment of writing proficiency relied predominantly on objec-tive tests of language usage skill. Evidence of this fact can befound in the language skills tests included in standardizedachievement batteries offered by publishers over the past 40years, as well as in the language skills sections of the majornational college entrance examinations. However, changingteacher attitudes and research and development efforts ledby Educational Testing Service (ETS) and the National As-sessment of Educational Progress (NAEP) have combined toshift the foci!, of writing assessment away from objectivetests, toward the use of writing samples as the basis for judg-ing proficiency. This new emphasis has been made possiblein part through development of writing sample scoring pro-cedures capable of producing valid and reliable results in anefficient, often cost effective manner.

The direct writing assessment techniques pioneered byETS and NAEP are already being adopted by school dis-tricts, state education agencies, postsecondary institutions,and test publishers. Further, acknowledging the desirabilityof directly assessing writing proficiency, professional asso-ciations of English teachers are urging adoption of writingsample-based testing.

As a result of these developments, many educators areseeking information on direct writing assessment. Thismonograph has been prepared to help meet their needs. It

1

offers the interested educator the basic information requiredto use educationally sound assessments of writingproficiency. It does not, however, present step-by-step in-structions on how to measure writing skill. Those steps varygreatly from situation to situation, and should, wheneverpossible, be planned with the assistance of an experiencedwriting assessment consultant. The monograph does, how-ever, describe general procedures for planning and conduct-ing an assessment, and strategies for tailoring that assess-ment to local needs. Sources of additional information arealso provided.

This introductory chapter offers a brief comparison of di-rect and indirect writing assessment thuds, highlightingthose features of direct. assessment th t make it the mostpopular approach. The status of writin assessment inAmerican education is then summarized with emphasis oncurrent patterns and developmental trends.

Chapter 2 presents an overview of direct writing assess-ment procedures, touching on considerations in maximizingtest quality, strategies for exercise development and alterna-tive scoring approaches. Chapter 3 discusses selection of awriting assessment approach to suit a specific educationalcontext.

A Comparison of Direct and Indirect WritingAssessment

There are two viable approaches to the assessment of writ-ing proficiency. One is the direct method. It relies on actualsamples of student writing to judge writing proficiency. Thesecond is the indirect method, which relics on objective tests.Research on the correlation between the two reveals a consis-tent and relatively strong relationship at various educationallevels. Summarized here are six studies that correlated objec-tive language usage test scores with scores obtained on writ-ing sample-based assessments.

The results of these studies suggest that the two ap-proaches assess at least some of the same performance fac-tors; yet each deals with some unique aspects of writing skill.These similarities and differences relate to assessment focus,practical aspects of testing, characteristics of test exercisesand aspects of test qualityFor a more detailed discussion of these fa,-tors as thei, relate to direct and mdtrect

aSSCSSMent. see Stiggins HSI)

2

10

Researchers Students Tested N Correlation

Godshalk, Swineford tL2of f man (1966) High school 646 46-.75

Breland. Colon & Rogosa(1976) College 96 42

Breland & Gaynor (1979) College 819 .63895 63517 58

Huntley, Schmeiser & Stiggins(1979) College 50 .43-.67

Hogan & Mishler (1980) Third graders 140 68Eighth graders 160 65

Mors, Cole & Khampahkit Fourth graders 84 20-.68(1981) Seventh graders 45 .60- 67

Tenth graders 98 72- 76

Assessment Focus. Direc- and indirect writing assess-ments focus on different components of writing. Direct as-sessment measures actual composition skill. Indirect testsability to useor recognize proper use ofthe conventionsof effective writing: grammar, punctuation, sentence con-struction, organization, and so on. Direct assessment pro-vides necessary and sufficient information for drawing con-clusions regarding a student's writing proficiency. Indirectassessment, on the other hand, provides necessarybut notalways sufficientinformation for evaluating a student'swriting proficiency.

An examination of traits measured in the two approachesreveals that indirect assessment tends to cover highly explicitconstructs in which there are definite right wad wrong re-sponses (e.g., grammar is either correct or it is not). Directassessment, on the other nand, tends to measure less tangi-ble skills (e.g., persuasiveness), for which the concept of rightand wrong is less relevant.

Practical Testing Considerations. Several important prac-tical matters related to testing sugvst additional differencesbetween direct and indirect assessment.

For example, effective assessment requires appropriateattitudes on the part of test users. With direct assessment,users of the test results must he willing to invest the time,money and effort to conduct a writing assessment that calls

3

11

En. complex, often time consuming testing procedures. In thecase of indirect assessment, users must be willing to accept aproxy measure: that is, a test that covers component skills ofNN riting without actually requiring students to wr'te. Giventhe appropriate attitudes, either direct or indirect assess-ment will most probably have its desired impact. If those atti-tudes are lacking, problems can be anticipated.

In either direct or indirect assessment, the examiner hastwo choices for test acquisition: (a) selecting an already exist-ing test or (b) constructing a new test.

If one decides to use previously developed exercises andscoring criteria for direct assessment, then the followingskills will be required of those conducting the assessment: (1)technical expertise in writing, to specify h writing skills%%41 be assessed; (2) test evaluation skills to investigate availa-ble options and select test items that measure the skills to beassessed; and (3) org niiational skills to set up, administer,score and report the results of the assessment.

Selecting an already developed objective tst requires theexpertise to determine the information needs of the test userand to re%iew and select a valid and reliable test. In mostcases, the user will also have to be skilled in interpreting andusing norm-referenced standard scores.

Developing a new direct instrument, which involves creat-ing new set of exercises and criteria for scoring, also de-mands organizational skills and technical writing expertise.In addition, however, psychometric expertise is required inorder to e% aluate the validity and reliability of the assessmentprocedures, and refine exercises and criteria as necessary.

Developing a new indirect assessment or objective test in-strument requires (1) technical expertise in writing to planthe assessment; (2) skill in item writing or selection; (3) or-ganizational skills to pilot test, analyze and select the newitems; and (I) psychometric expertise to evaluate the test'sreliability and the validity.

In short, developing new instruments for either testing ap-proach requires substantially more expertise and staff timethan does using existing assessment instruments.

Characteristics of Test Exercises. There are some funda-mental differences the kinds of test exercises used in directand indirect writing assessment First, the exercises differ inform Direct assessment exercises generally take the form of

1

a short paragraph that invites the examinee to respond to aquestion, state an opinion, resolve an issue, explain a proc-ess, recount an event, or simply express hislher feelings. Theexercise, if well constructed, identifies for the examinee the(1) form of writing to be produced, (2) audience to be ad-dressed, and (3) purpose forth' writing. Indirect assessmentitems frequently follow a multiple choice format, though fill-in questions are sometimes used. Various interlinear forms,as well as sentence combining items, are common.

As a result of differences in format, direct assessment ex-ercises are considerably more flexible than indirect. With di-rect assessment, the stimulus can be auditory or visual andcan be quite true to life (e.g., writing a job application letter).Indirect test items, on the other hand, are generally con-strained by the multiple choice (or other) format. Therefore,while direct assessment exercises can be made to closely ap-proximate "real world" writing, objective test items aresomewhat more artificial.

Judging Test Quality. The factors commonly considered injudging the psychometric adequacy of a test are reliabilityand validity.

Reliability and validity considerations for direct and indi-rect measures are quite similar. In the case of direct meas-ures, score stability is important over time, across exercises,across test forms and across raters. Consistency across ra-ters is not an issue with indirect measures, however, sincescoring is totally objective. In both cases, sources of inaccu-rate scores include poor test items and improper test ad-ministration. Sources of score inaccuracy unique to each ap-proach include: (I) guessing on indirect measures, and (2)poor or inconsistent scoring of direct measures.

Validity considerations are similar. Content validity is rele-vant to both types of writing tests and should be verified inboth via expert judgment. Criterion related validity, also im-portant in both cases, can be verified through correlationswith other indicators of writing proficiency.

Comparing Assessment Options. Direct and indirect ap-proaches to writing assessment are perhaps best comparedin terms of their relative advantages and disadvantages, andthe primary ways in which each can be used.

The major advantages of the direct assessment option are

ra

13

(1) the extent of information provided about examinees'writing proficiency, (2) potentially high fidelity (authenticity)of the exercise and response, (3) the adaptability of exercisesto a variety of relevant real world writing circumstances, (4)high face validity, and (5) relatively low test developmentcosts.

The major advantages associated with the indirect assess-ment are ( 1) high score reliability, (2) relatively low test scor-ing costs. and (3) high degree of control over the nature of theskills tested.

The disadvantages of the direct method include (1) highscoring costs, and (2' the potential lack of uniformity amongexaminees regarding the proficiencies assessed.

The disadvantages of the indirect method are (1) lack offidelity to real world writing tasks, (2) heavy reliance on ex-aminees' reading rather than writing proficiency, and inmany cases (3) lack of face validity in the objective measure.

Writing assessment program developers would do well tokeep these differences clearly in mind when planning an as-sessment program.

A Status Report on Writing Assessment ProgramsIn recent years, two national surveys of large -scale writing

assessment programs have been conducted. The first wasconducted by Frederick (1979) under the auspices of the Wis-consin Pupil Assessment Program of the Wisconsin Depart-ment of Public Instruction, and the second was conducted in1981 by McCready and Melton of Louisiana TechnologicalUniversity with support from the National Institute of Educa-tion. Each survey, at the time it was conducted, provided veryuseful insights into the status of large-scale assessment, andtaken together, the two surveys provide valuable perspec-tives regarding trends in writing assessment.

In 1979,18 states were conducting writing assessment pro-grams These assessments spanned Ole full range of gradelevels. The typical assessment at that time covered aboutthree grade levels, relied solely on a writing sample, or on a

nting sample in combination with an objective test to judgeproficiency, and involved holistic and/or primary trait scor-ing of the writing sample.

The status of writing assessment in 1981 is summarized inTable 1. Note that 24 states arc currently conducting writingassessments relying predominantly on writing samples

14

Table 1Evolution of Writing Assessments

1979 1981 1981

State State CityAssessments Assessments Assessments

Conducting Assessments 18 24 20

Testing in GradesK 1 11 1 1 72 1 1 83 1 7 104 7 7 95 3 5 106 1 7 117 2 5 98 10 10 129 4 11 15

10 2 6 1111 13 11 1312 5 5 9

Mean Grades Tested PerState 2.7 3.2 6.3

Testing StrategyGbjective only 1 6% 1 4% 3 5%Writing Sample only 7 39% 12 50% 9 45%Combination 10 55% 11 46% 8 40%

Scoring MethodHolistic 6 33% 15 65% 8 50%Analytical 1 6% 1 5% 5 31%Primary Trait 6 33% 4 17% 0 0%Combination 5 28% 3 13% 3 19%

scored holistically. Brief profiles are presented for state%Aeand large-city school district writing assessment programs.More detailed profiles ale presented in the Appendix. Theprofiles are summarized in various ways in Table 2, whichcompares 1979 and 11;81 assessments.

Several dimensions of this comparison are of interest.First, note that six more states have added assessment pro-grams over the past two years. Note also that while statewideassessments in both 1979 and 1981 tended to begin testing ingrades three or four, city schools tend to conduct a good deal

7

15

Table 2Overview of Large-scale Assessment Programs

STATEGrade(s)Tested

Alabama 3, 6, 9

Delaware 1-0, 11

California 3, 6, 12

Florida 5, 8, 11

HawPii 4, 8,11

Idaho 9

Louisiana 3, 7, 10

Maine 4, 8, 11

Maryland 9-12

Massachusetts 7,8, 9, 12

Michigan 4, 7. 10

Minnesota 4, 8, 11

Nevada 3.6, 9-12

NewHampshire

5, 9, 12

New Jersey 9

New Mexico 10

N'rthCarolina

11

Ohio S, 12

Writing SampleType of Scoring

Test Procedure

Writing Sample Holistic

Objective Test Primary TraitWriting Sample

Objective Test HolisticWriting Sample

Objective Test Analytic"!Writing Sample



Objective Test Primary TraitWriting Sample

. Writing Sample Holistic


Writing Sample HolisticAnalytical

Writing Sample Primary Trait

Writing Sample Primary Trait

Objective Test Holistic(3, 6)Writing Sample




Writing Sample HolisticAnalytical


816

STATEGradeis)Tested

Type ofTest

Writing SampleScoring

Procedure

Oregon 4, 7, 11 Objective TestWriting Sample

Holistic

Pennsylvania 5, 8, 11 Objective Test

Rhode Island 4, 6, 8, 10 Objective TestWriting Sample

Holistic

SouthCarolina

6, 8, 11 Writing Sample HolisticAnalytical

Texas 3,5,9 Objective TestWriting Sample

Holistic

Wyoming 6, 9 Writing Sample Holistic

CITY

Little Rock,AR

141 Objective Test

Phoenix, AZ 9-12 Objective TestWriting Sample

Analytical

Monterey, CA 1-12 Writing Sample Holistic

Tallahassee,FL

1-8 Objective TestWriting Sample

Analytical

Atlanta, GA 142 Objective Test

Des Moines,IA

9 Writing Sample HolisticAnalytical

Chicago, IL 9-12 Writing Sample Analytical

Boston, MA 2, 5, 8 Writing Sample Holistic

Wichita, KS K-12 Writing Sample Holistic

Baltimore,MD

1-9 Objective TestWriting Sample

Analytical

Detroit, MI 10-12 Objective TestWriting Sample

Holistic

Raleigh, NC 1-12 Objective TestWriting Sample

Teacher Option

17 9

STATEGrade(s)

TestedType of

Test

Writing SampleScoring

Procedure

Albuquerque,NM

4, 6, 9-12 Objective Test(4, 6, 9)Writing Sample

Holistic

Santa Fe, NM 7-12 Writing Sample HolisticAnalytical

New York, NY 8, 11 Writing Sample Holistic

Portland, OR 3-9 Objective Test

Austin, TX 3, 9 Objective TestWriting Sample

Holistic

Madison, WI 5, 8, 11 Objective TestWriting Sample

HolisticPrimary Trait

Seattle, WA 3, 6, 9-11 Writing Sample Analytical

Laramie, WY 6, 9 Writing Sample Holistic

of writing assessment as far down as grades one and two.Most assessment, however is conducted in junior and seniorhigh school. The average number of grade levels tested is onthe increase in statewide assessments. But neither the 1979nor the 1981 averages on this variable compare to the cityschools' average of 6.3 grade levels in each assessment.

With regard to writing assessment method, there is a rela-tively constant pattern over time and across settings. Large-scale assessments tend to rely on writing samples alone orwriting samples in combination with objective tests, Sole reli-ance on objective tests is rare.

Procedures for rating writing perfornia ice have changedmarkedly over the past two years. In 1979, assessors tendedto rely about equally on holistic and primary trait scoring.Little attention was paid to the analytical approach. In 1981,however, in both state and city programs, there has been asignificant decline in the use of primary trait scoring, and amarked increase in the use of both holistic and analyticalmethods.

In sum, significantly more writing assessment is being con-ducted in 1981 than in 1979, and that assessment relies heav-il, on holistically scored samples of student writing as the

It) 18

basis for judging proficiency. For more detail on assessmentprograms ,:onsult the Appendix.

StillNo "Best" AnswerThese are but a few of the many instances in which writing

assessment is being successfully conducted on national, state,and local levels. The remainder of this monograph describes(1) some of the procedures used in various assessment con-texts and (2) key measurement issues in the testing of writingskill.

The assessment of writing skill is a very complex task, be-cause of the broad range of potentially relevant writing com-petencies and the difficulties in setting standards of accept-able performance. There is not now, nor will there ever be, asingle best way to assess writing skill. Each indiv;dual educa-tional assessment and writing circumstancepresents uniqueproblems to the developer und user of writing tests. Therefore,great care must be taken in selecting the approach and themethods to be used in each writing assessment. Methods usedin one context to measure one set of relevant writing skillsshould not be generalized to other writing contexts withoutvery careful consideration of writing circumstances.

11

19

Chapter II: An Overview of DirectWriting Assessment Procedures

The development and implementation of a high qualitywriting assessment program can be complex and expensive.This chapter outlines procedures for managing that com-plexity and ensuring sound assessment.

Ensuring High Quality AssessmentTwo key considerations in determining the quality of writ-

ing assessment arc the reliability and validity of the scoresgenerated by the assessment. The exercise development andscoring procedures outlined in the following two sections ofthis chapter have been developed and refined specifically toensure' score reliability and validity. However, beforedescribing those procedures, it may be useful to explain relia-bility and validity as they relate to direct writing assessment.

Reliability. To be useful for educational decisions, testsmust yield scores that are consistent or reliable. When scoresare unreliable, the assessment results can lead to erroneousconclusions or decisions. In writing assessment, score incon-sistency can take any of several forms.

For example, suppose a writing assessment were adminis-tered to the same students tw ice, the second administrationfollowing a two- to three-week interval. And suppose thateven though no writing instruction took place, the scores ob-tained the second time were totally different from thoseachieved the first time for nearly every examinee. The exam-

13

20

mer would not know which score (if either) to depend on asthe true reflection of the students' proficiency. Or supposetwo w riting exercises were developed to measure exactly thesame skills and vet when both were administered to a stu-dent, the exercises resulted in totally different c,timates ofproficiency. Again, the examiner would not know whichscore was the better indicator of proficiency. Or, from a thirdperspective, suppose two judges read and evaluated a writingsample from the same student and drew totally different con-clusions regarding the student's proficiency. In this case, aswith the others, the examiner would not know which judg-ment to rely on. These three examples show ho unreliabilitycan manifest itself in the assessment of riting skill with writ-ing samples.

When scores arc unstable over time. differ across ()sten-sit& equix alent writing exercises and or differ across inde-pendent c% aluations of proficiency, there is reason to ques-tion the usefulness of the assessment procedures. However,w hen the procedures employed yield scores that are stable0%er time, across exercises and across independent evalua-tors. those scores can be confidently used for educational de-cisions. The test de% eloper is responsible for (1) employingassessment development procedures that maximize score re-liability. and (2) presenting systematic evidence of score reli-ability for re% Wyk. by users

Three factors are important in developing reliable tests.First, the writing skills to be measured must Ix clearly andconcisely defined by w riting experts. Only then is it possibleto ( I) demonstrate to users, exercise developers, and othersprecisely w hat skills are to be assessed; (2) judge exerciseappropriateness; and (3) inform judges about the criteria foracceptable performance.

Second. there must be a clear and unambiguous link be-tween the skills to be tested and the exercises developed. Thisinterrelationship ensures that exercises give the competentw titer the stimulus and opportunity to demonstrate w hateverskink) the user wants to measure.

And third, judges must be carefully trained to conduct thec% Amnion according to prespecificd criteria and agreedupon standards If these three guidelines arc followed.t fiances are that stores will be consistent mer time, acrossexercises. and across liners. If scores are found to be incon-

11

sisterit, assessment procedures should be re-examined inlight of these guidelines and revised accordingly.

Validity. Even if a developer of a direct writing assessmentis successful in achieving store stability through careful skillidentification, exercise development and evaluator training,the writing assessment developmental task is only partlycompleted. Attention must also be given to the validity of theassessment scores. The validity of a score depends on (1) thetest used to generate that score, and (2) the intended purposefor that score. Intended purpose can be identified in a varietyof ways, each of which can be considered a dimension ofvalidity. Cronbach (1971) has identified a number of suchdimensions that can be applied to the direct assessment ofwriting proficiency. For example, a test may be designed tomeasure a specific set of writing skills. If review of that test byqualified experts reveals that the exercises do indeed coverthose skill, then the test is said to cover the intended contentvalidly. It has achieved its content coverage purpose.

From a different but related perspective, a test that plays asignificant role in educational decision making (e.g., providesa basis for placement or selection) should inspire confidenceamong users. The exercises must appear to assess truly im-prtant skills. If this face validity is missing, the test will notbe usedregardless of the actual appropriateness of the ex-ercises. It is important that the exercises seem appropriateeven to the least sophisticated of the intended users.

There are other ways of revealing whether a test is achiev-ing its intended purpose. For example, a test of writingproficiency is only one of many potential indicators of writingskill. If a test is valid, then scores should be consistent with (orreflect the same level of proficiency as) other indicators ofwriting skill: for example, performance on job-related orreal-world writing tasks, amount of formal training in writ-ing, grades received in writing courses, and/or scoresach;,!ved in other objective or writing sample-based tests ofwriting skill. To the extent that the writing assessment devel-oper is able to show that performance on a newly developedwriting assessment is consistent with performe.nce on otherwriting-related tasks, the assessment has achieved its goal ofreflecting writing proficiency.

Test purpose largely determines the requirements for doc-umenting validity. For example, a direct writing assessment

15

22

may be very general, or it may be narrowly focused to beprecise and diagnostic. Suppose, for instance, that onewished to measure students' letter VS rising skills. A generalexercise might present the student with these directi:ms:

Pretend that you are applying for a job as a salespersonwith Acme. Inc. Write a letter to Acme explaining yourinterest and qualifications.

Because these instructions are very broad, responses canonly be judged on general merit. Raters will likely considersuch factors as word choi,:e, sentence structure. organization,mechanicsin short, the kinds of things one would considerin judging any piece of writing. And the A.esult NN ill be a gener-al profile of overall student writing performance. But sup-pose one wished to measure students' performance on ex-plicit letter writing skills, in order to diagnose individualstudents' strengths and weaknesses. This would call for sonicmodification in the item so that it might read as follows:

Pretend that you arc applying for a job as a salespersonith Acme. Inc Write a business letter addressed to Ms.

Jones, Sales Manager of Acme, 2525 Main, Huntsville.New York 20201. Explain your interest andqualifications. Attempt to convince Acme that You're thebest person for the job. Use proper business letter form.

These specific directions w ill allow responses to be judgedaccording to explicit criteria: students' ability to be convinc-ing and use proper business letter format. Responses to thefirst item could not be scored in this manner because the in-tended audience, purpose and expected letter format werenot specified in the instructions. In summary, if diagnosticinformation is desired, items must be carefully structured toelicit the appropriate type of response. Evidence of success inachieving the desired level of pcecision should be included invalidation research.

The purpose for testing may also be considered in terms ofthe specific educational decision in question. That is, a testmay be intended to rank order examinees in terms ofproficiency for selecting the most able for further training orthe least able for remediation. Or the assessment may beintended to provide information for masteryinonmastery de-cisions with regard to specific writing objectives. Becausethese arc different purposes, the assessment strategies used

16

23

to achieve them will differ. It is up to the developer to deter-mine the usefulness and appropriateness of assessment pro-cedures for meeting each specific decision-oriented purpose.

The essential point is that validity is a reflection of successin achieving the testing purpose. As with reliability, the testdeveloper has two primary responsibilities: to maximize va-lidity through careful test development and to report evi-dence of validity for users. Strategies for maximizing validityare similar to those for maximizing reliability. The writingskills to be assessed should be clearly and unambiguouslydefined. Both the skills and exercises developed to reflectthose skills should carefully be reviewed by subject experts toensure appropriateness. And once the test is administeredand scored, scores should be related to other relevant writingproficiency indicators to be sure the assessment is focused onthe desired dimensions of writing skill.

Developing ExercisesIn the discussion that follows, a writing exercise is consid-

ered to comprise all stimulus materials and instructions usedto define the writing task. Developing exercises for direct as-sessment of writing involves five carefully conducted steps.The first two steps are crucial for any writing assessment: (1)assessment planning and (2) exercise development. The re-maining three steps, while very important, are not alwaysimplemented. depending on the resources available and theseriousness of the decisions to be made. These are (3) testspecification and exercise review, (4) exercise pretesting and(5) final revision. Each of these five developmental steps isdiscussed in detail in the following paragraphs.

Assessment planning. The ultimate quality of any assess-ment is influenced more by the thoriughness and detail of itsoriginal blueprint than by any other factor. Several very im-portant test design questions must be thoroughly considered.If each is not individually considered, the chances of creatinga valid and reliable assessmentespecially a writing assess-mentare greatly reduced.

The first planning question concerns purpose. The solereason for conducting any educational assessment is to pro-vide information to facilitate some educational decision.Therefore, the primary step in writing assessment planningis to state precisely the specific educational decision to be

17

24

influenced by the resulting scores. Potential decisions include( 1 ) diagnosing individual student proficienc.' in specific writ-ing skill areas: (2) rank ordering examinee with regard togeneral writing proficiency for selection or placement: and(3) assessing specific or general w riting proficiency to evalu-ate the impact of an instructional program. (Additional deci-sions will be presented later.) Specific assessment strategiesvary according to purpose. Therefore, the decision(s) to befacilitated must be clearly specified at the outset.

Second, test developers must determine the specific formof w citing to be produced (e.g., essay, business letter, fiction),the audience to be addressed. and the purpose to be served inaddressing that audience. Any given student's level ofproficiency w ill vary as a function of writing form.

A third planning step calls for identifying the traits to bejudged in evaluating IA riting skill and criteria or standards ofacceptable performance for each trait selected. For example.organisation, style, tone and sense of audience are typicaltraits: that is, elements of writing skill. In order to judge per-formance, however, evaluators need more than a list oftraits. They need guidelines or criteria for determining good,poor or mediocre organisation. style, and so on. The com-plexity of traits and criteria is a function of assessment pur-pose. A broad assessment of overall writing skill allows someflexibility in the specification of criteria. For a diagnostic as-sessment, on the other hand, both traits and scoring criteriamust he delineated with great precision.

In summary. the writing assessment blueprint must in-clude (1) the educational decision(s) to be facilitated, (2) thew thing context (purpose, audience and type of writing to berequired), and (3) the specific traits or skills to be judgedalong with criteria for evaluating performance.

Exercise development. Once planning is completed, thedeclopmental goal becomes quite apparent. the design ex-ercises that provide the competent student with the necesst.rystimulus and w riting conditions to demonstrate his ,her levelof competency. In othei words. the writing tasks must informstudents of the purpose for the w ruing. the audience to beaddressed and the type of vt riting expected (necessary condi-tions). vt fill(' at the same time allowing students the latitude(e.g , sufficient exercises and time) to demonstrate their ca-pabilities. it should be apparent that unless careful planning

18

25

4

has preceded this step, appropriate exercise developmentwill be difficult at best.

Here are some specific guidelines to b- observed in con-structing writing exercises: First, the exercise developershould recognize the impossibility of covering all possible in-,stances of relevant writing. A realistic objective is to constructand include in the assessment an appropriate sample of rele-vant exercises. Based on student performance on that sam-ple, one can generalize about expected performance in paral-lel contexts. To insure the appropriateness of thesegeneralizations, however, samples must be carefully select-ed. For example, if one wishes to know whether students canwrite expository prose for an academic audience, one exer-cise is probably not enough; two or three similar exercisesmay be necessary to ensure that the sample is sufficientlyrepresentative. At the same time, ability to construct otherforms for other audiencese.g.. an entertaining piece offiction for young children=is irrelevant to the testing pur-pose at hand.

To use another example. suppose the purpose of an assess-ment is to determine mastery of a single clearly focused writ-ing objective: ability to present map directions efcectively inwritten form. Enough examples of student performanceshould be gathered to ensure that addition of another exer-cise would not significantly alter any conclusions about stu-dent performance. In other words, exerciszs must be clearlyfocused and sufficient in number.

The reader may recognize that this issue of skill sampling isrelated to both reliability and validity, as described earlier.For example. it is important to provide enough samples ofstudent writing to allow for stable scores (reliability), and tofairly and adequately sample the skill domain the test is in-tended to cover (validity).

Certainly the key question in all writing assessment is:How much writing is enough? There is no hard and fast an-swer. The number of exercises required and the length ofthose exercises are functions of the range of skills to be evalu-ated and the level of precision at which those skills aredefined. Broader assessments covering many skills generallyrequire more samples than precisely focused, narrow assess-ments. Recent research on this topic (Steele. 1979 and Bre-land, 1977) offers some guidance. The Steele research in-volved a broad assessment of end-of-college writing

26 19

proficiency via three 20- to 30-minute writing exercises.Analysis of score consistency revealed that the use of onlyone or two exercises yielded unreliable scores. However, theuse of all three exercises raised score consistency to an ac-ceptable level. Further, the study revealed that the additionof more exercises beyond the original three w ould notsignificantly increase reliability. These results were sup-vorted s,:v Breland's research which revealed that, in a simi-lar college-level assessment, a single 20-minute exercise warincapable of yielding consistent scores.

Braddock, Lloyd-Jones and Schorr (1963) Gi.fer guidancefrom a different perspective as to the amount of writingneeded to judge proficiency:

Even if the investigator is primarily interested in nothingbut grammar and mechanics. he should afford time forthe w riters to plan their central ideas, organization, andsupporting details, otherwise their sentence structureand mechanics will be produced under artificial circum-stances Furthermore. the writers ordinarily shouldhave time to edit and proofread their work after theyhave (lime to the end of tin 'r papers. . . Investigatorsshould consider permitting primary grade children total,e as much as 20 to 30 minutes, intermediategraders asmuch as 35 to 50 minutes, junior high school students 50to 75 minutes, high school students 70 to 90 minutes, andcollege students two 'tours (to demonstrate proficiency).(Emphasis added.(

Exercises should frame a clear and concise writing task sothat students fully understand what is requiredwhether ornot they can fulfill the requirements. Time pressure is unde-sirable. it is an artificial imposition that may not replicate thecircumstances in which real life writing occurs. Items shouldoffer the %,__,er a realistic, sensible challenge so as to main-ta;ri interest. Varied stimulus materials (written, auditory, orvisual) should Le used. Most important, examinees must begiven time to think, organize, wr'te, reread and revise.

Some writing assessments have attached great importanceto revision. As Rivas (1977) notes:

Rew riting skills are often considered to be the essence ofgood w riting All of us can express ourselves in someform. however ambiguous or inappropriate, but a goodIA rite knows how to revise such preliminary statementsso that they become less ambiguous and more appropri-ate.

20

27

Part of NAEP's 1974 w riting assessment called for w rilingand rew riting the same copy in an attempt to get at re \ ision(Rivas, 1977). Students were asked to w rite a class reportabout the moon, given certain facts. They w ere given 15 min-utes to w rite the first draft, using a pencil. Upon finishing,they were given 13 minutes to revise the first draft, using ablue pen so that any changes would stand out clearly. Theywere told to make any changes they w fished, including cross-ing out words or rewriting if necessary: rewriting was notrequired, how ever. Papers were scored for overall organiza-tion (based on the quality of the revision), and were`catego-rized to indicate the kinds of revisions attempted: cosmetic(improved legibility), mechanical, grammatical, transitional,informational, holistic (complete rewriting), and so on.Though some educators tight feel the test was not a truemeasure of revision skills (mans students, for reasons un-known, attempted no revision), the NAEP moon test repre-sents at least a step toward development of a proper revisiontest.

Clearly, attention must be given to editing and revision aspart of any w riting assessment, whether by providingsufficient time and opportunity for the examinees to revise ontheir own, or by providing specific instructions to revise, asNEWT did. If extensive revision (beiond proofreading forspelling and other mechanical errors) is desired, it Al benecessary to construct ti,' assessment to allow students timefor proper reflectionjust as in a real-life writing situotion.It will not be sufficient merely to give students an addit on,ilfive or ten minutes at the end of a writing exercise to "fixthings up." A better approach might be to allow studentstime to write one day, time to revise on a subsequent day. Thiskind of provision may increase administration time andcosts. However, it will also provide a more relevant (i.e.. trueto real life) test of revision skills than one-session i ,essment.

Review of specifications and exercises. Whenever possi-ble, the writing and assessment personnel responsible forassessment specifications and writing exercises shouldpresent their work to an independent group of writing andmeasurement specialists for review and formative evalua-tion. This review should cover-

1. The purpose for the assessment (decision to be made).

21

28

2. The definition of the assessment context (form of writ-ing, audience and reason for writing).

3. The criteria (skills to be assessed) and standards of ac-ceptable performance.

4. Relevance of exercises in terms of skills to be assessed.

5. Representativeness of exercises in terms of the domainof possible exercises.

6. Sufficiency of the exercises in providing students withthe opportunity. in terms of time and tasks, to demon-strate proficiency.

7. Clarity and conciseness of prescribed writing tasks.

8. Level of interest and challenge conveyed in stimulus ma-terials and writing instruction.

9. Adequacy of instructions and opportunity for revision,if that is a desired part of the assessment.

As the importance of an educational decision and!or as thenumber of students to be included in the writing assessmentincreases. the importance of independent review increasesalso. Ti"-us, n view is less critical with small - scale. local orclassroom assessments than with large-scale assessments onwhich selection decisions are often based.

Exercise pretesting. Whenever possible, exercises shouldbe administered to a sample of students prior to actual full-scale administration so that potential problems can beidentified and corrected. Pretesting procedures shouldclosely approximate actual administration in terms of type(though not number) of pretest students, conditions (e.g., fa-cilities, time limits, methods for providing directions) andscoring procedures. Developers should then independentlyevaluate results, attending to (1) the level of proficiency dem-onstrated (and whether that level seems to fluctuate from ex-ercise to exercise), (2) the nature of the responses produced(in terms of qu, lity, appropriateness, length and enthu-siasm) (3) the consistency of ratings across independentevaluations, and (4) the apparent clarity of instructions tostudents. Exercises that appear to yield inconsistent or re-peatedly low quality results can be identified and the reasonsfor apparent problems discuss 2d. Often. exercises can be ad-

22

29

justed. As with independent exercise review, the importanceof pretesting increases with the scope and importance of theassessment.

Final exercise revision. The final step in exercise develop-ment is to revise exercises on the basis of the review andpretest results. As final revisions are made, developersshould continue to ensure reliability and validity of scoresthrough careful use of test specifications, exercise develop-ment and preparation for scoring.

Procedures for Scoring Writing SamplesMany forms of objective tests can be machine scored. Writ-

ing tests that rely on w riting samples, however, require indi-vidual hand scoring by 4-Allied persons trained to applyagreed upon criteria and performance standards. Severaldifferent methods have been devised for scoring writingsamples depending on the assessment purpose. The most ap-propriate method in any given situation depends upon whatinformation one wishes to gain through scoring, how thatinformation, will be used, and what res rces are available.Some scoring methods are more complicatedand there-fore more costlythan others. The purpose of this section isto present a comparative overview of the general advantagesand disadvantages inherent in each of five approaches: ho-

,ic scoring, analytical scoring. primary trait scoring, scor-ing for mechanics and grammar, and T-unit analysis.

Holistic scoring. In holistic scoring, raters review a paperfor an overall or "w hole" impres1;ion. Specific factors such asgrammar, usage, style, tone and vocabulary undoubtedly af-fect the rater's response. but none of these considerationsis directly addressed As w ith all rating methods, raters mustbe carefully trained to conduct the evaluation The purpose oftraining is to minimize (at least temporarily) the effects ofindividual biases by helping raters internalize an agreedupon set of scoring standards. It is generally recommendedthat raters be experienced in language arts, familiar withpertinent terninology and practiced in rating student papersat the level for ,vhich they w ill bt. scoring. Consistencybothamong raters and among scores assigned by a single rateris very important in holistic scoring. Initial training takesabout half a day, but it is also necessary to build in time for

23

"refresher" sessions thrwighout the course of any scoring ac-tivity.

Papers are rated on a numerical scale. NAEP has usedboth 4-point and 8-point scales. Four-point scales are mostcommon. An even-numbered scale is recommended becauseit eliminates the convenience of a mid-point "dumpingground" for borderline papers.

Prior to actual scoring, the trainer and the most qualifiedor-experienced raters review a subset of the papers to bescored in order to identify "range finders." These arc papersthat are representative of all the papers at a given scoringlevel. With a four-point scale, for example, there would berange finders for the 4,3,2 and 1 levels. Range finder papersmust be so typical of papers at a given level that virtually allreaders agree on the assigned score. This is vital becauserange finders are used in training, and later used as modelsto assist raters during scoring. Trainers and their assistantsmay have to read dozens of papers in order to find the "typi-cal" range finder papers %sith which everyone is satisfied. Fortraining purposes, it is advisable to have at least two (prefer-ably more) range finders at each level.

Trainers do not work from any predetermined set of crite-ria in identifying range finders. They may, of course, discusstheir findings and observations during the process. But it isimportant to realize that in holistic scoring, there is no pre-conceived notion of the "ideal" paper. A paper assigned ascore of 4 w ill simply be a relatively high quality paper withina given group. it may or may not be an e.wellent paper in itsown right. As Brown (1977) notes, "It is possible that all ofthe papers at the top of the score are horribly written. Theymay be better than the rest, but ,till may be unacceptable tomost teachers of composition." If one has in mind somespecific criterion of performance that students must meet,holistic scoring will not be appropriate. Scoring levels are. setfrom within, irrespective of external standards.

Despite personal preferences, the holistic approach quick-ly produces marked consistency among ratersin virtuallyany group. This may be partly the result of peer pressure. Butmore likely it suggests that language arts people can agreethough the bases for their conclusions may differon whatconstitutes a relatively g,od and a relatively poor paper. In-terrater reliability (that is, agreement between any two ra-ters) can be expected to run from about .60 to .80 (Diederich,

24 31

1974). It may be higher in a few cases, depending upon thebackground of the raters and the amount of training timeallowed (so that raters can internalize the system),

All papers should be read by at least two raters to minimizethe chance of error resulting from rater fatigue, prejudice orother extraneous factors. ACT has achieved an interrater re-liability of .75 using two raters and three writing samples(ACT, 197%. Increasing the number of -aters beyond twodoes not seem to enhance score reliability (Steele, 1979).

Scores may be added or averaged across raters to deter-mine a final score. Disagreements of more than one ratingpoint should be resolves' by a third reader or through discus-sion by the disagreeing raters. Such disagreements can typi-cally be expected to occur in fewer than 5 percent of all casesif careful assessment planning and rater training is con-ducted.

Holistic scoring is rapid and efficient. Depending on thelength of student responses, experienced raters can usuallygo through 30 to 40 papers per hour (though inexperiencedraters cannot be expected to match this rate). Six hours ofscoring per day is considered about maximum to maintainhigh reliability. Scoring is intensive work; short hours withfrequent break periods yield the best results.

Because scoring levels are never defined, holistic scoringdoes not permit the reporting of specifics on student per-formance. After reading hundreds of papers, however, ra-ters typically have a suprerr.ely clear notion of what factorsinfluenced them to assign particular scores. F6r reportingpurposes they may translate those observations into leveldefinitions. Suppose, for example, that students were askedto write a job application letter. One might then say that a"typical" 4 paper used proper business letter format, usedvocabulary and tone appropriate to the occasion, describedthe student's qualifications in a %Nay that reflected a clear un-derstanding of job requirements (as presented in the item),and reflected consistently good sentence structure, correctmechanics, and so on. Such a definition would not necessarilyapply in total to every 4 paper, but would certainly capturethe essence of papers at that level and help make resultsmeaningful to parents and other audiences. Presentation ofsuch definitions in conjunction with sample student paperscan be an extremely effective reporting technique.

25

32

Analytical scoring. Analytical scoring involves isolatingone or moire characteristics of w riting and scoring them indi-vidually. Analytical scoring is most appropriate if one wantsto measure (and report) students' ability to deal with one ormore specific conventions of writing: punctuation, organiza-tion, syntax, usage, creativity, sense of audience, and so on.Traits must be explicit and well defined so that all raters un-derstand and agree upon the basis for making judgments. Inaddition, it is necessary to delineate in advance specific andcomplete criteria for judging each trait. In analytical scoring,rater:, rely on written guidelinesnot range findersto as-sist them in assigning scores. Ideally, raters should have achance to participate in selecting traits and establishing crite-ria. This promotes understanding of and agreement with cri-teria, and ultimately enhances interrater reliability. Exceptfor the setting of criteria, training and administration proce-dures are similar to those for holistic scoring.

Analytical scoring provides data on specific aspects of stu-dent riting performance. But does it really reveal hether,in general, students w rite yell? The answer depends on (1)

hether enough traits are analyied to provide a comprehen-sive picture, and (2) hether those traits analyzed aresignificantthat is, cc hether they actually contribute to good

riting. In an effort to identify those characteristics thatseem most to influence a reader's judgment about the qualityof a piece of writing, Diederich (1974) pc-formed a contentanalysis on a sai.Tle of student essays scored holistically.Marginal comments w ere invited (as would not be the case ina traditional holistic session), and later tallied to isolate thosefactors that seemed to iniluence experienced raters' scoresmost. Here, in order of significance, arc the factors Diederichisolated through that study:

1. Ideas

2. Mechanics (including usage, punctuation and spelling)

3. Organization

4. Wording

5. Fla%or (or style)

Of course, individual examiners may identify other traitsthey wish to score. However, this list of traits permits a rea-sonably comprehensive analysis of NA riting.

26 33

Factor-by-factor analysis of writing elements is more timeconsuming than holistic scoring. Depending on how manyfactors one looks at, it requires two to three times as long (ormore) to rate a paper analytically as it does holistically.

Analytical rating has been criticized becausethere is someindication it produces a "halo" effect; that is, students whoare rated high on one trait will tend to be rated high on alltraits. Page (1968) explains,

A constant danger in multi-trait ratings, is that they mayreflect little more than some general halo effect, and thatthe presumed differential traits will really not be mean-ingful.... We find (in our research) a very large halo, ortendency for ratings to agree w ith each other.

Despite these disadvantages, however, analytical scoringhas one great advantage: it provides potential for trait-by-trait analysis of students' writing proficiency.

Primary trait scoring. Primary trait scoring is similar toanalytical scoring in that it focuses on a specific characteristic(or characteristics) of a given piece of writing. However,while analytical scoring attempts to isolate those characteris-tics important to any piece of writing in any situation, pri-mary trait analysis is rhetorically and situationally specific.The most importantor primary trait(s) in a letter to theeditor will not likely be the same as that (those) in a set ofdirections f, assembling a bicycle.

The primary trait system is based on the premise that allwriting is done in terms of an audience, and that successfulwriting will have the desired effect upon that audience. Forexample. a good mystery story will excite and entertain thereader; a good letter of application will get the interview. In ascoring situation, of course, papers must be judged on thelikelihood of their producing the desired response.

Because they are situation-specific, primary traits differfrom item to item, depending on the nature of the assign-ment. Suppose a student were asked to give directions fordriving from his,her home to school. The primary trait mightthen be sequential organization, for any clear, unambiguousset of directions would necessarily be well organized with de-tails presented in proper order. As Millis (1974) points out,"Successful papers will have that [primary] trait; unsuccess-ful papers will notregardless of how well written they may

3427

be in other respects."Raters determine that some traits arc essential to success

in a given assignment. However, additional traits that con-tribute but arc not necessarily essential to the success of apaper arc termed -secondary" traits and may also be in-cluded in the c% aluation. if they can be clearly defined andexemplified for raters Scores may be weighted to show therelative importance of various traits, if desired. then totalledto indicate the overall quality of the paper.

The first step in primary trait scoring is to determine v Inchtrait or traits w ill be scored. The second is to develop a scor-ing guide to aid raters in assigning scores. To illustrate. con-sider the following guide developed by NAEP for scoring"letters to the principal on solving a problem in school." Itwas determined that a good letter would identify the prob-lem, present a solution, and explain how that solution wouldimprove the school Here arc NAEP's criterion lex els:

1. Respondents do not identify a problem or give no evi-dence that the problem can be solved or is worth solving.

2. Respondents identify a problem and either 01 how tosolve it or tell how the school would be improved if itwere solved.

3 Respondents identify a problem. explain how to solvethe problem. and tell how the school v ould be improvedif the problem were solved.

4. Respondents include the elements of a "3" paper. In ad-dition. the elements arc expanded and presented in asystematic structure that reflects the steps necessary tosolve the problem (Mullis. 1974).

Range finder papers may be used addition to the scoringguide. This practice is not common, iowever, for many ratersfind it cumbersome to rely on two points of reference.

All raters should be familiar with tie rationale underlyingthe primary trait system, and with the !evel definitions to beused in scoring. Raters must accept the fact that they will belooking for specific, well-defined traits. and be cautious aboutallowing extraneous criteria to influence scoring, NAEP rec-ommends that raters prescore (for practices at least I 0 sam-ple papers at each level during training in older to becomecomfortable with applying the criteria (Mullis. 974).

28 35

As with analytical scoring, defining criterion levels is themost time consuming step. It may be necessary to "test" nu-merous definitions on sample papers in order to come upwith a set that works. Herein lies a strong argument for keep-ing the list of traits to be scored brief. On an average, counton a day of trial and error, discussion and debate for eachtrait to be defined. This may sound time consuming, but thequality and clarity of the final definitions, and the ease withwhich they can be applied, will readily justify the time spent.

Like analytical scoring, primary trait scoring can allow thereporting of student performance with respect to specificcharacteristics: e.g., organization, awareness of audience.For this reason, primary trait scoring is greatly favored overholistic scoring in contexts where more precise information isneeded. But this advantage should be carefully weighedagainst the time and effort required to set up a .-..;:kableprimary trait scoring system. Aside from adopting alreadywritten criteria (e.g., from NAEP), there are no known short-cuts.

Scoring language usage and mechanics. Of the types ofscoring mentioned thus far, the scoring of writing mechanicsis the most time consuming, and the most complex approachfor which to provide training. This realization often comes asa great surprise to inexperienced raters, who may look onmechanics as a rather cut and dries affairuntil faced withthe prospect of ,etting up a scoring system.

The fact is, the standards of appropriate usage are subjectto continual change through popular usage. So rapid has thatchange become now that even usage textbooks sometimesreflect different notions of what is appropriate. For the sakeof consistency in scoring mechanics, a is necessary that afairly comprehensive guide be developed. It is possible, ofcourse, to use a standard referencean English hand-book-- for this purpose. But raters must agree to abide by thedocument, and if there are too many areas of disagreement,it may be simpler to design their own. Whatever the decision,it is imperative that everyone agree to s"orc according to therules of the guide, regardless of personal preference. Other-wise, the inconsistency will render the scores useless.

Several other decisions must be made as weli:1. Whether to count errors of commission and errors of

omissibn equally.

29

2. Whether to require formal usage, or to base guide ruleson informal usage.

3. Whether to count errors involving concepts or rules withwhich students may not be familiar (e.g., seventh grad-ers may not have been taught proper use of colon andsemicolonshould this be considered?).

4. Whether to count every identifiable error or to focus onspecific for easier reporting of results.

In addition, raters must establish a workable rating scale. Ifthey choose to retain a 4-point scale, for example. it will benecessary to determine how many errors will be allowed in a4 paper. how many in a 3 and so on.

One additional step necessary in scoring writing me-chanics is obtaining an accurate word count for each paper.Errors can then be tabulated per 100 words. Analyzing er-rors in this way does not penalize those who write long re-sponses. or give unfair advantage to those who write verylittle.

Test administrators should be cautioned about scoring me-chanics as one trait within a primary trait system. As theforegoing discussion Aicates, it is far more time consumingto score than other tt , and demands a number of specialconsiderations. Thereiore. test administrators should weighcarefully the advantages and disadvantages of such a com-bined approach.

Educators considering using the direct assessment ap-proach to evaluate mechanics should remember that under-standing of such usage elements as punctuation, grammar,diction, and sentence structure can be very efficiently, validlyand reliably assessed using available indirect assessmentmeasures. For mechanics or usage assessment, very carefulconsideration should be given to the objecti,,e test because itforces examinees to demonstrate explicit ability to deal effec-tively with the precise elements being tested. If a writing sam-ple is used to assess these elements. examinees will typicallyavoid language constructions which they are unable to useeffectively. Further, inconsistencies in usage patterns willmake comparisons among examinees. on the basis of me-chanics, difficult if not impossible. Such comparisons aregenerally possible with objective usage tests. In addition, be-cause a writing sample taps but a small, arbitrary portion of

30

37

an examinee's proficiency in writing mechanics, results 'an-not appropriately be used in diagnosis, whereas objectivetest results may be quite suitable for this purpose.

T-unit analysis. The concept of T-unit analysis was intro-duced in the 60s, and has gained popularity ever since as ameans of measuring writing sophistication. A T-unit may bethought of as an independent clause plus whatever subordi-nate clauses or phrases accompany it. In simple terms, a T-unit is the smallest group of words in a piece of writing thatcould be punctuated as a sentence (T stands for "termina-ble"). Consider the following passage:

I yelled at my cat Manfred and he ran away, but he camehome when he got hungry,

This passage has only one terminal mark of punctuation aswritten, but actually contains three T-units:

I yelled at my cat Manfred

and he ran away,

but he came home when he got hungry.

Each of these T-units is an independent clause that could bepunctuated as a sentence. Note that T-unit analysis is inde-pendent of punctuation; a writer may or may not punctuateT-units as sentences.

Studies have shown that T-unit length tends to increasewith the age and skill of the writer* (Hunt, 1977). In addition,it has been demonstrated that with increased skill, writerscan incorporate a greater number of distinct concepts into asingle T-unit. Consider the following example, using six shortsentences, each of which consists of one T-unit, abstractedfrom a longer piece:

1. Aluminum is a metal.

2. It is abundant.

3. It has many uses.

4. It comes from bauxite.

*There are notable exceptions. therefore. this tenden, , cannot be applied as a gen-eral rule Highly expenenced. sophisticated wnters may consistently use short T-turts Conversely. the use of lengthy T-units does not of itself render one a skillfulw mei%

31

3 8

Table 3A Comparison of Scoring Methods for

Direct Writing Assessment

DESCRIPTOR HOLISTIC ANALYTICAL

GENERALCAPABILITIES

Comprehensive, p neral Thorough, trait by traitpictiv e o: student Perform- analysis of writing, providesancc, writing viewed as a comprehensive picture ofunit ed coherent whole performance if enough transApplicable to any writing arc analyzed, traits are thosetask important to any piece of

w riling in any situation (e g .organization. wording.mechanics)

RELIABILITY High reliability if standardsare carefully established andraters are carefully trained

High reliability if criteria andstandards are well defined,and careful training isconducted

PREPARATIONTIME

READERS

Up to one day per item toidentify range finder (model)papers, up to one-half day totram readers using 4-pointscale, full day to train with 8-point scale

One full day to identify traits.one day per trait to developscoring criteria (unless traitsand criteria are borrowedfrom another source), one totwo days to review results ofpilot test and refine traits orcriteria as necessary, one-halfday to train raters

Qualified language artspersonnel recommended,high reliability can beachieved with non-languagearts readers giver. a efficienttraining

Qualified language artspersonnel recommended

SCORING TIME One to two minutes per paper(experienced readers mayread faster)

One to two minutes per paperper trait

CLASSROOM May be adapted for use in

USE class

REPORTING Allows reporting on students'overall writing skill

May be adapted for use inclass

Allows reporting of studentperformance on wide rangeof generalizable traits (I e ,the qualities consideredimportant to all good writing)

GROUP!SAMPLE SIZE

Primarily usable with a largersample. with a small sample,resnonses may be difficult to

ale

Best with smaller samples.extensive scoring time maymake costs prohibitive svthlarger groups

These arc very general quidehnes Due to the nature of the scoring-cost/amount-of-information trade-off across scoring methods, readers am urged to seek the technical assistanceof a qualified writing assessment specialist if there is a question regarding tnc appropriate use ofavailable scoring resources

32

39

PRIMARY TRAIT WRITINGMECHANICS

T -pNITANALYSIS

Highly focused analysis ofsituation-specific primarytrait (and possibly secondarytraits), provides specificinformation on a narrowlydefined writing task g .ability to recount details inchronological order)

Can provide either a general -Provides a measure ofor a specific profile of theme syntactical soptustica-student's abdity_to use bonmed .....i_gucirprOperly.

High reliability if criteria andstandards are well defined.and careful training isconducted.

High reliability if givensufficient training time andauthoritative, complete,acceptable guidelines (e g anEnglish handbook).

High reliabilityprovided trained andexperienced raters areused

One full day to identify traits,one day per trait to developscoring criteria (unless traitsand criteria are borrowed'from another source); one totwo days to review results ofpilot test and refine traits orcriteria as necessary, one-halfday to train raters

One to two days to set up ascoring system (unlessborrowed from anothersource). MI111=1171(4 one dayto internalize the scoringsystem and practice scoring

Half day to full day,depending on raters'previous experience

Qualified language artspersonnel recommended,non-language arts staff maybe able t3 score some traits.

Qualified language artspersonnel recommended.

Raters must beexpen need languagearts personnel,preferably thosealready familiar withthe concept of 1-unitanalysis

One to two RIM oer paperper trait

Five minutes or more per Vanes greatly,paper. depending on number depending on raters'of criteria



May be adapted foruse in class

Allows reporting of studentperformance on one or moresituation-specific traitsimportant to a particular task

Allows reporting of group orindw.dual data on students'general strengths orweaknesses in mechanics

Allows group orindividual reportingon syntacticalsophistication

Generally more cost-effectivewith smaller samples,depending on the number oftraits to be scored (with onetrait, sample size is not an

(issue)

Best with smaller samples,extensive scoring time maymake costs prohibitive withlarger groups

Sect with smallerse t... 5. extensivevonog time maymake costs prombnivewith larger groups

40

33

5. bauxite is an ore.

6. Bauxite '..00ks like clay.

Here's how a fourth grader rewrote the passage:

Aluminum is 3 metal and it is abundant. It has manyuses and it comes from bauxite. Bauxite is an ore andlooks like clay. (6 sentences to 5 T-units)

The r&vision of a typical eighth grader:

Aluminum is an abundant metal, has many andcomes from bauxite. Bauxite is an ore that looks likeclay. (6 sentences into 2 T-units)

And finally, the revision of a skilled adult, a professionalwriter:

Aluminum. an abundant metal with many uses, comesfrom bauxite. a claylike ore 'G vtntences into 1 T-unit)

T-unit analysis and review of conversions (from simplesentences into T-units) provide a good measure of sentencematurity and of a student's ability to consolidate multiplethoughts.

Sophisticated, condensed writing has undeniable appeal.T-unit analysis used in conjunction with holistic scoring islikely to reveal that the highest scored papers (i.e., those thatappealed most to readers) were in fact those with the mostsophisticated use. T-units.

T-unit analysis is still in the experimental stages it is ';meconsuming and costly to conduct. Moreover, it can .done by highly trained language arts specialists. Furtlsearch and use may. however, reveal more widespread appli-cability than has so far been anticipated. Two interestingfootnotes: syntactical maturity is apparently reflected in oralspeech as well as in writing, and such maturity can beenhanced through a sentence combining curriculum (Hunt,1977).

A Comparison of Scoring MethodsTable 3 offers a comparative ov rview of the scoring pro-

cedures discussed in this section, focusing on several key de-scriptors.

34

41

Chapter III: Adapting WritingAssessment to Specific Purposes

Educational tests have only one function: to facilitate edu-cational decision making. A test should not be administered,therefore, until the decision or decisions that rest on the re-sults of that test have been clearly articulated. This applies toall tests, including writing tests.

In many z.ducational contexts, writing tests can be and arebeing used effectively. For example, tests can play a role ininstructional management decisions. Such decisions include( I) the diagnosis of individual learner strengths andweaknesses for instructional planning, (2) the placement ofstudents into the next most appropriate level of instruction,and (3) educational and vocational planning as part of stu-dent guidance and counseling.

Tests can also be administered at key points in an educa-tional program to check student development in order to (I )screen the admission to an advanced or remedial program,or (2) certify minimum proficiency (e.g., for high school grad-uation).

And finally, tests can be used for program evaluation pur-poses such as in ( I) large-scale survey assessment, (2) forma-tive program evaluation, and (3) summativc program evalua-tion:

In the discussion that follows, each of these eight contextsis described in terms of the decision to be made, the primarydecision makers, and the type of writing skill informationneeded to make the decision. Decision makers include stu-

35

42

dents, parents, teachers, administrators (including specificproject or program administrators, as well as building-, dis-trict- and state-level administrators), guidance counselors,and the public (including taxpayers and elected officials).

Using Tests to Manage InstructionDiagnosis. Teachers often use tests and other perform-

ance indicators to track each student's level of development,thereby determining where that student is in the instruc-tional sequence, and anticipating the next appropriate levelof instruction. Diagnostic data gathered via direct writing as-sessment can help individualize instruction by simplifyingstudent grouping or instructional scheduling decisions. Inaddition, diagnostic writing skill data gathered over timemay provide a basis for grading or communicating progressto parents.

Placement. Decision makers such as teachers and educa-tional administrators must place each student at the level ofinstruction best suited to his /her skills. Typically, they usesuch performance indicators as writing skill tests, previouscourses completed, and grades to rank order students alonga continuum of writing skill development, then place them inthe appropriate course.

Guidance and Counseling. In deciding their future educa-tional or vocational activities, students need to know howtheir writing skill compares to that of other students withwhom they could comkte. Performance indicators like writ-ing tests can help provide such information. Writing tests canindicate the probability that a given student will find successand satisfaction in a program or professional position forwhich writing skill is a prerequisite. More specifically, nor-mative test data can help students, their parents and theirgu:dance counselors answer students' typical questions:Should I pursue advanced training in a postsecondary educa-tional program in which writing is a key element? In whichschool or job am I most likely to be successful? Though testscores should never serve as the sole basis for answeringsuch questions, they can play a valuable role.

Using Tests to Select StudentsAdmission. It is not uncommon to have more candidates

than, program openings. When this happens, teachers, coun--A.:ors and administrators must select students for admis-

3 ti

43

sion. Performance indicators such as writing tests can beused to rank order examinees to facilitate selection. Selectiondecisions most often affect those at either end of the skillcontinuum. That is, more able students are selected for inclu-sion in advanced writing prow ams. while less able studentsare selected for remedial writing programs.

Certification. Tests tailored to a specified certification do-main are often used to verify and document a student's mas-tery of specific knowledge or skills. For example. teachersmight use writing tests to certify mastery of beginning writ-ing skills for purposes of grading or promotion. Or districtand state administrators might use minimum writing compe-tency tests as criteria for high school graduation. Both exam-ples show how certification may be accomplished throughtesting.

Using Tests to Evaluate ProgramsSurvey Assessment. Survey assessment refers to the collec-

tion of group achievement data to determine general educa-tional development (e.g., in r 'ing). Data may be gatheredby administering a writing test to a carefully selected randomsample of students in the target population. Survey assess-ment is often cyclical, thus allowing for the examination oftrends in writing skill development over time. Decisionmakers include (1) building-, district- or state-level adminis-trators who allocate resources for special instructional needspinpointed by the assessment. zit- (2) the public, which makesvalue judgments regarding perceived and reported levels ofstudent writing skill development.

Formative Evaluation. In the context of formative pro-gram evaluation, program administrators and teachers at-tempt to determine which components of instruction arefunctioning as intended and which need further refinement.They may test students on ..ach of the intermediate and finaloutcomes of a writing program, for example. Assessment forformative evaluation may also involve mu .tiple test adminis-trations to determine the effectiveness of ongoingmodifications in a writing program.

Summative Evaluation. Summative evaluation reveals aprogram's overall merit, suggesting whether that programshould be continued or terminated. Tests designed to assessstude performance on final learning outcomes are an im-portant part of such an evaluation. Teachers, program.

37

44

building or district administrators, and the public (includingthe board of education) may be involved in summative evalu-ation decisions. As with survey assessment and formativeevaluation. rrdtiple test administrations are common. Testsmay be given prier to as well as following instruction, withretention testing after a given time interval.

Selecting Examinees as a Function of PurposeIn the three program evaluation contexts just cited (survey

assessment. formative evaluation. and summative evalua-tion). testing costs can be significantly reduced through ran-dom sampling. If the student population is very large, thendata summarized across a carefully selected random subsetof students will reflect group performance every bit as accu-rately as if every student were testedoften at a fraction ofthe cost. It is not within the scope of this paper to present allthe important considerations in sampling. as each specificeducational sitmoion is un;que. The intent is to point out thepotential financial ad% antagc of sampling and to urge its con-sideration

It should be apparent that sampling is not feasible withinstructional management or student screening decisions be-cause in these contexts. individual student data are neces-sary.

Developing Exercises as a Function of PurposeGenerally. the process for de\ eloping writing assessment

exercises remains constant across all eight educational as-sessment contexts. Careful planning is essential in all cases,and attention must alw ays be given to designing exercisesthat give the examinee sufficient opportunity (in terms oftime. a)propriate stimulus and range of tasks) to demon-strate Further. in all cases. the type of audienceand purpose for communication should be made clear to thestudent. In addition, exercises should frame challengingtasks based on aried and directly reit.% ant stimulus mate-rials. And finally, ;n all cases. clear and concise instructionsarc essential.

A few factors val y according to context and the nature ofthe decisions to be made As a general rule. the specificity ofan exercise level of detail in instructions) should increasealong w ith the specificity of the skills to be assessed. In otherwords. exercises to be used in broad survey assessment need

:38

45

not be quite so focused as exercises to be used in, say, adiagnostic test.

The amount of w riting required might also vary, depend-ing on the decisions to be made. For example, it might bepossible to rank order students in terms of general writingproficiency (via holistic scoring) on the basis of three or fourgeneral, relatively short writing samples. However, it wouldprobably he very difficult to use those same three or fourshort writing samples to reliably and validly determinewhether a student had mastered 10 to 15 specific, indepen-dent writing_skills. Generally the more precise and numer-ous the criteria and standards of acceptable performance,the more writing necued to evaluate performance.

And finally, exercises developed for use in a large-scalestatewide assessment or where important selection decisionsare prnding must be (1) independently reviewed by writingand assessment experts and (2) pretested. Pretesting and re-view are less critical with writing assessment exercises usedin inst uctional classroom management.

Selecting Scoring Procedures as a Function of PurposeSelection of scoring procedures is, in effect, part of assess-

ment planning. since this decision is influenced by the pur-pose fir the assessment and criteria to be used in judgingwriting proficiency. Though it is possible to conceptualize in-stances within each of the eight educational assessment con-texts in which any given scoring approach could be em-ployed, the actual scoring approach most commonly used willvary by context.

To illustrate, diagnosis of individual student strengths andweaknesses demands the level of specificity providedthrough analytical. primary trait or mechanics scoring.Placement and guidance. on the other hand, may only re-quire holistic ratings because the objective of assessment issimply to rank order students on a continuum of writing skill.

Consider measurement of student status. While selectionmay require a holistic ranking of students, certification maybe done through holistic ratings or analytical or primary traitscoring, depending on the specificity of the minimum compe-tencies to be certified.

Holistic scoring procedures are well suited to the relativelybroad, unfocused nature of large-scale survey assessment.However. analytical scoring may serve as well if the desire for

39

46

ContextDiagnosis

Table 4Writing Assessment Procedures

as a Function of Assessment Context

Assessment ContextDecision

to be made

Determineand trackeducationaldevelopment

Assessment ProcedureDecision Examinees Exercisemakers assessed specificity

Teacher Individual SpecificStudent

Placement Match levelof studentdevelopmentto level ofinstruction

TeacherCounselor

Individual General

Guidance Rank orderfor educa-tionalplanningdecisions

Administrator Individual GeneralCounseiorTeacherParentStudent

Selection Rank orderexamineesfor selec-tion intoinstruction

Administrator Individual GeneralCounselorTeacher

Certification Determinemastery ofspecificcompetencies

TeacherStudent

Individual Specific

SurveyAssessment

FormativeEvaluation

Policydecision restatus ofstudenteducationaldevelopment

Determinecomponentsofinstructionalprogram inneed ofrevision

Administrators Sample GeneralPublic

ProgramDeveloperTeacher

Sample Depends onprogramobjectives

Summative Program Administrator SampleEvaluation continuation

40

Depends onprogramobjectives

47

Context

Assessment Procedure

Holistic Analytical Primary trait Mechanics T-unitDiagnosis X X X

Placement X X

Guidance x x

Selection x x

Certification x x x xri

SurveyAssessment

X X

X XFormativeEvaluation

X

Summative X XEvaluation

484I

individual data justifies the additional time required.Scoring procedures for formative evaluation depend on

the specificity of the enabling and terminal objectives thatguide instruction. If overall writing proficiency is the focus ofthe program. analytical scoring may be selected. However, ifinstruction focuses on situation-specific rhetorical skills. pri-mary trait scoring may be most appropriate. Similarly, em-phasis on mechanics indicates selection of a correspondingscoring approach. In most instances, formative evaluationdemands scoring procedures more specific than holistic.

With summative evaluation, holistic assessment may pro-vide sufficient data to judge program viability. However, ifstated program goals subdivide writing skill into componentparts, analytical scoring may be appropriate. Instructionalprograms in w Tieing seldom focus on a single rhetorical cir-cumstance Rather. they deal with writing of many types, formany purposes. Therefore. primary trait scoring will havelimited value in this context.

Ensuring Efficient, Effective, and High QualityAssessment

The keys to successful direct riting assessment are carefulplanning, thoughtful and creative exercise development. andconsistent application of performance criteria during scor-ing. If these factors are given meticulous attention, the as-sessment will :ield data that are ( I ) sufficiently precise tosupport necessary decisions. (2) reliable, (3) valid for the in-tended purpose, and (4) maximally cost-effective.

The preceding discussion is intended to acquaint the in-terested educator with available assessment strategies and tohighlight some of the Issues involved in selecting a scoringprocedure appropriate for a specific context. Table 4 pro-vides an overall summary of the key points made in that dis-cussion

12

The reader is encouraged to refer to :fe list offollowing this section and to the APP! ND1X,

which names con:act persons in many states u ho canoffer further information on writing assessment ap-proaches and contingencies. In addition. ( API wel-comes further ',mune.% regarding rig assessment

49

REFERENCES

American College Testing Program. Alternative strategies for theassessment of writing proficiency. Iowa City, IA: Author, 1979.

Braddock, R., Lloyd-Jones. R.. and Schoer. L. Research in writtencomposition. Urbana. IL: National Council of Teachers of Eng-lish, 1%3.

Breland. H.M.. Conlon, G.C., and Rogosa, D. A preliminary studyof the writing ability. New York. NY: College Entrance Examina-tion Board, 1966.

Cronbach, L.J. Test validity. In R.L. Thornlike. Educational Meas-urement. Washington. D.C.: American Council on Education.1971.

Diederich, P.B. Measuring growth in English. Urbana, IL: NationalCouncil of Teachers of English. 1974.

Fredrick, V. Writing assessment research report: A national sur-vey. Monograph published by the Wisconsin Department of Pub-lic Instruction, Madison, WI: 1979.

Godshalk. F.I.. Swineford, F.. and Coffman, W.E. The measure-ment of writing ability. New York. NY: College Entrance Exami-nation Board. 1966.

Hogan. T. P.. and Mishler, C. Relationships between essay tests andobjective tests of language skills for elementary school students.Journal of Educational Measurement, 1980. 17, 219-227.

Hunt, K.W. Early blooming and late blooming syntactic structures.In C. Cooper and L. Odell (Eds.). Evaluating writing. Urbana, IL:National Council of Teachers of English, 1977.

Huntley. R.M., Schmeiser. C., and Stiggins, R. The assessment ofrhetorical proficiency: The role of objective tests and writingsamples. Paper presented at the annual meeting of the NationalCouncil on Measurement in Education, 1979.

Mt on, V., and McCready, M. Survey of large-scale writing assess-ment programs. Ruston, LA: Louisiana Technological Universi-ty. 1981.

Moss. Pamela A., Cole, Nancy S., and lc ampalikit, Choosak. Acomparison of procedures to assess written language skills ingrades, 10, 7 and 4. Paper presented at the Annual Meeting ofthe American Educational Research Association, Los Angeles.CA: 1981.

Mullis. I. The primary trait system for scoring writing tasks. Den-ver, CO: National Assessment of Educational Progress, 1974.

National Assessment of Educational Progress. Writing mechanics1969-1974: A capsule descr'ption of changes in writing me-chanics. Denver. CO: Author. 1975.

43

Page, E.G. The analysis of essays by computer (Final Report, U.S.Office of Education Project 6-1318). Storrs, CT: University ofConnecticut, 1968.

Rivas, F. Writelrewrite: An assessment of revision skills (WritingReport No. 05-W-04). Denver, CO: National Assessment of Edu-cational Progress, 1977.

Steele, J. The assessment of writing proficiency via qualitative rat-ings of writing samples. Paper presented at the Annual Meetingof the National Council on Measurement in Education, 1979.

Stiggins, R.J. A comparison of direct and indirect writing assess-ment methods. Research in the Teaching of English (in press)

44

APPENDIX:Large-Scale Writing AssessmentProgram Profiles(from Melton & McCready, 1981)

52

TESTING METHOD

StateGradesTested

SampleSize

(X 1000)Objective

TestWritingSample

ExercisesDeveloped By

Alabama 3,b,9 40-60 X Sta:,DepartmentUniversity FacultyTeachers

Delaware 1-8,11 <5 X X StateDepartmentNAEP Exercises

California 3,6,12 <5 X X Local Districts

Honda 3.5.8.11 <5 X X StateDepartmentTeachers

Hawaii 4.8,11 <5 X Committee

Idaho 9 10-20 X StateDepartment

Louisiana 3,7,10 <5 X X Teachers

Maine 4.8,11 10-20 X NAEP Exercises

Maryland 9-12 >60 X X StateDepartmentContractorTeachersLocal Districts

Massachusetts 7,8,9.12 >60 X StateDepartmentTeachers

Entire Population Tested

46

53

WRMI 3 SAMPLE DESCRIPT!ON

Kind of Scoring ResultsWriting Method Used By Contact

Narration Holistic Local Districts Dr William BerrymanSchools State Dept of EducationTeachers Room 607, State Office Bldg.

Montgomery. AL 36130Narration Primary Trait Local Districts Mr. Robert BigelowPersuasion Schools State Dept. of Public I n str

Teachers Townsend Bldg.. Box 1402State Dover, DE 19901DepartmentPublic Report

Varies by Holistic Local Districts Dr Dale CarlsonDistrict Schools State Dept. of Education

Teachers 721 Capitol MallSacramento, CA 95814

Special Task Analytical State Summary Dr Thomas H. Fisher

NarrationExpositionDescriptionPersuasion

Disseminated State Dept. of Educationon request Knott Building

Tallahassee, FL 32301Holistic Local Districts Dr Selvin Chin-Chance

State Dept of EducationQueen Liliuokalou Bldg1390 Miller StreetP O. Box 2360Honolulu. HI 96804

Narration Holistic Local Districts Ms. Karen UnderwoodExposition Schools State Dept of Educatior,Description Len B Jordan Office BldgPersuasion Boise. ID 83720Narration Primary Trait Local Districts Mr Joseph WilliamsExposition Schools Bureau of AssessmentPersuasion Teachers State Dept. of Education

State P 0 Box 44064Dr prrtment Baton Rouge, LA 70804

Narration Holistic Local Districts Dr Horace P Maxey, JrExposition NAEP State Dept. of Educational andDescription Cultural Services

State Office BuildingAugusta, M E 04333

NarrationEx positionDescription

Holistic State Dr William GrantDepartment State Dept. of EducationLocal Districts BWI AirportSchools P 0 Box 8717

Baltimore, MD 21240Description Holistic Local Districts Dr Allan HartmanPersuasion Analytical Schools State Dept of Education

31 St JamesBoston. MA 02116

47

54

TESTING METHOD

StateGradesTested

SampleSize Objective

(X 1000) TestWritingSample

ExerfasesDeveloped By

StateDepartmentUniversityFacultyNAEP Exercises

Michigan 4.730 <5 X

Minnesota .1 <5 X StateDepartmentTeachersUniversityFacultyLocal Distncts

Nevada 3,6.9 -12 5-10 X X StateDepartmentTeachers

NewHampshire

5,9.12 <5 .- StateDepartment

New Jersey 9 >60 X X Contractor

New Mexico 10 Unspecified X StateDepartmentTeachersLocal Districts

NorthCarolina

11 <5 X Contractor

Ohio 8,12 <5 X X StateDepartmentUniversityFacultyTeachers

Oregon 4.7.11 <5 X X StateDepartment

Pennsykani 5.8.11 Unspecified X

E ntire Population TestedAsseacment Under Development

48 55

WRITING SAMPLE DESCRIPTION

Kind of Scoring ResultsWriting Method rised By Contact

Narration Pnmary Trait To be specified Dr Edward RoeberExposition Michigan Dept of EducationDescription 620 Michigan National Tower

P 0 Box 30008Lansing. MI 48909

Narration Primary Trait Statewide Dr William McMillianExposition Reporting State Dept of EducationDescription Capitol Square, 550 Cedar StPersuasion St Paul. MN 55101

ExpositionDescriptionPersuasion

Holistic Local Distncts Dr R Harold MothersSchools State Dept of Lducation14xchers 400 West King StreetParents er. Carson City. NV 89701Students

NarrationExposition

Holistic State Dr. halt!! V CarrDepatment State DepLof EducationLocal Distncts 64 North Main Street

Concord, NH 03301Narration Holistic Local Districts Dr Stephen Koffler

School Department of EducationTeachers 225 West State StreetStudents Room 200

Trenton, NJ 08625DescriptionPersuasion

H ihstic Local Distncts Dr Carroll L HallSchools State Dept of EducationTeachers Education Building

Santa Fe, NM 87503Unspecified I ailis,... To be specified Dr William J Brown

Analytical State Dept of PublicInstructionitaleigh, NC 27611

NarrationExpositionDescriptionPersuusion

NarrationExpositionDeur)ptionPersuasion

Holistic State Mr Jim PaytonDepartment State Dept of EducationLocal Districts 65 South Front StreetSchools Room 804

Columbus. OH 43215Holistic State R B. Ctemmer

Department Oregon Dept of Education700 Pringle Parkway SESalem, OR 97310

State Dr Robert ColdironDepartment State Dept of Education

PO Box 911Ilarnsburg. PA 17126

49

56

TESTING METHOD

StateGradesTested

SampleSize

DC 1000)Objective

TestWritingSample


Rhode Island 4.6.8.10 <5 X Y, Contractor

SouthCarolina

6.8.11 >60 X StateDepartmentUniversityFacultyContractorTeachersLocal Districts

Texas 3,5,9 A0 X X StateDepartmentUniversityFacultyContractorTeachers

Wyoming 6.9 ' <5 X StateDepartmentUniversityFacultyTeachers

Citybilk Ruck.AR

1 - 1 1 Notspecified

X

Phoenix. AZ 9-12 5-10 X X TeacherLocal Dist nets

Monterey. CA 1-12 <5 X Teacher

Tallahassee.FL

1-8 5-10 X X UniversityFacultyTeachersLocal Districts

Atlanta, GA 1-12 x

Des Moines,IA

9 < 5 X TeacherLocal Districts

Entii s Population Tested

50 57


Kind u Scoring ResultsWritin Method Used By Contact

Narration Holistic Local Distncts Ms Martha HighsmithPersuasion Schools State Dept of Education

Teachers 199 Promenade StreetState Suite 204Department Providence, RI 02908

Narration Holistic Local Districts Dr Vana MeredithExposition Analytical Schools State Dept of Edu:ationDescription Teachers 1429 Senate Street, Room 604Persuasion State Columbia, SC 29201

Depar.---

Narration,DescriptionPersuasion

Holistic Local Districts Mr Keith L CruseTexas Edut ation Agency201 East 11th StreetAustin, TX 78701

NarrationDescription

Holistic Local Distracts Dr Mark FoxSchools Sta e Dept of Education

Hathaway BuildingCheyenne. WY 82002

Local Districts Dr Carolyn WeddleSchoois Little Rock School DistrictTeachers West Markham & Lzard

Little hock AR 72201NarrationExpositionDescription

ExpositionDescription

Analytical Schools Mr Ge: ale De GrowTeachers t-Sormx U! 'S District 210

2524) W Osi orn RdPhoenix, AZ 55017

Holistic Schools Dr Lloyd SwansonTeachers Monterey Peninsula Unified

School DistrictPO Box 131Monterey, CA 93940

Nor-rationExpositionDescription

Aaalyticai Local Districts Mr F W AshmoreSchools Leon Co Public SchoolsTeachers P 0 Box 246

Tallahassee. FL 32302

Schools Mr Alonzo Cnmfeathers Int School District 203

124 Central Avenue S WAtlanta, GA 30303

Persuasion Holistic Schools Mr Dwight M DavisAnalytical reacher Des Moines Int Comm Dist

1800 Grand AvenueDes Moines, IA 50307

51

TESTING METHOD

CityGradesTested

9-12

SampleSize

(X 1000)-,60

ObjectiveTest

WritingSample

X


TeacherChicago. IL

Boston. MA 1,5,8 5-10 X StateDepartmentTeachers

Wichita. KS K -12 10.20 X TeachersCoGrd of L A

BaltimoreMD

1-9 -.60 X X TeachersLocal Distracts

Detroit. MI 10-12 10-20 X X ContractorDept fl, A

Raleigh NC 1-12 - 5 X X ContractorTeachersLocal Districts

AlbuquerqueNM

13).9-12 5-10 X X StateDepartmentTeachers

Santa Fe N'-1 7-12 - 5 X Teachers

New York NY 8.11 b0 X StateDepartment

Portland. OR 3-9 Notspecified

X

5j ,-



Narration Analytical Local Districts Mr James RedmondExposition Cook Co Pubhc SchoolsDescription 228 North La Salle StreetPersuasion Chicago, IL 60601Narration Holistic Local Districts Mr William LearyExposition Schools Boston Puhlic School DistDescnption Teachers 15 Beacon Street

Boston, MA 02108Narration Holistic Local Districts D. Alvin E Moms

Schools Wichita Sedgwick UnfdDist 2594285 BroadwayWichita Falls, KS 67202

Narration Analytical Schools Mr Roland PattersonPersuasion Teachers Baltimore Co Public F.:.-hools

3 E 35th StreetBaltimore, MD 21218

Exposition Holistic Mr Charles WolfeWayne Co Public Schools5057 WoodwardDetroit, MI 48202

Narration feather Option Schools Mr C L HooperExposition Teachers Raleigh Dist Public SchoolsDescription Parents 601 De% ereux St

Students Raleigh, NC 27605Exposition Holistic Local Districts Mr E StapletonDscriptos 'schools Per,a1z110 Co Publi,. SchoolsPersuasion Teachers Box 1927

State A0, i-que. NM $7103DepartmentReported toSOdiaReport toStudent

Description Holism St hook Mr Philip Bch°Analytu al Santa he Co Public Schools

OM Alta VistaSanta I e NM 87501

Exposition Holistic Lot al Districts Mr Calvin E GrossPersuasion Schools New York City Schools

Teas hers 110 Livingston StreetState Brooklyn, NY 11201Department

Local Districts Dr Walter HathawaySchools Portland Public Schoolsfeat hers P 0 Box 3107Parents Portland. OR 97208Students

6053

TESTING METHOD

SampleGrades Size Objective Writing Exercises

City Tested (X 1000) Test Sample Developed ByAustin. TX 3.9 <5 X X State

DepartmentUniversityFacultyContractorTeachers

Madison, WI 5,8.11 <5 X X StateDepartmentUniversityFacultyParent/BusPeople

Seattle. WA 3.6.9-11 X TeachersCurr Specialists

Laramie, WY 6,9 < 5 X Com of localand state univmembers

54

61



Narration Holistic Local Districts Dr Jack DavidsonExposition Schools Austin. ES9Persuasion Teachers 6100 N Guada lope

State Austin. TX 78752Department

Narration Holistic Not specified Mr D S RitchieExposition Pnmory Trait Dane Co Public SchoolsPersuasion 59t5 W Dayton

Madison. WI 53703

Analytical Schools Mr Forbes Bottom lySeattle School Dist 1815 Fourth Ave NSeattle. WA 98109

ExpositionDescription

Holistic Local Districts Dr Joe LutieharmsSchools Laramie Co Public SchoolState District 1Department Cheyenne. WY 82001

Ii ,

ti55

DOCUMENT RESUME - ERICDOCUMENT RESUME ED 213 035 CS 206 751 AMOR Spandel, Vicki; Stiggins, Richard...

Documents

Transcript of DOCUMENT RESUME - ERICDOCUMENT RESUME ED 213 035 CS 206 751 AMOR Spandel, Vicki; Stiggins, Richard...