Testing Writing Specifications for the E8-Standards ...€¦ · 19 E8 Writing Test Specifications...

Testing WritingSpecifications for the E8-Standards Writing Tests

Technical Report 2011

Otmar Gassner Claudia Mewald Rainer Brock Fiona Lackenbauer Klaus Siller

AnwenderRechteck

Testing Writing

Specifications for the E-8 Standards Writing Tests

Technical Report 2011

Otmar GassnerClaudia MewaldRainer BrockFiona LackenbauerKlaus Siller

Bundesinstitut für Bildungsforschung, Innovation & Entwicklung des österreichischen Schulwesens Alpenstraße 121 / 5020 Salzburg

www.bifie.at

Testing Writing. Specifications for the E8-Standards. Writing Tests. Technical Report 2011.BIFIE Salzburg (Hrsg.), Salzburg, 2011

Der Text zu den Bildungsstandards (Kompetenzbereiche usw.) sowie die Aufgabenbeispiele können für Zwecke des Unterrichts in österreichischen Schulen sowie von den Pädagogi-schen Hochschulen im Bereich der Lehreraus-, Lehrerfort- und Lehrerweiterbildung in dem für die jeweilige Lehrveranstaltung erforderlichen Umfang von der Homepage (www.bifie.at) heruntergeladen, kopiert und verbreitet werden. Ebenso ist die Vervielfältigung der Texte und Aufgabenbeispiele auf einem anderen Träger als Papier (z. B. im Rahmen von Power-Point Präsentationen) für Zwecke des Unterrichts gestattet.

Autor/innen:

Otmar Gassner Claudia Mewald Rainer Brock Fiona Lackenbauer Klaus Siller

AnwenderLinie

AnwenderText-BoxWerbung für Microsoft!

Contents

3 EmbeddingtheE8WritingTestinanationalandinternationalcontext

3 ThePlaceofwritinginAustrianlowersecondaryschool

3 ValidityaspectswithregardtotheE8WritingTestconstruct

4 Testtakercharacteristics6 Cognitivevalidity6 Writingtheoryinbrief7 CognitiveprocessinginE8Standards9 Contextvalidity9 Setting:task10 SettingadministrationofE8Writingtests10 Lingusticdemands:Taskinputandoutput12 Scoringvalidity12 Criteriaandratingscale14 Ratercharacteristics14 Ratingprocess15 Ratertraining17 Post-examadjustments18 Reportingresults18 Consequentialvalidity

19 E8WritingTestSpecificationsVersion03(October2010)

19 1.Purposeofthetest19 2.Descriptionoftesttakers19 3.Testlevel19 4.TestConstruct20 E8ConstructSpace22 5.Structureofthetest22 6.Timeallocation22 7.Itemformats22 8.LanguagelevelforInstructionsandPrompts23 9.WritingRatingScale23 WritingRatingScale(October2010)26 10.Promptsandperformancesampleswithjustifications

35 Scaleinterpretations

35 Scaleinterpretation–Taskachievement37 Scaleinterpretation–Coherenceandcohesion39 Scaleinterpretation–Grammar40 Scaleinterpretation–Vocabulary

43 Literature

AnwenderText-BoxKET and PET p.11

45 Appendix

45 Promptinterpretation:Longprompt45 TaskAchiemement46 CoherenceandCohesion46 Grammar46 Vocabulary47 Promptinterpretation:Shortprompt47 TaskAchiemement47 CoherenceandCohesion48 Grammar48 Vocabulary

3Specifications for the E8-Standards

EmbeddingtheE8WritingTestinanationalandinternationalcontext

TheplaceofwritinginAustrianlowersecondaryschool

Thereseemstobesomeagreementthatspeakingandlisteningaretheskillsmostneededwhentryingtosucceedinaforeignlanguageenvironmentandthatbeingable to read is next in priority. This leaves writing as the skill least necessary forsurvival.Nevertheless,writingistrainedfromyearoneofsecondaryeducationonaregularbasis.Insomecoursebooksitstartsoffwithmodelparagraphsthatareper-sonalisedbythelearnersandleadsontoopenwriting,mostlybasedonthecontentofthecoursebookunitinprogress.Itgoeswithoutsayingthatlowerabilitylearnersaregivenmoreguidance,withsomeofthemhardlyeverattemptinganopenwritingtask.

Inpost-beginnerclassestheimportanceattributedtowritingincreases.Itseemstobeawide-spreadbeliefamongteachersofEnglishthatwhenwritingskillsareassessed,otherdimensionsoflanguagecompetencelikevocabularyandgrammarknowledgecanbeassessedautomaticallyatthesametime.Therefore,thewritinggradegoesalongwaytowardstheoverallEnglishgradeforthatparticularstudent(asspeakingishardlytakenintoaccount).

Whereasthisbeliefmightberesponsibleforthehighregardteachershaveforwri-ting, theawarenessof thecomplexityof assessmentprocedures forwriting is stilllimited.Thereisnoperceivedneedforsharedstandardlevels,thereisnoagreementonhowwritingshouldbetested,markedandweightedinrelationtotheotherskills(reading, listening, speaking),1 there are a great number of idiosyncratic markingschemesinplace(evenwithinoneschool),andthereisnoagreementonanythinglikepassmarksorcut-offscoresforgrading.

In this situation there is roomforconstructivewashback in thecourseof the in-troduc-tionofE8standards.ItishopedthatthewaythetestsareconstructedandassessedwillimpactonthewaywritingistaughtandassessedinAustrianschools.

Althoughmuchofwhathasbeensaidabovewasformulatedforthefirsteditionofthistechnicalreportin2008,itisstillrelevantandwecancertainlyseesignificantsignsofchange.Aprogrammetotrainfourhundredwritingratersisinplaceandspreads expertise across the country; test specifications and a number of pilotingphases have led to visible adaptations in the course books used; train the trainerprogrammesonhowtoassesswrittenperformancesfunctionasstartingpointsforschool-basedprofessionaldevelopment.Finally, thereorganisationofacentralisedapproachtotheassessmentofwrittenperformancesatE12level(Matura)hascon-tributedalottoraisingawarenessofthecomplexityofassessingwrittenscripts.

ValidityaspectswithregardtotheE8WritingTestconstruct

Shaw&Weir(2007)havedesignedacleargraphictoillustratetheir“frameworkforconceptualisingwritingtestperformance”(seefigure1,p.5).Ittakesalltherelevant

1 ThislackofagreementisnoticeabledespiteaclearstatementintheAustriancurriculumaboutallfourskillstobetaughtandtrainedequallyintheclassroom;unfortunatelythecurriculumdoesnotsayanythingonweigh-tingintests.(LehrplanderHauptschule.2008,p.2)

AnwenderRechteck

AnwenderUnterstreichen








AnwenderRechteck


4 Specifications for the E8-Standards

parametersintoaccountandcanserveastheblueprintforthedescriptionoftheE8WritingTestsandthetheoreticalframeworkonwhichtheyarebased.Withinthisframework the focusof thediscussionwillbeon the followingaspects: test takercharacteristics,cognitivevalidity,contextvalidity,scoringvalidity,andconsequentialvalidity.

Figure 1: Adapted from Shaw & Weir 2007, 4

Testtakercharacteristics

Itisobviousthattesttakercharacteristicshaveaninfluenceonthewayataskispro-cessedandatextiswritten.Threecategorieshavebeenidentified:physical/physiolo-gical,psychologicalandexperientialcharacteristics(Shaw&Weir2007,5).

Asregardsthefirstcategory,anyprovisionsmadeforschoolingcanbeconsideredsuf-ficientfortheE8testsituationasalltesttakersarepupilsinthelastformoflowerse-condaryschoolsinAustria.Toputitsimply,anypupilwhoisfitenoughtoattendEng-lishclassesatanAustriansecondaryschoolandtobeassessedisfittotaketheE8test.

6

Test-taker Characteristics

Cognitive Validity

Context Validity

Setting: Task

Response format Purpose Knowledge of criteria Weighting Text length Time constraints Writer/reader relationship

Setting: Administration

Physical conditions Uniformity of administration Security

Linguistic demands: (Task input and output)

Lexical resources Structural resources Discourse mode Functional resources Content knowledge

Response

Scoring Validity

Score

Cognitive Processes

Macro/planning Organisation Micro/planning Translation Monitoring Revising

Consequential Validity

Physical/Physiological Psychological Experiential

Rating

Criteria and rating scale Rater characteristics Rating process Rating conditions Rater training Post/exam adjustment Grading and awarding

Washback on individuals in classrooms Impact on institutions and society Avoidance of test bias


Psychologicalfactors,however,arealmostimpossibletocontrol.Mostcriticalismo-tivationasE8Standardsisalow-stakesexamthathasnoinfluencewhatsoeverontheindividualtesttakers’marksorontheirschoolcareer.Wecanexpectlowachieverstobemoreaffectedbylackofmotivation.Forthisreason,testresultsmightnotfullyrepresenttheactuallanguagecompetenceofthesestudents,buttheymightappeartobeatasignificantlylowerlevelbecauseafairnumberfromthisgroupoftestta-kersmaychoosenottoshowwhattheycandoinEnglish.Aslongasthetesthasnopracticalimplicationsfortheindividualtesttaker,itwillbedifficulttogeneraterealinterestandmotivationinthosethatdisplaya“could-not-care-less”attitude.

In2013theE8writingtestwillbeadministerednation-wideforthefirsttime.Thishasalreadyhadsomeimpactonteacherattitudeandmightalsohaveapositiveinflu-enceonlearnermotivation.Preferredlearningstylesandpersonalitytraitsareotherfactorsthatarerelevant,butcannotbecateredforinthegiventestsituation.

Thethirdgroupoffactorsareexperientialcharacteristicsreferringtofamiliaritywiththe test format.Whereas the test takers are all new to this particular type of te-sting, they shouldgenerallybe familiarwith the typeofpromptsused in theE8WritingTest.Asdetailsfromthetestspecificationsbelowconfirm(cf.pp.21–23),promptsusedarebasedontheAnlage zur Verordnung der Bundesministerin für Un-terricht, Kunst und Kultur über Bildungsstandards im Schulwesen (BGBl.II Nr.1/2009 v. 2.1.2009)2, theCEFRandtheAustriancurriculum(LehrplanderHauptschule2008,4–7undLehrplanderAHS2006,x–y).

Learners who have only done tasks that are heavily scaffolded will find the E8promptschallenging.Learnerswhohaveneverfacedopenwritingtasksintheirlear-ninghistory cannotbe expected toperformwell in theE8StandardsTestsor ininternationaltests.Wewouldconsideritimportantwashbackifcoursebookauthorsand,consequently,alsoteachersweretorethinktheissuesinvolvedandalsoattemptunscaffoldedwritingtaskswithALLpupils.AfterfouryearsofEnglishatsecondaryschooland some(very limited)writingatprimary level amounting tomore than500lessons,anystudentshouldbeabletodoataskliketheonebelowsuccessfully:

Figure 2: BIFIE Item archive (http://www.bifie.at/sites/default/files/items/writing_short_task_2010_2.pdf )

2011isthefirstyearwithanewgenerationofcoursebooksavailableforAustrianschoolstochoosefrom.Whatwasformulatedaboveasexpectedwashbackin2008hasmaterialised:ThenewcoursebooksincludewritingtasksthataregearedtotheE8writingspecificationswithanumberofthemextremelyclosetoactualE8writingprompts.Eventhetimeconstraintsandthespecificationsregardinglengthhavebeentakenonboard.Anothersalientfeatureistheattempttoactuallyteachthestudentsaboutusingparagraphswhenproducing(longer)texts.

2 Thisdocumentwillsubsequentlybereferredtoas“BIST-Verordnung”inthisreport.

You have come back from a one-week stay with a host family in Cambridge. At home you remember that you left your mobile phone in your room in Cambridge. Write a short e-mail to your host family.

Tell them where you are now. Tell them about your mobile. Ask them for the mobile. Tell them how you liked your stay.












Cognitivevalidity

“ThecognitivevalidityofaWriting task isameasureofhowclosely it representsthecognitiveprocessing involved inwritingcontextsbeyondthetest itself, i.e. inperformingthetaskinreallife”(Shaw&Weir2007,34).Whereasitisnotoriouslydifficulttodescribethecognitiveprocessesastheyarenotdirectlyaccessible,itseemsimportanttodescribeageneralwritingmodelthataccountsforwritinginareal-lifecontextaswellas inanexamsituation.However,onedifferenceshouldbenotedat the outset, namely that there is no time-constraint in most real-life situationswhereasintheE8testingsituationtime,topic,genre,andlengthofoutputarepre-determined.Thismightimposelimitationsontheplanningphaseaswellasonthewritingandrevisionphases.

Writingtheoryinbrief

Inthegivencontext,onlysketchyreferencesshallbemadetovarioussourcesthatpresentanddiscussthewritingprocessandmodelsofL1andL2writingindetail.AccordingtoGrabeandKaplan(1996,230–232),theplanningphase,whichtheycall“goalsetting”,involvesthesefivefactors:

�� anassessmentofthecontext�� apreliminaryrepresentationofthewritingproduct�� anevaluationofpossibleproblemsintaskexecution�� aninitialconsiderationofthegenrerequired�� anorganisationalplan

ShawandWeir(2007,37)makeapointofemphasizingtheadvantagesofamorepsycholinguisticallyorientedmodelofwritingovertheGrabeandKaplanmodelandrefertoField(2004)andKellogg(1994,1996).Interestedreadersmaywishtocon-sultthedetaileddiscussionthere.TheFieldmodel(Field2004,329–331)involves

�� macro-planning�� organisation�� micro-planning�� translation�� execution�� monitoring�� editingandrevising

AreferencetoScardamaliaandBereiter(1987)isessentialhereastheyhavedescri-bedtwodifferentstrategiesusedbyskilledandlessskilledwritersintheplanningphase: knowledge telling and knowledge transformation.

Inknowledgetelling,novicewritersplanverylittle,andfocusongeneratingcontentfromwithinrememberedlinguisticresourcesinlinewiththetask,topic,orgenre.Knowledgetransformingbytheskilledwriterentailsaheightenedawarenessofpro-blemsasandwhentheyarise–whetherintheareaofideas,planningandorganisation(content),orinthoseofgoalsandreadership(rhetoric)[…](Shaw&Weir2007,43).

Whereas thisholds true for allwriting,L2writingposes additional cognitivede-mandsonthewritersasField(2004)argues.Attentiondirectedtowardslinguisticaspectslikelexicalretrieval,spelling,andsentencestructurescanimpedethefluencyofwritingandthecapacitytoorganiseandstructurethetextasawhole.Someideasmighthave to be abandoned in the executionphase on the grounds of languageconstraintsandlimitations.





AnwenderRechteck


CognitiveprocessinginE8Standards

IntheE8contextwesuggestusingamodifiedGrabe/Kaplan-Fieldmodeltoillustra-tethewritingprocess,whichwillclearlybebasedonknowledge tellingandthushasaverybriefplanningphasemainlyconsistingofselectingandconsideringrelevantcontentpoints.

Thismodelincludesthefollowingphases:

�� assessmentofthecontext(whowritesaboutwhattowhomandwhy?)�� characteristicfeaturesofthegenrerequired�� preliminaryrepresentationofthewritingproduct�� selectionofcontentpoints�� evaluationofpossibleproblemsintaskexecution�� micro-planningatparagraphandsentencelevel�� translation�� monitoring�� revising

IntheE8testsituation,theplanningphaseisclearlynotelaborateorextensive.Afteranassessmentofthecontext,whichincludesidentifyingthetopic,thesituationofthewriter,thetextpurposeandtheaddressee,mosttesttakerswillmovestraighttotheconsiderationofthegenrerequiredanddevelopa“preliminaryrepresentationofthewritingproduct”.Thenthebulletpointswillpre-structurethecontentelementstobeincluded.Anorganisationalplanisnotnecessaryasthetasksarefairlyshortandguidedbycontentpointswithlittleopportunityfordeviation.Especiallywiththeshorttask(40–70words)planningeffortswillbereducedtothebareminimumandberestrictedtothedecisiononwhichcontentpointstoelaborateandhowtoproceedinthatdirection.

Figure 3

Thewritingismorecloselyguidedthaninreallifeasanumberofcontentpointsaregivenintheprompt.Ontheonehand,thismakesthewritingprocesssomewhateasierthaninreallife,ontheotherhand,itisanecessityifwewanttoensureinter-raterreliabilityforthedimensionoftaskachievement.Inadditiontothis,providingacontentschemaforcandidatesatthislevelisnecessarybecausethecognitiveloadforsimultaneousactivitiesonamacroandmicrolevelwouldbetoogreatandthetasktoodemanding.

Ithasbecomeclearfromthepresentdiscussionthatmacro-planningandorganiza-tionplaynoroleinthegivenwritingcontextandthattheproductdeliveredwillbefirmlysetintheareaofknowledgetelling.

Planning

ContextGenre

Preliminary representationContent

Problems

Writing

Micro-planningTranslationMonitoring

Editingand

Revising

AnwenderRechteck

AnwenderRechteck









AnwenderText-Boxplanning --> writing --> editing --> revising the text



Themicro-planningphase,thenextstepofthewritingprocess,mightbethepointwherepossibleproblemsintaskexecutionwillbeidentifiedbeforetheactualwritingbegins.Theproblemswillbecontent-relatedandhavetodowithknowledgeoftheworldandwhat(abstract)ideastousewiththecontentpointsgiven;theymightalsobeconnectedtotheattempttorecalltherequirementsofthegenreinquestionandwiththelanguagenecessarytoexpresstheideas.

Thisstageofidentifyinglanguageresourcesandtheirlimitationsisonlyafractionaway from actually putting pen to paper and undoubtedly is a central aspect ofmicro-planningfocusingonthepartofthetextthatisabouttobeproduced.Here,theplanningtakesplaceonatleasttwolevels:thegoaloftheparagraph,itselfalignedwiththeoverallgoalof thewritingactivity;withintheparagraph, the immediateneed to structure an upcoming sentence in terms of information (Shaw &Weir2007,39).

Micro-planningmergeswiththe translationphasewherepreviouslyabstract ideasonlyaccessibletothewriterhim/herselfaretranslatedintothepublicspacedefinedbyasharedlanguage.IncontrasttoShaw&WeirandField,weseemicro-planningandtranslationastwostagesthatareinterlinkedasthewritermightoscillatebetweentheoneandtheotheratsentenceleveloratparagraphlevel(Shaw&Weir2007,39–40).

Itisinthetranslationstagethatlanguagedecisionshavetobemadeandplanningdecisionshavetobeimplemented.Theactualproductionoftextwillbetakingplaceunder the constraints of content schemata, genre restrictions and the limitationsoflinguisticresourcesathandinL2.Whathasbeencalled“avoidancebehaviour”(e.g.avoidinglexisorstructuresthatseemunsafe)and“achievementbehaviour”(e.g.using simpler structures,paraphrasing)byField (2004,66–67)needs tobe takencareofinthemarkingscheme,asdoestheabilitytoproducecoherentandcohesivetexts.

Thenextstepismonitoringalthoughthisisnotnecessarilysequentialandmightbeoscillatingwithphasesoftranslation.“Atabasiclevelmonitoringinvolvescheckingthemechanicalaccuracyofspelling,punctuationandsyntax”(Shaw&Weir2007,41).AtE8levelthis iswhatcanbeexpected, ifnotinthelowestsegmentoftesttakers.Inaddition,betterwriterswillalsocheckbackoncontentandgenrerequire-ments.Thesemonitoringactivitieswillleadtoeditingandrevisingifsomepartsofthetexthavebeenfoundunsatisfactory.Thismightinvolveadding,deletingormo-difyingacontentpoint,addingcohesivedevices,replacingpoorwordsandphraseswithbetterones,orsimplycorrectingmistakesinspellingandstructure.

IntheE8context,writingiscertainlybasedontheknowledge-telling model (Scar-damalia&Bereiter1987);Hyland’ssummaryofthemodelepitomisesE8writingperformances:

A knowledge-telling modeladdressesthefactthatnovicewritersplanlessoftenthanexperts,reviselessoftenandlessextensively,andareprimarilyconcernedwithgene-ratingcontentfromtheirinternalresources.Theirmaingoalissimplytotellwhattheycanrememberbasedontheassignment,thetopic,orthegenre(Hyland2002,28).


















AnwenderRechteck


Contextvalidity

Testsshouldbeascloseaspossibletoauthenticreal-lifesituations.Writingisanacti-vitythatisnormallyperformedbyindividualsatatimesetasideforit.Writershaveapurposeandanaudience;theyhavethefreedomtointerruptthewritingprocessandresumeitatatimeoftheirchoice,especiallyforeditingandrevising;andtheycannormallyusedictionaries andother resources. In thegiven test setting, someconstraintswillbeoperative.

Shaw&Weir2007(64–142)discussanumberofaspectsofcontextvalidityrelatedtothreeareas:

�� Setting:task�� Setting:administration�� Linguisticdemands:Taskinputandoutput

ThesepointswillstructurethediscussionofcontextvalidityoftheE8WritingTests.

Setting:task

Theaspectstobediscussedhereareresponseformat,purpose,knowledgeofcriteria,weighting, text length, timeconstraints, andwriter-reader-relationship. In theE8WritingTestsauthenticityisoneofthemostprominentaimsofpromptconstruc-tion.However,incontrasttoreal-lifewritingthereisnoprovisionfortheuseofanyresourcematerialssuchasdictionaries.

ThewritingtasksaretargetedatpupilsofAustrianschoolsinyear8andnormallyagedfourteen.ThetasksaredesignedtoappealtothisagegroupandtoelicitscriptsthatshowwhattesttakerscandowithintheframeworkdefinedintheBIST-Verord-nung.Thedomainsandgenreshavebeencarefullyselectedfromthisframework,whichisbasedontheCEFR,andfilteredfurtheronthebasisoftheAustriancurriculum.

Astheresponseformatmaywellplayasignificantroleintestperformance(Aldersonetal.1995),thedecisionhasbeentakentoincludetwoformatsintheE8WritingTest.Thereisashorttask(40–70words)andalongtask(120–180words),whichareassessedseparately.Bothareopenwritingtasks.Goodwritershaveabetterchancetoshowtheirbestinthelongtask,whichisbasedonaB1descriptor,takenfromtheBIST-Verordnung.Lowerachieversareexpectedtodobetterintheshorttask,whichislimitedinscope,morecloselyguidedandbasedonanA2descriptor.However,bothgoodandweakerwritersareexpectedtoaddressbothtasks,theyarenotsup-posedtochooseonlyoneofthetasks.

Instructions,deliveredbothorallyandinwritingtothetesttakersbeforetheactualtestbyatestadministrator,andrubrics thatgowitheachtaskpresentcandidateswithinformationregardingtextlength(seeabove)andtimeconstraints.Forbothtasksthetesttakershave30minutesofwritingtimeplus5minutesforeditingandrevisinginall.After35minutesthereissometimeforwordcountbythecandidates.Theactualpromptscontextualisethetaskbydefiningthewriter-reader-relationship,statingpurposeandgenre,andgivingcontentpointstobeincludedinthetext.Theshorttaskcontains3–4contentpoints,thelongone5–8.

Informationonthescoringcriteriausedandtheirweighting,includingtheratingscaleused,scaleinterpretationsandbenchmarkedsamplescripts,ispublishedinthisreport(cf.pp.XX).Furthermore,sampleprompts,therating-scaleandbenchmar-kedtextsarepubliclyavailableontheBIFIEwebsite.







Setting:administrationofE8Writingtests

Initspresentform,thewritingtestwasfirstpilotedonasampleofca.800testta-kersin2007andin2009abaselinestudywascarriedout.Consequentlydetailedinformationonthe“pilotphase”between2006and2008andonthebaselinetestsin2009werepublishedinaTechnicalReport(Breit&Schreiner2010).Startingin2013,theE8writingtestswillbesetnationwideeverythreeyearsandallAustrianschoolchildreningrade8willbetested.OnlySENpupils,i.e.thosewithspecialeducationalneeds,willbeexemptedfromdoingthetests.

Inordertoensurereliabletestresults,thecircumstancesunderwhichtheE8writingtesttakesplacemustbesimilar.Thestepsdiscussedinmoredetailhereconcernphy-sicalconditions,uniformityofadministrationandtestsecurity,basedontheideasbyShaw&Weir(2007).

AsthevenuesofE8writingtestsareclassroomsinAustrianschools,physical testconditionsareofverysimilarstandardsandtesttakersshouldfindappropriatecon-ditionsfortakingthetest.

Inordertogranttheuniformityofadministration,thetestmustbeconductedac-cordingtostandardizedinstructionsbytrainedtestadministrators.Anextensivetestadministrator’smanualisprovidedduringthetestadministratortraining.Thema-nualincludesinformationonthebackgroundofthetest,checklistsandToDo’sbothforthepreparation,theactualsettingofthetest(e.g.startingtheexam,completingdifferentlists,standardisedverbal instructionsforthetestadministratoretc.),andtheconclusionoftheexamination.

Inanation-wideexamtherearesomeadministrativeconstraints:apoliticaldecisionhasbeentakenregulatingtestadministrationintheyearstocome:in90%oftheschoolstheE8testswillbeadministeredbytheteachersoftheschool(internaltestadministration).Infurther3%oftheschoolsthetestswillalsobeadministeredin-ternally,buttherewillbeexternalqualitymonitorstoassurethecorrectandstandar-disedadministrationofthetests.7%oftheschoolswillbetestedexternally.Alltestadministrators,bothinternalandexternalones,aretrainedtoadministerthetestsaccordingtoagreedstandardisedprocedures.However,itiswithintheresponsibilityoftheschools’headteacherstotakecareofacorrectandstandardisedtestadmini-stration,asthisistheonlywaytogetreliablefeedbackregardingtheperformanceoftheirpupilsandtoplanlocalmeasuresofqualitydevelopment.

Thepromptsusedinfuturewritingtestshaveallbeenwrittenbytheprospectivera-ters,moderated,edited,andscreenedbythewritingratertrainerteamatBIFIESalz-burg,pre-testedandstoredintheitemarchive.ThetestbookletsaredesignedbythesamegroupincooperationwiththepsychometricdepartmentatBIFIE.TheactualdistributionofalltestpaperstotheschoolsishandledcentrallybyBIFIESalzburg.

MoredetailedinformationontheadministrationoftheE8testswillbepublishedinatechnicalreportafterthefirstnation-widetestingin2013.

Linguisticdemands:Taskinputandoutput

IntheAustrianteachercommunitythecommunicativeapproachtolanguagelear-ning(Canale&Swain1980asanimportantprecursoroftheBachman1990modelof communicative language ability) iswidely accepted, and it is also setdown inwritinginthenationalcurriculum.Asthelearningtasksaremodelledonreal-lifecontexts, the learning environment aims to mirror real life as closely as possible.





ExamssetintheAustriancontextneedtosharethesepremisesandtoreflecttheminthetasksset.

Shaw&Weir(2007,91),Alderson(2004,13)andotherscomplainthattheCEFRremainsvagueandwithholdsanydetailswhenitcomestostructuresorvocabulary,usingtermslike“simple”inthedescriptors.Whilethisistrue,readingtheCEFRextensively rather than focusingonlyon the sections containing the scalesproveshelpful.Inchapter3,thedevelopmentofthecommonreferencelevelsisexplainedanditismadeclearthattheyprogressinaverycoherentwayfrom“thelowestlevelofgenerativelanguageuse”(CEFR2001,33)tosocialfunctionsand“descriptorsongettingoutandabout”(CEFR2001,34)basedonWaystage (A2)andasimplifiedversionofsometransactionallanguagefrom“’TheThresholdLevel’foradultslivingabroad”(CEFR2001,34).A2+doesnotsomuchincreasetherangeoftopics,butfocuseson“moreactiveparticipationinconversation”and“significantlymore[onthe]abilitytosustainmonologues”.B1reflectstheThresholdlevelandinvolves“theabilitytomaintaininteractionandgetacrosswhatyouwantto,inarangeofcon-texts”aswellas“theabilitytocopeflexiblywithproblemsineverydaylife”(CEFR2001,34).B1+makes increaseddemandson thequantitiesof information tobehandled.

As this is theway the levelshavebeen constructed (i.e. fromWaystage toA2), itseems legitimate tomove fromA2 specificationsback toWaystage.Andherewehaveavocabularylistandalistofstructuresconsideredcharacteristicofthatlevel.AsUCLEShavealsoorientedthemselvesonvocabulary lists fromtheCouncilofEuropePublications(LexicalInventoryinWaystage,1980,45–62;andinThreshold, 1979,85–115),itcanbeconsideredausefulshortcuttopickupthevocabularylistspublishedonthewebforKET(A2)andPET(B1),especiallyas thesehavebeenupdatedonthebasisoffrequencyandusagedatafromlanguagecorpora.General-ly,“[the]languagespecificationsofKETarethesameasthosesetoutinWaystage1990”(KETHandbook2007,1).

Toresumethediscussionofthevaguenessofdescriptorsusingwordslike“simple”,“basic”or“sufficient”,itmaysufficetosaythatthisvaguenessneedstobecontextua-lised.IftheA2descriptoronGrammaticalAccuracyreads“Usessomesimplestruc-turescorrectly,butstillsystematicallymakesbasicmistakes”(CEFR2001,112),wecanexpectlearnerstousetherangeofstructureslistedinthe Structural Inventory of Waystage (63–83)ortheKET Handbook(8–9)withseverelyrestrictedaccuracy.Inthissense,evenvaguetermslike“simple”arereasonablywell-definedsothatpromptwritersandratersknowwhattolookfor.

TheE8writingpromptsdonotrestricttesttakersintheiruseofspecificlexicalorstructuralresources,butgivethemtheopportunitytodemonstratetheirlinguisticabilitieswithinthetaskset.Theextentoftheirsuccessindoingsoisassessedaccor-dingtothegradeddescriptorsintheassessmentscale.

Whathas tobenoted,however, is thebasicorientationof theCEFRtowardsanadultlearnerandadominanceoftouristaspectsoflanguagelearning.ThisiswhytheAustrianE8Standardshavealso integrated the specifications setdown in theAustriancurriculumandadaptedtheCEFRdescriptorstotheagegroupofthetestpopulation.Thismainlyreflectstheselectionofdomainsandtransactionalsituati-ons.Ithasnoinfluenceonthestructuresincluded,thoughithassomeinfluenceonthewordlist.Generally,theschoolbooksusedinAustriatakethisintoaccount.AsthetestisexplicitlybasedontheAustriancurriculum,thelinguisticdemandsofthetestarefairforalltesttakers.



AnwenderText-BoxKET (A2) andPET (B1)

AnwenderRechteck





AnwenderErläuterung"simple", "basic", "sufficient" refers to KET and PET


Thewritingpromptsusedoftenspecifyparticularlanguagefunctionstobeperfor-med,e.g.“invite…,apologise…,askfor…,giveadvice…”.Alistofthesefunctionshasbeenmadeavailabletotheteacherspreparingthetesttakerssothattheycanbeexpectedtobeawareofthem.

Severalresearchpapershaveobservedaninteractionorevenaninterdependenceofcontentknowledgeontheonehand,andwritingperformanceandtestscoresontheother(Read1990,Papajohn1999,Weir2005).Provisionsforthishavebeenmadeby restricting topics toareas that can safelybeassumed tobe familiar to the testtakersastheyaresetdownintheAustriancurriculumandmusthavebeenincludedintheirEnglishlessons.However,thisstillleavesthefactthatsometesttakersmightfeel indisposed todealwith aparticular topic for anumberof reasons, themostcommonprobablybeinglackofmotivationandinterest.

Moredetailedinformationondiscoursemode(i.e.texttypes),functionalresources(i.e.intention/purpose),andcontentknowledge(i.e.topicarea)canbefoundinthetableonpp.XX

Scoringvalidity

Scoringvalidityisconcernedwithalltheaspectsofthetestingprocessthatcanim-pactonthereliabilityoftestscores.[…]Itiscriticalbecauseifwecannotdependontheratingofexamscriptsitmatterslittlethatthetaskswedeveloparepotentiallyvalidintermsofbothcognitiveandcontextualparameters.Faultycriteriaorscales,unsuitableratersorprocedures,lackoftrainingandstandardisation,poororvariableconditionsforrating,inadequateprovisionforpostexamstatisticaladjustment,andunsystematicorill-conceivedproceduresforgradingandawardingcanallleadtoareductioninscoringvalidityandtotheriskofconstructirrelevantvariance(Shaw&Weir2007,144–145).

Inthissectionweexamineeachoftherelevantparametersinsomedetail:criteriaandratingscale,ratercharacteristics,ratertraining,ratingprocess,ratingconditions,post-examadjustments,andgrading.

Criteriaandratingscale

Before the actual construction of the rating scale, information on existing scaleswascollectedandtheusefulnessofthescales intheframeworkofE8Testingwasanalysed:Jacobsetal.scoringprofile1981(Weigle2002,116);TEEPattributewri-tingscales,Weir1990(Weigle2002,117);FCEScoringrubric1997(Weigle2002,152);TOEFLwritingscoringguide2000(Tankó2005,125);IELTSbands2002(Weigle2002,159);AnalyticwritingscaledevelopedbytheHungarianSchool-Lea-vingEnglishExaminationReformProject2005(Tankó2005,127).

LumleyreportsfindingsfromWeigle1994,whousedananalyticscaletohave30compositionsassessedbynoviceandexpertraters.Weiglefocusedonnoviceraters,whichisrelevanttotheE8situationinAustriawherearatingcultureisonlyjustevolving.

Shefoundthatraterreliabilityincreasedasaresultoftraining,andthattheimprovedagreementwas the resultof ratersgainingbetterconsensualunderstandingof thetermsandlevelsrepresentedinthescale.Shefoundevidencethattraininghelpedclarificationoftheratingcriteria(Lumley2005,44).












Thissupportstheviewofthetestingteamthatinthegivencontextananalyticscalewouldbepreferabletoaholisticscale.ThisviewisalsosupportedbyWeigle2002,whomentionsseveraladvantagesofanalyticoverholisticscoring:

�� Itismoreusefulinratertrainingasinexperiencedraterscanmoreeasilyunder-standandapplythecriteriainseparatescales.�� Itisparticularlyusefulforsecond-languagelearners,whoaremorelikelytoshowamarkedorunevenprofile.�� Ascoringschemeinwhichmultiplescoresaregiventoeachscripttendstoim-provereliability(Weigle2002,120).

AnotherreasonforrulingoutaholisticapproachwasthefactthatratingproceduresforscriptswithintheAustrianschoolsystemarenotregulated,showgreatvarietyandaretoalargeextentholistic,evenimpressionistic.Asassessmentproceduresforwriting inAustrian schools cannotbe taken as abasis for adisciplined approachtowardsratingscripts,breakingwiththistraditionseemedtobestguaranteeafreshapproachtoassessment.

TakingthegeneralbackgroundofAustriantraditionsinassessingwritingintoac-count and inspiredby theHungarian scale (Tankó2005, 127), thedecisionwastaken to design an analytic scale measuring four dimensions:Task Achievement,CoherenceandCohesion,Grammar,andVocabulary.Whereasthreeofthesefourdimensionshave a strong recognitionvalue forAustrian teachers,Coherence andCohesionmightappearunusualandreflectsthehighimportancegiventothisdi-mensionbytheCEFR.Thesefourdimensionspromisedtoyieldenoughdetailforaconstructivefeedbackprofileonindividualtesttakers’performance,informationforinstructionaswellasinformativedataforsystemmonitoring.TheassessmentscalewasconstructedbearinginmindthefactthattheoverallmajorityofperformancescouldbeexpectedtobearoundA2/B1.ThismeantthatA2andB1descriptorsnee-dedtobeincludedwhileanythingatandaboveB2couldbeneglected.WeareawareofthefactthatthiskindofscalecannotmeasureB2orC1performancesandwehavesettledforstatingthatperformancesabovetheupperendofthedescriptorsintheE8scalearecalled“aboveA2”forshorttasksand“aboveB1”forlongtasks.But,generally, theapplicabilityofaparticulardescriptordoesnotautomatically signalthatascriptisatthatCEFRlevel.Firstly,bandsconsistofmorethanonedescriptor,andsecondly,linkingwrittenperformancestotheCEFRisacomplexprocedurethatisbeyondthescopeofthisreportandwillbediscussedinaseparatepublication.

Thesecondconsiderationinscaleconstructionwasthecognitiveloadthatraterscanmanageintheratingprocess.Thedecisiontousefourdimensionsisalsoinagree-mentwiththeCEFRrecommendationtoreducethenumberofpossiblecategoriesto“afeasiblenumber”as“morethan4or5categoriesstartstocausecognitiveover-load”.(CEFR2001,193)Wetakeitthatthiswarningalsoappliestothenumberofbandsanddescriptors that raters canhandle, sowehaveopted for fourbandssuppliedwithdescriptorsandthreeemptybandsinbetween,makingitaseven-bandscaleplusazeroband.ThescalehasbeencondensedtoonepagewithanextendedscaleeachforTaskAchievementShortandTaskAchievementLong.Asthisdeflatedscalemightnotcarryenoughinformationfortherateratthebeginningoftheirtrai-ning,scaleinterpretationshavebeenprovided(seepp.11–11).Thescalesthemselveshavebeenfine-tunedinthetrainingprocessinanongoingdialoguewiththeraters.Itfollowsfromthisthatthescalesarewhathasbeencalledassessor-oriented(Weigle2002,122;CEFR2001,38).

At thatpoint thescalesconsistedof threecolumns:Thefirstbeingadeflatedde-scriptorforeachofthefourbands,thesecondbeingextendedandcontainingmore





AnwenderRechteck


AnwenderHervorheben


detail,andthethirdquotingtherelatedCEFRdescriptor.AnimportantdecisionintheprocessofscaleconstructionwastheremovaloftheCEFRlevelsattheendoftheCEFRdescriptorsand,inasecondstep,theremovaloftheCEFRdescriptorsaltogether.Thiswasthelogicalsteptotakewhensomeratersawardedband7toascriptandarguedthatthescriptwasaB2performance.However,suchanargumentisinadmissibleasthepromptsusedinthetestarewrittenonthebasisofA2orB1descriptors and responses to these prompts simply cannot measure performancesaboveA2orB1respectivelyasonebasicfactoristhescopeofaperformancetogetherwith thegiven limitationsofdomainsandgenres.SowhentheB2descriptor forgrammaticalaccuracy“Showsarelativelyhighdegreeofgrammaticalcontrol.Doesnotmakemistakeswhichleadtomisunderstanding.”(CEFR2001,114)describestheperformancewell,itdoesnotmeanthatitisB2,butthattheA2/B1taskhasbeencarriedoutverywellandthatthe(grammar)performanceisaverygoodA2orB1performancerespectively.

Thewritingscriptsareassessedonfourdimensions:TaskAchievement,CoherenceandCohesion,Grammar,andVocabulary.WhereasthelastthreearebasedontheCEFRandtheAustrianBIST-Verordnung,theCEFRdoesnotcontainanythingontaskachievement.Inourview,however,thecontentaspectofwritingiscentralandlargelyresponsiblefortheoverallqualityofascript.Nevertheless,thereisnooverallgradeforwriting,butallfourdimensionsareratedseparatelyandarereportedasaprofile,whichmoreoftenthannotisunevenormarked.

Ratercharacteristics

Ithasbeenreportedthat“SubjectspecialistsandlanguagetrainedEFLteachersde-monstrateatendencytoemployratinginstrumentsdifferently”(Elder1992,inShaw&Weir2007,169).InthisrespectthepresentsituationinAustriaisuncomplicatedasallratersareteachersofEnglishwhoteachinlowersecondaryschools.SomeofthesearenativespeakersnowlivingandworkinginAustria,somehaveauniversitybackground,otherswereeducatedatUniversityCollegesofTeacherEducation.

Althoughtheratersgo througha specific training that familiarises themwith theratingscalesandtheratingprocedures,differencesintheirexperientialbackgroundandintheirprofessionaltraininganddevelopmentmayleadtodifferingassessmentsofscripts.Inordertomakeratersawareofthisandtostartaprocessofself-reflection,allratersgetdetailedfeedbackontheirratingbehaviour,bothafterthelasttrainingsessionandaftertheadministrationofawritingtest.Theyareinformedabouttheirinter-raterreliabilityandraterseverity.Eventually,harshnessandleniencyofratersistakencareofthroughRaschmodelling.

Ratingprocess

Milanovicetal.(1996)identifiedanumberofapproachesraterstakeintheprocessofratingascript.Inourtrainingsessionswegenerallyadviseagainstthe‘readthrough’andthe‘provisionalmarkapproach’,bothofwhicharebasedononereadingofthescript.Ratersareencouragedtoadopta‘principledtwoscan/readapproach’totheprocesswithafocusontaskachievementandcoherenceandcohesioninthefirstreadingandongrammarandvocabulary in the second.The lengthof the scriptsseemstosupportthisapproach.

WeareawareofgroupeffectsonraterreliabilityasdescribedbyShaw&Weir(2007,174–175)andhavemadeanefforttousethemtoouradvantageinthestandardi-sationmeetingsatthebeginningofthetrainingsessionsandtheratingsession.Inaddition to theprocedures recommended for standardisationmeetings (Alderson,


AnwenderText-BoxTA is not mentioned in the CEFR








Clapham&Wall1995,112–113)aconsiderableamountoftimeisspentonthedetailedinterpretationoftheprompts(seeappendix,pp.38–40)andanopendiscus-sionofanyquestionsthatmightberaisedbytheraterstakingintoconsiderationthatallratershavealsobeeninvolvedinthewritingofpromptsandtheirpiloting.Anadditionalsetoftenbenchmarks,gainedinanextensivebenchmarkingconferencewithtenbenchmarkers,playsavitalroleinthestandardisationsessions.

Ratingconditions

Theon-siteratingsessionwilltakeplaceinAprilorMayandlaststwotothreedays.Itstartswithastandardisationmeetingandthencontinueswithratingsessionsofthescriptsinratingbooklets,whichhavebeencarefullycompiledbytheBIFIESalzburgpsychometricdepartment.Thiswillprovidethemwiththerelevantdataneededfortestanalysisandfeedback.

In2013thewholeE8populationofsome90.000pupilswillbetested.Raters,whohavebeentrainedatdifferentintervalssince2006,willtakepartinstandardisationsessions inwhich theywillundergosupervisedratingwith thenewtestprompts.Itwilltakehalfaday’sworktodealwitheachpromptandgiveratersenoughtimeforon-site ratingandclarificationof ratingproblemsbasedon theparticular testprompts.Theremainingunratedscripts,approximatelythreequartersintotal,willberatedoff-sitewithinsixtoeightweeksattheraters’convenience.

Ratertraining

AccordingtoAlderson,Clapham&Wall,ratertrainingisoneofthemostessentialaspects inanefforttoobtainreliableassessments(1995,105).Lumleyrefers toa growing body of work that shows the effectiveness of the training process in allowing raters to develop adequate reliability or agreement in the use of individual scales in relation to specified test tasks (2005,62).

ThishasbeentakenveryseriouslybytheBIFIEwritingratertrainerteam,whohavedevelopedaneight-monthstrainingprogrammeforratersstartinginOctoberandpreparingtheratersforthemockratingsessioninMayorJune.Thisprogrammeisdescribedinsomedetailbelow.

RECRUITMENTIntherecruitmentphaseteachersinAustrianlowersecondaryschoolsareapproa-chedtobecomewritingraters.Asthetesttakerscomefromthethreedifferenttypesof lowersecondaryschools, theGeneralSecondarySchool(Hauptschule)andtheAcademicSecondarySchool(AHS)andtheNeueMittelschuleasanewcombina-tionofthesetwoformertypes,carehasbeentakentoensureintakeofratersfromallthreeoftheseschooltypes.WhilerecruitmentwasoriginallycarriedoutbybifieSalzburguntil2009,theadministrationoftherecruitmentprocesshassincebeenoutsourcedtotheregionalUniversityCollegesofTeacherEducation.

TRAINING PHASE 1: OCTOBER (1 DAY; FACE-TO FACE SESSION)AstheCEFRisthemostrelevantbackgrounddocumentfortheE8Standards,thestartingpointofthefirsttrainingsessionisThe Common European Framework ingene-ralandtheOverall Writing Scales for Production and Interactioninparticular.Thefami-liarizationwiththeCEFRisimplementedonthebasisoftherecommendationsmadeintheManual on Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF) (2003, 70–77),inclu-dingsortingtasks.Itismadeclearatthispointthatoneaspectofwritingisrelatedtocommunicativeactivitiesandstrategies,anotheronetolinguisticcompetences.


TheTestSpecificationsarepresentedanddiscussedindetail:domains,genres,modelofwriting,promptformat.Inthisphasethereisafocusonpromptproduction.Thereare Guidelines for prompt writers thatprovideassistanceintheprocessofpromptwriting.

Eachprospectiveratertakesonthetaskofwritingonelongoroneshortpromptinatrainingtandemduringtheweeksfollowingthefirsttrainingsession.Allpromptsaresentintothetrainerteamformoderation.Oncethepromptshavebeenscreened,theyarepilotedbythepromptwriters.Behindthisprocedurethereisthebeliefthatratersneedtoknowaboutthequalitiesofpromptsandwhatelementstheymustcontain.Thispreparesthemforbetterinteractionwiththetestpromptsintheactualratingphase.

Withregardtodifferentialvalidityitisimportanttostatethatallprecautionsweretakenattheearlystageofpromptwritingtoavoidtestbias.Varyingculturalback-groundsandknowledgeoftheworldweretakenintoaccountaswellasthegivenvarietyofcognitivecharacteristics,mothertongue,ethnicityandgender.

AfteranintroductiontotheAustrianE8Standardstheratingscaleispresentedandexplainedonthebasisofwrittenscaleinterpretations(seepp.39–47).Unfoldingthesevenbandswithfourofthemdefinedandworkingthroughthefourdimensionstakestime.Theprocedureadoptedistolookatthesevenbandsofonedimension,whichleadstosometheoreticalunderstandingofthescales,butconnectionstoac-tualscriptsarestilltenuous.Sotheratersgettwoscriptseachforindividualmarkingon thefirstdimension (task achievement).Theydiscuss their assessment in smallgroupsandthetrainerdisclosesthebenchmarkedassessmentsandarguesthecase.Thisprocedureisrepeatedfortheotherthreedimensions.

In the secondphase of the trainingworkshop theparticipants get sets of bench-markedscriptswhichtheyrateononedimensionbeforetheygetafinalsetofscriptswhichisratedonallfourdimensions.Afterdiscussionsandargumentationofthejudgementsforthebenchmarkedscriptstheparticipantshavearoughideaofthecomplexityoftheratingprocessandtheeffortittakestostandardisejudgements.Theratingsheetsfilledinbytheparticipantsprovideafirstsetofdatathathelpstomonitorintra-raterreliability.

TRAINING PHASE 2: OCTOBER – DECEMBERThesecondtrainingphaseisanopenonewithafairlyloosestructure.Allpartici-pantsfirstwritetheirpromptsandgetthembackfromthetestingteamasscreenedprompts(insomecasesthepromptsarereturnedtothewritersforrepair).Asafirstmeasuretowardsqualityassurancethepromptwritersproducearesponsetotheirownprompt.Thisshouldmakepromptwritersawareofthemoreobviousflawstheirpromptsmighthave.Thepromptsarethenpilotedinoneoftheirclassessothatallparticipantshavearound20scriptsbasedontheirprompt.

TRAINING PHASE 3: JANUARY – MARCHOncethepromptwritingandpilotingisfinished,theonlineratingphasestartsinJanuarywiththetrainerteamsendingoutscriptstotheratersforindividualratingoneachofthefourdimensions.Theratershaveaboutfourweekstodothisandsendintheirratings.Whenallratingsheetshavebeentransmittedtothetestingteam,the benchmarks are sent out to the raters. In February/March the raters practisetheirratingskillsontheirownpilotscripts.Theyselecttwoscriptstobefedintothetrainingprocess.Theyratethesescriptsandwritejustificationsfortheirratings.Thedigitalisedscriptsaresenttothetrainertogetherwiththeratingsandthejustifica-tions.ThetrainergoesthroughthesescriptsandselectsinterestingsamplesfortheupcomingtrainingworkshopinMarch/April.



TRAINING PHASE 4: APRIL/MAYTrainingPhase4startswithadiscussionofopenquestionsfromprevioustrainingphases.Thenthereare twostandardisation sessionswith recycled scriptsandnewbenchmarksandthetimeisspentratingscriptsanddiscussingparticularproblemsarisingintheprocess.

Afterthesetwostandardisationsessionsthefirstpromptwritertandempresentstheirpromptandthewholegrouprates2–4scripts.Experiencefrompreviousratingses-sionshasshownthat,astheratershavetohandleanumberofdifferentpromptsinthisphase,theyneedmoreguidanceintheanalysisofthepromptsandthereforearealsoprovidedwithpromptinterpretations.Thepromptwritersthendisclosetheirjudgementsanddefendtheirscoresinadiscussionwiththewholegroupmonitoredbythetrainer.Thisprocedureisrepeatedsothatthemajorityoftheratershavethechancetodiscusstheirscriptsandtheirjudgementswiththewholegroup.Inter-raterreliabilityandintra-raterreliabilityaremonitoredandpertinentdataoneachrateriscollectedsystematically.

TRAINING PHASE 5:RATING SESSION: IMMEDIATELY AFTER TRAINING PHASE 4Therating sessionbeginswitha standardisationmeeting,which involves ratinganumberofbenchmarkedscriptsbasedonfamiliarprompts.Thesecondimportantpartofthismeetingistheanalysisandinterpretationofnewpromptsthatwerepi-lotedonarepresentativesampleofprospectiveAustriantesttakersandwhichmightactuallybeusedinafuturetest.Ratersaregivendetailedinformationandareinvitedtodiscussanyissuesthatarestillunclear.

Then the actual ratingbegins.The raters receive fourbooklets of 25 scripts eachwhichwerewritteninresponsetoashortoralongprompt.Thereisaratingplanwithoverlapformultiplerating.Afterabouttenscriptshavebeenrated,theratersmeetwiththesupervisortodiscussanycriticalissuesthatmayhavecomeupduringtherating.Thentheyproceedtoratetheotherscriptsofthatbooklet,whichinvolvessomefreetimemanagementfortheraters.Thisprocedureisthenrepeatedfortheotherthreebooklets.

Theratingsheetsfilledinbytheparticipantsprovideasetofdatafortheanalysisofratingbehaviour.Thedataareusedtogiveextensivefeedbacktoallratersontheirinter-raterreliabilityandraterseverity.

TRAINING PHASE 6: UPDATE SESSION IN THE YEAR OF THE ACTUAL TESTThereareregionalstandardisationmeetingsforallraterswhomarkscriptsfromtheE8writingtests.Inthesesessionsratersareupdatedon,forexample,anychangesre-gardingtheassessmentscaleused.Thentimeisspentontheanalysisandinterpretati-onofthepromptsusedintheactualtest,andtheyratebenchmarkedscriptsbasedontheseprompts.Aftertheupdatetheon-siteratingsessionstarts,asdescribedabove,withslightadaptationsdependingonthenumberofpromptsusedinthetest.

Post-examadjustments

Althoughconsiderableeffortsaretakeninthetrainingprogrammetominimisedis-crepanciesinraterbehaviour,theratingsareadjustedforanyremainingdifferencesinraterseveritybymeansofmultifacetedRaschanalysisafterthescriptshavebeenmarked.Thisbecomespossiblebyhavingacertainproportionofscriptsmarkedbytwo(doublerating)andanotherproportionofthescriptsbyallraters(multiplera-ting)sothatraterbehaviourcanbeassessedintermsofmodelfitaswellasseverity.


Extremeraterbehaviour,rareatthisstage,isanalysedandcommunicatedtothera-tersaswellasdiscussedinratertrainingsessionsforsubsequenttestadministrations.

Reportingresults

ThepurposeoftheE8StandardsisgivingfeedbackonthewritingcompetenceofAustrianpupilsingrade8.Theaim,therefore,issystemmonitoringratherthancer-tificationorselectionatthelevelofindividualtesttakers.Consequently,whilethetestresultsarelinkedtotheCEFR,criticalcutgradesonwhichtobaseselectionde-cisionsneednotbeestablishedbythetestconstructors.Itishopedthatbyprovidingresultstoindividualteachersandschoolsthisfeedbackwillinstigateaqualitativede-velopmentthatwillradiatebeyondregionsandspreadthroughoutthewholeschoolsystem.Thewayfeedbackontheresultsisgiventotesttakersandotherstakeholdersisbeingdevelopedatthemoment.Incompliancewithpoliticalrequirements,onlythetesttakersthemselveswillhaveaccesstotheirindividualresultsthroughacodetheywillbegivenwhensittingtheexam.Teachersandschoolprincipalswillreceiveaggregateddataforthegrouprelevanttothem(class,school)viaaninternetplat-form.Educationalauthoritieswillreceiveprintedreports.

Theinformationthatresultsfromthewritingtestisreportedonthefourdimensi-onsoftheWritingScale(TaskAchievement,CoherenceandCohesion,Grammar,Vocabulary).The results for eachdimensionare reportedona scale from0 to7,whichenablesreferencetotheCEFRuptoB1.RatingsareadjustedfordifferencesinraterseverityandtaskdifficultybymeansofmultifacetedRaschanalysis.There-sultsarethereforecomparableacrossalltesttakersregardlessofwhichraterratedtheperformanceandwhatparticularprompttheperformanceisbasedon.TheprocessofstandardsettingandCEF-linkingwillbedescribedinmoredetailinatechnicalreportaftertheactualtestin2013.

Consequentialvalidity

Shaw&Weir(2007,218)taketheterm‘consequentialvalidity’fromMessik1989andinterpretitinthelightofrecentliteraturetoincludewashback(influencesonteaching, teachers, learning, curriculumandmaterials) and impact (influencesonthecommunityat large).TheE8Standardscanbeenvisagedasan instrumenttoinitiatechangesinthedirectionofpositiveorbeneficialwashback.

Inandaround2008newcoursebooksforteachingEnglishtothetargetgrouphavebeenlaunchedandanumberofthemclaimtobeinformedbytheCEFRandtheE8Standards.ThismeansthattextbookwritersarewellawareoftheE8StandardsTestsandareadaptingtheirmaterialstowardsthem.

Therequirementsforthewritingtestareclearlylaiddowninthisreportanddemon-stratewhatkindsofwritingourlearnersareexpectedtodeliver.

Second edition, 2011:The expectations of the test designers formulated in2008havebeenlargelyfulfilled3.ThreeyearslateragreatnumberofwritingtasksinthenewcoursebooksusedinAustrianschoolshavechangedinthedirectionindicatedintheTechnicalReport4of2008.Thereismuchlessscaffoldedwriting;thetasksarerealisticandauthentic;texttyperequirements,variationintexttypes,textlengthandtimeconstraintsareallinlinewiththepresenttestspecifications.Somecoursebooksalsoemphasisetheuseofparagraphsinwriting,givehintsonhowtowrite

3 Firstedition,2008:Itishopedthatthiswillleadtolessscaffoldedwriting,thusenhancinglearnerempower-ment.Theemphasisgiven tocoherenceandcohesion in theCEFRand theE8Standardsmightalso focusteacherattentiononthisareaandentailimprovements.




AnwenderRechteck

AnwenderRechteck


AnwenderText-BoxKonsequenzen


goodparagraphs,andprovidecorrespondingexercises.ThismeansthatAustriantesttakerswhosittheE8testsafter2011willbefamiliarwiththetestformat,theparti-cularrequirements,andtherubrics.

E8WritingTestSpecificationsVersion03(October2010)

TheguidingdocumentsforthedevelopmentofthewritingtestspecificationsfortheE8standardtestsaretheAustriancurriculum(AHS2006;APS2008),theAnlage zur Verordnung der Bundesministerin für Unterricht, Kunst und Kultur über Bildungs-standards im Schulwesen (BGBl.II Nr.1/2009 v. 2.1.2009)andthe Common European Framework of Reference for Languages (CEFR,2001).ThesedocumentslistwritingcompetencesatdifferentproficiencylevelsintermsoftheCEFR.

1.Purposeofthetest

Themainpurposeofthewritingtestistofindoutaboutstrengthsandweaknessesintesttakers‘writingcompetenceandtousethisinformationbothfortheimpro-vementofclassroomproceduresandsystemmonitoring.Whatismore,individualanddetailedtestresultsarereportedtothetesttakers,whichisofinteresttothetesttakersthemselvesandtheirparents.

2.Descriptionoftesttakers

Thetest takersareAustrianpupils in the threedifferent typesof lower secondaryschools,theGeneralSecondarySchool(APS)andtheAcademicSecondarySchool(AHS)andtheNeueMittelschule(APS)asanewcombinationofthesetwoformertypes towardstheendofgrade8(8.Schulstufe).PupilsfromallthreeabilitygroupsinAPSwillbetested.Themajorityoftesttakerswillbeaged14.

3.Testlevel

ThedifficultylevelofthetestissupposedtoencompasslevelsA2toB1intheCom-mon European Framework of Reference.

4.TestConstruct

In E8 testing the most significant competences needed for writing are identifiedforassessmentpurposes.Thisisfirstandforemostthecommunicativecompetencedemonstratedinanappropriateresponsetothetask.Inpracticaltermsthismeansthatallexpectedcontentpointsofthepromptsaretobeclearlyandmeaningfullymentionedby the test takers. For thehigher bands, elaborationof some contentpointsisrequired.Secondly,itistheabilitytoproducefluenttextbyusingadequatedevicestocreatecoherenceandcohesiononparagraphandtextlevel.Thirdly,agoodknowledgeofarangeofgrammaticalstructuresandtheabilitytousethemaccura-tely,andfourthly,thechoiceofvocabularythathasacertainrange,isaccurateandrelevanttothecontent.

Thereforethetestisdesignedtoelicitlanguagesamplesthatallowthecandidatestobeassessedinfourareas:TaskAchievement,CoherenceandCohesion,Grammar,andVocabulary.Thetwotasksareassessedseparatelybytrainedraters,usinganana-lyticratingscalebasedonthesefourdimensions.Multiple-ratinganddouble-ratingofasufficientlylargesampleofscriptsensurereliability.Differencesinraterseverityareadjustedforintheprocessofmulti-facetedRaschanalysis.

AnwenderRechteck

Anwenderwichtig







Pro

mp

t

Typ

e

CE

FR

-Lev

el

CE

FR

Des

crip

tor

Des

krip

tor

aus

BIS

T-V

O:

Sch

üler

/inn

enk

önn

en…

Top

icA

rea

Text

Typ

esIn

tent

ion/

Pur

po

se

Pri

mar

y

Aud

ienc

e

LongPrompt

B1

¡

Can

writ

eac

coun

tso

f

ex

perie

nces

,des

crib

ing

fe

elin

gsa

ndre

actio

nsin

si

mpl

eco

nnec

ted

text

.

¡

Can

writ

ea

desc

riptio

nof

an

eve

nt,a

rece

nttr

ip–

real

or

imag

ined

.

¡

Can

nar

rate

as

tory

.

¡

Can

writ

epe

rson

alle

tter

s

desc

ribin

gex

perie

nces

,

fe

elin

gsa

nde

vent

sin

som

e

de

tail.

¡

Erfa

hrun

gsbe

richt

e

sc

hrei

ben,

ind

enen

Gef

ühle

un

dR

eakt

ione

nin

ein

em

ei

nfac

hen,

zus

amm

en-

hä

ngen

den

Text

wie

der-

ge

gebe

nw

erde

n

¡

eine

Bes

chre

ibun

gei

nes

re

alen

ode

rfik

tiven

Ere

ig-

ni

sses

,z. B

.ein

erR

eise

,

ve

rfass

en

¡

eine

Ges

chic

hte

erzä

hlen

¡

ausf

ührli

cher

eK

arte

n,

pe

rsön

liche

Brie

feu

ndE

-Mai

ls

sc

hrei

ben

und

darin

auc

hüb

er

E

reig

niss

e,E

rfahr

unge

nun

d

G

efüh

leb

eric

hten

¡

Fam

ilieu

ndF

reun

de

¡

Woh

nen

und

Um

gebu

ng

¡

Ess

enu

ndT

rinke

n

¡

Kle

idun

g

¡

Kör

per

und

Ges

undh

eit

¡

Jahr

es-

und

Tage

sabl

auf

¡

Fest

eun

dFe

iern

¡

Kin

dhei

tund

Erw

achs

enw

erde

n

¡

Sch

ule

und

Arb

eits

wel

t

¡

Hob

bys

und

Inte

ress

en

¡

Um

gang

mit

Gel

d

¡

Erle

bnis

seu

ndF

anta

siew

elt

¡

Ged

anke

n,E

mpfi

ndun

gen

und

G

efüh

le

¡

Ein

stel

lung

enu

ndW

erte

¡

Um

wel

tund

Ges

ells

chaf

t

¡

Kul

tur,

Med

ien

und

Lite

ratu

r

¡

Inte

rkul

ture

lleu

ndla

ndes

kund

liche

A

spek

te

¡

(Fic

tiona

l)bi

ogra

phie

s

¡

Sim

ple

(tech

nica

l)de

scrip

tions

¡

Dia

rye

ntrie

s

¡

Dire

ctio

nsa

ndin

stru

ctio

ns

¡

E-M

ails

¡

Lette

rs(p

erso

nal;

advi

ce,a

pplic

atio

n)

¡

Mag

azin

ear

ticle

¡

Nar

rativ

ere

port

¡

Leng

thy

Pos

tcar

ds

¡

Sta

tem

ento

fper

sona

lvie

ws

and

op

inio

ns

¡

Sto

ries

(cre

ate

ane

ndin

g;g

iven

an

en

ding

–c

reat

ea

stor

y;u

sea

vi

sual

impu

lse

toc

reat

ea

stor

y;

pe

rson

al)

¡

Toc

onve

y

em

otio

ns,

fe

elin

gs

¡

Toin

form

¡

Toc

onvi

nce,

pe

rsua

de

¡

Toe

nter

tain

,

pl

ease

¡

Tok

eep

into

uch

¡

Tod

escr

ibe

¡

Tog

ive

di

rect

ions

and

in

stru

ctio

ns

¡

Sel

f

¡

Oth

ers

E8

Co

nstr

uct

Sp

ace

Tabl

e 1


Pro

mp

t

Typ

e

CE

FR

-Lev

el

CE

FR

Des

crip

tor

Des

krip

tor

aus

BIS

T-V

O:

Sch

üler

/inn

enk

önn

en…

Top

icA

rea

Text

Typ

esIn

tent

ion/

Pur

po

se

Pri

mar

y

Aud

ienc

e

ShortPrompt

A2+

¡

Can

writ

eab

oute

very

day

as

pect

sof

his

/her

en

viro

nmen

t,e.

g.p

eopl

e,

pl

aces

,ajo

bor

stu

dy

ex

perie

nce

inli

nked

s

ente

nces

.

¡

Can

writ

eve

rys

hort

,bas

ic

de

scrip

tions

ofe

vent

s,p

ast

ac

tiviti

esa

ndp

erso

nal

ex

perie

nces

.

¡

inF

orm

ver

bund

ener

Sät

ze

et

was

übe

rda

sal

ltägl

iche

U

mfe

lds

chre

iben

,wie

z. B

.

üb

erF

amilie

,and

ere

M

ensc

hen,

Ort

e,S

chul

e

¡

Fam

ilieu

ndF

reun

de

¡

Woh

nen

und

Um

gebu

ng

¡

Ess

enu

ndT

rinke

n

¡

Kle

idun

g

¡

Kör

per

und

Ges

undh

eit

¡

Jahr

es-

und

Tage

sabl

auf

¡

Fest

eun

dFe

iern

¡

Kin

dhei

tund

Erw

achs

enw

erde

n

¡

Sch

ule

und

Arb

eits

wel

t

¡

Hob

bys

und

Inte

ress

en

¡

Um

gang

mit

Gel

d

¡

Erle

bnis

seu

ndF

anta

siew

elt

¡

Ged

anke

n,E

mpfi

ndun

gen

und

G

efüh

le

¡

Ein

stel

lung

enu

ndW

erte

¡

Um

wel

tund

Ges

ells

chaf

t

¡

Kul

tur,

Med

ien

und

Lite

ratu

r

¡

Inte

rkul

ture

lleu

ndla

ndes

kund

liche

A

spek

te

¡

(Fic

tiona

l)bi

ogra

phie

s

¡

Sim

ple

(tech

nica

l)de

scrip

tions

¡

Dia

rye

ntrie

s

¡

Dire

ctio

nsa

ndin

stru

ctio

ns

¡

E-M

ails

¡

Lette

rs(p

erso

nal;

advi

ce,a

pplic

atio

n)

¡

Mag

azin

ear

ticle

¡

Not

es

¡

Not

ices

¡

Pos

tcar

ds

¡

Sta

tem

ento

fper

sona

lvie

ws

and

op

inio

ns

¡

Toc

onve

y

em

otio

ns,

fe

elin

gs

¡

Toin

form

¡

Toc

onvi

nce,

pe

rsua

de

¡

Toe

nter

tain

,

pl

ease

¡

Tok

eep

into

uch

¡

Tod

escr

ibe

¡

Tog

ive

di

rect

ions

and

in

stru

ctio

ns

¡

Sel

f

¡

Oth

ers

A2

¡

Can

writ

esh

ort,

sim

ple

fo

rmul

aic

note

sre

latin

gto

m

atte

rsin

are

aso

f

im

med

iate

nee

d.

¡

Can

writ

esh

ort,

sim

ple

im

agin

ary

biog

raph

ies

and

si

mpl

epo

ems

abou

tpeo

ple.

¡

kurz

e,e

infa

che

Not

izen

und

M

ittei

lung

ens

chre

iben

,die

si

cha

ufu

nmitt

elba

re

B

edür

fnis

seb

ezie

hen

¡

einf

ache

Tex

tez

. B.z

u

B

ildim

puls

eno

der

S

chlü

ssel

wör

tern

(key

wor

ds)

sc

hrei

ben

¡

kurz

e,e

infa

che

Bio

grafi

enu

nd

a

nder

eei

nfac

hefi

ktio

nale

Te

xte

schr

eibe

n

E8

Co

nstr

uct

Sp

ace

Tabl

e 2


Thetablesbelowsummarisetheconstructspacerelevantforitemdesign.ItliststheprompttypesusedtotestthewritingcompetencesasspecifiedintheBIST-Verord-nung, targetedatlevelsA2,A2+,andB1oftheCEFR.Thetasksattheselevelsaskfor(mostly)concretecontent.ThereforetopicsarerestrictedtoareasthatcansafelybeassumedtobefamiliartothetesttakersastheyaresetdownintheAustriancur-riculumandmusthavebeenincludedintheirEnglishlessons.

More specifically, the tasks display various text types and writing intentions/pur-poses.Table1providesanoverviewoftherangeoftexttypesandwritingintenti-ons/purposesfortheproficiencylevelstested.Fortheactualconstructionofwritingitemspromptwritersaregivenspecialprompt-design-specifications,whichclearlylistwhatkindofprompt–intermsofprompttype,level,BIST-Descriptor,topicarea,andtexttype–thepromptwriterissupposedtocreate.

5.Structureofthetest

Thetestcontains2sections.Section1consistsofashortwritingtaskwithanex-pectedresponseof40to70words.Section2consistsofalongwritingtaskwithanexpectedresponseof120to180words.

ThetwotaskswillbeassessedseparatelyonthebasisofthefourdimensionsoftheWritingScale.

6.Timeallocation

Totaltestingtimeavailable:45minutes.Timeforadministrationatthebeginning(handingouttestbooklets):5minutes.Time for administration at the end (word count and collecting test booklets): 5minutes.

Workingtime:35minutes.Theshorttaskshouldtakeabout10minutes,thelongtaskabout20,with5minutesforrevision.

7.Itemformats

Thecandidates’scriptswillbehandwrittenonthepagesprovidedinthetestbook-let.Thewritingtaskisguidedbypromptsthatensurethatthecandidatesproduceenoughlanguagethatmakesreliableandvalidassessmentpossible.

Thepromptsmaycontainblackandwhitepicturesordrawings.Theyneedtobeap-propriatefortheageandatalanguagelevelnohigherthanA2.Inputtextsshouldbeauthentic,ifatallpossible,andaslongasnecessarytocontextualisethetask.Ideally,theyshouldnotbelongerthan50words.

Promptsaredevelopedtobefreeofstereotypes.Theyoffertheopportunitytowritefromexperience,butaredesignednottointrudeonthestudents’personalfeelings.

8.LanguagelevelforInstructionsandPrompts

AllinstructionsandpromptsareinEnglish.However,theymustbeformulatedinlanguage that is well within reach of the candidates’ expected language level andtherefore easily understandable for all test takers.Test takers must not be put ata disadvantagebecause theyhavedifficultyunderstanding the instructions or theprompts.ThereadingcompetenceexpectedisCEFRlevelA2.

AnwenderRechteck


Promptsneedtogivethereasonforwriting,theaudienceandtherequiredtexttype.Thenumberofwordsfortherequiredlengthofthetextswillbeindicated.

9.WritingRatingScale

Pages23–25includethefourdimensionsoftheanalyticratingscaleforwritingandmoredetailedscalesforTaskAchievement.

TaskAchievement CoherenceandCohesion Grammar Vocabulary

7

¡ completetask achievement

¡ cohesiononboth sentenceandparagraph levelusingalimited numberofcohesivedevices¡ clearandcoherenttext

¡ goodrangeofstructures¡ relativelyhighdegreeof grammaticalcontroland fewinaccuracieswhichdo notimpaircommunication¡ messageclear

¡ goodrangeofvocabulary communicatingclearideas¡ generallyaccurate vocabulary¡ formulationssometimes variedtoavoidrepetition

6

5

¡ goodtask achievement

¡ goodsentencelevel cohesionasalinear sequenceonasimple level¡ someparagraphlevel coherenceandcohesion¡ fairlyclearandcoherent text

¡ generallysufficientrange ofstructures¡ occasionalinaccuracies whichcanimpair communication¡ messageclear

¡ sufficientrangeof vocabularycommunicating clearideas¡ occasionallyinaccurate vocabulary¡ majorerrorspossible whenexpressingmore complexideas

4

3

¡ sufficienttask achievement

¡ somesimplesentence levelcohesionusing simpleconnectorslike ‘and’,‘but’and‘because’¡ frequentlackof coherenceandcohesion onparagraphlevel¡ textoftenlacksclarityand coherence

¡ limitedrangeofsimple structures¡ frequentlyinaccuratewith basicmistakes,generally withoutcausingbreak- downofcommunication¡ messageusuallyclear

¡ limitedrangeof vocabularymostly communicatingclear ideas¡ frequentlyinaccurate vocabularycontrollinga narrowlexicalrepertoire¡ tendencytousephrases fromtheprompt

2

1

¡ sometask achievement

¡ basiclinearconnectors (‘and’,‘then’)onwordor wordgrouplevel¡ textnotcoherent

¡ extremelylimitedrangeof simplestructures¡ limitedcontrolcausing frequentbreakdownof communication¡ messageseldomclear

¡ extremelylimitedrangeof vocabularycommunicating fewclearideas¡ mostlyinaccurate vocabularyfrequently causingbreakdownof communication¡ severalchunksliftedfrom theprompt

0¡ notask achievement

¡ notenoughassessable language

• notenoughassessable language

¡ notenoughassessable language

WritingRatingScale(October2010)

AnwenderText-BoxAktuell (2016): bands 0 bis 4


Texttyperequirements:• Itisexpectedthattexttyperequirementsaremet.Iftheyarenotmet(missing/inappropriatesalutationorclosingformula;inappropriateregister),thereisdowngradingbyone band(twobandsifrequirementsarenotmetatall).

Textlength:• 120–180words:anythingbelow110wordswillbepenalised(downgradingbyone band).Fewerthan80words-downgradingbytwobands(ifthegeneralscoreisband 3orabove;otherwise,downgradebyoneband.).• Below-lengthanswers (50–79words):assessmentisconfinedtobands1and2. Answerscontainingfewerthan50wordsreceive0.• Over-lengthanswers(morethan180words):thewholeanswerisassessed.

ExtendedScales

7¡ completetaskachievementwith

¡ allcontentpointsmentioned&threeormoreofthemelaborated

6

5¡ goodtaskachievementwith

¡ 85%ofthecontentpointsmentioned&twoorthreeelaborated¡ orallcontentpointsmentionedandoneortwoelaborated

4

3¡ sufficienttaskachievementwith

¡ 65%ofthecontentpointsmentioned&oneortwoelaborated¡ orallcontentpointsmentionedwithoutelaboration

2

1¡ sometaskachievementwith

¡ 50%ofthecontentpointsmentioned&noelaboration

0 ¡ notenoughassessablelanguage

TaskAchievementLongTasks




Texttyperequirements:• Itisexpectedthattexttyperequirementsaremet.Iftheyarenotmet(missing/ inappropriatesalutationorclosingformula;inappropriateregister),thereisdowngrading byoneband(twobandsifrequirementsarenotmetatall).

Textlength:• 40–70words:anythingbelow30wordswillbepenalised(downgradingbyoneband).• Below-lengthanswers (fewerthan30words):assessmentisconfinedtobands1and2. Answerscontainingfewerthan20wordsreceive0.• Over-lengthanswers(morethan80words):thewholeanswerisassessed.

ExtendedScales

7¡ completetaskachievementwith

¡ allcontentpointsmentioned&oneortwoofthemelaborated

6

5¡ goodtaskachievementwith

¡ allcontentpointsmentioned&possiblyoneelaborated

4

3¡ sufficienttaskachievementwith

¡ 70%ofthecontentpointsmentioned&noelaboration¡ or50%ofthecontentpointsmentioned&oneelaborated

2

1¡ sometaskachievementwith

¡ 50%ofthecontentpointsmentioned&noelaboration

0 ¡ notenoughassessablelanguage

TaskAchievementShortTasks




10.Promptsandperformancesampleswithjustifications

Thetaskprototypesbelowaretakenfromthe2007E8WritingTest.ItisimportantthatthetasksarestructuredandcontainanumberofcontentpointssothatTaskAchievementcanbemeasured.4

Therubricsbelowhavebeenreducedtoaminimumbecausemost thingsarean-nouncedbythetestadministratorinGermanbeforethetesttakersopentheirtestbooklets.The rubrics inEnglish are to ensure that theykeep themainpoints inmind,butalsothatlearnerswithanL1languageotherthanGermanhavethesamefairchancetodothetask.

10.1Longtask

Instructions

LongpromptfromE8Testing2007

10.1.1Script1

4 YoufindmorepromptsontheBIFIEwebsite:http://www.bifie.at/freigegebene-items

Read the instructions carefully and then write your text on the answer sheet.Time: 20 minutesText: 120–180 words. Use paragraphs.In your text, try not to use language from the task below.

You have just moved to another town/village.

Write a letter to your American/English friend in which you tell him/her about your new situation.

Inform him/her about ● your new place of living ● the reason for movingDescribe ● the town/village you’re living in now (buildings, people,…) ● your new homeTell him/her about ● the first days of your ‘new life’ (new school, teachers,…) ● how you feel about your new situation

Dear Bill, how are you? Now I’m living in Vienna. That is in Austria. It is a very big city with nice people. There is also a fun fair called “Prater”. My parents got divorce and so I’m living here with my mother. There are wonderful buildings in this city like the animal park “Schönbrunn” and many castles. I like the river “Donau” very much, because I often go swimming there. My new house is very big and next to it is a forrest. I like that. The first day of my “new life” was not so good. When I came into my class most of the pupils laughed at me but the teacher was nice. I hope you will write back.

Yours, Raphael

(124 words)

AnwenderOval



AnwenderRechteck

AnwenderText-BoxTA: 5CC: 4G: 5V: 5


Justifications

TaskAchievement 5

Thetextmeets the text type requirements, iswithin theword limit set, andusesan informal register suitable fora letter toa friend.Thecandidateworkshiswaythroughthecontentpoints,onlyjusttouchingonthelastone.HowhefeelsneedstobeinferredfromstatementslikeThe first day of my ‘new life’ was not so good, most of the pupils laughed at me, and the teacher was nice.Alltheothercontentpointsarementioned.Whereas content point 3 shows good elaboration, the elaboration ofcontentpoints1and4islesssuccessful.ThepassageNow I’m living in Vienna. That is in Austria.canbetakenasmentioningcontentpoint1andthefollowingtwosen-tences(It is a very big city with nice people. There is also a fun fair called “Prater”.) canbeseenasanattemptatelaboration.

Withcontentpoint4elaborationisjustasthin.Thestatement My new house is very bigmentionscontentpoint4,butdoesverylittleinthewayofdescribingthenewhome(likeforinstancehowmanyrooms,whathis/herroomlookslike,etc.)andthesentenceadded(next to it is a forrest. I like that.)isnotreallydescriptive.Sothewritercannotbegivenmuchcreditforthisattemptatelaboration.

Thismeetsthedescriptorintheratingscaleforband5: good task achievement with all content points mentioned and one or two elaborated.

CoherenceandCohesion 4

Fromtheverystarttheideasdonotconnectwell.Theintroductoryquestionhowareyou?islefthangingtobefollowedbyNow I’m living in Vienna. Thethreeshortsen-tencesthatfollowaddsomeinformationaboutVienna.Butthenextidea(My parents got divorce)meetsthereaderunpreparedandthereisnolinktosmooththetransition.AtthispointitbecomesclearthatthethreesentencesaboutViennashouldactuallyhavebeenmovedtocontentpoint3.Thiswaythefirstunitofthetextcomprisingcontentpoints1and2wouldhaveflowedbetter.Textorganisationisbasedonthesequenceofthecontentpointsprovidedwiththewriter’shandpracticallyinvisible.

Thetextiscertainlycohesiveatsentencelevelbuthardlyatparagraphlevel.Movingfromoneideatotheother(tellingaboutthenewplace–thereasonformoving–describingthenewtown)maybeimpliedbytheorderofthecontentpoints,buttheabruptwaythishasbeendoneshowsthewriter’slimitations.

However,thetextis“fairlyclearandcoherent”.Sentencelevelcohesionisgoodandsomebasicconnectorsareusedtodelivera“linearsequence[ofpoints]onasimplelevel”.Thereissomeparagraphlevelcoherence,buttherearenotransitionsorlin-kingdevicesbetweenthevariousideaspresented.Whileband5couldbeconsideredforthisperformance,the lackofparagraphorganizationleadstodowngradingbyonebandtoband4.

Grammar 5

Atfirstsightthestructuresusedaregenerallysimple,manyofthesentencesareveryshort.Therearetwoidenticalcasesofthepresentprogressive(I’m living),fourusesofis, there is and there are,fourcasesofthepresentsimple(I like, I often go, I hope), threesimplepasttenseforms(was, came, laughed)andonefuture(will write).


Althoughallofthesetenseformsarebasic,theyseemsufficienttoexpressthewriter’sideasandtheyhavebeenusedcorrectly.Moreover,thewriterusescomplexsentencescorrectly,whichhintsatmorecomplexlanguagecompetences.

Thereareactuallythreesubordinateclauses (because, when), butthefinalsubordi-natorthatisomittedin I hope THAT you will write back. Andifonetakesacloselookattheothersentences,wefindgoodpost-modificationofnounphrases(big city WITH…, fun fair CALLED…, buildings…LIKE…). Even the coordinate clauseshaveatwisttothem…and SO…, and NEXT TO IT….Sothereisdefinitelysuffici-entcomplexitytojustifyband5.

Vocabulary 5

Thecandidatehasasufficientrangeofvocabularytoexpresshimself.Ontheonehand,thereisratherunexpectedvocabularyused(nearly)correctly(got divorce, fun fair, laugh at, forrest)asevidenceofagoodrangeofvocabulary,ontheother,thereisalsorepetitionofverysimpleexpressionssuchasnice, big and like,andtherearesomeoccasionalinaccuracies (got divorce, forrest). ThemajorityofwordsbelongtothemostfrequentlyusedbasicEnglishvocabulary,buttheideascommunicatedarealwaysclear.

Agoodrangeofvocabularycanonlyrarelybeseen,whilemostpartsofthetextdis-playasufficientrange.Whatisdefinitelylackingistheabilitytovaryformulationstoavoidrepetition(band7),sothetextisband5.

10.1.2Script2

Justifications

TaskAchievement 4

ThetextmentionsallcontentpointswiththeexceptionofCP2,wherereasonsformovingshouldbestated(=85%).ThereissomeelaborationofCP4bydescribingthesizeofthegardenandCP5bysupplyingthereaderwithsomeadditionalinfor-mationabouthisnewfriends(theyspeakverygoodEnglish)andhisachievementsatschool(lotsofEnglishlessonsandhimdoingwellinEnglish),althoughthisonlyimplicitlyreferstothecontentpoint“thefirstdaysofyournewlife”.CP6hasbeenliftedfromthepromptaddingthewordgood.

AsCPs1and6haveonlybeendealtwithinanextremelybasicwaybymerelymen-tioningSalzburgandliftingaphrasefromthepromptandtheelaborationofCPs4and5isratherweak,adowngradetoband4istheconsequence.

Hi Steven!In Salzburg it is very cool and I’m living in a small flat with 5 rooms two bedrooms a kitchen a livingroom and a bathroom. Our garden is not so big, but big enough for us. The building very beautiful and it give no skyscrapers and it is very hot. I always go in the garden and I lie in the sun. The people are very funny and they accept that I speak english. I have got two new friends and they speak very good english. The teachers are very good and we have a lot of english and I’m the best one, but in Deutsch I’m very bad. I feel very good with my new situation and I wish all my old friends and the teachers a good luck for the next time and I hope you always wish me a good next time.Yours Olav! (149 words)

AnwenderOval




Textorganizationisquitelowwithsimpleadditionasthedominatingstructuringprinciple.Thesimplest connectorandoccurswithundue frequency,proving thatideasaremostlystrungtogetherwithoutexpressinglogicalrelations.Apartfromthesecondsentence (Our garden is not so big, but big enough for us)thetextdoesnotreadwellbecausesomesentencesthatfolloweachotherhavelittleornoconnectionatcontentlevel,sothetextlackscoherence.Somechunksoflanguagethathavelittleincommonareoftenjoinedinonesentence,e.g.The building very beautiful and it give no skyscrapers and it is very hot.

Therefore,thistextischaracterisedbyanoticeablelackofclarityandcoherenceandsomeratherbasicsentencelevelcohesion.Thiswouldpointtowardsaweakband3.Astherearenoparagraphsthetexthastobedowngradedtoband2.

Grammar 3

Thewriteruses aquite limited rangeof simple structures correctly, repeating thesamebasicpatternwith littlevariation.Heonlyusespresenttensestructures,alt-houghCPs2and5wouldinvitetheuseofPastTense.Therefore,therangeofstruc-turescannotbeconsideredsufficientforthepurposesofthetask.EvenwithinthenarrowframeofPresentTensesentencestructuresthereisinappropriateuseofthecontinuous form in I’m living in a small flat. Thesimplemessage isusually clear,althoughanerrorsuchasit give no skyscrapers causesbreakdownofcommunication.Similarly, in the building very beautiful thisphrase leavesusundecidedwhether itshouldrefertothehousewherethewriterlivesorthebuildingsinSalzburg.Allthiswouldsuggestaveryweakband3,which,however, issupportedbytherelativelyhighdegreeofcorrectness.

Vocabulary 4

Thetextshowssuccessfulcontrolofalimitedrangeofvocabulary,withsomegoodphrasesstickingoutsuchasbig enough for us, skyscrapers, orthey accept that I speak english. Thesimplevocabularyusedinthefirstpartofthetextcommunicatesmostlyclear ideas,but in the last sentence thewriter seems tobeattempting toomuch,leavingthesafearea,andthisresultsinseveralbreakdowns(I wish them a good luck; L1:for the next time; I hope you always wish me a good next time)demonstratingthelimitations,asdoestheuseofDeutschforthesubjectGerman.Thenarrowlexicalrepertoireandalsothetendencytoliftphrasesfromtheprompt(I feel very good with my new situation)wouldindicateaband3,buttheoccasionalneatexpressionandthefactthatsevereproblemsappearonlywhentryingtoexpressamorecomplextrainofthought(toleratedatband5)justifyaweakband4.


10.2ShortTask

Instructions

ShortpromptfromE8Testing2007

10.2.1Script3

Justifications

TaskAchievement 7

Theregisterandthelayoutareclear indicationsthatthistext isanemail.Thesa-lutationandclosingformulasaremostappropriateandforthesereasonstexttyperequirementsareperfectlymet.Allcontentpointshavebeenmentionedandthereiselaborationofcontentpoint1 It was so great!, I liked the games we played, the food was excellentandcontentpoint4asthecandidatemakesenquiriesaboutanupco-mingeventinthenearfuture I nearly forgot it: Tanja’s birthday party is in two weeks, she invited me, are you invited too?,sowehavecompletetaskachievement–band7.


ThetextadmirablyincorporatesqualitiesofspokenEnglishLet’s talk about, Oh and, I nearly forgot, Okay I have to, whichonewouldexpectinaninformalemail,thatmakesitflowwell.Anumberofcohesivedevicesareusedtoconnectgroupsofsen-tencestogetherverywell,suchaslexicalcohesionparty-it-party,conjunctionbecause-and,backwardandforwardreferencingwe played-tell your mum that,andthere isevidenceofgood linearsequencingofpointsmaking itaclearandcoherenttext,pointing it towardsband7.Thereare,however, twoabruptchanges in the linearsequenceofthetextWhat are you going to doandI nearly forgot,butasparagraphsarenotexpectedinshorttextsitremainsaband7.

Read the instructions carefully and then write your text on the answer sheet.Time: 10 minutesText: 40–70 words. In your text, try not to use language from the task below.

Your friend’s birthday party was a few days ago. Write an e-mail to tell him/her that you liked the party.

• Tell him/her why you liked the party.• Tell him/her what you liked best.• Ask your friend when you are going to meet again.• Suggest something for the next weekend.

Dear Daisy, how are you? Let’s talk about your party. It was so great! I liked the party best, because Lukas was there. And I liked the games we played. Oh and tell you mum, that the food was excellent! What are you going to do on Sunday? Maybe we can go to cinema or swimming. Tell me please if you have time. I nearly forgot it: Tanja’s birthday party is in two weeks, she invited me, are you invited too? Okay I have to help my mum with dinner.Love you big kissYours,Aida (95 words)

AnwenderOval



Grammar 7

Thereisarelativelyhighdegreeofgrammaticalcontrolwithonlyoneslightslipinaccuracy.However,theomissionofthedefinitearticlegotocinemadoesnotimpaircommunication and themessage throughout the text is clear, suggestingband7.Thecandidate’sgooduseof thepresent,past,going-to-future,conditional,Saxongenitive,andasubordinateclausetoaddressallthepossiblefunctionssolicitedfromtheprompt:toinform,toaskhow are you?, What are you going to do on Sunday? are you invited too?,tosuggest Maybe we can go to cinema,indicatesthegoodrangeofstructuresthecandidateisabletouseaccurately,thussupportingastrongband7.TheverycasualsentenceanEnglish‘native-speaker’woulduseOkay I have to help my mum with dinner,inwhichthereisnouseofthedefinitearticle,isfurtherevidencethatclearlypointstoaband7.

Vocabulary 7

Thevocabulary,elicitedfromtheprompt,pointstowardsband7.Notonlydoesitcontainagoodvarietyofappropriateandaccuratecontentwordsexcellent, invite, dinner, butalsomanycollocationsthatareequallyasappropriateandaccuratewhichlendacertainnaturalnesstothetextthe food was excellent, going to do on Sunday, have time, in two weeks, I nearly forgot, love you, big kiss.Anotherindicatorforband7isthecandidate’schoiceofwordswhichenablehertogethermessageacrossveryclearlythroughoutthetext.ThephrasesLet’s talk about, Oh and, I nearly forgot, Okay I have tobestowacertain“chattiness”tothetext.Furthermore,theuseofLove you big kiss, asanalternativeoradditiontothecommonclosinglineYours, whichexemplifyhowthecandidatecanvaryformulationst

Testing Writing Specifications for the E8-Standards ...€¦ · 19 E8 Writing Test Specifications...

Documents

Transcript of Testing Writing Specifications for the E8-Standards ...€¦ · 19 E8 Writing Test Specifications...