22669 Rv Research Notes 47

8/20/2019 22669 Rv Research Notes 47

1/54

Research Notes

Issue 47February 2012

ISSN 1756-509X


2/54

Research NotesIssue 47 / February 2012A quarterly publication reporting on research, test development and validation

Guest Editor

Dr Jayanti Banerjee, Program Manager , Cambridge Michigan Language Assessments

Senior Editor and Editor

Dr Hanan Khalifa, Assistant Director , Research and Validation Group, Cambridge ESOLDr Fiona Barker, Senior Research and Validation Manager , Research and Validation Group, Cambridge ESOL

Editorial Board

Dr Nick Saville, Director , Research and Validation Group, Cambridge ESOLAngela ffrench, Assistant Director , Assessment and Operations Group, Cambridge ESOLRon Zeronis, Assessment Group Manager , Assessment and Operations Group, Cambridge ESOLDr Ivana Vidaković, Senior Research and Validation Manager , Research and Validation Group,Cambridge ESOL

Production Team

Caroline Warren, Research Support Administrator , Cambridge ESOLRachel Rudge, Marketing Production Controller , Cambridge ESOLJohn Savage, Editorial Assistant, Cambridge ESOL

Printed in the United Kingdom by Océ (UK) Ltd.


3/54

CAMBRIDGEESOLRESEARCHNOTES ISSUE FE BRUARY

© UCLES – The contents of this publication may not be reproduced without the written permission of the copyright holder

ResearchNotes

Editorial notes

WelcometoissueofResearchNotesourquarterlypublicationreportingonmattersrelatingto

researchtestdevelopmentandvalidationwithinUniversityofCambridgeESOLExaminations

Thisissue–thefirstof–presentstheresearchoutcomesfromthefirstroundofCambridge

ESOL’sFundedResearchProgrammeundertakeninItbenefitsfromtheguesteditorshipofDr

JayantiBanerjeeProgramManageratCambridgeMichiganLanguageAssessments

FollowingDrBanerjee’sguesteditorial(seethefollowingpage)whichdescribestheprojectsand

suggeststheirimpactforCambridgeESOLandmorewidelytherearefourarticlesbasedonthe

CambridgeESOLFundedResearchProgrammewhichcoverarangeoftopicsandcontextsrelevant

totheteachingortestingofCambridgeEnglishThereportedresearchincludesinvestigationsofthe

validityoftestitemsandcandidates’outputandtheimpactanduseofvariousCambridgeEnglish

testsintwospecificcontextsSuchstudiesenableCambridgeESOLtosupportresearchthatgoes

beyondthenormalrangeofstudiesweareabletocommissionorundertakeourselvesthereby

enhancingourunderstandingofthenatureandimpactofthelanguagetestsweworkwithonadaily

basisandadditionallyprovidingimportantoutsiderviewpointsfrombothestablishedandnewer

researchersinthelanguagetesting–orteaching–fieldsThesecondroundofresearchfundedbythisprogrammeisclosetocompletionandthethird

roundisalreadyunderwaysowelookforwardtoreportingonthesestudiesinfutureissuesof

ResearchNotesForthosereadersinspiredtosubmittheirownresearchproposalstheCallfor

ProposalsforthefourthroundisexpectedtobeavailableinAugustontheCambridgeESOL

ResearchandValidationwebsitesoforfurtherdetailsvisitwwwresearchCambridgeESOLorglater

thisyear

WefinishthisissuewithanupdateonALTEeventsfromMartinNuttalloftheALTESecretariat

theannouncementofthewinnersoftheCarolineClaphamIELTSMastersAwardand

theCambridgeILTALifetimeAchievementAwardanddetailsofthethvolumetobe

publishedintheStudiesinLanguageTestingseries

Withthenewcalendaryearwearethinkingofintroducingvariousinnovationsto Research

Notesandareplanningareadersurveylaterthisyeartohelpinformthefuturedirectionofthis

publication

Editorialnotes

Guesteditorial

JayantiBanerjee

Investigatinglearners’cognitiveprocessesduringacomputer-basedCAEReadingtest

StephenBaxandCyrilWeir

Investigatingfigurativeproficiencyatdifferentlevelsofsecondlanguagewriting

JeannetteLittlemoreTinaKrennmayrJamesTurnerandSarahTurner

TheattitudesofteachersandstudentstowardsaPET-basedcurriculumataJapanese

university

JunNagaoToruTadakiMakikoTakedaandPaulWicking

FCEexampreparationdiscoursesinsightsfromanethnographicstudy

DinaTsagari

ALTEbriefing

CarolineClaphamIELTSMastersAward

WinneroftheCambridgeILTALifetimeAchievementAward

StudiesinLanguageTesting

Contents


4/54



Guest editorialJAYANTI BANERJEE CAMBRIDGEMICHIGANLANGUAGEASSESSMENTSUSA

EnglishlanguagetestsmatterTheymatterforthechildrenwhoarecompilingtheirlanguageportfoliosaswellas

foryoungadultshopingtostudyinanEnglish-medium

universityTheymatterforuniversityadmissionspersonnel

oremployerswhoareselectingthebestcandidatesfortheir

degreeprogrammesorjobsEnglishlanguagetestshave

tremendoussymbolicpower(Shohamy)because

theyconferaccesstoprivilegescertifyandbyextension

delimitknowledge

AsaresultprovidersofEnglishlanguagetestshave

agreatresponsibilitytostakeholdersTestusersrely

ontestdeveloperstoprovidehigh-qualityteststhat

meetprofessionalstandardsTheyalsoexpecttestingorganisationstopresentevidencetosupporttestscore

interpretationsandusesCambridgeESOLtakesthese

professionalresponsibilitiesseriouslyandhasdevelopeda

PrinciplesofGoodPracticebooklet(wwwCambridgeESOL

orgaboutstandardspogphtml)thatencapsulatesthe

organisation’scommitmenttofiveessentialprinciples

validityreliabilityimpactpracticalityandquality

Aspartofthiscommitmentinlatetheorganisation

launchedtheCambridgeESOLFundedResearchProgramme

ThefirstCallforProposalsencouragedstudiesofits

CambridgeEnglishexamsinthefollowingareas

• testvalidationissues• issuesrelatingtocontextsoftestuse

• issuesoftestimpact

ThisissueofResearchNotesshowcasesthefourprojects

thatwerefundedinthefirstroundandwhichtookplace

inEachstudyprovidesinsightintooneormore

CambridgeEnglishexaminationsinaspecificcontextorfrom

aspecificperspective

BaxandWeir(thisissue)haveinvestigatedthecognitive

processesemployedbyparticipantsonacomputer-based

CambridgeEnglishAdvanced(CAE)Readingtestinorderto

checktheextenttowhichtheitemselicittherangeandlevelofcognitiveprocessesexpectedofanadvancedlevelReading

testwhichseekstoemulatereal-worldacademicreading

processesTheyusedeye-trackingtechnologytocollectdata

intheformofGazePlotsandHeatMapswhichindicateboth

howthevolunteertesttakers’eyesmovedwhenreading

theinputtextsandansweringthequestionsaswellashow

longthetesttakerslookedatparticularsectionsofthetext

BaxandWeiralsoadministeredquestionnairestocapture

immediateretrospectionsfromtesttakersTheresultingdata

confirmedthatthetesttakersemployedanappropriaterange

andlevelofcognitiveprocessesastargetedbyCAE items

Thepapernotonlyprovidesevidenceforthevalidityofthe

CAE Readingsectionbutitalsodemonstratesthevalueofeye-trackingtechnologyintestvalidation

LittlemoreKrennmayrTurnerandTurner(thisissue)have

analysedasubsetofexamscriptsfromtheCambridgeLearner

Corpustoinvestigatethefeaturesofmetaphorthatdistinguish

performancesatdifferentlevelsoftheCommonEuropean

FrameworkofReference(CEFRCouncilofEurope)

UsingtheMetaphorIdentificationProcedure(MIP)developed

bythePragglejazGroup()Littlemoreetalfoundthat

metaphoruseincreaseswithproficiencylevelMetaphor

clustersemergeonlyattheintermediatelevelsLittlemoreet

alalsofoundthatthetypesofmetaphorsusedchangeswith

proficiencylevelaswellasthefunctionsthesemetaphorsperformThesefindingssuggestthatdescriptorsformetaphor

usecouldfeasiblybeincorporatedintoratingscalesforwriting

NagaoTadakiTakedaandWickingandTsagari(thisissue)

havefocusedontestuseinspecificcontextsNagaoetal

haveinvestigatedattitudestowardstheCambridgeEnglish

Preliminary(PET)inJapananemergingmarketforthetest

ThisstudyisparticularlyinterestingbecausethePET is

relativelynewinJapanandthestudyhascapturedknowledge

abouttheexamaswellasattitudestowardsitataveryearly

stageofitsintroductionThestudyshowsthatthetestdoes

meetlearners’needsbutislesspopularwithteachersIt

identifiestheneedforteachersupportprogrammesandit

alsoshedssomelightonthePET ’sfitnessforpurposeintheJapanesecontext

TsagarihasstudiedCambridgeEnglishFirst(FCE)test

preparationclassesinCyprusThroughacombinationof

classroomobservationsandteacherinterviewsTsagari

amassedarichdescriptionofthelearningactivitiesand

teachertalkShefoundconsiderableinfluenceofthetest

uponthelearningactivitiesintheclassroomandalsoin

theteachertalkparticularlytheadvicethatteachersgave

totheirstudentsSomeofthisinfluencewasverypositive

buttherewerealsobarrierstopositiveimpactTsagari

pointsoutthattheteacherswerenotanopenconduitof

informationabouttheexamRathertheimpactoftheFCE upontheclassroomwasmediatedthroughtheteachers’

knowledgeandbeliefsabouttheexamtheirprofessional

skillsandtheirownlanguageabilityAssuchinadditionto

providingawindowintoFCE preparationclassesthisstudy

hasidentifiedstakeholderneedsinGreece

Anastasi()andCronbach()remindusthat

theprocessofgatheringvalidityevidenceisnevercomplete

Indeedthemoreimportantandinfluentialatestthegreater

theneedforcollectingongoingevidenceforthevalidityofits

useTogetherthesepaperscontributetothegrowingbody

ofvalidityevidencefortheCambridgeESOLGeneralEnglish

examinations

ForpromotionalpurposesCambridgeESOLincreasingreferstoitsexamsbytitlessuchasCambridge EnglishKey Preliminary FirstAdvancedandProficiency althoughthenames

of theexamsthemselveshavenotchangedOurauthorsfrequentlyrefertotheexamsbytheiracronyms–KET PET FCE CAE andCPE respectivelyFormoreinformationsee

wwwcambridgeesolorgexams


5/54



ReferencesAnastasiA()EvolvingconceptsoftestvalidationAnnualReview

ofPsychology –

CouncilofEurope()CommonEuropeanFrameworkofReferencefor

LanguagesLearningteachingassessmentCambridgeCambridge

UniversityPress

CronbachLJ()Fiveperspectivesonthevalidityargumentin

WainerHandBraunHI(Eds)TestValidityHillsdaleNJLawrence

ErlbaumAssociates–

PragglejazGroup()MIPAmethodforidentifying

metaphorically usedwordsindiscourseMetaphorandSymbol

()–

ShohamyE()ThePowerofTestsHarlowEssexPearson

EducationLimited

Investigating learners’ cognitive processes during acomputer-based CAE Reading testSTEPHEN BAX CRELLAUNIVERSITYOFBEDFORDSHIREUK

CYRIL WEIR CRELLAUNIVERSITYOFBEDFORDSHIREUK

Introduction

Thisstudyinvestigatesthecognitiveprocessesemployedby

participantsonacomputer-basedCAE Readingtestwitha

viewtoassessingthecognitivevalidityoftheReadingtest

itemsIttakesasitsstartingpointthecognitiveprocessing

approachwithitssetofcognitiveprocessesdescribedby

KhalifaandWeir(Chapter)Inadditionitdrawson

themethodsforinvestigatingthoseprocessesadoptedin

WeirHawkeyGreenandDevi’sstudyofacademicreading

intheUK()andcomplementsandextendsthemusing

onscreenrecordingandeye-trackingtechnologyaswellasdevelopingotheraspectsofthemethodologyThecentral

questionwastowhatextentthetestitemselicitedtherange

andlevelofcognitiveprocessesexpectedofanadvanced

Readingtestwhichseekstoemulatereal-worldacademic

readingprocesses

Intheeventinsightsfromeyetrackingcombinedwith

questionnairedatatoprovideconvincingevidencethateven

thislimitedsetofCAE testitemssucceededinelicitinga

widerangeofappropriatecognitiveprocessesincluding

thosehigherlevelreadingprocessesnecessaryforreal-world

academicreadingDataincludingGazePlotsandHeatMaps

illustratingparticipants’eyemovementsindicatedthattest

takerssuccessfullyemployedanappropriaterangeandlevel

ofcognitiveprocessesastargetedbytheCAE itemsInthe

processtheprojectalsodemonstratedthateye-tracking

technologyincarefulcombinationwithmoretraditional

methodsofanalysishasthepotentialsignificantlyto

improveourcapacitytovalidateReadingtestitemsinfuture

Rationale

Itisaxiomaticthatlanguagetestsassessingtheacademic

languageproficiencyofoverseasstudentsiftheyareto

beappropriateforuniversityadmissionshouldreflectthe

demandsoftheacademiccoursesthesestudentsareaimingtofollowInadditioninternationalexaminationboardshave

adutytoprovidevalidinformationforstakeholdersandto

demonstratequality

Oneaspectofsuchlanguagetestswhichshouldbe

demonstrablyvalidistheextenttowhichtheyassess

thecognitiveprocessesrequiredinacademicstudyFor

exampleifanadvancedReadingtestistobeaccepted

asvalidbyacademicinstitutionsitshoulddemonstrably

testtherangeandlevelofcognitiveprocessestypically

expectedinacademicstudycontextsincludingcognitive

processingatlowerandhigherlevelsIfitfailstodoso–for

exampleifittestsonlyalimitedrangeofprocessesoronly

low-levelcognitiveprocesses–thenitcannotclaimtobe

anappropriatetoolforassessingtheacademiclanguage

competencerequiredatuniversitylevel

Thisistoinsistonwhatisknownascognitivevalidity

Sincethesithasbeenarguedthattestsassessing

complexcognitiveconstructsshouldestablishthissort

ofvalidity(GlaserBaxterandGlaser)since

cognitiveinterpretativeclaimsare‘notforegoneconclusions

[but]needtobewarrantedconceptuallyandempirically’

(Ruiz-PrimoShavelsonandSchultz)Bythesame

tokenWeirhasarguedthatthoselanguagetestswhich

implicitlyorexplicitlyclaimtomatchreal-worldbehaviour

shouldalsobecognitivelyvalid(Weir)Inshortifa

languagetestdoesnotelicitfromtesttakersthesametype

andlevelofcognitiveprocessingasisusedandexpectedinthereal-worldtargetsituationthenitisnotavalid

instrumentforassessingthatareaoflinguisticbehaviourIt

istheseissuesconcerningtherangeandtypeofcognitive

processinginCAE onscreenreadingtestswhichthecurrent

projectsoughttoinvestigate

Traditionallyresearchintoreaders’cognitiveprocesses

hasdependedheavilyonretrospectiveorconcurrentverbal

reportingasameansofunderstandingwhatreadersare

thinkingastheycompleteReadingtestitemsRecent

improvementsineye-trackingtechnologyhoweverfurnish

additionalopportunitiestogaininsightsintoreaders’actual

asopposedtoreportedbehaviourpermittingsignificantly

enhancedinsightsintotheirongoingsecond-by-secondreadingactivityandhenceagreaterinsightintotheir

probablecognitiveprocessing


6/54



Researching cognitive processes inacademic reading

InWeirHawkeyGreenandDevi’sstudy()ofreading

foracademicpurposesinUKuniversitiesanumberofearlier

modelsofreadingprovedtobehelpfulespeciallythose

thataccountedforthepurposefulandstrategicactivitiesof

readersinanacademiccontextandthosewhichspecifiedthetypesofreadingrelevanttothatacademiccontext

(seeWeiretalforafulldescriptionofthese)Asthe

authorsnote

ingeneraltermsthereadingtypescovered[inanacademiccontext]are

expeditiousreadingie quickselectiveand efficientreadingto access

desiredinformationina text(scanningskimmingandsearchreading)

andcarefulreadingieprocessingatextthoroughlywiththeintention

toextractcomplete meaningsfrom presentedmaterial (Weir Hawkey

GreenandDevi)

UrquhartandWeir’s()distinctionsbetweenglobal

localandcarefulexpeditiousareofparticularimportanceto

thedesignofthisstudyastheyofferataxonomyofdifferent

typesofreadingwhicharerelevanttoreadingacademic

EnglishGlobalcomprehensionreferstotheunderstandingof

informationbeyondthesentenceincludingmainideasthe

linksbetweenideasinthetextandthewayinwhichthese

areelaboratedItinvolvesintegratinginformationinthe

textmentalmodelbuildingandunderstandinghowmacro

propositionsinthewholetextfittogetherThereaderin

carefulglobalreadingattemptstoidentifythemainidea(s)

byreconstructingthemacro-structureofatextLogical

orrhetoricalrelationshipsbetweenideasarerepresented

incomplexesofpropositions(seeVipond)often

representedbythewriterbymeansofparagraphingglobalreadinginvolvesattemptingtoreconstructthese

complexesLocalcomprehensionconcernstheunderstanding

ofpropositionswithinthesentence(individualphrases

clausesandsentences)Localcomprehensioninvolves

wordrecognitionlexicalaccessandsyntacticparsingand

establishingexplicitpropositionalmeaningatthephrase

clauseandsentencelevelCarefulreadinginvolvesextracting

completemeaningfromatextwhetherattheglobalorlocal

levelAsnotedabovethisisbasedonslowcarefullinear

incrementalreadingforcomprehensionExpeditiousreading

incontrastinvolvesquickselectiveandefficientreadingto

accessrelevantinformationinatext

Carefulreadingasanumbrellatermencompasses

processingatsentenceintersententialtextandmulti-text

levelsItisimportantthattestsdesignedtopredictthe

abilitytoreadinEnglishatuniversitylevelhavearangeof

itemswhichextendbeyondcomprehensionatthesentence

levelietheyshouldcontainahighproportionofitemsthat

testreadingatthemorecomplexstagesofprocessing(see

KhalifaandWeirfordiscussionofthese)Inacademic

lifereadersfindthemselveshavingtoreadandlearnfroma

wholetextaswellasintegratinginformationfromvarious

textsespeciallyforthepreparationofassignmentsTests

whichfocusonsentence-levelprocessingalonearetherefore

notthebestindicatorsofacademicreadingabilityTypicallyinthepastmodelsofreadinghaveusually

beendevelopedwithonlycarefulreadinginmind(seefor

exampleHooverandTunmerRaynerandPollatsek

)Howevercarefulreadingmodelshavelittletotell

usabouthowskilledreaderscopewithotherexpeditious

readingbehaviourssuchasskimmingforgist(Raynerand

Pollatsek–)Carver()andKhalifaand

Weir()suggestthatthespeedandefficiencyofreading

isimportantaswellascomprehensionInrelationtoreading

foruniversitystudyWeiretal()foundthatintheirsampleofuniversityundergraduates‘formanyreaders

readingquicklyselectivelyandefficientlyposedgreater

problemsthanreadingcarefullyandefficiently’Khalifaand

Weir()distinguishthreetypesofexpeditiousreading

skillrelevanttoacademicstudyscanningskimmingand

searchreadingScanningisaformofexpeditiousreading

thatoccursatthelocallevelItinvolvesreadinghighly

selectivelytofindspecificwordsfiguresorphrasesinatext

Skimmingisgenerallydefined(UrquhartandWeir

Weir)asreadingquicklybysamplingtexttoabstract

thegistgeneralimpressionandorsuperordinateidea

skimmingrelatesexclusivelytoglobalreading

Unlikeskimmingsearchreadinginvolvespredetermined

topicsThereaderdoesnotnecessarilyhavetoestablisha

macro-propositionalstructureforthewholeofthetextbutis

ratherseekinginformationthatmatchestheirrequirements

Howeverunlikescanning(whereexactwordmatches

aresought)thesearchisnotforexactwordmatchesbut

forwordsinthesamesemanticfieldasthedesiredtarget

informationSearchreadingcaninvolvebothlocaland

global-levelreadingWherethedesiredinformationcanbe

foundwithinasinglesentencethesearchreadingwould

beclassifiedaslocalandwhereinformationhastobe

constructedacrosssentencesitwouldbeseenasglobal

Searchreadingatthegloballevelisthekeyexpeditiousreadingskillforuniversitystudents

KhalifaandWeir’s()exegesisaddsafurtherlayer

tothisdepictionbyidentifyingthecognitiveprocesses

thatunderliethetypesofreadingrelevanttotheacademic

contextandthecognitiveloadimposedonthatprocessing

bythevariouscontextualparametersofthetextitself(in

termsoflexicalandsyntacticcomplexityandcohesion)

Theyarguethatreadingproficiencyisafunctionofboth

thelevelofprocessingrequiredbythereadingtaskand

thecomplexityofthereadingtextitiscarriedoutonIn

thisstudywearefocusingonthenatureoftheprocessing

requiredbyreadingtasksattheitemlevelFordetailsoftext

complexityiecontextualparametersinreadingthereader

isreferredtoKhalifaandWeir(Chapter)

Ourresearchstudyinvestigatesparticipants’processing

ofasmallnumberofCAE ReadingtestitemsOurinterestis

intheextenttowhichtheitemselicittherangeandlevelof

cognitiveprocessesrelevanttoacademicstudyinEnglishIf

theitemsonlyelicitcognitiveprocessesatalowerlevelof

complexity(wordrecognitionlexicalaccessandsyntactic

parsingandestablishingexplicitpropositionalmeaningat

thephraseclauseandsentencelevel)thentheirvalidity

forassessingacademicreadinginEnglishisinquestionif

howeverourrelativelysmallsampleofCAE Readingtest

itemsdemonstrablyelicitawiderrangeofprocessingintermsoftheKhalifaandWeirprocessingmodelincluding

integrationofinformationbuildingamentalmodelofa


7/54



textaswellastext-levelcomprehensionthisisapositive

starttoestablishingcognitivevalidityevidenceforthe

claimofCAE tobeanappropriatetestofreadinginEnglish

foracademicpurposesObviouslywewouldeventually

needtolookatalargersampleofCAE Readingitemsin

termsoftheir cognitive validityandalsoinvestigateCAE

Readingtexts intermsoftheircomparabilitytothosemetin

academiclifetoestablishmoresubstantialevidenceofthetest’sconstructvalidity

KhalifaandWeir’s()modelaccountsforthedifferent

typesofreadingthatreadersmightchoosetocarryoutin

academiclifethedifferentlevelsofprocessingthatmight

beactivatedandtheknowledgebasenecessarytocomplete

anassignedreadingtasksuccessfullyThismodelprovided

uswiththetheoreticalframeworkonwhichouronscreen

retrospectionquestionnairewasbasedandalsoonwhich

ouranalysisoftheeye-trackingdatawasstructured

A processing approach to investigatingreading

Weiretal(–)reviewtheliteratureonwhathas

beencalledasubskillsapproachtotestingreadingwhichis

basedontheassumptionthatitispossibletotargetparticulartypesof

itemortesttasktospecifictypesofreadingsothatoneitemmighttarget

theabilitytounderstandthemeaningofanindividualwordinatextand

anothermighttargettheabilityto extracttheoverallmeaningofatext

withinaverylimitedtimeframe(skimming)

Theyalsonotethat‘[t]hedebateoversubskillscentred

ontheabilityofexpertjudgestoarriveataconsensus

aboutwhatwasbeingtestedandtheessentialroleofthecandidatewaslargelyoverlookedThemajorityofstudies

paidsurprisinglylittleattentiontothecognitiveprocessing

requiredforcandidatestocarryouttesttasks’(Weiretal

)andthenciteAlderson()whoargues

that

[t]he validity of a test relates to the interpretation of the correct

responsestoitemssowhatmattersisnotwhatthetestconstructors

believeanitemtobetestingbutwhichresponsesareconsideredcorrect

andwhatprocessunderliesthem

Inshortunderstandingofthetraitbeingmeasured

requiresaninsightintothecognitiveprocessingrequiredfor

completionofthetask

Eye tracking in the study of cognitiveprocesses in reading

Inanattempttogaininsightintoreaders’cognitive

processesmanyresearchershaveadoptedproceduresin

whichparticipantsreportretrospectivelyonthelinguistic

processwhichtheyhaveengagedinHowevergiventhe

doubtssometimesexpressedabouttheuseofretrospective

reportingforexamplebyAfflerbachandJohnston()

andCordonandDay()andsinceeye-trackingtechnologyhasimprovedconsiderablyinrecentyears

wedecidedtomakeuseofeyetrackingtechnologyinthis

studyinordertogainbetteralbeitstillindirectinsightinto

cognitiveprocessingincombinationwitharetrospective

questionnaireinwaystobedetailedbelow

Theuseofeyetrackinginthestudyofreadingisnot

newRayner()reviewsyearsofresearchinto

readingusingeyetrackingofvarioussortsdividedinto

threeperiodsbeforewereachwhatDuchowski()

hascalledthecurrent‘fourthera’distinguishedbythe

possibilityofinteractivityRaynerhighlightssomeofthemaininsightswhicheyetrackinghasofferedforour

understandingofreadingFirstlywhenreadingEnglishitis

notedthateyefixations(whentheeyedwellsmomentarily

onaparticularpoint)typicallylastabout–

millisecondsandthemeansaccadesize(iewhenthe

eyemovesfromonepointtoanother)is–letterspaces

(Rayner)Thisisofinterestinthepresent

studyparticularlywhenidentifyingindividualwordsina

textwhichconstitutetheanswertoatestitemSecond

eyemovementsareinfluencedbynumeroustextualand

typographicalvariablesforexample‘astextbecomes

conceptuallymoredifficultfixationdurationincreases

saccadelengthdecreasesandthefrequencyofregressions

[wheretheeyemovesbackratherthanforwards]

increases’(ibid)whichcouldpotentiallybeusefulin

comparingbetterandworsereadersalthoughthisisnota

focusofthecurrentstudy

ImportantlyforthecurrentprojectRayneralsonotesthat

thebasicthemeofhishistoricalreviewinparticularofthe

thirderafromthesonwards‘isthateyemovement

datareflectmoment-to-momentcognitiveprocesses’

(Rayner)Heexpandsthepointasfollows

A crucial point that has emerged recently is that eye movement

measures canbeusedtoinfermoment-to-momentcognitiveprocesses

inreading andthatthe variability inthe measures reflectson-line

processing For example there is now abundant evidence that the

frequencyofafixatedwordinfluenceshowlongreaderslookattheword

(Rayner)

MorerecentstudiesconcurwithRaynerastothevalue

ofeyetrackingforresearchingcognitiveprocessesSpivey

RichardsonandDale()offeradetaileddiscussion

ofhowandwhyeyemovementscanbetakentobe

goodindicatorsofcognitiveprocessesandtermthem

‘awindowintolanguageandcognition’()The

samemetaphorisusedbySalvucciandGoldbergwho

seeeyetrackingas‘awindowintoobservers’visualand

cognitiveprocesses’(seealsoAnsonRashidHornandSchwegler)Someresearcherssuchasde

Greef Botzer and VanMaanen()takethistoextremes

suggesting–toquotethetitleoftheirarticle–that‘Eye-

TrackingReadingtheMind’butthisisarguablyover-

confidentItisourpositionthatalthoughthetechnology

offerspossiblythebestavailableinsightintocognitive

processeseye-trackingdatashouldbetreatedasmerely

indicativeofcognitiveprocessingratherthanatrueandfull

reflectionofit

Intermsofdevelopmentsineye-trackingtechnology

recentadvanceshaveimprovedimmeasurablyourability

todetectwhatreadersarelookingatsecondbysecondallowingthedetailedanalysisofindividualdifferences

betweenreadersataveryhighlevelofdetail(seeeg

BertramBuscherBiedert HeineschandDengel


8/54



EgerBallStevensandDodd)aswellastheanalysis

ofhighlyprecisefixationandsaccadepatterns

Research methodology

Inthissectionweoutlinetheresearchdesigninstruments

usedeye-trackingsoftwareandhardwarealongwiththe participants

Researchinstruments

Oneresearchtoolusedinourstudywastheretrospective

reportAsWeiretal()note

[a]process-oriented approachto defining readingactivityin language

tests seeks anexperimental method which permits comment onthe

actualreadingprocessitself

Participantsintheirparticularstudyweregivenonepart

ofanIELTSReadingtestchosenbyanexpertfocusgroup

soastoincludearangeofitemsrequiringbothexplicitly

statedandimplicitinformationlocatedacrosssentences

andallowingbothexpeditiousandcarefulreadingtypesand

werethenaskedtocompletearetrospectionformAmong

otherthingsthisforminvestigatedtheprocessesthat

participantsengagedinwhilelocatingthecorrectanswerto

eachindividualitemoftheReadingtest

Onelimitationofthatapproachisthattesttakersneeded

tocompletethefullsetoftestitemsbeforecompleting

theretrospectivequestionnairesothattheirrecallofthe

cognitiveprocessestheyhademployedwasnecessarily

delayedTheliteratureonstimulatedrecall(egGassand

Mackey)emphasisesthefactthatthesoonerafter

theexperiencetherecalliselicitedthemorelikelyitistobeaccurateForthisreasonourstudymakesuseofthe

flexibilityofcomputer-baseddeliveryinordertoelicitrecall

ofthecognitiveprocessesimmediatelyaftercompletion

ofeachtestitemGiventhefactthattheirretrospection

isthereforealmostimmediateitwasanticipatedthat

thisapproachwouldaffordgreaterreliabilityinterms

ofparticipants’introspectionconcerningthecognitive

processestheyemployedforeachtestitem

Design

TheoriginalCAE testusedproducedbyCambridgeESOLin

AdobeFlashformatwasreproducedsoastobeidenticalin

everyway(usingAdobeFlash)andallowmaximumcontrol

overfontsizeinteractivityanddesignandwaslinkedtoa

localdatabasetoallowformoreefficientdataanalysisand

processingTheonlydifferencefromtheoriginalwasthe

insertionofabriefinteractivequestionnairebetweeneach

testitemforreasonsdescribedaboveTesttakerstherefore

hadanear-identicalexperiencetothatofreal-worldCAE

CBTtesttakers

Eyetrackingtechnicalspecifications

TheeyetrackerusedwasaTobiiTUnlikemosteye

trackersthesenewdevicesdispensewithchinrestshelmets

andotherdistractionsandinadditionthetrackingcamerasarehiddeninthemonitorcasingensuringthatusers’

behaviourisasnaturalaspossiblewithoutunwarranted

intrusionontheirmentalprocessingTheTsamplerate

isHzpersecondwhichallowsdetailedtrackingof

normalreadinganditwassettoascreenrecordingrate

offramespersecond(Fulltechnicalspecificationscan

befoundatwwwtobiicom)Inadditionthedevicewas

furnishedwithbinoculartracking(ratherthantrackingon

one eyeonly)ausercameraandspeakersforplayingthe

tutorialsoundtrack

Participants

Onehundredandthreemultinationalparticipants

studying at aUKuniversityrepresentingmorethan

nationalitiesandlanguagegroupsandranginginacademic

levelfrompre-universityFoundationyearstudents

(n)toYear(n)andYearundergraduate

students(n) completedthetestitemsfromtheCAE

computerised ReadingtestdescribedbelowAgesranged

from–(n%)–(n%)and

–(n%)

Thetest-takingactivityofasampleofthese(n

%)wasrecordedusingTobiiscreenrecordingsoftware

whichcapturedeverykeypressmousemovementeye

movementandfacialexpressionThesampleselectedfor

eyetrackingwasweightedtoensuregoodrepresentation

acrossallacademiclevelssothattheeye-trackingdata

coveredstudentsatFoundationlevelYearandYear

undergraduatelevelsApartfromthatselectionwasrandom

Allstudentssignedappropriateethicsformsandpersonal

informationformsInadditiontheywereaskedtoratetheir

ownfamiliaritywithcomputersingeneralandonscreentests

inparticularAswasexpectedwiththisyoungandeducated

groupallreportedextensivefamiliaritywithcomputer

technologyandonscreentestsofvariouskinds

Testitems

TheoriginalCAE testconsistedofsixtextsandatotalof

multiple-choice(MC)itemsTimeconstraintsandtechnical

constraints(describedbelow)ledtotheselectionoffourof

thesetextswithatotaloftestitems(Partsandwith

items–and–)

IntheoriginalCAE testPartconsistedofthreeshort

textswithtwoMCitemsoneachatotalofsixitemsallof

whichwereincludedinourtestPartoftheoriginal CAE

test(withquestions–)consistedofataskinwhichtest

takersdraganddropcorrectpartsofatextintoplaceto

completethewholeThiscouldinprinciplebeeyetracked

foreachparticipantbutgiventhehugevariationinscrolling

anddraggingbehaviouritwouldbecomplextocompareany

twoparticipants’behaviourthroughaneye-trackingdevice

soforthisreasonPartwasomittedPart(items–)

consistedofasinglelongtextwithasidescrollbarand

althoughthispresentedsimilaranalyticalproblemsinterms

ofcomparingeye-trackingbehaviouracrosscandidates

itwasnonethelessincludedowingtotheimportanceof

testingparticipants’readingoverlongerstretchesthan

theshorttextsinPartWithrespecttoPartgiventhe

inadvisabilityoftrackingeyemovementsovertoolengthy

aperioditwasdecidedtoomitthislastsectiontoensurethatthewholetestwouldtakenomorethanapproximately

minutes


9/54



Onscreenquestionnaire

Theretrospectivequestionnairewhichappearedaftereach

testitemwascompletedaimedtoelicitfromparticipants

theirownideaofhowtheyhaddealtwiththatitemInterms

ofcontentanddesignitdrewonthepaperquestionnaire

usedbyWeirHawkeyGreenandDevi’sstudy()

butinthelightofdiscussionwithtwooftheauthorsofthat

paper(WeirandGreen)itwasmodifiedandshortenedinan

attempttomakeitclearer

TheversionusedconsistedofthreepartsThefirstasked

abouthowtheyhadapproachedthetextandquestionsand

offeredthreeoptionsasfollowsfromwhichparticipantshad

tochooseone

BeforereadingthequestionI

a–readthetextorpartofitslowlyandcarefully

b–readthetextorpartofitquicklyandselectivelytogeta

generalideaofwhatitwasabout

c–didnotreadthetext

Partpresentedfivequestionsaskingaboutparticular

cognitivestrategiesParticipantscouldchoosemorethanone

iftheywished

TofindtheanswertothequestionItriedto

–matchwordsthatappearedinthequestionwithexactly

thesamewordsinthetext

–matchwordsthatappearedinthequestionwithsimilaror

relatedwordsinthetext

–searchquicklyforpart(s)ofthetexttoanswerthe

question

–readpart(s)ofthetextslowlyandcarefullytogetthe

answertothequestion

–readrelevantpart(s)ofthetextagaincarefully

Partpresentedtwooptionsaimingtodistinguish

betweenlocalandglobalprocessingParticipantshadto

chooseone

Ifoundtheanswer

–withinasinglesentence

–byputtinginformationtogetheracrosssentences

Thisgaveatotalofuptosevenresponsespercandidate

amaximumofoneinPartfiveinPartandoneinPart

Thefocusofthequestionnairewasthereforeonvarious

aspectsoftheprocesseswhichthereadershadused

aimingtogaininsightsastowhethertheyhadreadglobally

orlocallycarefullyorexpeditiouslyhadusedword-

searchstrategiesforexamplehadattemptedtocombine

informationacrosssentencesandsoon

Procedure

Afterallpersonalinformationformsconsentformsand

computerfamiliarityformshadbeencompletedtheproject

proceededasfollows

StageForthoseusingtheeyetrackerparticipants’individual

eyefixationsandsaccadeswerecarefullycalibratedusing

theTobiicalibrationtoolwhichidentifieseachperson’s

individualpatternofgazeandsaccadebehaviourandensures

theaccuracyofthesubsequenttrackingoftheirreading

duringthetestThiscalibrationwascarriedoutindividually

foreachparticipant

Stage

EachparticipantwatchedashortvideotutorialmodelledcloselyontheCAE CBTtutorialsexplainingeachaspect

oftheprocesstheywereabouttofollowThisvideoalso

explainedtheretrospectivequestionnairewhichappeared

betweeneachtestitem

Stage

ParticipantsthencompletedtheCAE readingitemsonscreen

Theyweregivenatimeindicationofminutesforthe

questionsAsnotedabovethetestexperiencefollowed

theCAE CBTproceduresexceptthatimmediatelyafter

answeringeachtestitemparticipantswerepresentedwith

aninteractivescreenelicitingtheirretrospectiverecallofthe

cognitiveprocessestheyhadusedtoanswerthatquestion

Thescreenalsoshowedthequestionitselfagainsoasto

stimulatemoreaccuraterecallAllanswersandresponses

weresavedtoadatabase

Analysis

Whenthetestshadbeencompletedtheprocessofanalysis

wasinitiatedwhichconsistedofthefollowingthreestages

itemselectionparticipantselectionandtheanalysisofthe

eye-trackingdata

Itemselection

Inordertoinvestigatewhethertheparticipantshad

employedtherangeofcognitiveprocessingtypesidentified

inKhalifaandWeir()asdiscussedabovethefirst

stepintheanalysiswastoselectitemsfromtheCAE test

whichcoveredtherangeofcognitiveskillsTothisendthe

testitemswereexaminedbyanexpertfocusgroupso

astoidentifythecognitiveprocessingoperationswhich

eachitemaimedtoelicitForexampleitemswhichwere

devisedsoastotestareader’sabilitytofindandmake

useofalexicalitematalowerlevelofcomplexitywere

distinguishedfromitemsdevisedtotestareader’sability

tomakeconnectionsatahighertextlevelandsoonOn

thisbasisfiveitems()wereselectedwhich

coveredtherangeofcognitiveprocessesinKhalifaand

Weir’smodel()fromthelowest(atthelexicallevel

item)tothehighest(drawingonthewholetext)as

setoutinTable

Thesefiveitemswerethenanalysedonthebasisof

scores fromthewholecohort(n)toensurethatthey

werefunctioningwellsofarasthissamplesizecouldtell

us attheappropriatelevelofdifficultyAscanbeseen

inTableitwasconfirmedthatfacilityvaluesofthefive

items fellwithintherange–anddiscrimination

indicesoftheseitemswereallorgreaterbothofwhichHenning()suggestsasacceptablerangesofthese

valuesrespectively


10/54



TableSpecificsofitemsselectedforfurtheranalysis

ItemnumberinCAEtest

Targetareaofeachitem

Acrosstwopara-

graphs

Withinone

paragraph

Withinone

sentence

Particularlexis

(withinsentence)

Acrosswholetext

Facilityvalue(fromn)

Discriminationindex(item-totalcorrelation)

Participantselection

Thesampleofparticipantswhoserecordingswouldbe

analysedwasdrawnfromthosewhoseCAE testswere

eyetracked(noutoftheoriginal)Giventhe

aimsoftheprojectthesamplewasfurtherrestrictedto

thestrongercandidatessinceouraimwastoinvestigate

howtheCAE itemsperformedwhentakenbycandidates

attheappropriatelevelandnotbycandidatesbelow

thatlevelAs alltheparticipantshadalsotakenasetof

IELTSonscreenreadingitemsofdifferenttypesonthe

samedaythisadditionalyardstickofstudents’onscreen

readingabilitieswasavailableand‘strong’candidateswere

thereforedefinedasthosewhohadscoredhighlyonboth

theCAE itemsandtheIELTSitemscombined(iethose

withmorethan%ofthepossible)Thisgaveatotal

poolofparticipantswhoweredemonstrablyproficient

onscreenreadersingeneraltermsandnotonlyonCAE test

itemsinparticularsincetheyhadalsoperformedwellon

theIELTSitems

Ofthesemoreproficientparticipantsnotallhadcorrectlyansweredallofthefiveitemsselectedforanalysis

soinadditionforeachtestitemparticipantsfromthepool

wereidentifiedwhohadthatparticularitemcorrectApart

fromitemwhichonlyfourofthepoolhadanswered

correctlysixparticipantswerechosenforanalysisforeach

item(Ofcoursethesewerenotthesamesixparticipants

foreachitem)Theupshotofthiswasthattheeye-tracking

datatobeanalysedconsistedofatotalofrecordings

ietheresponsesofsixstrongparticipantswhoanswered

correctlyforeachoffouritemsandthefourparticipantsfromthepoolwhohadansweredquestioncorrectlyIn

thereportonthefindingswhichfollowsthesearegiventhe

initialsA-Fforreasonsofanonymitythoughagainitshould

benotedthatparticipantAisnotthesamepersonineach

itemanalysed

Analysisofeye-trackingdata

Theonscreenrecordingswerethenanalysedthrough

boththeTobiiStudiosoftwareandthroughdetailed

visualandstatisticalanalysisInordertofocusthis

analysisninequestions(setoutinTable)wereposed

foreachparticipantandeachtestitemThesequestions

weredesignedtoexamineallkeyaspectsofthereaders’

processingincludingthosecoveredintheironline

questionnairetoallowforlatercomparisonAlongsideeach

questioninTablecanbeseentheapproachusedorthe

softwaretoolemployedininvestigatingthatquestionthese

willbefurtherexplainedandexemplifiedbelow

Thesequestionspermittedinsightintothekindsof

cognitiveprocesseswhichparticipantshadusedwhen

successfullyansweringeachtestitemForexampleitem

intheCAE testrequirestesttakersspecificallytoread

thewholetext(the‘target’inourterms)soinvestigationof

therangeofquestionsinTablepermittedustoascertain

whetherparticipantshadinfactdonesoOurapproachthereforeallowedunprecedentedinsightsintoreaders’

moment-by-momentreadingbehaviourastheyresponded

TableAnalysisofeye-trackingdata

Questions Analytical tools

Visualanalysisofeyemovements

(videodata) –seeegFigure

VisualanalysisofGazePlotdata–seeegFigures

–

HeatMapdata–seeegFigure

Automatedstatisticalanalysisof

fixations–seeAppendices–

Didtheparticipantreadthequestion?(Definedasatleastalignedfixations)✔ ✔ ✔ ✔

DidtheparticipantreadthequestionBEFOREcarefullyreadingthetext?

✔ ✔ — —

Didtheparticipantuseexpeditioussearchstrategiestolocatethecorrectsiteoftheanswerefficiently?

— ✔ — —

Didtheparticipantreadall questionoptions? ✔ ✔ ✔ ✔

Didtheparticipantreadthequestionoptionscarefully ?(minfixationsperoption)

✔ ✔ ✔ ✔

Didtheparticipantskimoptions(fewerthanfixations) — ✔ ✔ ✔

(Qs)Didtheparticipantfocusmostheavilyonthetargetarea?(seeTableforhowthiswasdefinedforeachitem)

✔ ✔ ✔

(fornon-scrollingitemsQand

Q)

✔

(fornon-scrollingitemsQand

Q)

Didtheparticipantreadmore than one paragraphcarefully? ✔ ✔ ✔(fornon-scrolling

itemsQandQ)

✔

(fornon-scrolling

itemsQandQ)

(Qonly)Didtheparticipantscrollandsamplevariouspartsoftext?

✔ ✔ — —


11/54



toeachtestitemandunprecedentedinsightintowhether

eachitemwasfunctioningcorrectlyintermsofthecognitive

processesitwaseliciting–animportantpartofitsvalidity

asarguedabove

Twoanalystsindependentlyexaminedtheeye-tracking

dataforeachoftheonscreenrecordingsinthelight

oftheninequestionsinTablethentheanalyseswere

comparedOfthetotalofjudgementsmade(eightquestionsx)theratersagreedonanddisagreedon

onlyanagreementof%Thehighlevelofagreement

isexplainedbythefactthattheeye-trackingdataoffers

aremarkabledegreeofclaritytotheanalystwithfew

areasofdoubtThedisagreementswerethenresolved

throughdiscussiontogivetheresultssetoutintheFindings

section below

Analytical tools

Beforeexaminingtheresultsitisimportantforthesakeof

claritytoexplaineachofthetoolsofanalysisoutlinedin

Tablesoastoclarifyhowtheyfacilitatedtheanalysts’

judgements

Visualanalysisofeyemovements(videodata)

TheTobiisoftwareallowstheanalysttofollowthemoment-

by-momentreadingoftheparticipantplottedonscreenby

aseriesoflines(indicatingsaccades)andcirclesofvarious

sizes(representingfixations–smallercirclesforshorter

fixationsandlargeronesforlongeronesSeeFigure)

FigureExampleofthreefixationandtwosaccaderepresentations

Thistoolallowsforthedetailedobservationandanalysis

ofvariousaspectsofthereadingprocesssincetherecording

canbeslowedforeasierobservationHoweverwiththistool

itcanbedifficulttoseelargerpatternsofbehaviourwhich

arebetterobservedwithGazePlotandHeatMaptools

describedbelow

VisualanalysisofGazePlotdata

TheGazePlottoolallowsfortheanalysisofpatternswhich

mightbemissedonthevideosinceitillustratesgraphically

thefixationsandsaccadesofeachreaderforaselected

segmentnumberedinorderTheGazePlotillustratedin

Figureforexampleshowsacomparisonbetweentwo

readerscolouredlightanddarkrespectivelyonthesame

screendemonstratingthedetailedpicturewhichthetoolcangiveofreaders’patternsofreading

FigureExampleofGazePlotdatashowingtworeaders’eyemovementssuperimposed

InanotherexampleofGazePlotdataFigureshowsa

readerwhoreadtwoparagraphsofatextwhereasFigure

bycontrastshowsareaderwhochosetoreadonlythe

firstparagraphThisisparticularlyusefulwhenanswering

questioninTableabovetoidentifyhowmuchofeach

texttheparticipantscovered

FigureParticipantcompletingquestion–notethecoverageofthewholetext

FigureParticipantcompletingquestion–notethefocusonparagraphoneonly

FigureshowshowsaccadescanidentifiedthroughGaze

PlotdataThistoolisvaluableinansweringquestionin

Tableregardingexpeditiousreadingsinceitcanshowfor

examplewhenthereaderusessearchreadingexpeditious

strategiestofindthecorrectpartofthetextInFigurethereaderhasjustreadthequestionthen(sincethequestion


12/54



mentionsParagraph)usesexpeditiousreadingskillstofind

andlocatethecorrectpartofthetextjumpingfromfixation

numbertothecorrectpartofthetextatnumber

FigureExampleofasaccadeindicatingexpeditiousreadingafterreadingthequestion(therelevantsaccadeisbetweenfixationnumbersto)

VisualanalysisofHeatMapdata

TheTobiiHeatMaptoolallowsfortheanalysisoffrequency

andlengthoffixationintheformofaheatmapgivinga

viewoftheareastowhichthereadergavethemostvisual

attentionTheHeatMapinFigureforexampleshowsthat

thereaderexaminedparagraphmostcloselyandprecisely

whichpartsofparagraphtheyexaminedandshowsthat

theyalsolookedatallpartsofthequestionandoptions(Theoriginalisincolourwhichcannotbereproducedhere)In

conjunctionwiththestatisticaltools(seebelow)thistool

canthereforegiveaclearsenseoftheareastowhichthe

participantgavemostattention

FigureExampleofHeatMapdata(theoriginalisincolour)

Automatedstatisticalanalysis

TheTobiiStudiosoftwarefacilitatesdetailedstatistical

analysisofreaderbehaviourExamplesaregivenin

Appendices–toillustratethekindofdatathetoolcan

produceTheillustrationsintheappendicesaretakenfrom

CAE questioninAppendixisdataregardingthenumber

oftimeseachreaderfixatedonthequestionitselfwhile

Appendixshowshowlongittookinsecondsbeforeeach

participantlookedatthequestionAppendixshowshow

frequentlyeachreaderfixatedoneachquestionoptionin

itemandAppendixshowshowlongeachreaderspent

oneachoptionHereitisnoteworthyforexamplethat

formanyoftheparticipantsbutnotalloptionseemed

tobemoredistractingTheseillustrationsdemonstrate

the kinds ofnumericaldatawhichwereavailablein

the analysis

Findings and discussion

TheresultsoftheanalysisaresummarisedinTableRow

ofthetablesetsoutthequestionswhichwereevaluatedby

theanalystsusingthetoolsoutlinedinTableaboveRows

–setouttheresultsoftheanalysisforeachtestitemin

turnRowsetsoutthetotalsforeachquestionandrow

setsoutthepossiblemaximumratingforeachquestionRow

thensetsoutthepercentages

Themainfindingsareasfollowsforeachquestion

• ItwasclearfromcolumnAthat%ofparticipants

hadreadeachquestioncarefully(aswewouldexpect

ofproficientandcomputer-literatestudentsthoughit

isworthnotingthatsomelessproficientstudentsnot

examinedinthisstudydidnotdoso)

• FromcolumnBitisapparentthatallparticipantsonevery

questionbaronereadthequestionbeforereadingthe

TableSummaryofanalysis

Row

I tem Target area No of

partici-

pants

A Did

participants

read the

question?

(at least

fixations)

B Did

participants

read the

question

BEFORE

reading

the text

carefully?

C Did

participants

use expedi-

tious search

strategies

to locate

the correct

place of

the answer

efficiently?

D Did

participants

read all

question

options?

E Did

participants

read

question

options

carefully ? (

fixations per

option)

F Did

participants

skim options

(fewer than

fixations)

G (Not

Q) Did

partici-

pants fixate

or focus

most heavily

on target?

H Did

participants

read more

than one

paragraph

carefully?

I (Q

only) Did

partici-pants

scroll and

or sample

various

parts of

text?

Q Acrosstwopara-

graphs

QWithinone

paragraph

Q Withinonesentence

Q Particularlexis

(within

sentence)

Q Acrosswholetext

Max

% % % % % % % % % %


13/54



text(%theexceptionbeingoneparticipantonQ

columnB)

• ColumnCshowsthatasizeablemajority(%)used

appropriateexpeditiousstrategiestofindthecorrectpart

ofthetextforeachanswer

• IncolumnDweseethateveryparticipantreadallthe

optionsonalltestitemsexceptinoneinstance(ieQin

columnD)

• ColumnEshowsthatallparticipants(%)hadread

alltheoptionscarefullywithoneexceptionwherethe

participanthadmerelyskimmedthroughoneoptionThis

canalsobeseenincolumnF

• ColumnGisthemostpertinenttothisstudysinceit

showsthatinalmostallcasesparticipantshadfocusedon

theareatargetedbythetestitemmeaningtheitemshave

astrongclaimtocognitivevalidityItisworthexamining

thiscolumninsomedetail

• ColumnGrowshowsthathalfofthesixstudents

answeringquestionfocusedentirelyonthetwotargetparagraphstogettheanswerasanticipatedHowever

onestudentreadthewholetextcarefullyinsteadandthe

remainingtwomerelyskimmedthroughthefirstparagraph

(withonlythreeandfixationsrespectivelylastingonly

andseconds)andthenfocusedheavilyonthe

secondparagraphGiventhattheyallansweredthisitem

correctlythissuggeststhatthesetwocandidatesobtained

enoughinformationinoneparagraphtosatisfythemor

elsewerefortunate

• ColumnGrowshowsthatallstudentsperformedon

questionasanticipatedfocusingheavilyonthetarget

paragraphasawholeFigureistakenfromthisquestion

andshowsvividlyingraphicformpreciselywherethe

studentfocusedattention


questionasanticipatedfocusingheavilyonthetarget

sentence


questionasanticipatedfocusingheavilyandrepeatedly

onthetargetlexisThemeanfixationdurationonthe

relevantlexiswassecondsforallparticipantswith

anaveragefixationcountofIfwecomparethis

withanotherrandomlyselectedpieceoflexisfromthe

sameparagraphwhichreceivedameanofseconds

ofattentionandameanofoffixationcountsitis

clearthatthetargetlexiswassuccessfullyidentifiedand

receivedahighleveloffocusamongtheseproficienttest

takersThisinturnimpliesthattheitemwassuccessfully

targetingtheappropriatecognitiveprocessingactivity

(focusingonlowerlevellexicalareas)

• ColumnHalsomeritscarefulconsiderationforeach

test item

• ColumnHrowshowsagainasdiscussedabove

thatforquestiontwostudentsdidnotreadthetwo

paragraphsfullyforreasonsalreadydiscussed

• ColumnHrowshowsthatalthoughthetargetanswer

forquestionwastobefoundwithinoneparagraphtwo

studentsneverthelessreadmorethanthatoneparagraph

Theotherfourparticipantswerehighlyfocusedintheir

reading–anexamplecanbeseeninFigureThetwo

whoreadmorethannecessarywerepresumablychecking

thattheirexpeditioussearchreadinghadworkedproperly

andthattheyhadnotmissedanything

• ColumnHrowimpliesalsothatmoststudents

(threeoutoffour)alsoreadbeyondthetargetsection

readingmorethanoneparagraphcarefullyeventhough

theanswerwasfoundwithinaparticularsentence

Sincetheyhadalreadyallusedexpeditiousskillstofind

thecorrectsentenceasseenincolumnCthiswas

presumablyforchecking

• ColumnHrowsuggeststhatasallstudentsquickly

foundthecorrectlexistoanswerthequestiontheydidnot

need(exceptone)toreadanyotherparagraphsagaina

signoftheirefficiencyandconfidenceasreaders

• ColumnHrowconcernsquestionwhichrequired

agraspofthewholetextItwasinterestingtosee

differentstrategiesforthisquestionThreestudents

asisclearfromColumnHrowreadmorethanone

paragraphcarefullybutthreedidnot–infacttheyread

almostnothingbeforeidentifyingthecorrectanswer

presumablybecausebythatstagetheyhadalready

builtupasufficient ideaofthewholetexttochoosethe

correct response

• ToillustratethisfurtherTableshowstheamountoftime

spentbyeachofthesixparticipantsonquestionon

thetextandthequestionoptionsrespectivelyThisshows

–perhapssurprisingly–thatallparticipantsapartfromB

spentlongeronthequestionsthanonthetextameanof

secondsontheformerandsecondsonthe

TableComparisonoftime(seconds)spentonQtextversusQquestions

Participant TotalVisitDurationQQuestions(Mean)(seconds)

TotalVisitDurationQQuestions(Sum)(seconds)

TotalVisitDurationQText(Mean)

TotalVisitDurationQText(Sum)

A

B

C

D

E

F

Allrecordings


14/54



latterSomeparticipantsspentalmostnotimeatallonthe

text(egEtooksecondsandFtook seconds)

whichstronglysuggeststhattheyhadalreadyconstructed

astrongandconfidentsenseofthetext’soverallsense

ThisisgraphicallyillustratedinFigurewhichshows

participantE’seyemovementsconcentratingheavily

onthequestionsandalmostnotatallonthetextbefore

answering

FigureParticipantE’seyemovementsonquestion

ReturningtothefullsummaryinTablecolumnrelates

onlytoquestionandshowsagainthefactthatthree

participantsusedthescrollbarandreadbackthroughthe

textreadingcarefullythroughseveralparagraphswhile(as

notedabove)threeothersscarcelyreadthetextatallThis

isaninterestingfindingsinceratherthanreflectingbadlyon

thetestitemitdemonstratesthatwithitemstestingglobal

understandingsomecandidatesmightadoptamorecarefulapproachselectingtore-readsomepartsexpeditiouslyand

readcertainpassagescarefullywhereasotherparticipants

mightalreadybeclearandconfidentenoughnottoneed

tore-readanyofthetextatallBothbehaviourscanbe

characteristicofproficientreadersandbothimplyhigher

levelcognitiveprocessingskills

Student questionnaires evaluated

AsnotedintheMethodologysectionparticipantswere

askedaftercompletingeachitemtoreportretrospectivelyon

theirrecentlycompletedprocessingoperationsWhenthe

eye-trackingdatahadbeenanalysedindetailasdiscussed

aboveitwasthenpossibletocomparetheeye-trackingdata

withthisparticipantquestionnairedataandthentocompare

thetwo

Datafromthestudentquestionnaireswastherefore

examinedalongsidethedatagatheredfromeyetracking

discussedaboveIntotalthereweresevenquestionnaire

optionsforeachtestitemandatotalofeyetracking

recordingsgivingresponsesTheparticipants’

responseswerethenexaminedinthelightoftheeye-

trackingdataandmarkedasaccurateorinaccurateFor

exampleifaparticipantsaidshehadreadthetextbeforereadingthequestionthiscouldeasilybecheckedagainst

theeye-trackingdataIfthestudentdidnotinfactdoso

thenherresponsewouldbemarkedasinaccurateTo

takeanotherexampleifaparticipantrespondedbysaying

shehadnotreadthetextcarefullybuttheeye-trackdata

suggestedotherwisethenthatanswertoowasadjudged

inaccurateTheanalysiswascarriedoutbytwoadjudicators

independentlywithanagreementratioof%Doubtful

caseswerediscussedandagreementreached

Itwasfoundthatofthepossiblechoicesparticipants

hadbeenaccurateintheirself-reportin(%)casesandinaccuratein(%)casesThiscouldbecause

forcelebrationinthataclearmajorityoftheparticipants’

self-assessmentswereaccuratebutgiventhattheir

retrospectivefeedbackwaselicitedimmediatelyafterhaving

completedeachtestitemtheycouldsurelybeexpected

tobemoreawareofwhattheyhadjustbeendoingIt

couldthereforebearguedthatthefactthattheiraccuracy

inretrospectionissolowcastsdoubtuponstudieswhich

dependheavilyonretrospectivereportingforgaininginsights

intocognitive processing

ThereareotherpartialexplanationsfortheseresultsItis

possiblethatthewordingofsomepartsofthequestionnaire

confusedsomeparticipantswhichmayexplainwhymany

statedthattheyhadreadthetextbeforethequestion

whentheyhadclearlydonetheoppositeItisalsopossible

thatparticipantfatigueplayedapartNonethelesssince

theseaspectscannotinthemselvesaccountforsucha

highnumberofinaccurateself-reportsitwouldappearthat

retrospectivereportsincognitiveprocessingresearchcould

belessreliablethanhasoftenbeensupposedandthat

eyetrackingcouldofferamorereliableguidetocognitive

processinginfutureresearchintoreading

Conclusion

ThisprojecthasresearchedasetofCAE onscreentestitems

withaviewtoinvestigatingtheircognitivevalidityThrough

theonscreentestingofstudentstheeyetrackingof

%ofthemastheycompletedthetestandthenthe

selectionofasampleofmoreproficientonscreenreaders

formoredetailedanalysiswehaveshownthattheitems

analysedperformedeffectivelyintermsofelicitingfrom

testtakersboththerangeofcognitiveprocessingidentified

inKhalifaandWeir’smodel()andalsothedifferent

levelsofprocessingfromlowerareastomorecomplexlevels

includingwholetextcomprehension

Detailedanalysisofeachitemthroughavarietyof

approachesusingthegraphicvideoandstatisticaltools

affordedbyeye-trackingsoftwareaswellascarefulvisual

analysisdemonstratedthewaysinwhichthesetest

itemswereperformingintermsofthecognitiveprocesses

theywererequiringofreadersThesetofitemstogether

demonstrablytestedcognitiveprocessingatthelower

levels(egoflexis)thesentenceleveltheparagraphlevel

acrossparagraphsandatwhole-textlevelThesetofitems

canthereforeclaimwithsomeconfidencetohavecognitive

validityinKhalifaandWeir’sterms

Inadditiontheresearchdemonstratedthevalueofusing

eyetrackingtoassistinthevalidationoftestitemsandthepossiblelimitationsoftraditionalretrospectivereportson

participants’cognitiveprocessesInourviewthisresearch

opensexcitingnewwindowstocontinuethemetaphoronto


15/54



boththecognitiveprocessesofreadersundertestconditions

andalsoontothewaysinwhichtestitemscanperform

whenelicitingparticularcognitiveprocessesinreading

References and further readingAfflerbachPandJohnstonP()Ontheuseofverbalreportsin

readingresearch JournalofReadingBehaviour ()–AldersonJC()AssessingReadingCambridgeCambridge

UniversityPress

AnsonCRashidHornSandSchweglerR()ThePromiseof

Eye-TrackingMethodologyforResearchonWritingandReading

OpenWordsAccessandEnglishStudies ()–

BaxterGandGlaserR()Investigatingthecognitivecomplexity

ofscience assessmentsEducationalMeasurementIssuesand

Practices()–

BertramR()Eyemovementsandmorphologicalprocessingin

readingTheMentalLexicon()–

BuscherGBiedert RHeineschDandDengelA() Eye-tracking

analysisofpreferredreadingregionsonthescreeninMynattED

SchonerDFitzpatrickGHudsonSEEdwardsWKandRodden

T(Eds)ProceedingsofthethInternationalConferenceonHumanFactorsinComputingSystemsCHIExtendedAbstractsVolume

AtlantaGeorgiaUSA–April–

CarverR()ReadingRatetheoryresearchandpractical

implications JournalofReading()–

CordonLandDayJ()Strategyuseonstandardizedreading

comprehensiontests JournalofEducationalPsychology –

deGreefT BotzerAandVanMaanenP-P()Eye-tracking

ReadingthemindinProceedingsofthethAnnualEuropean

ConferenceonCognitiveErgonomics(ECCE)DelftNetherlands

August–NewYorkNYACMPress–

DuchowskiA()ABreadth-FirstSurveyofEye-tracking

ApplicationsBehaviorResearchMethodsInstrumentsandComputers

(BRMIC)()–

EgerNBallLStevensRandDoddJ()CueingRetrospective

VerbalReportsinUsabilityTestingThroughEye-MovementReplay

inProceedingsofHCIThestBritishHCIGroupAnnual

ConferenceUniversityofLancasterUKAvailableonlinehttpwww

bcsorgserverphp?showConWebDoc

GassSandMackeyA()StimulatedRecallMethodologyinSecond

LanguageResearchMahwahNJLawrenceErlbaum

GlaserR()ExpertiseandassessmentinWittrockMCand

BakerEL(Eds)TestingandcognitionEnglewoodCliffsPrentice

Hall–

HenningG()AGuidetoLanguageTestingDevelopment

EvaluationResearchCambridgeMANewburyHouse

HooverWAandTunmerWE()Thecomponentsofreading

inThompsonGBTunmerWEandNicholsonT(Eds)ReadingacquisitionprocessesClevedonUKMultilingualMattersLtd–

KhalifaHandWeirC()ExaminingReadingResearchandpractice

inassessingsecondlanguagereadingStudiesinLanguageTesting

volumeCambridgeUCLESCambridgeUniversityPress

PressleyMandAfflerbachP()Verbalprotocolsofreadingthe

natureofconstructivelyresponsivereadingHillsdaleNJLawrence

Erlbaum

RaynerK()Eyemovementsinreadingandinformation

processingyearsofresearchPsychologicalBulletin()

–

RaynerKandPollatsekA()Thepsychologyofreading

EnglewoodCliffsNJPrenticeHall

Ruiz-PrimoMShavelsonRLiMandSchultzS()Onthe

ValidityofCognitiveInterpretationsofScoresFromAlternative

Concept-MappingTechniquesEducationalAssessment()

–

SalvucciDandGoldbergJ()Identifyingfixationsandsaccades

ineye-trackingprotocolsinProceedingsoftheEye-trackingResearch

andApplicationsSymposiumPalmBeachGardensFLUSA–

NovemberNewYorkACMPress–

SpiveyMRichardsonDandDaleR()Themovementofeye

andhandasawindowintolanguageandcognitioninMorsellaE

andBarghJ(Eds)OxfordHandbookofHumanActionNewYork

OxfordUniversityPress–

UrquhartAandWeirC()Readinginasecondlanguageprocess

productandpracticeLondonLongman

VipondD()Micro-andmacro-processesintext

comprehension JournalofVerbalLearningandVerbalBehaviour

–

Weir C () LanguageTestingandValidationanevidencebased

approach BasingstokeNewYorkPalgraveMacmillan

WeirCHawkeyRGreenTandDeviS()Thecognitive

processesunderlyingtheacademicreadingconstructasmeasured

byIELTSinBritishCouncilIDPAustraliaIELTSResearchReports

volume()–Availableonlinefromhttpwwwielts

orgresearchersresearchaspx

Appendices

Appendices–illustratethekindofdatathetoolcan

produceTheillustrationsintheappendicesaretakenfrom

CAE questionAppendixshowsthenumberoftimeseach

readerfixatedonthequestionitselfwhileAppendixshows

howlongittookinsecondsbeforeeachparticipantlookedat

thequestionAppendixshowshowfrequentlyeachreader

fixatedoneachquestionoptioninitemandAppendix

showshowlongeachreaderspentoneachoptionHereitis

noteworthyforexamplethatformanyoftheparticipants

butnotalloptionseemedtobemoredistractingThese

illustrationsdemonstratethekindofnumericaldatawhich

wasavailableintheanalysis

AppendixNumberoffixationsonquestion

AppendixTimetofirstfixationonquestion


16/54



AppendixNumberoffixationsonquestionoptions

Participant Fixationcount

Option(correct

response)

Option Option Option

A

B

C

D

E

F

AllRecordings

AppendixVisitdurationforquestionMCoptions(amountoftimeinsecondsspentoneachoptionbyeachparticipant)

Participant Totalvisits(seconds)

Option(correct

response)

Option Option Option

A

B

C

D

E

F

AllRecordings

Investigating figurative proficiency at different levels ofsecond language writingJEANNETTE LITTLEMORE UNIVERSITYOFBIRMINGHAMUK

TINA KRENNMAYR VUUNIVERSITYAMSTERDAMTHENETHERLANDS

JAMES TURNER UNIVERSITYOFBIRMINGHAMUK

SARAH TURNER UNIVERSITYOFBIRMINGHAMUK

Introduction

Inverybroadtermsmetaphorinvolvesdescribingone

thingintermsofanother(egwhenwomen’scareersare

describedas‘hittingaglassceiling’)Metonymyinvolves

theuseofoneentitytorefertoarelatedentity(egtheuse

oftheterm‘Hollywood’torefertotheUSfilmindustry)

Studiesofmetaphor(andtoalesserextentmetonymy)

haveshownthattheyperformkeyfunctionssuchasthe

signallingofevaluationagendamanagementmitigation

andhumourtechnicallanguagereferencetoshared

knowledgeandtopicchange(Semino)Anabilityto

usethemappropriatelycanthuscontributetoalanguagelearner’scommunicativecompetence(LittlemoreandLow

aandb)andisthereforelikelytobeakeyindicatorofalanguagelearner’sabilitytooperateatdifferentlevelsof

proficiencyasdefinedbytheCommonEuropeanFramework

ofReferenceforLanguages(CEFR)TheCEFRwhich

formspartofawiderEuropeanUnioninitiativeisaseries

ofdescriptionsoflanguageabilitieswhichcanbeapplied

toanylanguageandcanbeusedtosetcleartargetsfor

achievementswithinlanguagelearningIthasnowbecome

acceptedasawayofbenchmarkinglanguageabilityallover

theworldTherearesixlevels(AABBCandC)

EachlevelcontainsaseriesofCanDostatementswhich

describethevariousfunctionsthatonewouldexpecta

languagelearnertoperforminreadingwritinglisteningand

ThisarticlehasbeenacceptedforpublicationinAppliedLinguisticspublishedbyOxfordUnivesityPress


17/54



speakingatthatlevelTheCanDostatementsforwriting

ability(thefocusofthisstudy)canbefoundat

wwwcoeinttDGPortfolio?LEandMdocuments

introDatabankdescriptorshtml

Inthesestatementsthereisaclearprogressioninterms

ofthecomplexityoffunctionsthatalearnerisexpectedto

performandwemightthusexpecttheiruseofmetaphor

tobothchangeandincreaseacrossthedifferentlevelsForexampleatLevelAlearnersareexpectedtobeableto

‘writeashortsimplepostcardforexamplesendingholiday

greetingsandfillinformswithpersonaldetails’Wewould

expectverylittleuseofmetaphorhereexceptperhaps

fortheoddmetaphoricallyusedprepositionwhereasat

LevelClearnersareexpectedtobeableto‘writeclear

smoothly-flowingtextinanappropriatestyle write

complexlettersreportsorarticleswhichpresentacase

withaneffectivelogicalstructurewhichhelpstherecipient

tonoticeandremembersignificantpoints[and] write

summariesandreviewsofprofessionalorliteraryworks’

Herewewouldexpectlearnerstousemetaphortoconvince

andpersuadeaswellastolinktheirideastooneanother

Todatetherehasbeennodetailedinvestigationintohow

alearner’suseofmetaphordevelopsacrossthesedifferent

levelsNorhastherebeenanyinvestigationintotheways

inwhichalearner’sLbackgroundinfluencestheiruseof

metaphorandmetonymyatdifferentlevelsofproficiencyin

learners’writingSuchastudywouldbeusefulasitcould

contributedescriptorspertainingtotheuseofmetaphorand

metonymywhichcouldthenbeusedintrainingmaterials

Thefindingsofsuchastudywouldalsobeusefulfor

organisationssuchasCambridgeESOLwhichareinvolved

inlanguageassessmentastheycouldbeincorporatedinto

themarkingcriteriafortheirwrittenexaminationsInthisarticlewedescribeastudyfundedbyCambridge

ESOLwhichusedtheCambridgeLearnerCorpus(CLC)

auniquecorpusofexamscriptsateachoftheselevelsto

meetthefollowingaims

• toidentifyfeaturesofmetaphorthatdistinguishthe

differentCEFRlevelsasmeasuredbytheCambridge

exams

• toprovidedescriptorsrelatingtometaphorusethatcould

beincorporatedintothedifferentCEFRdescriptorsfor

eachlevelofwritingforEnglish

Thefocusofthestudywasonmetaphorasthishas

reasonablyrobustidentificationtechniqueItalsolooked

tosomeextentatmetonymybutforreasonsmentioned

belowourquantitativefindingsfromthemetonymypart

ofthestudyarenotreportedhereWelimitedourstudyto

thetopfivelevels(A-C)afterobservingthatvirtuallyno

metaphorwasproducedatLevelAOurfirstobjectivewas

tomeasuretheamountofmetaphorproducedacrossCEFR

levelsAtoCThemostwidelyusedmaximallyinclusive

approachtometaphoridentificationisthePragglejaz

Group()metaphoridentificationprocedure(MIP)

Thisprocedureinvolvesidentifyingasmetaphorany lexical

unitthathasthepotentialtobeprocessedmetaphorically

Theanalystbeginsbyidentifyingallthelexicalunitsinthe

text(inmostbutnotallcasesa‘lexicalunit’referstoa

‘word’butseetheMethodologysectionbelow)Thenfor

eachlexicalunittheyestablishitsmeaningincontextand

decidewhetherithasamorebasiccontemporarymeaning

inothercontextsandifsowhetheritsmeaninginthetext

canbeunderstoodincomparisonwiththismorebasic

meaningInthemajorityofcasesthedecisionwastaken

toregardasinglewordascomprisingthelexicalunitevenwhentheanalyst’sintuitionmightbetoclasscertainuses

asphrasesoradictionarymightrecordtwoormorewords

asmakingupaphraseologicalunitThereasoningbehind

thisdecisionisoutlinedinSectionBasicmeaningstendto

bemoreconcreterelatedtobodilyactionormoreprecise

Ifthisisthecasethenthelexicalunitismarkedasbeing

‘metaphoricallyused’Weusedaslightlyadaptedversion

ofthistechniqueinspiredbySteenDorstHerrmann

KaalKrennmayrandParma’sMIPVU()Someuseful

featuresoftheMIPVUforourparticularprojectarethatit

includes‘directmetaphors’(iesimilesandthelike)aswell

as‘implicitmetaphors’suchastheuseof‘this’and‘that’or

pronounssuchas‘it’or‘one’toreferbacktometaphorically

usedwords(egThepathshetookwasindeedtheright

one)and‘possiblepersonifications’(suchas‘thedepartment

needstoact’ )Allofthesefeatureshavebeenfoundtovary

acrosslanguagesandpresentconsiderablechallengesto

learnersHoweverittreatsphrasalverbsandmultiword

itemsassingleunitsforanalysisLanguagelearnersoften

makemistakeswithinphrasalverbsandmultiworditems

suggestingthattheymaynotalwaysbelearningthemas

fixedphrasesandthattheymayattimesbetreatingthemas

novelcompoundsInordertogetattheseitemswetherefore

electedtosplitanyphrasalverbsandmultiworditemswhose

meaningsweredeemedtobepartiallymotivatedbythebasicsensesoftheirconstituentsWealsoincludeditemsthat

involvedachangeinwordclassso‘snaked’wouldcountas

ametaphoreventhoughithasadifferentwordclassinits

basicsenseHerewefollowDeignan’s()workwhich

showsthatmetaphoricalsensesoftendifferformallyfrom

theirliteralcounterpartsThetechniquethrowsupitems

thatsomepeoplemightnotconsidertobemetaphorFor

exampleinourdatatheword‘in’inthefollowingsentence

meninthereallyhighpositions

wouldbemarkedasmetaphorbecauseitcontrastsand

canbeunderstoodincomparisontoitsmorebasicspatial

meaning(insideacontainerroombuildingetc)Forsome

analystsmarkingthisuseof‘in’asmetaphorwouldbe

somewhatcounter-intuitiveasitisthemostconventional

wayofexpressingthisconceptanditisverydifficulttothink

ofanalternativeItisclearlyverydifferentfromtheuseof

theterm‘blackhole’inthefollowingsentencewhichalso

comesfromourdata

managerstendtofallinablackholewhentheyretire

TheMIPdoesnotmakeanyclaimsaboutwhetherthelexical

unitisactuallyprocessedasametaphoronlyidentifieslexical

MetaphoricallyusedlexicalunitsareindicatedbysolidunderliningwhereasmetonymicallyusedlexicalunitsareindicatedbyitalicsInourexamplesonlythosemetaphorsand

metonymiesthatarerelevanttotheparticularpointthatwearemakingareunderlined


18/54



unitsthathavethepotentialtobeprocessedasmetaphor

Thisisimportantforstudiesofmetaphorusedbylanguage

learnersasprepositionsmaybeusedindifferentways

inthelearner’sownlanguageafactwhichmakestheir

metaphoricityinthetargetlanguagemuchmoreapparent

(LittlemoreandLowb)Forexamplethecorresponding

sentenceinRussianwouldbesomethinglike

мужчинынавысокихпозицияхпостах--menon

high positions

sothemetaphoricityofthe‘in’mayinfactbemoresalient

foraRussianlearnerofEnglishthanitisforanativespeaker

whomaybelesssensitivetothemetaphoricityunderlying

conventionalexpressionsWealsobegantodevelopa

methodologyformetonymyidentificationbasedonasystem

proposedbyBiernacka(forthcoming)butbecausethis

techniqueisstillunderdevelopmentwedonotreportour

findingsinthisarticle

Most‘dead’orperhapsmoreappropriatelytermed

‘sleeping’metaphors(Müller)tendtobefoundwithin

thecategoryofclosed-classitemsandmost‘novel’or

‘creative’metaphorstendtoinvolveopen-classitemsItis

thereforeinterestingtolookathowlearnersmakeuseof

openandclosed-classitemsrespectivelyastheymayreflect

differentwaysofusingmetaphorOursecondobjectivewas

thereforetoexploretheextenttowhichtheusethatlearners

makeofopen-classmetaphoricalitemsresemblesthatwhich

theymakeofclosed-classmetaphoricalitemsacrossthe

differentCEFRlevels

IthasbeenobservedthatnativespeakersofEnglishtend

toproducemetaphorinclustersthattheseclustersserve

importantcommunicativefunctions(CameronandLow)andthatsomeofthemostcommunicativelyeffective

clustersarethosethatcontainmixedmetaphorsdespitethe

factthattraditionalwritingguidesoftentellwriterstoavoid

mixingtheirmetaphors(Kimmel)Onewouldalso

expectsomedevelopmentintheproductionofmetaphor

clustersinlearnerwritingatthedifferentlevelsThethird

objectivewasthereforetolookatthesizethedistribution

andthenatureofthemetaphorclustersproducedbylearners

ateachofthelevels

Itisimportanttolooknotjustattheamountofmetaphor

thatisbeingusedbutatwhatlearnersusemetaphorforin

theirwritinginotherwordswhatfunctionsitisbeingused

toperformOurfourthobjectivewasthereforetoassessthe

waysinwhichthelearners’useofmetaphorcontributestoa

learner’sabilitytoperformtherelevantfunctionsateachof

theCEFRlevelsWelookedatmetaphorsthatoccurredboth

withinandoutsideclusters

Aswellasdiscoveringhowmuchmetaphorthelearners

useateachlevelandwhattheyuseitforitisworth

investigating(forteachingpurposes)theextenttowhich

theyareabletouseitaccuratelyIflearnersareparticularly

likelytousemetaphorinaccuratelyatoneofthelevelsthis

isusefulforteacherstoknowastheycanthenaddressthe

issueatthatparticularlevelItmightalsobethecasethat

whenlearnerstryoutnewmetaphoricalexpressionsthey

usetheminaccuratelyatfirstandthendevelopaccuracy

atalaterstageItisthereforeusefultoknowifthereisa

particularstageoflearningatwhichtheystarttodothis

asteachersandexaminerscouldthenbemorelenientin

theirerrormarkingtoallowforexperimentationOnemight

alsoexpectmetaphorerrorstobeduetosomeextent

toLinfluenceandonemightexpecttheamountofL

influencetodecreasegraduallyacrossthedifferentlevelsasthelearnersacquireanunderstandingofthewaysinwhich

metaphorisusedinthetargetlanguageAlternativelyas

Kellerman(aandb)hasshownforidiomsLinfluence

inmetaphorusemaypeakatthebeginningandadvanced

stagesoflearningThefifthobjectiveofourstudywasto

exploretheextenttowhichtheuseofmetaphorinthe

transcriptsappearedtobeinfluencedbytheLbackground

ofthelearnersateachofthelevels

Research questions

Theobjectiveslistedabovetranslateintothefollowing

researchquestions

IntwosetsofCambridgeESOLexamscripts(one

producedbyGreek-speakinglearnersofEnglishandone

producedbyGerman-speakinglearnersofEnglish)

Inwhatwaysdoestheamountofmetaphorproduced

varyacrossCEFRLevelsAtoC?

Inwhatwaysdoestheusethatlearnersmakeofopen-

classmetaphoricalitemsresembleordifferfromthat

whichtheymakeofclosed-classmetaphoricalitems

acrossthedifferentCEFRlevels?

Inwhatwaysdoesthedistributionofmetaphorclusters

varyacrossCEFRLevelsAtoC?

Inwhatwaysdothefunctionsperformedbythe

metaphorclustersvaryacrossCEFRLevelsAtoC

andhowcloselydothesefunctionsrelatetotheCEFR

descriptors?

Towhatextentdolearnersusemetaphor‘incorrectly’

andhowistheiruseofmetaphorinfluencedbytheirL

background?

Methodology

OnehundredessayswrittenbyGreeklearnersofEnglish

(ateachlevel)andessayswrittenbyGerman

learnersofEnglish(ateachlevel)wereselectedfromthe

CambridgeEnglishexams(KET PET FCE CAE andCPE )in

theCambridgeLearnerCorpusAsfaraspossibleattempts

weremadetoextractessaysonrelatedsubjectsinorder

tominimisetheimpactoftopictypeinourresultsWe

thereforeusedthesamesearchtermstoextractessaysfrom

thecorpusateachofthefivelevelsWechosethewords

‘politician’‘politics’‘government’‘economy’‘measures’and

‘environment’Searchtermssuchasthesereflectdomains

thathavebeenshowntoinvolveasubstantialamountof

metaphor(Semino)Theyarealsobroadenoughto

WewouldliketothankAnnaEyngornforthistranslation SeewwwCambridgeESOLorgexamsfordetailsoftherangeoftopicsandformatofeachexamstudiedhere


19/54



encompassawidevarietyofessaysallowingustoextract

sufficientdataateachofthelevelsBecausethedifferent

CEFRdescriptorsinvolvetheabilitytoperformverydifferent

functionsthegenresoftheessaysthatstudentsareaskedto

writefortheCambridgeESOLexaminationsvaryconsiderably

acrossthedifferentlevelsAtthedifferentlevelsstudents

areaskedtoproducearangeofdifferentgenresincluding

lettersemailsnarrativesaswellasargumentativeessaysThustheterm‘essay’isinterpretedverybroadlyinthis

researchprojectInourdatatheAessaysconsistedentirely

oflettersthatwerewrittentoserveverybasictransactional

ordescriptivefunctionssuchasmakingarrangementsto

attendanimaginaryclassmakingholidayarrangements

describingobjectsorrecentpurchasesTheBessaysalso

includedalargenumberoflettersbuttheyrequiredmore

evaluationandthusincludedtopicssuchasdescriptionsof

excitingeventsatschoolgivingadviceondilemmasand

describingabirthdaypartyTherewerealsoasmallnumber

ofshortstoriesinourdatasetatthislevelTheessaysat

Blevelweremorelikelytotaketheformofargumentative

essaysorothertypesofevaluativeandorpersuasivewriting

suchasnewspaperarticlesTheyincludedpolemicaltopics

suchastheenvironmentmediaintrusioninventionsthe

importanceofforeignlanguagesandthebenefitsdrawbacks

ofpublictransportandthecarAtCleveltherewasa

widervarietyofgenresdesignedtoelicitpersuasiveand

evaluativelanguageandtheessaysincludednominationsfor

awardsdescriptivediscursivepersuasiveandcomparative

academicarticlesAtCleveltheessaypromptsrequired

thewriterstoproduceandmarshalcomplexargumentsin

favourofparticularactionsortoshowadeepunderstanding

ofabstractconceptsThegenreswereevenmoremixed

includingawardnominationsforpeopleandorganisationsproposalsforurbandevelopmentlettersofcomplaint

discursivecomparativeandpersuasiveacademicarticles

andphilosophicaltreatisesonthevalueofeducation

Thesedifferentgenresareagoodreflectionoftherangeof

functionsthatlearnersaresupposedtobeabletoperformat

eachCEFRlevel

Theessayswerethendividedintolexicalunitsandentered

intoanExcelspreadsheetwithonelexicalunitoneach

lineAswesawabovethedecisionwastakentoregarda

singlewordascomprisingthelexicalunitevenwhenthe

analyst’sintuitionmightbetoclasscertainusesasphrases

oradictionarymightrecordtwoormorewordsasmaking

upaphraseologicalunit(forexample‘growup’)Studies

ofsecondlanguagewrittenandspokenproductionhave

shownthatlanguagelearnersoftenusethewrongverb

prepositionparticlecombinationsinunitssuchasthese

(Alejo)Thesefindingsindicatethatlearnersmay

attimestreatsuchchunksinamorecompositionalway

thannativespeakers(NSs)relyingonwhatSinclair()

referstoasthe‘openchoice’principleastheylacksufficient

collocationalknowledgetoemploythe‘idiomprinciple’

Thusalthoughcertaincombinationsmayhavethestatusof

phrasesforlexicographerslinguistsorNSsgenerallywe

cannotmakeanyassumptionsabouttheirstatusasphrases

forNNSs(seeMacArthurandLittlemoreforthcomingforanin-depthdiscussionofthisissue)

Inordertoidentifyallpotentiallymetaphoricallyused

lexicalunitsintheessaysweusedanadaptedversionof

theMIPVUMetaphorIdentificationProcedure(Steenetal

)whichisbasedonthePragglejazGroup’s()

MetaphorIdentificationprocedureintroducedabove

Wealsoattemptedtouseasimilartechniquetoidentify

metonymyinwhichwelookedforcontiguityratherthan

comparisonbetweenthebasicsenseofthelexicalunitand

itsmeaningincontext(Biernackaforthcoming)However

becausethistechniqueisstillunderdevelopmenttheresultsfromthisstrandoftheresearcharenotreportedThe

metaphorswerethencategorisedintoopenandclosed-class

itemsWeusedthesefigurestocalculatetheproportions

ofmetaphorusedateachlevelandtheproportionsof

metaphorsthatcomprisedopenandclosed-classitemsat

eachlevel

Asearchformetaphorclusterswasthenconductedusing

atimeseriesanalysisThistechniqueisnormallyusedto

chartthemovementsofstockpricesovertimeappearingin

thefinancialsectionofanewspaperandthesameprinciple

canbeusedtocalculatethe‘movingmetaphoricdensity’of

aspanofdiscourseTocalculateitaspansizeofsay

wordsisselectedThemetaphoricdensityacrossthewords

inthisspan(wordsto)iscalculatedThisisequalto

thenumberofitemsidentifiedasmetaphordividedby

(thenumberofitems)Theresultisplacedatthemid-point

(thethword)Thespanisshiftedoneworddownandthe

metaphoricdensitycalculatedforthenext-wordspan(

to)Theresultisplacedatthemid-point(thethword)

Themetaphoricdensityofthenextspan(wordsto)

iscalculatedandplacedatthemid-point(thethword)

andsoonuntiltheendofthetextisreachedThetechnique

allowstheresearchertoproducemetaphoricdensitycharts

suchasthefollowing

FigureIllustrationofamovingmetaphoricdensitychartforaCAEessaywrittenbyaGermanlearnerofEnglish

ThemetaphorclusterthatappearsatpointBintheabove

chartwasasfollows

Ifagirldevelopsinawaytolikedollsandlanguages

andhatecomputerandmathsthisisjustfine–but

oneshouldnot‘push’herinanydirectionThiswidely

spreadpatternofthinkingismirroredinGermanpolitics

(GermanlearnerofEnglishCAE C)

Thisisausefulgraphicaltechniqueforidentifying

metaphorclusterswithindiscourseItcaninformqualitative

analysisbyallowingtheresearchertoidentifystretchesof

textwithhighlocalisedmetaphoricdensity(ieclusters)Thenextstagewastodecidewhatpercentageofmetaphor

touseasa‘cut-off’pointinourdefinitionofametaphor

clusterPreviousstudies(egCameronandStelma)


20/54



haveusedthe‘suddenonset’ofmetaphorastheirmain

identificationcriterionforametaphorclusterUnderthis

approachthespikethatappearsatwordsinFigure

wouldbeacandidateforconsiderationasametaphor

clusterbecauseitfollowsalongperiodofrelativelylow-

levelmetaphoruseeventhoughtheactualmetaphoric

densityofthisspikeisrelativelylow(%)Howeverinour

studywewantedtocomparetheuseofmetaphorclustersacrosslevelssoweneededtoidentifyastandardstarting

pointintermsofmetaphoricdensityInordertodothiswe

conductedmanualexaminationsofthemetaphoricdensity

chartsforanumberofessaysateachofthefivelevelsand

analysedthemalongsidetheessaysthemselvesWelooked

atclustersat%intervalsuntilwereachedalevelwhere(a)

wecoulddiscernvisiblemetaphoruseaboveandbeyond

thesortsofhighlyconventionalisedmetaphoricalusesof

prepositionsandthelikeand(b)thenumberofclusters

wasnotsogreatastobemeaninglessWeagreedthatthe

most‘meaningful’leveltostartatwas%sowelooked

atclustersof%%%%%and%(there

wereveryfewclustersatthislevelsoitmadenosenseto

lookforclustersof%andabove)Taking%density

asourstatingpointwethencalculatedthenumberand

distributionofclustersthatappearedateachlevelinboth

datasets

Thenumberofclustersproducedateachlevelwas

calculatedusingthistechniqueandmeasuresweremadeof

thedensitiesoftheclustersWethenconductedamanual

searchofthemetaphorsthatappearedbothwithinand

outsidetheclusterstoestablishhowlearne

22669 Rv Research Notes 47

Documents

Transcript of 22669 Rv Research Notes 47