Diagnostic Testing of ESL Reading Skills:Diagnostic ... · diagnostic assessment of reading...
Transcript of Diagnostic Testing of ESL Reading Skills:Diagnostic ... · diagnostic assessment of reading...
AAAL 2012 Annual Conference
Diagnostic Testing of ESL Reading Skills:Diagnostic Testing of ESL Reading Skills: Relating Test-taking Strategies to
R di AbilitiReading Abilities
Alan UrmstonHong Kong Polytechnic Universityg g y y
DELTA is …
an online English language proficiency assessment system designed specifically for the Hong Kong academic context
d d h d k
designed specifically for the Hong Kong academic context which can
diagnose students’ strengths and weaknessestrack and report on students’ language gainsbe linked to curriculum/teaching/materialsbe linked to curriculum/teaching/materials
help students to plan their language learning
inform curriculum developmentinform curriculum development
DELTA Track and Component Skills Profile DELTA Track and Component Skills Profile
‘It is common to assert that diagnostic tests are i d d b h h d k f intended to probe the strengths and weaknesses of learners, but there is virtually no description, much less discussion of what the underlying much less discussion of what the underlying constructs might be that should be operationalized in valid diagnostic tests’ (p.28). g p
(Alderson, 2007)
Validating the DELTA: Th di The reading test
Jang (2009) identifies two key issues in the diagnostic assessment of reading comprehension: ‘the characteristics of reading skill profiles ‘the characteristics of reading skill profiles … and the usefulness of the resulting diagnostic information for students and teachers’ (p 39) information for students and teachers (p.39), reiterated by Alderson (2007) who suggests that ‘central to diagnosis must be the provision of usable feedback either to the learners themselves or to the diagnoser’ (p.30).
The DELTA reading subskillsDrawing on taxonomies of reading subskills (Munby, 1978; Weir, 1983; Alderson & Lukmani, 1989; Jang, 2009), knowledge of the Hong Kong tertiary-level English language learning and teaching context the DELTA Team initially
Identifying specific information
level English language learning and teaching context, the DELTA Team initially identified eight (8) reading subskills to be tested in the DELTA:
y g pInterpreting a word or phrase as used by the writerUnderstanding main ideas and supporting ideasUnderstanding information and making an inferenceUnderstanding information and making an inferenceUnderstanding an argument made by the writerInterpreting an attitude or intention of the writerUnderstanding grammatical relationships of words or phrases across a textUnderstanding grammatical relationships of words or phrases across a textIdentifying text type
The DELTA reading componentg pconsists of
four testlets (reading text + test items)increasing difficulty level (e.g. easy, medium, difficult)items of calibrated difficulty level (Winsteps)reading texts of the type (topic, theme, genre) that undergraduate students would be expected to read (both undergraduate students would be expected to read (both academic and non-academic)
Component Diagnostic Report: Reading
‘What a test of reading tests is not simply what its i h f j d constructors say it tests, nor what a set of judges
considers it to test. It must surely and crucially relate to what happens inside a test-taker’s head when he to what happens inside a test-taker s head when he or she responds to an item’ (p.478).
(Alderson 1990)(Alderson, 1990)
Test-taker stimulated recallsPost-test interviews were conducted with 10 students who had taken the DELTA a few days ypreviously.
Students were asked to describe the strategies they had used to answer the questions.
In some cases students did live think-aloud while answering reading questions.
Strategies used were identified and coded using the di h d i d b C h d U t ( )coding scheme devised by Cohen and Upton (2007).
Identifying specific information (Ability to locate and recognize information in one spot/in more
Text level: Easy (DELTA 1 65) Item
(Ability to locate and recognize information in one spot/in more than one spot with or without rephrasing within specific context.)
Text level: Easy (DELTA 1-65) Item
Although the cities that the Maya built were beautiful and the people
Even though the workers built the cities, built were beautiful and the people
worked hard to build them, usually, only the priests lived in the cities and other people lived in small villages in
A. they were reserved only for priests.* B. the ordinary people didn't like the
________
other people lived in small villages in the forests. They lived in small huts with no windows with the walls made of poles covered with dried mud and
cities. C. it took them a long time to construct
the cities. l h h hof poles covered with dried mud, and
the roof made of grass or leaves. D. people thought they were not
comfortable enough.
Identifying specific information
S2. Annie (average ability): I read the question and the options and then I scanned through the q p gparagraph and I found the word 'priests'. I noticed that the first option has the same word and so I chose A.
S3 Joyce (high ability): S3. Joyce (high ability): I read the whole passage quickly and so I knew the answer should be A. It’s an easy passage so I understand very well.
S5. Bill (average ability): I read the whole passage. I understand it easily coz the sentences are short and the words are easy to understand. I eliminated the other short and the words are easy to understand. I eliminated the other options as I knew they don’t make sense.
C i f d if i ifi i f iCommon strategies for Identifying specific information
Code Strategy descriptionCode Strategy description
T5 Reads the question and then reads the passage/portion to look for clues to the answer, either before or while considering options.
R7 Reads a portion of the passage rapidly looking for specific informationR7 Reads a portion of the passage rapidly looking for specific information.
T24 Selects options through elimination of other option(s) as unreasonable based on paragraph/overall passage meaning.
R5 Reads the whole passage rapidly.
TW1 Uses the process of elimination (i.e., selecting an option even though it is not understood out of a vague sense that the other options couldn’t be correct)understood, out of a vague sense that the other options couldn t be correct).
R4 Reads the whole passage carefully.
TW3 Selects the option because it appears to have a word or phrase from the passage in it –ibl k dpossibly a key word.
R6 Reads a portion of the passage carefully.
T22 Selects options through vocabulary, sentence, paragraph, or passage overall meaning T22 p g y, , p g p , p g g(depending on item type).
So, for Identifying specific information (and for the other subskills) the other subskills) …
students use a combination of reading test management students use a combination of reading, test management and test wiseness strategies to answer the items.
So how do reading strategies translate into di kill b kill ? reading skills or subskills?
‘Reading strategies are deliberate, goal-directed attempts to control and modify the reader’s efforts to decode text understand words and construct meanings decode text, understand words, and construct meanings of text. Reading skills are automatic actions that result in decoding and comprehension with speed, efficiency, in decoding and comprehension with speed, efficiency, and fluency and usually occur without awareness of the components or control involved’ (p.368).
Afflerbach et al (2008)
Just reads from the start and straight to the end; reading for no purpose [she] wouldn't stopno purpose, [she] wouldn't stop or pause. If it's something [she] can't understand then [she'll] stop once in a while to check what [she’s] read.
Reads all the questions first then reads the relevant part of the text.
DELTA Reading Text Comparison
DELTA R di P It MDELTA Reading Person-Item Map
More-proficient Readers
GRO
Less-proficient Readers
OWTH
DELTA R di P It MDELTA Reading Person-Item Map
more proficient readers
what the more proficient readers proficient readers can do
less proficient readers
The challenge The challenge …
to provide our learners with usableto provide our learners with usable information to help them know what they can and cannot do nowthey can and cannot do now.
Component Diagnostic Report: Reading (revised)
Tentative descriptions of proficiency
Proficient readers (DELTA > 110) can:
Identify specific information in feature articles on less familiar topics
Understand the main and supporting ideas in academic articles in fields outside of their course of study
h d f f l Interpret the attitude or intention of a writer in fictional or biographical texts
The challenge … to predict growth in our learners learners.
200 DELTA TRACK
160
180DELT
113118 120
125120
140A
MEA 113
80
100S
U
R
E
60
08/2011 08/2012 08/2013 Graduation
DELTA Measure Predicted Measure Target Measure
The challenge …
to help our learners to grow gas readers.
ICOSAICOSAReading RequirementReading Requirement
ReferencesAfflerbach P Pearson P D & Paris S G (2008) Clarifying Differences between Reading SkillsAfflerbach, P., Pearson, P.D. & Paris, S.G. (2008). Clarifying Differences between Reading Skills
and Reading Strategies. The Reading Teacher, (61)5, International Reading Association Alderson, J.C. (1990). Testing Reading Comprehension Skills (Part 2): Getting Students to Talk
about Taking a Reading Test (A Pilot Study) Reading in a Foreign Language (7)1 pp 465‐about Taking a Reading Test (A Pilot Study). Reading in a Foreign Language (7)1, pp.465‐503.
Alderson, J.C. (2007). The Challenge of Diagnostic Testing: Do We Know What We Are Measuring? In J Fox M Wesche D Bayliss L Cheng C Turner & C Doe (Eds ) LanguageMeasuring? In J. Fox, M. Wesche, D. Bayliss, L. Cheng, C. Turner & C. Doe (Eds.) Language Testing Reconsidered. Ottawa: University of Ottawa Press, pp. 21‐40.
Alderson, J.C. & Lukmani, Y. (1989). Cognition and Reading: Cognitive Levels as Embedded in Test Questions Reading in a Foreign Language (5)2 pp 253 270Test Questions. Reading in a Foreign Language (5)2, pp.253‐270.
Jang, E.E. (2009). Demystifying a Q‐Matrix for Making Diagnostic Inferences about L2 Reading Skills. Language Assessment Quarterly, (6), pp.210‐238.
Munby J L (1978) Communicative Syllabus Design Cambridge University PressMunby, J.L. (1978). Communicative Syllabus Design. Cambridge University PressWeir, C.J. (1983). Identifying the language problems of overseas students in tertiary education in
the U.K. Unpublished PhD thesis, Institute of Education, University of London.
For more information on DELTA For more information on DELTA
http://gslpa.polyu.edu.hk/eng/delta web/http://gslpa.polyu.edu.hk/eng/delta_web/