How assessing reading comprehension with multiple-choice questions shapes the construct: a cognitive processing perspective

Andre A. Rupp, Humboldt University of BerlinTracy Ferne and Hyeran Choi, Uinversity of Ottawa

Language Testing 2006;23;441

Presenter: Betty Hsu

MC Questions Assess

Reading Comprehension?

MC questions Testing

Response process

Non-testing

Reading process

?Does the assessment offers evidence of an understanding of the passage?

IntroductionReading comprehension: the process of making sense of printed text in non-testing contexts is a complex, fluid and purpose-driven process.Reading comprehension Assessment : the process is unique to the testing context and is changed by the design of items and the selection of texts.

Assessment Specific

Assessment determined through item design and text selectionMultiple-choice questions as a problem solving task rather than a comprehension taskSelecting choices involving unconditional and conditional response strategiesSelecting choices as a combination of various mental resources

Outline

Section One: Theoretical foundations for response processes to MC items (Theory Review)1.construct and models of reading comprehension2..models of responding to MC questions (adults)

Section Two: participants, methodology and instruments (The Method)Section Three: Results and FindingsSection Four: Directions for future research and the strengths and limitations of this study

Deconstructing reading comprehension

1.Foundational component processes in reading comprehension

2.Effects of reading purpose on strategies and skills

3.Relationship between genre conventions and reading purpose

4.Levels of reading comprehension assessed with MC questions

5.Predicting MC item characteristics through cognitive psychometric approaches

6.Summary

1.Foundational component processes in reading comprehension

Model and theory of reading comprehension:1. Terms: bottom-up processes, ”Matthew Effect”2. Construction-integration (CI) model (Kintsch, 1998) 3. Structure-building framework (Gernsbacher, 1990)The key to successful comprehension is a reader’s ability to efficiently, accurately, and automatically extract and organize information from texts and to integrate it with existing knowledge to form a coherent mental representation of the text.

1.Foundational component processes in reading comprehension

Model for responding to MC reading comprehension questions by Embretson and Wetzel (1987):1. The model consists of 2 components: 1) text itself 2)the selection of an answer to an item, driven

by two processes: 1) fasification 2) confirmation 2. The theory predicts that it is possible to code different fea

tures of the text, the question stem, the correct answer and even the distractors. Especially, the item characteristics may decide which features affect response process in which manner.

All the mentioned theories and models assumed that readers engage in similar processes both in non-testing and test-taking situations.

2.Effects of reading purpose on strategies and skills

Purpose of reading: will guide readers in the selection of their strategies, the range and intensity of their skills

Readers adjust their strategies and comprehension process that most suit their purpose for readingsThe purpose of responding to MC questions: answer them correctly, so test-takers select their strategies accordingly to optimize their chances for successPotential factors of strategy selection in testing situations:1. the level of linguistic difficulty of the text 2. the content and phrasing of the questions 3. the location of information from the correct answers and the distrac

tors 4. the level of cognitive activity required of the respondent (Nevo, 198

9)5. specific testing format (e.g. MC, constructed response, paragraph sor

ting, oral cloze)

2.Effects of reading purpose on strategies and skills

Test-takers often rely on test preparation materials, which suggest the most efficient strategies for successful performanceClassification of test-taking strategies for reading comprehension test: 1. General 2. Text-related 3. Item-related strategies

Commonly recommended strategies for GRE and TOFEL 1. reading the text or the questions first2. looking for key words, or not

Diversity and inconsistency in recommended strategies imply that test-takers’ response processes likely differ from reading comprehension processes in a non-testing context

3.Relationship between genre conventions and reading purpose

The relationship between genres, text types, or registers represents differences in function, purposes, text organization and linguistic conventionsQuestions about a text from a given genre A are likely to engage learners in very different response processes than questions for a text from an alternative genre B due to the inherent characteristics of the genres.The interaction effect between learner competency and testing format highlights that the testing format and text types will change the construct and processes of reading comprehension

4.Levels of reading comprehension assessed with MC questions

Item formats (MC questions) on the response process: 1. the quality and intensity of the reading comprehension process can va

ry considerably across items.2. different levels of reading comprehension are assessed with different i

tems. structure and content of MC questions on reading comprehension test:1. TOEFL or the SAT : likely to assess a mixture of ‘local’ and ‘global

’ comprehension processes, which will force readers to draw on component and integrative processes to different degrees.

2. The newest version of the TOEFL: contains a wider range of item types and includes three basic comprehension, three inference, and one reading-to-learn item types

Sheehan (1999) differentiate between MC questions :1. to identify the main idea of a passage2. to directly search a specific piece of information3. to disambiguate vocabulary in context4. to resolve anaphoric or pronominal references

5.Predicting MC item characteristics through cognitive psychometric approaches

Findings across Psychometric research studies provide evidence :1. Variables coding item features such as the plausibi

lity of the distractors appear to be more predictive of item difficulty than the characteristics of the texts that the items refer to

2. The text itself is important for responding correctly to MC questions on standardized tests, which have shown consistent and systematic above-chance performance for conditions when items were answered alone

3. Responding to MC questions on a reading comprehension test might draw much more on verbal reasoning abilities relevant to a problem-solving context than on general higher-order comprehension abilities

6.Summary

Responding to MC questions on reading comprehension tests is a complex process, which might be consistent with reading comprehension in non-testing context in lower-level skills but might be very different in the engagement of higher-level skills Since testing provides a unique purpose for reading, it impacts the responding strategies which are mediated by characteristics of the test input such as text type and question type. Hypothesis: responding to MC reading comprehension questions on many standardized reading comprehension tests is much more a problem-solving process.

MC Questions Assess

Reading Comprehension?

MC questions Testing

Response process

Non-testing

Reading process≠

Objectives of the study

To develop response process profiles for examineesTo link the qualitative findings to empirical quantitative findings in the reading assessment literature on strategy selection, test methods effects and item characteristic predictionTo develop a strategy selection and response process inventory for administration with large-scale language tests of reading comprehension with MC format

Research Questions

What conscious strategies test-takers deliberately select when they respond to questions? What unconscious skills the test-takers draw on when they respond to questions? How characteristics of the passages and the questions influence these conscious choices and unconscious engagements?

Method-participants10 participants, 3 male and 7 female from second language courses at a large Canadian university in OntarioCharacteristics of participants: (they reported)1. having either recently taken the CanTEST or plannin

g on taking it soon, matching typical test-takers in Ontario

2. coming from Argentina, Brazil, Canada, China, Damaskus, Syria, and Sri Lanka;

3. 8-9 of them having ‘much’ or ‘very much’ experience with MC tests, and comfortable with MC testing format, but only 6 having ‘much’ or ‘very much’ experience with reading comprehension tests utilizing MC questions.

Method-participantsCharacteristics of participants: (they reported)4. preparing for reading comprehension tests

through test preparation materials or courses, and by reading authentic materials.

5. reading in their first language and English, but reading more in English, probably due to the immersion context

6. only a few of them working specifically on improving their general reading comprehension skills and primarily by learning vocabulary through the use of a dictionary.

Method-Instrument

The source material: 1. Adopting items from the CanTEST, a large-scale paper-and pe

ncil test developed in Canada, for people wanting to be admitted to undergraduate and graduate programs

2. 11 passages from 6 content domains with about six MC questions items per passage, the passages on the CanTEST are all expository texts and the items requiring the test-takers to find information, read for basic comprehension, and read to learn

A questionnaire:1. containing many open-ended questions to profile test-takers

in MC reading comprehension contexts2. a revised version in a selected-response format is to be admin

istered with the CanTEST in the future

Method- Procedure

Text What How1st Preparation Observing and recording the

responding behavior

2nd Interview(read and respond, and then being interviewed)

Asking question by question1 how they select their answer2. how they rate the difficulty of the question on a five-point Likert scale3. what make the questions difficult or easy

3rd Think Aloud asking participants to think aloud while responding to each of the questions and do Q&A for clarification

Questionnaire asking participants to do questionnaire

Method-Procedure

The participants are given three texts in different sequences, with the order of presentation partially counterbalanced. Each text contains two passages from two domains. Most items are rated by the participants as either very easy or easy even though an inspection of the answers shows that the difficulty ratings do not correlate highly with the answer scores.The participants are fluent and comfortable enough in English so that responding in English does not pose a problem for them.The sessions lasts approximately 2 hours and participants are nominally reimbursed for their time and efforts with $30.

Method- coding and analysis

Interviews are digitally recorded, transcribed, and read into the NVivo software (2005). Subsequently, responses are coded according to

1. the kinds of strategies used to answer the items2. the ratings of perceived difficulty3. sources of the perceived difficulty4. processes for selecting answers 5. related characteristics of their response processes

Based on the coding, holistic profiles of the participants are developed and supplemented with information from the questionnaire data to help with extracting process information from the data.

Results Results Strategy selection

Developing various general strategies for responding based on their previous experience and on their experience with teachers(The strategies mostly about the procedure of responding: whichone first: the first paragraph/ the entire text / the questions ?)

Two separable strategies: macro and micro-level strategies. Macro-level strategies are concerned with general approaches in taking a MC reading comprehension test, while micro-level strategies used in responding to each individual item.Unconditional or conditional strategies:

1. Unconditional strategy: a specific one major strategy that test-takers always employ for a MC test

2. Conditional strategies depending on the perceived difficulty of the text or the questions (the strategies is conditional on the perceived length of the text, the perceived familiarity with the text type/ the topic/the vocabulary, and the perceived complexity of the responding processes )

Results Results Strategy selection

Clear patterns of responding to MC questions

1. Responding to MC questions is rarely a linear process as suggested by the CI framework or the structure-building framework. More commonly, texts are merely scanned for key words in chunks or even after questions being read first.

2. The process of responding to MC questions relies on key word matching (a process which test-takers often facilitate by underlining or highlighting individual words or phrases considered to be pertinent for understanding the text or answering the questions.)

3. The key words relevant for responding to MC questions may not necessarily be coherently connected and mentally stored because the text is segmented in answering MC questions

Results Results The role of item order and perceived item difficulty

It is assumed that the questions are ordered sequentiallyResponse process is influenced by how difficult or easy an individual item is perceived (a complex assessment on the part of the test-taker) . If the item is perceived to be easy or the selected choice is much more highly activated than other competing choices, answer choice is quickly selected and not all response options is read, understood, and eliminated. One participant stated that she did not even attempt to eliminate choices more carefully because: the main idea was who can get this information I think when I read I got the idea that, the information was already . . . what I remembered from the paragraph on, it’s like an answer while I was reading this was much more quickly so I didn’t need to eliminate

Results Results The role of logical elimination of distractors and informed guessing

When items are perceived to be more and more difficult, the solution process becomes a continual back and forth between the question and relevant text sections in order to logically eliminate - i.e. “falsify” potentially incorrect choices or to “confirm” the potentially correct option One participant described why she decided to start eliminating choices with respect to the perceived difficulty: When I don’ t understand very well the text I start to eliminate, but if I have a clear idea of the

text I just go for the answer.

Results Results The role of logical elimination of distractors and informed guessing

Informed guessing: guessing was usually seen only as a last resort or only after knowledge-based or logic-based elimination The following quote illustrates the complexity and artificiality of the logical elimination process.

The participant responded to the following item: You might want to use census data if you were interested in . . .a) designing new homes for large families;b) building components for computers;c) moving your business to a new area;d) finding a more favorable climatic zone.I picked C, because A, designing new homes for large families , you

would need demographics data to do that … really not necessarily census data for new homes. …& Moving your business to a new area, yes. If you have a candy store… The reasoning process is unique to a testing context.

ResultsResults Affecting the perceived difficulty of items

Factors impacting the perceived difficulty of individual items 1. semantic similarity2. plausibility of the distractors,3. the length and vocabulary of the stem and choices4. the difficulty of the text,5. the item type.

1. The plausibility of the distractors appear to be more predictive of item difficulty than the characteristics of the texts

2. Items that asked for the main idea of a passage were usually perceived as easy

3. The text itself is important for responding correctly to MC questions on standardized tests. Longer and more difficult texts, which makes locating the appropriate passage section more difficult. Local features of a text become more or less relevant depending on the MC questions

4. Responding to MC questions on a reading comprehension test might draw much more on verbal reasoning abilities relevant to a problem-solving context than on general higher-order comprehension abilities

ResultsResults What type & level of reading comprehension is being

assessed with MC Question?

The Text

The Questions

Test-takers

Response Process

Testing contextNon-testing context≠

Logical reasoning

Time constraints

Question difficultyDistractors

MC Questions

Higher order reading comprehension

Lower-order component processes

Correct Response

ResultsResults What type & level of reading comprehension is being

assessed with MC Question?1. The response process is different from reading process, becaus

e the response process is determined by the interaction among characteristics of the questions, the text and the test takers.

2. The process of logically finding the correct option is influenced by time constraints, the perceived difficulty of a question type and the plausibility of distractors.

3. Distractors with very close meaning and high plausibility will make the text comprehension subordinate to logical reasoning, which changes the process of reading comprehension in a testing context compared to non-testing contexts.

4. Therefore, MC questions might function well as separable measures of how difficult different aspects of texts are for test-takers or of how well test-takers engage in lower-order component processes rather than as composite measures of higher-order reading comprehension, which they may be sometimes colloquially assumed to be.

Limitations and future directions

Limitations: The study is limited by the number of participants, the texts and questions chosen, and the way of the think-aloud and reflection questions.

Moreover, it is limited because it only compares the processes of responding to MC items to theoretical predictions about processes in non-testing contexts but does not compare the actual behavior of the participants in such contexts.

Future directions:Additional systematic analyses that link the text types, item types, test-taker characteristics, and response processes across MC reading comprehension tests for various populations of test-takers more closely would be desirable. It would also be important to investigate some of the hypotheses about the effects of text and item characteristics on response processes .

ConclusionsThe sequence and structure of MC questions influence response strategies and result in difference in response processes and reading comprehension in non-testing contexts. Test-takers frequently segment a text into chunks and focus predominantly on the microstructure representation of a text base rather than the macrostructure of a situation model. As a result, higher-order inferences that may lead to an integrated macrostructure situation model in a non-testing context are often suppressed or are limited to grasping the main idea of a text. Newer types of MC questions tend to focus more strongly on higher-level reading comprehensionIn sum, different MC questions create very particular comprehension and response processes. A bridge between response processes for MC items and reading comprehension processes generally needs to be built both rationally and empirically so that results from MC tests about some form(s) of reading comprehension can be more effectively and clearly communicated to test-takers and educational decision-makers.

® 全民英語能力分級檢定測驗GENERAL ENGLISH PROFICIENCY TESTADVANCED READING COMPREHENSION TEST

SECTION II: SKIMMING & SCANNINGPart IIn this part, there are 3 passages with a total of 6 multiple-choice questions. For each passage, please first read the question(s) and then read the passage quickly and selectively to find the answer(s). Please write your answers on your answer sheet.First read the following questions.23.What is the main subject of this passage?A.The style of a renowned glass artistB.The revival of glass architectureC.The uses of glass in theater designD.The glass buildings constructed in the 1950s24.What is the purpose of the third paragraph?A.To explain the reasons for a phenomenonB.To compare two different viewpointsC.To point out some potential problemsD.To describe a production process

Now skim the following passage to find the answers to the above questions.In the 1920s, the renowned Swiss architect Le Corbusier observed that the history of architecture is the history of the window. In the new generation of glass buildings now appearing in cities around the globe, "the window has broken out of the frame," says architect James Polshek, lead designer of the Rose Center for Earth and Space at the American Museum of Natural History in Manhattan.From Berlin to Tokyo, New York to London, a rash of striking new structures is giving glass construction new life. Not since the early 1950s, when sleek, green-tinted glass buildings like New York City's Lever House rose amid the stone canyons of countless major cities, has glass elicited so much attention. There are so many glass buildings currently on the drawing boards or under construction, in fact, that it is hard to keep track of them all. From the 357-foot-long barrel-vaulted, transparent roof of New York-based architect Rafael Vinoly's Kimmel Center for the Performing Arts in Philadelphia to the soaring glass dome of London architect Norman Foster's restoration of the historic German Reichstag, glass is assuming innovative, dramatic and surprising forms.This new trend is due primarily to the development of new construction methods and glass production techniques. Ingenious systems now allow expanses of glass to be suspended in front of a series of nearly invisible metal trusses and braces. In addition, advancements in glass production since the 1973 energy crisis have resulted in revolutionary changes in its performance. Glass is now being manufactured with special coatings that adjust the thermal intake for all four directions of the compass and even allow for the increased intensity of the sun at certain latitudes. The new glass is not only spectacular to look at and through, it is also s

afer, stronger and more energy efficient.