Art Graesser

Department of Psychology and the Institute for Intelligent Systems

Assessments of Reading at Deeper Levels of Comprehension

Main Messages

• We need assessments of reading comprehension that systematically tap deeper levels of comprehension.

• Good deep measures integrate reading skills/strategies with world knowledge and nonlinguistic components.

Levels of Language & Discourse

• Graphemes and phonemes• Morphemes and words• Syntactic composition • Linguistic style and dialect• Explicit propositions• Referents of referring expressions• Common ground • Discourse focus versus

presuppositions• Situation models & inferences• Embedded dialog• Configuration of multiple agents• Genre, registers, rhetorical

structures• Plot configurations • Local and global coherence• Point of message -- theme• Goals and attitude of author

Five major categories (Kintsch, 1998; Graesser, Millis, & Zwaan, 1997)

1)Surface code

2)Propositional textbase

3)Situation model

4)Text genre and rhetorical structure

5)Pragmatic communication

What do we mean by deep comprehension?

Cognitive Processes Bloom (1956)

Recognition

Recall

Comprehension (Inference)

Application Analysis

Synthesis

Evaluation

Abstractness of Content (Mosenthal, 1996)

(1) Most concrete: concrete person, action, thing

(2) Highly concrete: observable attributes (action, types, attributes, amounts)

(3) Intermediate: Procedures, goals, and manner

(4) Highly abstract: cause, effect, reason, evidence

(5) Most abstract: inferred theories & themes

Gates-MacGinitie Reading Test

0

0.1

0.2

0.3

0.4

0.5

Levels 1 & 2 Levels 4 & 5

Good Questions that Tap Pragmatic Communication

(Questioning the Author , Beck, McKeown, Hamilton, & Kucan, 1997)

• What is the author trying to tell you?

• Why is the author telling you that?

• Does the author say it clearly?

• How could the author have said things more clearly?

• What would you say instead?

Domain KnowledgeWisher & Graesser (2007)

People(Agents)

Taxonomies

Spatial Layout

Compositional Structure

Procedures & Plans

Causal Networks

Other

Deep-Level Reasoning Questions (Graesser and Person,1994)

LEVEL 1: SIMPLE or SHALLOW 1. Verification Is X true or false? Did an event occur?2. Disjunctive Is X, Y, or Z the case?3. Concept completion Who? What? When? Where?4. Example What is an example or instance of a category?

LEVEL 2: INTERMEDIATE 5. Feature specification What qualitative properties does entity X have?6. Quantification What is the value of a quantitative variable? How much? 7. Definition questions What does X mean?8. Comparison How is X similar to Y? How is X different from Y?

LEVEL 3: COMPLEX or DEEP9. Interpretation What concept/claim can be inferred from a pattern of data?10. Causal antecedent Why did an event occur? 11. Causal consequence What are the consequences of an event or state? 12. Goal orientation What are the motives or goals behind an agent’s action?13. Instrumental/procedural What plan or instrument allows an agent to accomplish a goal? 14. Enablement What object or resource allows an agent to accomplish a goal?15. Expectation Why did some expected event not occur?16. Judgmental What value does the answerer place on an idea or advice?

Analysis of 120 Multiple Choice Questions in Cognitive Psychology

• 120 MCQ items of cognitive psychology textbooks – 4 test banks– 30 items per textbook

• Items coded on the level in Graesser & Person taxonomy.

0

0.1

0.2

0.3

0.4

0.5

Shallow level Deep Level

Challenges

• How can we promote more diversity in the landscape of questions?

• How can we encourage deeper questions?

• How do we change:– Textbook writers– Professors &

teachers– Students

Cognitive and Behavioral Tests in Experimental Psychology

OFF-LINERecall SummarizationRetrospective think aloudRecognition testsCloze procedure Multiple choice testsSentence verificationQuestion asking &

answeringResponse signal

paradigmRatingsWord sorting

ON-LINEImmediate think aloud Self-paced reading timesEye trackingWord naming latencies Lexical decision latencies RSVP-SOAPhysiological recordings

fMRI Evoked potential

Open-ended Questions

The sun exerts a gravitational force on the earth as the earth moves in its orbit around the sun. Does the earth pull equally on the sun? Explain why?

EXPECTATIONSThe sun exerts a gravitational force on the earth. The earth exerts a gravitational force on the sun. The two forces are a third-law pair.The magnitudes of the two forces are the same.

MISCONCEPTIONSOnly the larger object exerts a force. The force of earth on sun is less than that of the sun on earth.

Virtues of the Multiple Choice Format

• Familiar to the educational enterprise– School systems– Textbook companies– Psychometric community

• Easy to score

• Objectively and systematically scored

• Distracters add complexity

Analytical Schemes

• Categories of distractors– Near miss with pedagogical point (versus obscurity)– Thematic versus unrelated distracters

• Relation to associated text– Local versus global– Explicit versus inference– Main rhetorical content versus details

• Qualitative causal analysis – How does a change in A affect B?– Increase, decrease, no change, versus indeterminate

What happens to the pins when the key is turned to unlock the door? A) they drop B) they rise C) they remain stationary

(Graesser & Olde, 2003)

Closing Comments

• Rand Reading Study Group (Snow, 2002) emphasized comprehension– Text– Reader– Activities– Sociocultural context

• Training effective strategies of reading comprehension requires analysis of world knowledge and deeper levels of processing (edited by McNamara, 2007)

?