The Problem of Comprehension:
description
Transcript of The Problem of Comprehension:
The Problem of Comprehension: The Problem of Comprehension:
A Potential Barrier to Reading Curriculum-Based A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Measurement as a Progress Monitoring and Screening Instrument Screening Instrument
Mark R. Shinn, Ph.D.
Professor and School Psychology Program
National Louis University, Skokie, IL
http://markshinn.org
February 21st, 2013
1 of 6 members of Technical Review Panel, National Center for Student Progress Monitoring, USDE/OSEP2003-2007
Editor and Contributor to 2 Major Texts on CBM
Author of More than 75 Refereed Journal Articles and Book Chapters on the Topic of CBM, Progress Monitoring, and Screening
My Area of Expertise
Mark R. Shinn, Ph.D. Serves as a Paid Consultant for Pearson Assessment for their AIMSweb product that provides CBM assessment materials and organizes and report the information from 3 tiers, including RTI. He provides technical support and training.
Mark R. Shinn, Ph.D. Serves as a Consultant for Cambium/Voyager/Sopris for their Vmath product, a remedial mathematics intervention but has no financial interests. He helped them develop their progress monitoring system.
Mark R. Shinn, Ph.D. Serves as a Consultant for McGraw-Hill Publishing for their Jamestown Reading Navigator (JRN) product and receives royalties.He helped them develop their progress monitoring system.
Mark R. Shinn, Ph.D. Serves as a Member of the National Advisory Board for the CORE (Consortium on Reaching Excellence) and receives a stipend for participation. He provides training and product development advice.
DisclosureDisclosure
QuickTime™ and aH.264 decompressor
are needed to see this picture.
Learning to Read is Critical for School (and Life) Success
Frequent Formative Assessment Like Weekly Use of Reading Curriculum-Based Measurement (R-CBM) is Among our Most Powerful Tools!
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.
And the Number 1 Most Powerful TEACHING Variable
What about comprehension?
What about “word callers?”
I know this kid who...reads “fluently” but
doesn’t understand a THING they read...
BUT...
Will Be
Available in
•pdf format
•iBook format
Presentation is Based on the Following White Paper in PRODUCTION
A “glossy” and official Pearson version will be finished soon and sent to you.
Shinn, M.R. (2013). The problem of comprehension: A potential barrier to Reading Curriculum-Based Measurement (R-CBM) as a progress monitoring and screening instrument. Minneapolis, MN: Pearson Assessment.
References on CBM Criterion Related Validity
Deno, S. L., Marston, D., Shinn, M. R., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling reading disability. Topics in Learning and Learning Disability, 2, 53-59.
Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239-256.
Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28.
Hamilton, C., & Shinn, M. R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.
Miura Wayman, M., Wallace, T., Ives Wiley, H., Ticha, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41, 85-120.
Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 458-478.
10
Accessing Reading Materials
markshinn.org1. Click on the Downloads for
Professionals Icon
2. Click on the Presentations and Handouts Folder
3.Click on AIMSweb Webinar R-CBM and the Problem of Comprehension (Sponsored by Pearson) 2013 Folder
10
Big Ideas
1. When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general
reading skills.
2. A strong empirical relation has been demonstrated between R-CBM and measures of general reading
and reading comprehension (RC).
3. Persuasive empirical data are not always persuasive of teacher opinion.
4. General reading skills are necessary, but not sufficient for comprehension.
5. RC is not a single “thing." Judgments are based on:
1. What a student is expected to read.
2. How the student will be assessed.
6. RC is influenced by a number of important variables, including
1. General reading ability
2. Language, especially vocabulary and familiarity with cultural idioms
3. Knowledge, especially content knowledge and knowledge of the world
4. Metacognitive strategies, including motivation and interest
2 Other Big Ideas We Can’t Discuss Today
1. If educators are concerned about specific student’s RC, then we must have a systematic
assessment process to address these concerns, starting with ruling out general reading skill
deficits, but also including interviews, observations, and diagnostic assessment.
See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A.
Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD:
National Association of School Psychologists.
• If students have low general reading skills, RC strategy intervention should be time limited and
the use of such strategies should be actively taught, expected, supported, and rewarded in Tier
1, general education and/or content area classes.
See: Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension
strategies. American Educator, Winter, 39-50.
2 Other Big Ideas We Can’t Discuss Today
1. If educators are concerned about specific student’s RC, then we must have a systematic
assessment process to address these concerns, starting with ruling out general reading skill
deficits, but also including interviews, observations, and diagnostic assessment.
See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A.
Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD:
National Association of School Psychologists.
• If students have low general reading skills, RC strategy intervention should be time limited and
the use of such strategies should be actively taught, expected, supported, and rewarded in Tier
1, general education and/or content area classes.
Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies.
American Educator, Winter, 39-50.
The Problem of “Fluency”
Experienced--and Often Powerful--Teachers Push Back When R-CBM is (Inappropriately) Interpreted and Communicated as “Fluency”
How Did This Happen?
– Errors in Judgment(s), Confusing Constructs with Behaviors Tested
– Tendency toward Reductionism
An Example of Reading Curriculum-Based Measurement (R-CBM)
It was a pretty good composition. I felt proud knowing it was the best one at my school. After I’d read it five times, I was impatient to start reading it out loud. I followed the book’s directions again. First I read the composition out loud without trying to sound impressive, just to hear what the words sounded like.
QuickTime™ and aH.263 decompressor
are needed to see this picture.
Billy, 4th Grader
Phonemic Awareness
Alphabetic Understanding
Fluency
Vocabulary
Comprehension
Oversimplifying the NRP
Strands are NOT BOXES-
Low Scores “in the Box” Led to
TEACHING the Things in the Box
SEPARATELYhttp://www.nationalreadingpanel.org/
Phonemic Awareness
Phonics
Fluency
Vocabulary
Comprehension
ALL These Skills
General Reading Skill
Consequences of (Mis) Interpretation as Fluency
Teacher Push Back and Shallow InterventionIf R-CBM Interpreted As
FluencyIf Interpreted R-CBM(Correctly) As
General Reading Ability
Goal is to Read Fast Goal is to Read WELL
Interventions Emphasize Speed
Interventions Emphasize Quality
Interventions are “Slices” or Bandaids
Interventions are Integrated or Bandages
Research Evidence of “Fluency-Driven” Intervention or
“Reading Faster”?
Our results indicate that repeated reading does not qualify as an evidence-based or promising practice for students with or at risk for learning disabilities
(p. 276)Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading
interventions for students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263-281.
Implementing repeated reading and wide reading interventions without more formative intervention is not likely to be valuable (p. 9)
Wexler, J., Vaughn, S., Roberts, G., & Denton, C. A. (2010). The efficacy of repeated reading and wide reading practice for high school students with severe reading disabilities. Learning Disabilities Research & Practice, 25, 2-10.
Potentially Severe Educational Need
A Student with a Significant Reading Discrepancy
Poor Reading Quality as Well as Quantity
Reads Accurately?
Reads Efficiently with Automaticity?
Reads with Expression (Prosody)?
Effective Strategy for Unknown Words?
Errors Distort or Preserve Meaning?
Self Corrects Errors (Comprehension Self-Monitoring)?
Adjusts Pace When Text Difficulty Changes?
QuickTime™ and aH.264 decompressor
are needed to see this picture.
A Poor High School Reader
55
Grade 8 Material < 10th percentile at beginning of Grade 8
It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming
Comprehension Measure Criterion Measure Correlation
R-CBM SAT Word Study .80
SAT Comprehension 0.91
Question Answering SAT Word Study .66
SAT Comprehension .82
Recall SAT Word Study .58
SAT Comprehension 0.70
Cloze SAT Word Study .71
SAT Comprehension 0.72
Fuchs, L.S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28.
It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming
Shinn, M.R., Good, R.H., Knutson, N., Tilly, W.D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21(3), 458-478.
.90.88 .85 .71 .72
But, I KNOW and HAVE SEEN Word Callers!!!!!!!
The Research
The idea of word callers has gained popularity despite a lack of evidence that applies “to an appreciable number of poor readers” (Stanovich, 1986, p. 372).
Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360-406.
But, I KNOW and HAVE SEEN Word Callers!!!!!!!
The Practice
31 of 75 Third Grade Teachers responded to a postcard put in their box asking them
“if you teach a third-grade student who can read fluently, but has difficulty comprehending text.”
41%!Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers'
judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.
But, I KNOW and HAVE SEEN Word Callers!!!!!!!
Teachers Identified PAIRS of Kids in Their Classrooms:
Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.
Word Callers (WC) Similarly “Fluent” Peers
Read Fluently, But Didn’t Comprehend
Read Equally as Fluently (Well) As WC, But DID COMPREHEND
Teachers Predicted BOTH Pair’s R-CBM, Maze, and Oral Comprehension Question Answering Skills--THEN Both Pairs WERE TESTED
The “Word Caller”
Teachers Were Partially Right!
WCs “Comprehended” Less Well
WC
Peers
But WRC Was ALSO MUCH Lower
WC
Peers
Almost 2 Standard Deviations Lower
Almost 1.3 Standard Deviations Lower
So Where Do This Leave Us...
1. Reading Comprehension is Complex and We Tend to Oversimplify and Overgeneralize
• Reading is Necessary, But Not Sufficient for Understanding
• Reading Comprehension Comes From Reading, But is MORE Than Reading
“Comprehension” is Oversimplified
A-Rod hit the cover off of the ball, but ended the game with a 6-4-3 double play.
Factual:
Who is A-Rod?
What does “6” mean?
Inferential:
Why would people from Beantown celebrate this?
Why would this event mean different things in June than October?
Another Common Example
173-word paragraph with 14 sentences of words that are very short (about 4 letters) and rated as late Grade 6 by Flesch-
Kincaid
The procedure is actually quite simple. First, you arrange things into
different groups. Of course, one pile may be sufficient depending on how
much there is to do. If you have to go somewhere else due to lack of
facilities that is the next step, otherwise you are pretty well set. It is
important not to overdo things. That is, it is better to do too few things at
once than too many. In the short run, this may not seem important, but
complications can easily arise. A mistake can be expensive as well. After
the procedure is completed, one arranges the materials into different
groups again. Then they can be put into their appropriate places.
Eventually, they will be used once more and the whole cycle will have to
be repeated. At first, the whole procedure will seem complicated. Soon,
however, it will become just another fact of life. It is difficult to foresee
any end the necessity for this task in the immediate future, but then one
can never tell.
Let Me Add Some Information
The procedure is actually quite simple. First, you arrange things into different groups. Of
course, one pile may be sufficient depending on how much there is to do. If you have to go
somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set.
It is important not to overdo things. That is, it is better to do too few things at once than too
many. In the short run, this may not seem important, but complications can easily arise. A
mistake can be expensive as well. After the procedure is completed, one arranges the
materials into different groups again. Then they can be put into their appropriate places.
Eventually, they will be used once more and the whole cycle will have to be repeated. At first,
the whole procedure will seem complicated. Soon, however, it will become just another fact of
life. It is difficult to foresee any end the necessity for this task in the immediate future, but
then one can never tell.
How to Wash Clothes
Reading Reading ComprehensionComprehension
KnowledgeKnowledge Fluency*Fluency*We Refer to It as We Refer to It as
General Reading SkillsGeneral Reading Skills
MetacognitionMetacognition
LanguagLanguagee
• • ProsodyProsody• • Automaticity/RateAutomaticity/Rate
• • AccuracyAccuracy• • DecodingDecoding
• • Phonemic AwarenessPhonemic Awareness
• • Oral Language SkillsOral Language Skills• • Knowledge of Language Knowledge of Language
StructuresStructures• • VocabularyVocabulary
• • Cultural InfluencesCultural Influences
•• Life ExperienceLife Experience• • Content KnowledgeContent Knowledge• • Activation of Prior Activation of Prior
KnowledgeKnowledge• • Knowledge about Knowledge about
TextsTexts
• • Motivation & Motivation & EngagementEngagement
• • Active Reading Active Reading StrategiesStrategies
• • Monitoring StrategiesMonitoring Strategies• • Fix-Up StrategiesFix-Up Strategies
*modified slightly from presentations by Joe Torgesen, Ph.D. Co-Director, Florida Center for Reading Research; www.fcrr.org
For Some, the Hardest Thing
They’ll Ever Do
The Easiest Thing To Teach
The Longer It Takes...
The Bigger Deficits Here
And Here
And the MOST Unmotivated
Here
Oral Reading is the EASIEST to Measure--Let’s Get This Down
and Add MORE Tools
Comes From Reading, But Is Not Reading
Reading Early, Well, and Wide (EWW) is Critical for Reading
Comprehension.
See: Hunter, P.C. (2012). It's not complicated! What I know for sure about helping our students of color
become successful readers. New York, NY: Scholastic.
Cunningham, A.E., & Stanovich, K.E. (1998). What reading does for the mind. American Educator, 8-15.
Denton, C.A., & Vaughn, S. (2010). Preventing and remediating reading difficulties: Perspectives from
research. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in
a three-tier model, including RTI (pp. 469-500). Bethesda, MD: National Association of School
Psychologists.
Linan-Thompson, S., & Vaughn, S. (2007). Research-based methods of reading instruction for English
language learners Grades K-4. Alexandria, VA: Association for Supervision and Curriculum
Development.
http://www.youtube.com/watch?feature=player_embedded&v=RiP-ijdxqEc
Comes From Reading, But Is Not Reading
Very Few Students Receive Evidence-Based or Sufficiently Explicit/Systematic
Comprehension Strategy Instruction, Especially in Navigating
Expository/Informational Text and Vocabulary, Learning the “Right Words” the
“Right Way.”
This is an INSTRUCTIONAL PROBLEM, Not An Assessment Problem!See: Feldman, K., & Kinsella, K. (2005). Narrowing the language gap: The case for explicit vocabulary
instruction. In Scholastic (Ed.), Read About: . New York, NYKamil, M.L., Borman, G.D., Dole, J., Kral, C.C., Salinger, T., & Torgesen, J. (2008). Improving Adolescent
Literacy: Effective Classroom and Intervention Practices: A Practice Guide. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education.
Marchand-Martella, N.E., Martella, R.C., Modderman, S.L., Petersen, H., & Pan, S. (2013). Key areas of effective adolescent literacy programs. Education and Treatment of Children, 36, 161-184.
Torgesen, J., Houston, D.D., Rissman, L.M., Decker, S.M., Roberts, G., Vaughn, S., . . . Lesaux, N. (2007). Academic literacy instruction for adolescents: A guidance document from the Center on Instruction. Portsmouth, NH: RMC Research Corporation, Center on Instruction.
We Need to Expand Our Assessment Repertoire
We Need to Add a Variety of Ways to Authentically Assess Comprehension in the
Materials We Expect Students to be Able to Read and UNDERSTAND
See:
Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A.
Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp. 671-698). Bethesda, MD:
National Association of School Psychologists.
Howell, K.W., & Nolet, V. (1999). Curriculum-based evaluation: Teaching and decision making (3rd ed.). Atlanta, GA: Wadsworth Publishing Company.
Big Ideas
1. When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general
reading skills.
2. A strong empirical relation has been demonstrated between R-CBM and measures of general reading
and reading comprehension (RC).
3. Persuasive empirical data are not always persuasive of teacher opinion.
4. General reading skills are necessary, but not sufficient for comprehension.
5. RC is not a single “thing." Judgments are based on:
1. What a student is expected to read.
2. How the student will be assessed.
6. RC is influenced by a number of important variables, including
1. General reading ability
2. Language, especially vocabulary and familiarity with cultural idioms
3. Knowledge, especially content knowledge and knowledge of the world
4. Metacognitive strategies, including motivation and interest