Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt...

Universal Screening of Academics and Behavior in an RTI Framework

A collaboration of Vanderbilt University and the University of Kansas

Funded by U.S. Department of Education. Office of Special Education Programs, Judy Shanley, Project Officer

Award No. H324U010004

Daryl Mellard

April 1-2, 2008Virginia’s RTI Institute

Fredericksburg, VA

Acknowledgements from previous presentations• Marcia Invernizzi, U. of Virginia,

November 2007 presentation at Roanoke

• Hugh Catts, U. of Kansas, April 2006 presentation at NRCLD National SEA conference on RTI, KCMO (available at http://nrcld.org/sea/index.html)

A Little Overview• General principles

about screening

• Early reading screening

• Behavioral screening

Screening Component in an RTI Framework1. Academic and behavioral prediction

2. Measures that are quick, low cost, repeatable, critical (predictive) skills, minimal administration training

3. Question: Student at-risk?

4. Affirmative answer: More attention (assessment/ intervention) to class or student

5. Criteria: Criterion benchmark or normative

Screening Accuracy

Three influences:• Base rate.

• Diagnosticity.

• Values of those setting the cutoff (criterion) score (i.e., resource priorities).

Constructing Screening Measures

• Match the curricular demands facing students

• Proximity of the screening test to the criterion performance (Closer should yield higher accuracy.)

• Should include multiple, related indicators

Screening AccuracyScreening Accuracy

Particular attention is given to the Particular attention is given to the accuracy of screening instrumentsaccuracy of screening instruments

Errors in identification can be costlyErrors in identification can be costly - over identification- over identification - under identification- under identification Accuracy typically quantified within a Accuracy typically quantified within a

clinical decision making modelclinical decision making model

Clinical Decision Making Clinical Decision Making ModelModel

At risk Not at risk

Norm

al

RD

Ou

tcom

e

True Positive

a

False Negativ

e

bTrue

Negative

d

False Positive

c

Screen

Sensitivity

a / a + b

Specificity

d / c + d

Positive Predictive

Power

a / a + c

Negative Predictive

Power

d / b + d

Accuracy of screening is determined by Accuracy of screening is determined by

…… How well your instrument separates How well your instrument separates

those who eventually will have a those who eventually will have a problem from those who will notproblem from those who will not

What you choose as a cut-off scoreWhat you choose as a cut-off score

The Ultimate ScreenThe Ultimate Screen

TP100

FN0

FP0

TN100

More Typical ScreenMore Typical Screen

TP80

FN20

FP20

TN80

More Typical ScreenMore Typical Screen

TP90

FN5

FP35

TN70

ROC CurveROC Curvehttp://www.anaesthetist.com/mnm/stats/roc/http://www.anaesthetist.com/mnm/stats/roc/

What to Measure?What to Measure?

What is the criterion?What is the criterion? Reading comprehension involves a Reading comprehension involves a

mixture of complex abilities mixture of complex abilities Role of each changes over time Role of each changes over time

Part 2:Early Reading Part 2:Early Reading ScreeningScreening

Predictive Predictive indicatorsindicators

Criterion Criterion measuresmeasures

Remember: Time Remember: Time interval is a big interval is a big influenceinfluence


Variables related to early reading Variables related to early reading - letter knowledge- letter knowledge- phonological awareness- phonological awareness- rapid namingrapid naming- vocabulary and grammar- vocabulary and grammar

- reading itself (non-word or word reading)- reading itself (non-word or word reading)


Variables related to later reading Variables related to later reading - word reading- word reading

- oral reading fluencyoral reading fluency - vocabulary and grammar- vocabulary and grammar - text comprehension- text comprehension

Reading Screening MeasuresHugh Catts, April 2006, http://nrcld.org/sea/index.html Texas Primary Reading Inventory (Foorman et al.,

1998). Dynamic assessment model (O’Connor & Jenkins,

1999). Catts, Fey, Zhang, & Tomblin (2001). Dynamic Indicators of Basic Early Literacy Skills

(DIBELS). Phonological Awareness Literacy Screening (Invernizzi,

Juel, Swank & Meier, 1997). CBM tools.

Early Screening ToolsEarly Screening Tools

Comprehensive Test of Phonological ProcessingComprehensive Test of Phonological Processing Test of Phonological AwarenessTest of Phonological Awareness Test of Early Reading AbilityTest of Early Reading Ability

All correlated with reading outcomes (moderate All correlated with reading outcomes (moderate range), but little data on sensitivity and specificityrange), but little data on sensitivity and specificity

Texas Primary Reading InventoryTexas Primary Reading Inventory(Foorman et al., 1998- www.tpri.org)(Foorman et al., 1998- www.tpri.org)

Designed to be used by teachers to Designed to be used by teachers to identify children at risk for RD and to identify children at risk for RD and to further evaluate their strengths and further evaluate their strengths and weaknesses in reading-related skillsweaknesses in reading-related skills

5 screens for K-25 screens for K-2ndnd grade grade Designed to hold false negatives to a Designed to hold false negatives to a

minimumminimum Includes an inventory of secondary Includes an inventory of secondary

measures to help rule out false positivesmeasures to help rule out false positives

TPRI (1998)TPRI (1998)K (Dec) predicting end of 1stK (Dec) predicting end of 1st

At risk Not at risk

Norm

al

RD

Ou

tcom

e 92 5

181143

Screen (shorten version)

Sensitivity

94.8%

Specificity

55.9%

Positive Predictive

Power

39.1%

Negative Predictive

Power

97.3%

Base rate 23%

Risk rate 55.8%

Dynamic Indicators of Basic Dynamic Indicators of Basic Early Literacy Skills (DIBELS)Early Literacy Skills (DIBELS)

Standardized and readily availableStandardized and readily available www.dibels.uoregon.edu www.dibels.uoregon.edu www.aimsweb.comwww.aimsweb.com Developed to monitor progress and inform Developed to monitor progress and inform

instructioninstruction

DIBELSDIBELSK (Fall) predicting end of 1st K (Fall) predicting end of 1st

At risk Not at risk

Norm

al

RD

Ou

tcom

e 8577 1824

122589345

Screen ( Initial sound fluency, Letter name fluency)

Sensitivity

82.5%

Specificity

56.7%

Positive Predictive

Power

47.9%

Negative Predictive

Power

87.0%

Base rate

32.5%

Risk rate

56.0%

CBM ToolsCBM Tools

Letter-Name FluencyLetter-Name Fluency Letter-Sound FluencyLetter-Sound Fluency Initial-Sound FluencyInitial-Sound Fluency Phoneme Segmentation FluencyPhoneme Segmentation Fluency Nonword Reading FluencyNonword Reading Fluency Oral Reading FluencyOral Reading Fluency Oral Retell FluencyOral Retell Fluency Maze FluencyMaze Fluency

CBM ToolsCBM Tools

Assessments given 3 or more times a Assessments given 3 or more times a year to evaluate growth in reading year to evaluate growth in reading (meeting benchmarks) (meeting benchmarks)

Each can be considered a screening Each can be considered a screening opportunityopportunity

O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)

Large battery of preliteracy skillsLarge battery of preliteracy skills- - rapid letter namingrapid letter naming (# of letters named from (# of letters named from

random list in 1 min)random list in 1 min)

- - syllable deletionsyllable deletion (say “baseball” without “ball”) (say “baseball” without “ball”)

- - segmenting phonemessegmenting phonemes (tell me how many (tell me how many

sounds in “saw”)sounds in “saw”)

- - phoneme repetition phoneme repetition (say “p” “I” “f” )(say “p” “I” “f” )

Chose cut-off scores to maximize sensitivityChose cut-off scores to maximize sensitivity

O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)K (K (Nov) predicting April of 1stNov) predicting April of 1st

At risk Not at risk

Norm

al

RD

Ou

tcom

e 15 0

19223

Screen (phonem seg, RLN, deletion)

Sensitivity

100%

Specificity

89.3%

Positive Predictive

Power

39.5%

Negative Predictive

Power

100%

Base rate

6.5%

Risk rate

16.5%

TPRI (1998)TPRI (1998)11stst (Oct) predicting end of 1st (Oct) predicting end of 1st

At risk Not at risk

Norm

al

RD

Ou

tcom

e 111 8

305175

Screen (letter-sound, blending, word reading)

Sensitivity

93.3%

Specificity

63.5%

Positive Predictive

Power

38.8%

Negative Predictive

Power

97.4%

Base rate

19.9%

Risk rate

47.7%

DIBELSDIBELSBeginning 1Beginning 1stst NWF predicting end 1st ORF NWF predicting end 1st ORF

At risk Not at risk

Norm

al

RD

Ou

tcom

e 7477 2956

165445067

Sensitivity

71.7%

Specificity

76.6%

Positive Predictive

Power

59.6%

Negative Predictive

Power

84.8%

Risk rate

39.1%

Base rate

32.6%

Dynamic AssessmentDynamic Assessment May have advantage over static assessmentMay have advantage over static assessment Measurement of ability over time in order to Measurement of ability over time in order to

monitor progress monitor progress Measurement of learners’ potential over the Measurement of learners’ potential over the

short termshort term Assessor actively intervenes during the Assessor actively intervenes during the

course of the assessment with the goal of course of the assessment with the goal of intentionally inducing changes in the intentionally inducing changes in the learner's current level of performance.learner's current level of performance.

““Mini-assessment” of response to Mini-assessment” of response to interventionintervention

O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)Oct 1Oct 1stst predicting April 1 predicting April 1stst

At risk Not at risk

Norm

al

RD

Ou

tcom

e 11 0

17826

Screen (phoneme seg, RLN, phoneme repetition)

Sensitivity

100%

Specificity

87.3%

Positive Predictive

Power

29.7%

Negative Predictive

Power

100%

Base rate

5.1%

Risk rate

17.2%

O’Connor & Jenkins (1999)O’Connor & Jenkins (1999)

Dynamic AssessmentDynamic Assessment- phoneme segmentation- phoneme segmentation- used Elkonin boxes to progressively - used Elkonin boxes to progressively

teach segmentation of a set of test items teach segmentation of a set of test items wordswords

- score based on the number of trials needed - score based on the number of trials needed to master the taskto master the task

O’Conner & Jenkins (1999)O’Conner & Jenkins (1999)Oct 1Oct 1stst predicting April 1 predicting April 1stst (dynamic) (dynamic)

At risk Not at risk

Norm

al

RD

Ou

tcom

e 10 1

1959

Screen

Sensitivity

90.9%

Specificity

95.6%

Positive Predictive

Power

52.6%

Negative Predictive

Power

99.5%

Base rate

5.1%

Risk rate

8.8%

Compton, Fuchs, Fuchs, Compton, Fuchs, Fuchs, & Bryant (2006)& Bryant (2006)

Screened in 1Screened in 1stst (Oct) predicting end of 2 (Oct) predicting end of 2nd nd

MeasuresMeasures

- CTOPP Sound Matching- CTOPP Sound Matching

- CTOPP Rapid Digit Naming- CTOPP Rapid Digit Naming

- WJPB-R Oral Vocabulary- WJPB-R Oral Vocabulary

- Word Identification Fluency (WIF)- Word Identification Fluency (WIF) Initial level, 5-week slopeInitial level, 5-week slope

Grade 1Grade 1Word-Identification Word-Identification

FluencyFluency

Teacher: Teacher: Read Read these wordsthese words..

Time: 1 minute.Time: 1 minute.

twotwo

forfor

comecome

becausebecause

lastlast

fromfrom ......

Compton et al. (2006)Compton et al. (2006)11stst (Oct) predicting end of 2 (Oct) predicting end of 2ndnd

At risk Not at risk

Norm

al

RD

Ou

tcom

e 35 2

15514

Screen (includes WIF level & slope – CTA)

Sensitivity

94.6%

Specificity

91.7%

Positive Predictive

Power

71.4%

Negative Predictive

Power

98.7%

Beyond First gradeBeyond First grade

Most common screening for Tier 2 has been Most common screening for Tier 2 has been oral reading fluency (ORF)oral reading fluency (ORF)

ORF strongly correlated with 3ORF strongly correlated with 3rdrd grade state grade state assessmentsassessments

Strong correlations do not necessarily Strong correlations do not necessarily translate into high sensitivity and specificitytranslate into high sensitivity and specificity

Measurement of level and slope may help Measurement of level and slope may help (e.g., dual discrepancy)(e.g., dual discrepancy) Must deal with potential scaling problemsMust deal with potential scaling problems

What does research tell us What does research tell us about screening?about screening?

Can identify children at risk for reading problems Can identify children at risk for reading problems Can be done as early as the fall of kindergartenCan be done as early as the fall of kindergarten Need to choose measures carefullyNeed to choose measures carefully Must match measures to curriculumMust match measures to curriculum - letter naming- letter naming - phonological awareness- phonological awareness - word reading- word reading - text reading- text reading Must not forget about other factors related to Must not forget about other factors related to

comprehensioncomprehension - oral language - oral language

What does research tell us What does research tell us about screening?about screening?

False positive rates are high and efforts need to False positive rates are high and efforts need to be in place to limit the cost of over prediction be in place to limit the cost of over prediction

Progress monitoring within a RTI framework may Progress monitoring within a RTI framework may serve this purposeserve this purpose

Need to equate forms to scaleNeed to equate forms to scale Dynamic assessment has potentialDynamic assessment has potential

Screening for Possible Reading RiskGrade CBM Probe Cut-off

Kindergarten Letter Sound Fluency < 10 letters/minute

Grade 1 Word IdentificationFluency

< 15 words on list/minute

Grade 2 Passage Reading Fluency

< 15 words in text/minute

Grade 3 Passage Reading Fluency

< 50 words in text/minute

Grade 4 Maze Fluency < 10 Maze replacements/2.5 minutes



Note: These figures may change pending additional RTI research.

Tier 1–Primary Prevention:Confirming Risk Status With PM• At the end of 5-8 weeks, student risk

status is confirmed or disconfirmed.

Grade

InadequateReading

Slope

Inadequate Math

ComputationSlope

Inadequate Math

Concepts and Applications

Slope

Kindergarten < 1 (LSF) < 0.20 < 0.20

Grade 1 < 1.8 (WIF) < 0.25 < 0.30

Grade 2 < 1 (PRF) < 0.20 < 0.30

Grade 3 < 0.75 (PRF)

< 0.20 < 0.50

Grade 4 < 0.25 (Maze)

< 0.50 < 0.50


< 0.50 < 0.50


< 0.50 < 0.50

Note: These figures may change pending additional RTI research.

Dr. Marcia InvernizziUniversity of Virginia

Universal Literacy Screening:

First Steps Toward Prevention & Intervention

Universal Literacy Screening

Screen routinely Fall – Mid-year – Spring

Why? Ensures that students who need additional

support do not go too long before receiving additional instruction/intervention.

Helps identify the “point of entry” into the tiers of RtI intervention & the kinds of supports needed.

Monitors student growth over time

Phonological Awareness Literacy Screening

The state-provided screening tool for Virginia’s EIRI Consists of three instruments:

PALS-PreK (for preschoolers) PALS-K (for students in kindergarten) PALS 1-3 (for students in grades 1-3)

Measures young children’s knowledge of important literacy fundamentals.

45

PALS Instrument Content AreasPALS-PreK PALS-K PALS 1-3

Print and Word Awareness X

Nursery Rhyme Awareness X

Name Writing X

Rhyme Awareness X X

Beginning Sound Awareness X X

Alphabet Knowledge X X X

Letter Sounds X X X

Concept of Word X X

Blending X X

Sound-to-letter X X

Spelling X X

Word Recognition in isolation X X

Fluency X


Purpose #1: Identification of children in need of further

assessment and/or intervention

Solution: PALS class reports available on the PALS

website: http://pals.virginia.edu


Purpose #2: Provision of feedback about how a class is

performing so that classroom-based curriculum or instructional issues can be identified as soon as possible.

Solution: PALS class reports available on the PALS

website: http://pals.virginia.edu


Purpose # 3: Diagnosis of children who may have had a poor

testing experience. Solution:

PALS individual student reports available on the PALS website: http://pals.virginia.edu

Online Score Entry & Reporting

Activities for TeachersPALS Web site Instructional resources

http://pals.virginia.edu

Fitzpatrick, J.. (1997). Phonemic Awareness. Cyprus, CA: Creative Teaching Press. (p. 30)

Please contact the PALS office if you have any questions or concerns!

1-888-UVA-PALS [email protected]

ReferencesReferences Catts, H.W., Fey, M.E., Zhang, X., & Tomblin, J.B. (2001). Catts, H.W., Fey, M.E., Zhang, X., & Tomblin, J.B. (2001).

Estimating risk for future reading difficulties in kindergarten Estimating risk for future reading difficulties in kindergarten children: A research-based model and its clinical implications. children: A research-based model and its clinical implications. Language, Speech, and Hearing Services in the Schools, 32, Language, Speech, and Hearing Services in the Schools, 32, 38-50.38-50.

Compton, D.L., Fuchs, D., Fuchs, L.S., & Bryant, J.D. (2006). Compton, D.L., Fuchs, D., Fuchs, L.S., & Bryant, J.D. (2006). Selecting at-risk readers in first grade for early intervention: A Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, Journal of Educational Psychology, 98, 394-40998, 394-409..

Foorman, B.R., Fletcher, J.M., Frances, D.J., Carlson, C.D., Chen, D., Foorman, B.R., Fletcher, J.M., Frances, D.J., Carlson, C.D., Chen, D., Mouzaki, A., Schatschneider, C., Wristers, K., & Taylor, R. (1998). Mouzaki, A., Schatschneider, C., Wristers, K., & Taylor, R. (1998). Technical Report Texas Primary Reading Inventory Technical (1998 Technical Report Texas Primary Reading Inventory Technical (1998 Edition). Edition). Houston, TX: Center for Academic and Reading Skills and Houston, TX: Center for Academic and Reading Skills and University of Houston. University of Houston.

O’Connor, R.E., & Jenkins, J.R. (1999). Prediction of reading O’Connor, R.E., & Jenkins, J.R. (1999). Prediction of reading disabilities in kindergarten and first grade. disabilities in kindergarten and first grade. Scientific Studies of Scientific Studies of Reading, 3, Reading, 3, 159-197. 159-197.

Part 3: Behavioral ScreeningPart 3: Behavioral Screening

Challenges in Challenges in screening behaviorscreening behavior

Recognize age Recognize age influences on influences on predictive and predictive and criterion datacriterion data

Alternative Alternative proceduresprocedures

Challenges in Behavioral Screeningvan Lier, Verhulst & Crijnen (2003)

• Screening was originally for detecting the presence or absence of highly specific medical conditions that could be detected in a benign presymptomatic stage and for which treatment was available.

• Disruptive behaviors lack this specificity and unitary underlying conditions; multi-factorial

• Lack a well-delineated onset affecting valid detection

Other Challenges• Referral by regular classroom teachers

is regarded as the most vulnerable to bias due to differing expectations (Hersh & Walker, 1983; Lloyd et al., 1991)

• Referral peak for students with behavioral problems occurs in grade 9 (academic problems grades 2 and 3)

Social-environmental Domains of Behavioral InfluenceMulti-factorial influences across

developmental periodsFamilyPeer groupSchool and NeighborhoodIncorporate into a multiple gating

procedure (e.g., Loeber, 1990; Lochman, 1995)

Behavioral DimensionsInternalizing

• Depression

• Phobias

• Social isolation

• Peer isolation

Externalizing

• ADHD

• Oppositional defiant disorder

• Conduct disorder

Extra-Ordinary ChallengesChallenges

• Stressful life events

• Exposure to community violence

• Maltreatment

• Poverty

• Divorce

• Maternal mental illness

Generalization

• Developmental consistency

• Gender consistency

• Social group consistency

Family Survey Items (Multiple item composite score)• History of maternal

mental illness• High maternal anxiety• Rigid parental

perspectives (attitudes, beliefs, values)

• Frequency of positive parental interactions, esp. with mother

• Stressful life events• Large family size• Minimal maternal

education level• Head of household

in unskilled occupations

• Disadvantaged minority status

• Single parenthood

Academic PredictorsUse an percentile

index of the number of --

• absences• school changes• failed courses• grades repeated• discipline referralsGrade x school level

data

Student Self-Ratings as Predictors

The degree to which the student perceives -- • School related

Staff cares about the students. Racial tension in school between staff and

students. Sense of being treated fairly.

• Peer related Work hard in school. Discuss schoolwork/intellectual topics. Cheat on tests. Pressure to use drugs.

Instruments for Multi-gating (1)Inattention/Overactivity with Aggression (IOWA

Conners) 10 items for teacher rating1. Fidgeting2. Hums and makes odd noises3. Excitable, impulsive4. Inattentive, easily distracted5. Fails to finish things, short attention span6. Quarrelsome7. Acts “smart”8. Temper outbursts (explosive/unpredictable)9. Defiant10. Uncooperative

Multi-gating screening (2)• Conners Abbreviated Symptom

Questionnaire (CASQ)

• 10 items• Inattention

• Overactivity

• Impulsivity

Results for consideration1.5 SD on the IOWA Conners were

judged as situationally disruptive in the classroom rather than pervasively disruptive

2.0 SD on the IOWA Conners and CASQ more conservative; students in most need of services

Systematic Screening for Behavior Disorders • Walker and Severson, 1990• Items on externalizing and internalizing factors• 1st gate: Classroom teacher nomination of 3

students• 2nd gate: Teacher rating of 3 highest on

external and internal factors; best for behavior problems

• 3rd gate: School staff assess on 2 measures of school adjustment with direct observations

• Critical Events Index: 33 External & internal behavior problems; Least expensive screen

Supplemental measures• Academic Engaged Time: 2, 20 minute

classroom observations; % engaged

• Peer Social Behavior: 2, 20 minute playground observations; level, quality, and distribution of behavior

School Social Behavior Scale (Merrill, 1993)Two broad domains

1. Social competence: interpersonal skills, self-management skills and academic skills

2. Anti-social behavior: hostile-irritable, violation of school rules, disruptions of school activities

• Teacher or staff student ratings; 65 items

• Not so great on internal dimension

Revised Behavior Problem Checklist (Quay & Peterson, 1987)• Teacher rating scale; ages 5 to 18

• Lack of representative national norms

• Spanish translation

• 89 items; 20 minutes

Psychosocial Constraints Severson, H.H., Walker, H.M., Hope-Doolittle, J.,

Kratochwill, T.R. & Gresham, F.M. (2007). Proactive, early screening to detect behaviorally at-risk students. Journal of School Psychology, 45, 193-223.

Sameroff. A.J., & Rosenblum, K.L. (2006). Annals of New York Academy of Science, 1094, 116-124.

Sameroff, A.J., Peck, S.C., and Eccles. J. S. (2004). Development and psychopathology, 16, 873-896.

Phelan, P., Yu, H.C., & Davidson, A.L. (1994). Navigating the psychosocial pressures of adolescence: The voices and experiences of high school youth. American Educational Research Journal, 31 (2), 415-447.

School-wide Information System• Horner and Sugai

• Web-based software system

• Recording, entering, ordering, and reporting Office Detention Referrals

• http://SWIS.org

Thank YouOn the web @ nrcld.org

Daryl [email protected]

Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt...

Documents

Transcript of Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt...