1

Best Practices in Assessment and Intervention for ELLMike Vanderwood, Ph.D.University of California-RiversideMike.vanderwood@ucr.edu

2

ELL Growth

9.6% of students in the US public schools are ELLs 25.2% of students in California schools are ELL, and

85.3% of these students speak Spanish ELLs’ literacy skills wane in comparison to non-ELL

students (NCES, 2005) ELLs have a higher risk of being placed into special

education across elementary grades, and are 40-50% more likely than their White peers to qualify for SLD. (Artiles, et al, 2005)

Schools often delay examination for Special Ed because they want to eliminate language interference

3

Current Practice Very few teachers and psychologists are

trained to work with students who have diverse language backgrounds

Some suggest that we have a national shortage of qualified staff and the shortage will continue to grow.

4

Foundation Of Practice Growing understanding that educational

practices should be guided by high quality research and standards of practice

New standards for determining whether a practice is “evidence based”

IES implementation of “What Works Clearinghouse”

AERA/APA/NCME Test Standards

5

Foundations: Test Standards and ELL

6

LEP Assessment (English language learners) Students are considered Limited English

Proficient when: they were not born in the U.S. and native

language is not English, or Come from environments where English is not

dominant language Bilingual individuals do not necessarily have

equal proficiency in both languages

7

Standards Must have validity evidence for the purpose

used Pg. 95. If student is not from the culture or

linguistic that produced norms, scores may not provide a valid comparison

Standard 9.3, bilingual students should be assessed to determine the proficiency in both languages

8

Standards (cont) Standard 9.10 determining language

proficiency needs to be based on a range of language features

Standard 9.11, interpreters need to be fluent in both languages, and have some knowledge of assessment

9

Test Bias for LEP (ELL) Significant flaws in research in this area Primary issue is determination of English

proficiency Studies should account for: acculturation, cultural

background, quality of instruction and educational history

Current evidence does not indicate bias, yet results are questioned by many

10

Foundations: Research Standards

11

Scientifically Supported Interventions Dept. of Education was reorganized to put a focus on

high quality research Institute of Education Sciences (IES) Created What Works Clearinghouse

http://www.whatworks.ed.gov/ WWC evaluates educational interventions Focused on applying randomized controlled

experiments as methodology to determine causation

12

What Works ELL Report

13

ELL Literacy Assessment

14

Prevention is Critical Several authors (Gersten, 2005; Vaughn,

2005; What works, 2007; Vanderwood, 2008) suggest RtI is an exceptionally appropriate service delivery approach for ELL

15

RTI Defined RtI is the practice of (1) providing high-

quality instruction/intervention matched to student needs and (2) using learning rate over time and level of performance to (3) make important educational decisions. Source: Response to Intervention: Policy

considerations and implementation (National Association of State Directors of Special Education-2005).

16

Quick Summary of What We Know for NS Early screening and Intervention impacts performance and

can reduce those who need special education Reading Performance is best described by performance in:

Phonological Awareness Phonics Comprehension Fluency

Progress monitoring with instruments that have strong reliability and validity can improve outcomes for students with academic problems

17

Does this literacy knowledge apply for ELLs? Additional Focus on oral language proficiency Influence of culture?

ELL Students are Often not included in literacy screens. Why?

Research suggests, we can accurately assess English early literacy skills as early as Kindergarten for ELLs.

18

Language Proficiency Degree of control one has over the language in

question Basic Interpersonal communication skill (BICS)

Communicating in socially related situations First area to develop in new language

Cognitive academic language proficiency skills (CALPS) Developed through academic activities

19

Language of Instruction Center for Research on the Education of Students Placed At

Risk (CRESPAR) Very few studies that meet current research standards Native language instruction appears to be most beneficial,

and does not harm English Language Dev. Early exit appears to work as well as late exit (research is not

exceptioinally clear).

http://www.csos.jhu.edu/crespar/techReports/Report66.pdf

20

Critical Point It is not necessary to wait to deliver English

Language literacy support/interventions until the native language is established

In fact, all ELL students’ literacy skills should be assessed as soon as possible (i.e., kindergarten).

21

Previous ELL Reading Research Chiappe, Siegel, & Wade-Woolley (2002)

858 Kindergarteners, 23 languages A significant relationship between PA and improved reading

was reported in this study for monolingual and ELL students Linan-Thompson & Hickman-Davis (2002)

Supplementary reading instruction with at-risk, low SES second grade monolingual and ELL students

30-minute supplementary instruction: phonological awareness, vocabulary, decoding, comprehension, and word analysis strategies

Positive outcomes for reading for monolingual and ELL students

22

Lesaux & Siegel (2003) Suggest that PA may be a better predictor of reading

development than oral language proficiency Development of reading for ELL students was not predicted

by English Language Proficiency PA instruction was effective for ELL students EL group developed quickly and by grade 2 were performing

as well or better on most tasks as EO students Metal-linguistic advantage for ELL may explain rapid

improvement PA is an important skill for EL. For poor performers, similar

profile exists to EO poor performers

23

Literacy Research with typical RtI measures What type of evidence to we need

Ability to identify students at risk Sensitivity and specificity

Sensitive to growth during intervention Use across curricula and interventions Minimally impacted by language proficiency?

24

Nonsense Word Fluency (NWF) Standardized, individually administered test of alphabetic principles Intended for children from mid to end of kindergarten through the

beginning of second grade Children are given 1 minute to read as many nonsense words as

possible Example:

Read whole word, such as “lut” or say the individual sounds, such as /l/ /u/ /t/

Psychometrics One month alternate form reliability is .83 Predictive validity in first grade with CBM ORF first grade

is .82, and CBM ORF second grade is .60

25

NWF Study # 1: Purpose of Study To examine the appropriateness of using the

NWF in assessing students’ reading readiness skills for an EL, urban population

26

Site Demographics 1 school site 100% Free/ Reduced Lunch Ethnicity:

83% Hispanic (n=165) 11% African American (n=21) 6% other (n=13)

Home language: 82% Spanish (n=186) 14% English (n=32) 5% other (n=12)

27

EL Status (Determined by CELDT): Beginning LEP N = 49 (21%)

Intensive/structured-English classes Early Intermediate Range N = 58 (25%)

Structured English class Intermediate Range N = 111 (47%)

structured-English/English only classes Early Advanced/Advanced range N = 16 (7%)

Excelled class

28

Mean Differences (All Signficant)

22.19 25.17 52.98

104 103 94

17.516 19.926 27.323

32.23 43.16 72.71

44 43 42

14.325 19.897 32.722

44.98 47.85 81.61

56 55 59

21.410 27.292 34.601

59.63 91.69 100.78

16 16 16

37.845 48.202 42.372

32.72 39.39 68.54

220 217 211

23.277 30.530 35.180

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

English LanguageLearner LevelBeginning LEP

Early Intermediate

Intermediate

Advanced

Total

Time 1 Median Time 2 Median Time 3 Median

29

Measuring Growth

31.5109 1.2220

92 91

23.90756 .92448

40.5000 1.4886

42 42

26.00211 .95634

37.5273 1.5337

55 53

26.37680 1.06314

41.1563 1.3290

16 15

41.86793 1.37047

35.7195 1.3679

205 201

26.82908 1.00874

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

Mean

N

Std. Deviation

English LanguageLearner LevelBeginning LEP

Early Intermediate

Intermediate

Advanced

Total

Total Growth from Time 1 to Time 3 Weekly Growth from Time 1 to Time 3

30

Results of Study NWF was able to significantly predict reading scores

on a state-mandated achievement test above and beyond EL level

There were significant mean differences on all NWF scores by EL Level.

NWF alone significantly predicted SAT9 reading composite at 11 of 12 opportunities

Growth was not significantly different by EL group

31

NWF Study # 2: SAMPLE METHODS 3-year longitudinal study Spanish-speaking ELL population

90% ELL in Grade 1 58% ELL in Grade 3

Assessed first-grade students: Nonsense Word Fluency SAT9

Assessed third-grade students: ORF MAZE CAT6

32

Correlation of Grade 1 Measures to Grade 3 Measures

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

NWF (Beginning)

NWF (Middle)

NWF (End)

SAT9

CAT6 ORF MAZE

.36

.38

.34

.17

.50

.57

.64

.25.26

.24

.14

.23

33

Specificity and Sensitivity Analysis VP = Valid positive; FN = False negative; FP =

False positive; VN = Valid negative; Pos. PV = Positive predictive value (VP/(VP + FP); Neg. PV = Negative predictive value (VN/(VN +

FN); Sensitivity = VP/(VP + FN); Specificity = VN/(VN + FP); Hit rate = (VP + VN)/(VP + FN + VN + FP).

34

Specificity and Sensitivity of NWF to ORFOutcome Measure NWF At Risk NWF Not at

RiskIndices

CBM ORF < 25th percentile VP = 19 FN = 15 Sensitivity = 55%

CBM ORF > 25th percentile FP =18 VN =82 Specificity = 82%

Pos. PV = 51% Neg. PV = 84% Hit Rate = 75%

35

Specificity and Sensitivity of NWF to MazeOutcome Measure NWF At Risk NWF Not at Risk Indices

Maze < 25th percentile VP = 30 FN = 40 Sensitivity = 43%

Maze > 25th percentile FP = 7 VN = 57 Specificity = 89%

Pos. PV = 81% Neg. PV = 59% Hit Rate = 65%

36

Specificity and Sensitivity of NWF to CAT6Outcome Measure NWF At Risk NWF Not at Risk Indices

CAT6 < 25th percentile VP = 16 FN = 18 Sensitivity = 47%

CAT6 > 25th percentile FP = 21 VN = 79 Specificity = 79%

Pos. PV = 43% Neg. PV = 81% Hit Rate = 71%

37

Progress Monitoring - Intervention Study #1: Research Question What is the impact of a phonological

awareness intervention implemented with at-risk ELL students?

How effective is progress monitoring in English with ELL.

Healy, Vanderwood, & Edelston. (2005)

38

Methods Participants

15 Low SES ELL first grade students (7 male, 8 female) with PSF and NWF baseline scores 30 and below

Materials Sounds and Letters for Readers and Spellers

(Greene, 1997) Progress Monitoring

PSF and NWF weekly

39

Methods (cont.) Procedures

Small group (max. 5) ½ Hour Sessions x2 per week Exited after PSF > 50 and NWF > 45 12 – 25 Sessions

40

Results 6 students exited at first exit point after 12 sessions 12 students (80%) exited by conclusion 2 students (13.3%) met exit criteria for PSF, but not

NWF 1 student (6.7%) met exit criteria for NWF, but not

PSF The group as a whole went from mean PSF and

NWF scores that were considered to be in the at risk range to mean PSF and NWF scores that were in the mastery level range

41

0

10

20

30

40

50

60

Assessment Dtaes

Co

rre

ct

Se

gm

en

ts

Goal Line

BaselineIntervention

Trendline

Example of PSF scores of a participant who was exited from the intervention

42

0

10

20

30

40

50

60

1/2

1/2

004

1/2

8/2

004

2/4

/200

4

2/1

1/2

004

2/1

8/2

004

2/2

5/2

004

3/3

/200

4

3/1

0/2

004

3/1

7/2

004

3/2

4/2

004

3/3

1/2

004

4/7

/200

4

4/1

4/2

004

4/2

1/2

004

4/2

8/2

004

5/5

/200

4

5/1

2/2

004

5/1

9/2

004

5/2

6/2

004

6/2

/200

4

6/9

/200

4

Assessment Dates

Co

rre

ct

So

un

ds

Goal Line

Trendline

Baseline Intervention

A participant’s NWF scores who met the exit criteria for PSF, but not NWF

43

0

5

10

15

20

25

30

35

40

45

50

Assessment Dates

Co

rre

ct

Se

gm

en

ts

Goal Line

Trendline

BaselineIntervention

A participant’s PSF scores who met the exit criteria for NWF, but not PSF

44

Progress Monitoring/Intervention Study #2: To examine the extent to which a direct and

explicit PA intervention impacts at-risk ELLs’ early literacy skills.

To replicate findings by Healy, et al., 2005 To examine the impact of a bilingual, direct

and explicit PA intervention with at-risk ELLs. To assess the extent to which progress

monitoring is effective with ELLs.

45

Method

10 Participants (8 males, 2 females) CELDT Levels (beginner, early intermediate, & intermediate levels) Three Pronged Screening Method

Bottom 25% of students on Developmental Reading Assessment DIBELS PSF & NWF (at-risk levels)

Intervention Curriculum Sounds and Letters for Readers and Spellers (Greene, 1997) with

modifications Progress Monitoring

One time per week using PSF and NWF tests

46

Procedures 2 groups of 5 students 30 minutes intervention 2x per week Reinforcement: Token Economy Treatment Integrity

A 10-item checklist was developed Assessed 30% of intervals with 90% accuracy Curriculum manual was used 100% of observations

Added Bilingual support and Spanish PA intervention

47

Results Baseline

PSF: 0 – 6 (M = 3, SD = 2.75) NWF: 0 – 32 (M = 10, SD = 9.75)

Final 3 Monitoring Data Points PSF: 19 – 72 (M = 56.8, SD = 20.6) NWF: 49 - 66 (M = 56.8, SD = 23.4)

Effect Size [Cohen’s d (1988); Pooled (Swanson & Saches-Lee (2000)]

PSF: 19.8; 1.9 (Large) NWF: 4.8; 1.3 (Large)

48

Conclusion In 20 sessions of supplemental English PA instruction, 8 of

10 students met or exceeded PSF and NWF goals This study provides further support for the use of RTI to

determine LD status among ELL students at-risk for reading failure.

3 of 3 students met PSF goals using the bilingual PA intervention

Trendline analysis indicates that the Bilingual PA intervention was as and more effective than English PA alone for this very small sample, but only for PA skills.

49

Oral Reading Fluency and Middle School Students, including EL Learners

50

CBM per Week Gains by English Fluency

00.10.20.30.40.50.60.70.80.9

1

EO FEP ELL

IL gain

GL gain

EO

FEP ELL

Grade level Word Per Week Gain

.57 N=1084

.60 N=294

.68 N=1415

Instructional level Word Per Week Gain

.40 N=841

.44 N=247

.56 N=1150

51

Assessment Conclusions Initial evidence that typical measures used for

monitoring progress for EO students can be used with ELL

Initial evidence that measures can pick up growth caused by an intervention

Initial evidence that rate of growth for ELL is similar to EO

52

Instructional Practices for ELL (Gersten & Geva, 2003) Explicit teaching Promotion of English Language Learning Phonemic awareness and decoding Vocabulary development Interactive teaching that maximizes student

engagement Opportunities for accurate responses with

immediate feedback

53

Vanderwood et al. English Intervention studies Used Dibels to select group Small group instruction (3 to 5) Explicit instruction in phonemic awareness and

phonics Frequent use of Choral responding Behavioral reinforcement techniques to maintain

high levels of engagement 30 minutes of intervention 2X per week Between 87-93% of students moved established

level on Dibels measures within 16 weeks

54

Experimental ELL PA Intervention All 12 Sessions include a Vocabulary Section,

and 5 Phonological Awareness Activities in the following order: Phoneme Production/Replication; Phoneme Segmentation and counting, Phoneme Blending, Phoneme Isolation, and Rhyming.

55

Sessions 1-3 Session 1: / t /; / s /; / m /; / b /; / k /; / f /,

Short / a /. Session 2: Review: / t /; / s /; / m /; / b /; /

k /; / f /, Short / a /. New : / r /; / h /; / j /; / n /; / l /; / p /.

Session 3: Review: / t /; / s /; / m /; / b /; / k /; / f /; / r /; / h /; / j /; / n /; / l /; / p /, Short / a /. New : / g /; / d /; / v /.

56

Spanish Intervention (Vaughn et al., 2006a) 69 ELL (Spanish) 1st graders at risk for

reading difficulty Screening was conducted in Spanish

(Woodcock Language Proficiency Battery-Spanish—LWID) and Experimental measure of Spanish word reading ability)

Randomly assigned by school (7 schools, 20 classrooms) to treatment or comparison.

57

Treatment: Systematic, explicit instruction in oral language and reading (Lectura Proactiva, Mathes, et al., 2003) for groups of 3 to 5 for 50 minutes a day.

In Spanish to match core literacy instruction Comparison: School’s standard reading

intervention (e.g., guided reading, reading recovery, tutoring)

58

Vaughn et al., 2006a Results Pre-Post design

No group differences in pre-test scores Assessed Spanish & English literacy & oral

language Treatment group scored higher on most

Spanish outcome measures No group differences on English outcome

measures

59

Intervention Starting Points (Vaughn, et al., 2006b) Started with assumption that ELL learn to read like

monolingual students: through phonological recoding and spelling sound patterns

Assumed that those students have problems learning how to read are struggling because they have not mastered the alphabetic principle.

Decodable text used throughout instruction Sight words were taught (words that are less

phonetically regular in English)

60

Instructional Design (Vaughn, et al., 2006b) Systematic and explicit instruction for

Phonemic awareness Phonemic decoding skills Word recognition fluency Construction of meaning Vocabulary Spelling Writing

Used the same intervention as others had used with EO students, but interspersed language support activities

61

Population 2 districts in Texas 4 schools

English intervention to at least 2 classes of ELL first graders (48 to 99% Spanish speaking)

At least 60% of school population was Latino 80% or more of 3rd grade students passed state

reading tests

62

Screening Assessment 14 first grade classrooms 2 tests in English, 2 in Spanish

Letter word Identification from Woodcock Language Proficiency Battery in English and Spanish

Word lists in English and Spanish Note: this is not currently an empirically supported

approach to screening 56 students met criteria in both languages, 48

participated in the study 24 treatment, 24 comparison

63

Intervention Structure 48 student Groups of 3 to 5 50 minutes per day from October to May Supplemental to core reading instruction (not as

replacement!) Teachers received 12 hours of professional development

before intervention and 6 hours after first 6 weeks Staff development throughout the year occurred and lessons

were videotaped Conducted intervention validity checks

64

Intervention implementation Fast paced, all students responding, followed

by individual turns Provided immediate feedback

65

Results Intervention students outperformed

comparison group on English letter naming, phonological awareness, other language skills and reading achievement

Differences were less significant on Spanish measures

66

Evaluating Interventions for ELL Structure: Direct Instruction is critical Size: Small group is essential Feedback process: immediate and often Content: similar to monolingual students with

enhanced vocabulary

67

Special Ed Eligibility Issues for ELL

68

Special Ed Assessment Issues Eligibility determination issues:

Non-verbal IQ tests

69

IQ/ Processing Assessment Techniques bilingual examiner Translator Translated version developed with norms translated version with English norms Translated version with foreign country norms non-verbal assessment Problem-solving

70

Testing Options English developed Test translated to foreign

language May not measure same characteristic in the foreign

language Questions may have changed in meaning Cultural groups may differ in conceptualization of

construct Violates Test Standards

Foreign norms with American translated test Different culture, issue of equivalence

71

Non-verbal for ELL Many lack appropriate psychometric evidence Predictive power is limited and not as strong

as traditional IQ tests Not normed with ELL population

72

Issues related to LD identification for ELL We need to improve our system at identifying and

intervening with ELL students who have academic problems. We need to identify students who are most at risk of

academic difficulties earlier than 3rd grade, preferably kindergarten.

We need to use resources in general and special education more efficiently.

We need to provide an integrated continuum of services that uses high quality data for entry to succeeding levels.

We need to re-conceptualize the construct of LD to integrate language proficiency.

73

We need to better connect our initial eligibility assessments to instruction and intervention (assessment to intervention link) and address issues of literacy development in L1 and L2

We need to integrate recent assessment science that suggests that progress monitoring can have substantial positive impact on academic outcomes.

74

RTI special education decision making Disability and Need Concept of Dual Discrepancy Disability:

Lack of response to high quality intervention Torgesen’s work shows about 6%

Need: Convergent Data District determined Norm referenced Individualized achievement test Large scale assessment Alternative assessment local norms Percentage for cutoff ranges from 6 to 10%

75

What about culture? MAMBI Acculturation surveys?

76

Conclusions Initial evidence early literacy measures can be

used to identify EL students at risk of literacy problems.

Initial evidence tools can be used assess growth during intervention

77

What we don’t know How does language proficiency affect the

ability of the measures to achieve their purposes?

How do we integrate culture into the decision making process?

How do we integrate native language knowledge into the process?