Fast Track Program Evaluation Using Assessments Diagnostically

Philip A. Streifer, Ph.D.Bristol Superintendent of Schools; UCONN Executive Leadership

ProgramYvel Crevecoeur

Doctoral Student, Department of Educational Psychology, program in Special Education

Primary Issues

• Bristol an urban leader in reform• Next phase of development: program

evaluation and focus on instructional excellence• Primary reading intervention program:

Fundamentals of Literacy Development (FOLD)• Rising CT AYP Standard for 2008: Need to track

student gains over time vs. NCLB/AYP year by year analysis

Pressures

• 181 First Days of School• Growing Poverty Rate• Manage Expectations in City

– Article for upcoming ‘Chamber Views’– Ongoing speaking and publications on challenges

• Focusing work of 1000 people• Rising AYP Standards and City confidence in

schools: can we ‘get over the bar’?

Challenges

• Migration• Economically Disadvantaged• Rising State NCLB Standards for 2008

Reading – 68 to 79% proficiency Math – 74 to 82% proficiency

• Special Education Placements and Costs

181 First Days of School One School’s Experience

Transfers In• July-Aug- Sep 53• October 16• November 11• December 4• January 24• February 12• March 11• April 1• May 0• June 0• TOTAL 132

Transfers Out• July-Aug- Sep 61• October 8• November 14• December 3• January 8• February 12• March 10• April 2• May 3• June 1• TOTAL 122

GRAND TOTAL for ONE YEAR: 254 Children in and out/School Population +/-350

Data Driven Reform and Improvement

Past, Present and Future

8

POWER STANDARDS

UNWRAPPED

STANDARDS

BIG IDEASESSENTIAL QUESTIONS

CURRICULUM &

PACING GUIDESEFFECTIVE

TEACHING STRATEGIE

S

GRADE/DEPT DATA

TEAM

ASSESSMENT OF

STUDENT LEARNING

SCHOOL DATA TEAM

DISTRICT DATA TEAM

IMPROVEDSTUDENT

ACHIEVEMENT

Data-Driven

Decision

Making

MAKING STANDARDS WORK

The Bristol Accountability Initiative

CT Mastery Test and CAPT

• CMT – new version for 2005-06• CAPT – new version for 2006-07• Both are tougher tests• Each year NCLB/AYP standard rises• NCLB/AYP standard for 2007-08 (spring 2008)

is rising dramatically

Matched Cohort Comparison on the Connecticut Mastery Test

Percent of Students Scoring At or Above ProficiencyGrade/Cohort Year Math Reading Writing

3 2006 82.7 68.5 85.3

4 2007 85.6 74.5 87.9

  Change + 2.9 + 6.0 + 2.6

4 2006 86.0 71.1 83.25 2007 83.8 74.8 89.2  Change - 2.2 + 3.7 + 6.05 2006 80.8 74.2 89.9

6 2007 84.0 75.5 80.9

  Change + 3.2 + 1.3 - 9.0

6 2006 81.7 74.5 79.37 2007 83.5 80.2 80.3  Change + 1.8 + 5.7 + 1.07 2006 79.6 80.0 78.0

8 2007 82.1 79.7 82.0

  Change + 2.5 - 0.3 + 4.0

Percent Increase

Matched Cohort Data (2006 to 2007) Writing*

Total MatchedStudents Tested

Grade /Cohort

Percent Moved FromBELOW Proficient toProficient and Above

Number ofStudentsMoved toProficient

and Above

590 Gr 3 - 4 48% 41 out of 85

634 Gr 4 - 5 57% 61 out of 107

619 Gr 6 -7 34% 43 out of 128

679 Gr 7-8 38% 56 out of 146

*Grade 5 to 6 matched cohort did NOT make gains in Writing.

Matched Cohort Comparison on the Connecticut Mastery TestPercent of Students Scoring At or Above Goal

Grade/Cohort Year Math Reading Writing3 2006 62.7 52.6 62.8

4 2007 66.4 58.4 68.9

  Change + 3.7 + 5.8 + 6.1

4 2006 68.6 58.6 61.75 2007 68.7 61.6 69.7  Change +0.1 + 3.0 + 8.05 2006 58.0 58.9 68.1

6 2007 63.7 63.1 59.3

  Change + 5.7 + 4.2 - 8.8

6 2006 60.2 60.5 55.37 2007 60.9 69.7 56.3  Change + 0.7 + 9.2 + 1.07 2006 56.3 67.3 51.7

8 2007 59.9 71.3 62.3

  Change + 3.6 + 4.0 + 10.6

Percent Increase

District CAPT 2007 ResultsDistrict CAPT 2007 Results

  Mathematics ScienceReading Across The

DisciplinesWriting Across the

Disciplines

 % At/Above

Goal% At/Above

Proficient% At/Above

Goal% At/Above

Proficient% At/Above

Goal% At/Above

Proficient% At/Above

Goal% At/Above

Proficient

Bristol 51.6 88.1 40.8 88.1 50.2 87.2 50.2 86.8

Male 54.3 87.6 44.9 86.8 45.6 81.3 40.1 81.7

Female 48.5 88.6 36.2 89.6 55.5 94.0 61.6 92.6

Black 28.3 71.7 20.8 77.4 46.2 71.2 36.5 71.2

Hispanic 25.4 69.8 10.8 70.8 26.6 79.7 27.0 73.0

White 57.1 91.7 47.1 91.3 53.4 89.6 54.0 89.7

F/R Meals 27.2 67.6 19.0 73.7 26.3 74.5 23.9 69.4

Full Price 58.2 93.6 46.8 92.0 56.7 90.7 57.2 91.4

Special Ed. 14.3 50.8 11.1 49.2 11.3 38.7 11.5 44.3

Not Special Ed. 55.7 92.2 44.1 92.4 54.4 92.4 54.3 91.3

Not ELL 51.9 88.7 41.2 88.4 50.7 87.5 50.5 87.3

*ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup.

Areas highlighted in yellow show where Bristol exceeded State averages.

Performance Level Graph – District vs. State AveragesPerformance Level Graph – District vs. State Averages

Gains Analysis: Matched Cohorts

• Grade 3 to 4 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Gain Beyond Expected– Writing: Expected Gain

• Grade 4 to 5 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Loss Below Expected– Writing: Expected Gain

Gains Analysis: Matched Cohorts

• Grade 5 to 6 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Gain Beyond Expected– Writing: Expected Gain

• Grade 6 to 7 (2006 to 2007)– Math: Expected Gain– Reading: Expected Gain– Writing: Expected Gain

Gains Analysis: Matched Cohorts

• Grade 7 to 8 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Gain Beyond Expected– Writing: Moderate Bain Beyond Expected

• Grade 8 to 10 (2004 to 2007)– Math: Moderate Gain Beyond Expected– Reading: Expected Gain– Writing: Expected Gain

Role of Program Evaluation

• FOLD: – working; needs to be expanded

• Read-180 vs. ReadAbout:– Gains Analysis

• ReadAbout Deployed in More Classes 2007-08 with Grant Funds

Return on Investment

• Bristol Per Pupil Expenditure is 125th out of 169 (upper end of lower third)

• CMT 2007: at or just above state averages• CAPT 2007: above state average• City of Bristol getting a good return on its

educational investment

New Pressing Issue• CT AYP standard rises dramatically in 2008 in reading and math• Projection of 2008 AYP indicates many schools will be cited under NCLB• Current district performance at state average on CMT; above state

average on CAPT– Poor AYP performance in 2008 will likely cause erosion of public

confidence and City support• Deeper analysis ongoing to determine:

– Areas of focus for each school– Safe Harbor status and why it is important– Gains analysis to show progress of students over time who remain in

the district; to shift focus and opinion and show staff that they are doing a good job

Key Strategies Going Forward

• AYP and Safe Harbor – achieving improvement for all children

• Program Evaluation – FOLD (first pilot)• Gains Analysis – Cohort Improvement Over

Time and Across Disparate Tests • Instructional Excellence

Preliminary 2008 AYP Analysis

23

FOLD Evaluation• Why were Hubbell Elementary School’s reading

scores so high?• Development of the Foundations of Literacy

Development (FOLD) evaluation process • Identifying Growth Model of Improvement for each

elementary school• “SWAT Team” and focused plans on other

intervnetions• Long term objective: Maximizing what really works

for students participating in FOLD.

24

FOLD Evaluation Process

• Semi-structured interview of principal who originally implemented FOLD:– Key design features– Developed pilot survey questions

• Reviewed and refined survey questions• Administered survey to literacy teachers and

principals

25

FOLD Evaluation Process• Preliminary method of analysis:

– Identified each school’s performance on indicators of DRA2, CMT, and history of achievement

• Ranked each school’s performance to identify:– Differences across high and low performing schools– Differences between respondents (i.e., principals and

literacy teachers) to identify features of implementation that need revision

Gains Analysis Logic

• AERA paper presentation 2007• Data must be matched pairs and equal interval

scales• Rescale data if using different tests• Must know the absolute possible range of each

dataset• Apply Modified Effect Size analysis• Interpret with reconceptualization of Cohen’s d

Gains Analysis – Different Scales• Rescale

– Reset to zero scale

– Ratio of one scale to the other

• Analysis– Cohen’s

denominator for pooled SD for populations (for unequal variances

range

range

rescalefinal2

122 T

TT T

2T T X -X

21

T1 T2sg

ES

Interpretation:

Less Overlap = More Difference in Scores +/-

We would expect little overlap and a positive Cohen’s d

Transformation Logic – Step 1

• Two Scales:– 0 to 10: range = 10– 20 to 40: range = 20

• Step 1: Put scales on same starting point:– 0 to 10: range = 10– 0 to 20: range = 20

• [Scores are reset to start at a value of zero. That is, tests with possible scoring ranges of A→B are rescaled such that scores range from 0→(B-A).]

| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

R1

R2

0 10 20 30 40 50

| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

R1

R2

0 10 20 30 40 50

Ratio of R1 to R2 = .5

| | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

| | | | | | | | | | 10| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

R1

R2

0 10 20 30 40 50

Step 2: Multiply R2 scores by ratio of the scales (.5) to reset to R1 scale

| | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

| | | | | 5 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

R1

R2

0 10 20 30 40 50

Rescale

• Range1/Range2 = Resets one score to the other• Example:

– R1/R2 = 5/10 = 0.5 (or…R2/R1 = 10/5 = 2.0)

Reconceptualizing Cohen’s d in this Context

Gains Analysis Findings…So Far

Read Mean SD Read Mean SD Math Mean SD Math Mean SD

3 to 4 3 4 3 4 4 to 5 4 5 4 5 3 to 4 3 4 3 4 4 to 5 4 5 4 5

Hi Poverty, Low Performing 0.68 215 237 30 34 -0.24 227 219 40 34 0.26 235 246 42 44 -0.11 238 232 54 55

Hi Poverty, High Performing 0.61 244 263 32 32 -0.21 261 252 43 43 0.42 259 276 43 39 -0.13 284 277 46 47

Low Poverty 0.43 238 258 43 50 -0.21 258 248 52 48 0.13 260 267 52 49 -0.15 277 268 55 63

Low Poverty 0.43 229 245 37 41 -0.30 256 243 43 42 -0.10 261 256 53 39 0.20 265 274 45 47

All Schools Improving the Same Over Time: Poorer Performing Schools Not Catching Up – Need Faster Growth – May be Unreasonable to Achieve

Using Data Mining to go Deeper - Variables

• School• Grade 3 Teacher• Grade 4 Teacher• Grade 5 Teacher• Gender• Ethnicity• SPED status• F-R Eligible• ELL status

• GR5 CMT Math Scale• GR5 CMT Read Scale• GR5 CMT Writing Scale• GR4 CMT Math Scale• GR4 CMT Read Scale• GR4 CMT Write Scale• GR3 DRA Reading Level

Results So Far:• Poverty not a significant factor in achievement; all

schools performing: one year gain (+/-) for one year instruction

• Teacher is associated with achievement – but not clear as to how– Strong teachers are assigned in many cases to lower

performing students which probably accounts for gains analysis

• Need to focus interventions on specific groups of students; identified through data mining and ‘teacher/administrator knowledge’

Summary• Past work by district on DDDM and Cultural

acceptance critical• Next Steps:

– Program evaluation and advanced data analysis– Focus on instructional excellence

• Marzano: Effective Teaching Strategies• Saphier & Gower: The Skillful Teacher

– Maintain City confidence – get to ‘safe harbor’ for 2008

• Change the Conversation from NCLB/AYP to Growth Over Time