DYS StARR Project March 2010 SREE Presentation, Washington DC Ohio Department of Youth Services...
-
Upload
austin-greer -
Category
Documents
-
view
215 -
download
0
Transcript of DYS StARR Project March 2010 SREE Presentation, Washington DC Ohio Department of Youth Services...
DYS StARR ProjectDYS StARR ProjectMarch 2010March 2010
SREE Presentation, Washington DCSREE Presentation, Washington DC Ohio Department of Youth ServicesOhio Department of Youth Services
Striving Readers Grant Funded by IESStriving Readers Grant Funded by IESProject Evaluation By Ohio State UniversityProject Evaluation By Ohio State University
Issues on Design and Impact from a Issues on Design and Impact from a Randomized Control Trials StudyRandomized Control Trials Study
William E. Loadman, PIWilliam E. Loadman, PI Raeal MooreRaeal Moore Jing ZhuJing Zhu Richard LomaxRichard Lomax
LocalLocal ContextContext
Incarcerated youth ages 12-21Incarcerated youth ages 12-21 7 High Schools (6 Male 1 Female)7 High Schools (6 Male 1 Female) Highly restrictive environmentsHighly restrictive environments Students with limited reading abilityStudents with limited reading ability
– 2/3 below grade level2/3 below grade level Average length of stay 10.5 monthsAverage length of stay 10.5 months Students with concomitant problemsStudents with concomitant problems No consistency on entry and exit of No consistency on entry and exit of
studentsstudents
Mobility IssuesMobility Issues ODYS release and return ODYS release and return
– 16% of Ineligible youth left and returned16% of Ineligible youth left and returned– 8% of Read 180 youth left and returned8% of Read 180 youth left and returned– 7.5% Traditional youth left and returned7.5% Traditional youth left and returned
Inter School movementInter School movement– 24% Ineligibles moved across schools at least once24% Ineligibles moved across schools at least once– 27% of Read 180 and Traditional youth moved across 27% of Read 180 and Traditional youth moved across
schools at least onceschools at least once– Ineligible youth moved as many as 5 times and Read Ineligible youth moved as many as 5 times and Read
180 and Traditional youth as many as three times180 and Traditional youth as many as three times
At any point in time there are approximately At any point in time there are approximately 1400 - 1700 youth across all DYS schools1400 - 1700 youth across all DYS schools
Targeted Intervention Logic ModelTargeted Intervention Logic ModelProgram
Inputs/ActivitiesClassroom Practices:
Intermediate Outcomes Short-Term Long Term
Student Outcomes
Leadership Training provided for Principals, Literacy Coaches, and District Staff [Initial and follow-up; 5 hours total
in Y1]
R180 Training for Teachers/Teacher Aides [initial 2 days and Semi-annual
follow-up; 15 hours total in Y1. Two 5 hour PD sessions (10 hrs
total) in Y2]
Scholastic Software for classroom including
supplemental materials. [including
Classroom supplies/materials]
Scholastic R180 Software Adapts instruction for
Students needs
A Maximum of 15 students are scheduled for 2-45
minute periods consisting of the 5 components of
Read180: whole group, individualized learning,
computer activities, small group (each 20 min), wrap
up (10 min)
Targeted Strategies Used include:
-Sequencing-Identifying main idea/detail
-Summarizing-Identifying cause/effect
-Making Inferences
Positive impact upon return to
home/community
Use of strategies to increase
comprehension/vocabulary
Reduce recidivism
Increased literacy, fluency, comprehension and confidence.
Reading at Grade Level
Increased engagement in
educational tasks.
Students are assessed using R-skills and Scholastic
Reading Inventory
Literacy Coach on classroom practices
Increased student self-efficacy
Implementation Data CollectionImplementation Data Collection
Daily implementation logsDaily implementation logs Weekly observation data (OSU)Weekly observation data (OSU) Quarterly feedback from Scholastic Quarterly feedback from Scholastic
observations observations Professional development attendanceProfessional development attendance
– Literacy coachesLiteracy coaches– Read 180 teachers/aidesRead 180 teachers/aides– All teachersAll teachers– PrincipalsPrincipals
Targeted Intervention Targeted Intervention Implementation ResultsImplementation Results
FacilityPD Attendance
Year 1PD Attendance
Year 2Instruction
Year 1Instruction
Year 2
1 Moderate High Needs Improv. Moderate
2 Moderate High Moderate Needs Improv.
3 High High Moderate High
4 High High High High
5 High High Moderate Moderate
7 High High Moderate Moderate
8 High High Moderate High
Total High High Moderate High
Targeted Intervention Targeted Intervention Implementation ResultsImplementation Results
OSU Observation ResultsOSU Observation Results– Read 180 implemented on ModelRead 180 implemented on Model– Variation across schoolsVariation across schools– Youth interested and engagedYouth interested and engaged– Differentiated instruction observedDifferentiated instruction observed– Youth disrupted class on 60% of early Youth disrupted class on 60% of early
visits, but youth not removed from visits, but youth not removed from classroomclassroom
– When either aide or teacher not in When either aide or teacher not in classroom, instruction very difficultclassroom, instruction very difficult
Scholastic observationsScholastic observations– Reported as on model/protocolReported as on model/protocol– Better able to meet 90 minute model in Better able to meet 90 minute model in
year two compared with year oneyear two compared with year one– Encouraged use of more strategies to Encouraged use of more strategies to
help keep youth engaged (year two)help keep youth engaged (year two)– Encouraged use of SAM reports to better Encouraged use of SAM reports to better
inform instruction (year two)inform instruction (year two)
Targeted Intervention Targeted Intervention Implementation ResultsImplementation Results
Impact Design Impact Design Targeted InterventionTargeted Intervention
Random assignment of student to Random assignment of student to condition (100% reading below grade condition (100% reading below grade level, 6 month stay, no GED/diploma)level, 6 month stay, no GED/diploma)– Condition 1 Read 180Condition 1 Read 180– Condition 2 Traditional English classCondition 2 Traditional English class
Project start-up baseline testingProject start-up baseline testing– CAT Reading and MathCAT Reading and Math– SRI Reading Lexile scoresSRI Reading Lexile scores
End of each term assessments on SRI (4 End of each term assessments on SRI (4 per year)per year)
Targeted Intervention Sample: Targeted Intervention Sample: Classrooms/TeachersClassrooms/Teachers
8-12 students per classroom across 8-12 students per classroom across facilities and quartersfacilities and quarters
One-on-one interaction was greater in One-on-one interaction was greater in Read180 relative to Traditional classesRead180 relative to Traditional classes
One Read 180 teacher and aide per One Read 180 teacher and aide per building/classroombuilding/classroom
All teachers certified in English/Language All teachers certified in English/Language ArtsArts
Targeted Intervention Sample:Targeted Intervention Sample:StudentsStudents
0
70.1
2.8
24
3.1
55
45
0.2
68.3
1.7
26.1
3.7
96.5
3.5
57.8
42.2
95.2
4.8
0
10
20
30
40
50
60
70
80
90
100A
sia
n
Bla
ck
His
pa
nic
Wh
ite
Mu
ltir
acia
l
Ma
le
Fe
ma
le
No
Ye
s
Race Gender SpecialEducation
R180 Traditional
Targeted Intervention Sample:Targeted Intervention Sample:StudentsStudents
0.3 0.3 15
0.3 0.3 1 0.3 0.30.4 0
42.2
0.84.5
0 0 0
29.1
0.4 0
17.2
30.1
43.7
22.5
0
10
20
30
40
50
60
70
80
90
100
R180 Traditional
12.8
13.1
2123.8 22.3
12.6
2.3 1 0.5 1.1
41.7
28.2
9.3
4.6
14.6
1.35
14.8
23.118.9
9.4
1.1 0.4 0.4 1.8
45.8
24.2
7.35.3
15.1
25.9
0
10
20
30
40
50
60
70
80
90
100
14 15 16 17 18 19 20 21 22
0 (Pr
e-enro
lled)*
*
8 9
10 11 12
gradu
ated
Age * Grade
R180 Traditional
Targeted Intervention AnalysesTargeted Intervention Analyses Treatment of the Treated (TTT)Treatment of the Treated (TTT)
– This study defines TTT based on being present This study defines TTT based on being present in class for at least 5 weeks in each of 2 or in class for at least 5 weeks in each of 2 or more quartersmore quarters
– Estimated 5 HLM analyses using the number of Estimated 5 HLM analyses using the number of quarters of actual treatment receivedquarters of actual treatment received
– The base model (grand mean centered) The base model (grand mean centered) included the following variables (white, age, included the following variables (white, age, Math CAT, Read CAT, disability, grade level, Math CAT, Read CAT, disability, grade level, institution and mobility) institution and mobility)
– The dependent variable was the SRI scoreThe dependent variable was the SRI score– Variables with p values of .08 or less were Variables with p values of .08 or less were
retained in each final model. retained in each final model.
TTT Descriptive ResultsTTT Descriptive Results
720
740
760
780
800
820
840
0 1 2 3 4 5 6 7 8
Number of Terms
SR
I Sc
ore
Comparison
READ 180
Overall
Time Plot of the Mean Responses for the READ 180 Group, the Comparison Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall Group, and the Overall including All Youthincluding All Youth
TTT Descriptive ResultsTTT Descriptive Results
500
550
600
650
700
750
800
850
900
0 1 2 3 4 5 6 7 8
Number of Terms
SR
I Sc
ore
Comparison
READ 180
Overall
Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall for and the Overall for Youth with Youth with Less Than Two Quarters of TreatmentLess Than Two Quarters of Treatment
TTT Descriptive ResultsTTT Descriptive Results
720
740
760
780
800
820
840
0 1 2 3 4 5 6 7 8
Number of Terms
SR
I S
co
re
Comparison
READ 180
Overall
Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall for and the Overall for Youth with Youth with At Least Two Quarters of TreatmentAt Least Two Quarters of Treatment
Overall Read 180 Comparison
n col % n col % n col %
Slope > 0 405 62.40% 248 72.09% 157 51.48%
Slope 0 244 37.60% 96 27.91% 148 48.52%
Total 649 100% 344 100% 305 100%
TTT Descriptive ResultsTTT Descriptive Results
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
0 2 4 6 80
500
1000
1500
# of Terms
SR
I S
core
Overa
llTra
dit
ion
al
Read
18
0
TTT HLM ResultsTTT HLM Results
Level 1:Level 1:
Level 2: Level 2:
Age is continuous; grand mean centeredAge is continuous; grand mean centeredMath CAT is continuous; grand mean centeredMath CAT is continuous; grand mean centeredRead CAT is continuous; grand mean centeredRead CAT is continuous; grand mean centeredDisability Status is dichotomized (0= not disabled; 1 = Disability Status is dichotomized (0= not disabled; 1 =
disabled)disabled)Grade level is continuous and ranges from 8-12; grand Grade level is continuous and ranges from 8-12; grand
mean centeredmean centered
iijiiij Njnijy 0,1,2,... and 1,2,..., for ,
iiii
iii
b0543
210
).GRDLVLGRDLVL()DISB().READCATREADCAT(
.)MATHCATMATHCAT().AGEAGE(
iiii b121 )TRTGRP().READCATREADCAT(
Youth (n = 542) who received between Youth (n = 542) who received between two and eight two and eight quarters of treatment. quarters of treatment.
TTT HLM ResultsTTT HLM Results
Fixed Effect Estimate SE t-ratio p-value Effect Size
Intercept α0 782.9 10.253 76.36 <.0001 --
Age α1 15.9294 5.4662 2.91 0.0037 0.02
MathCAT α2 10.4414 3.8851 2.69 0.0074 0.01
ReadCAT α3 37.6965 3.7074 10.17 <.0001 0.18
Disability α4 -28.3319 14.8666 -1.91 0.0572 0.01
Grade Level α5 17.3026 7.2463 2.39 0.0173 0.01
ReadCAT*Time β1 3.458 1.1505 3.01 0.0028 0.02
TRTGroup*Time β2 22.342 3.4899 6.4 <.0001 0.12
Random Effect b0 b1
b0 18440*
b1 -846.83 2181.18*
18820*
Estimates for the Fixed & Random Effects Estimates for the Fixed & Random Effects Using 2- 8 Quarters of TreatmentUsing 2- 8 Quarters of Treatment
Regression Adjusted Means Regression Adjusted Means 2006-2007 (Year 1)2006-2007 (Year 1)
2 Quarters or more 3 Quarters or more
READ 180
Comparison READ 180
Comparison
Baseline Mean
784.34 773.27 783.88 773.36
Estimated Post-test Mean
856.95 827.95 879.99 820.12
Difference score
+72.61 +54.68 +96.11 +46.76
Regression Adjusted Means Regression Adjusted Means 2006-2008 (Year 1 & 2)2006-2008 (Year 1 & 2)
2 Quarters or more
3 Quarters or more
4 Quarters or more
READ 180
Comparison
READ 180
Comparison
READ 180
Comparison
Baseline Mean
766.98 753.64 764.71 757.11 756.17 759.64
Estimated Post-test Mean
813.02 768.34 823.28 766.79 842.02 757.46
Difference score
+46.04 +14.7 +58.57 +9.68 +85.85 -2.18
a
Summary of IssuesSummary of Issues Staff turnoverStaff turnover Release datesRelease dates Time in treatmentTime in treatment Student attendanceStudent attendance Student mortalityStudent mortality Local building level perturbationsLocal building level perturbations Student cooperationStudent cooperation Variability of SRI measureVariability of SRI measure Psychometric properties of SRIPsychometric properties of SRI Use of CAT as an additional outcome Use of CAT as an additional outcome
measuremeasure
Summary of Methodological IssuesSummary of Methodological Issues
Tracking student entry, movement, exit Tracking student entry, movement, exit and returnand return
PowerPower SRI measure quality/variabilitySRI measure quality/variability Missing dataMissing data Losing dataLosing data Maintaining student confidentialityMaintaining student confidentiality Reporting issuesReporting issues Use of dataUse of data
Summary of Issues Cont’dSummary of Issues Cont’d
Student disruptive behavior in classStudent disruptive behavior in class Student’s placed in isolationStudent’s placed in isolation Student’s removed from classStudent’s removed from class Student receipt of treatmentStudent receipt of treatment Student assessment and cooperationStudent assessment and cooperation Teacher cooperationTeacher cooperation Student release dateStudent release date
Summary of FindingsSummary of Findings Read 180 being implemented on model and according Read 180 being implemented on model and according
to protocolto protocol Randomized control trials able to verify causal Randomized control trials able to verify causal
relationshiprelationship Approximately 72% of Read 180 students demonstrate Approximately 72% of Read 180 students demonstrate
a positive slope of increasing reading scores over a positive slope of increasing reading scores over time, compared with about 51% for the traditional time, compared with about 51% for the traditional group group
Read 180 adds about 22 Lexile points per quarter of Read 180 adds about 22 Lexile points per quarter of treatment over and above the traditional group treatment over and above the traditional group performance performance
Students in Read 180 significantly out perform Students in Read 180 significantly out perform students assigned to the traditional group students assigned to the traditional group
Effect sizes vary according to model, but most are Effect sizes vary according to model, but most are trivialtrivial
Most Read 180 students gain, but are still not reading Most Read 180 students gain, but are still not reading at grade levelat grade level
Future ResearchFuture Research Student perception of Read 180 content Student perception of Read 180 content
and structureand structure Impact analysis with CAT as an outcome Impact analysis with CAT as an outcome
measure.measure. Sub-analysis determining the influence of Sub-analysis determining the influence of
disability status on program intervention disability status on program intervention impacts. impacts.
Analysis of the influence of program on Analysis of the influence of program on recidivism.recidivism.
Sub-analysis of reason for incarceration, Sub-analysis of reason for incarceration, reading improvement, and recidivism. reading improvement, and recidivism.