School Selection and Randomization for a School Selection and Randomization for a School RCT of a Universal Social-Emotional School RCT of a Universal Social-Emotional

Learning and Literacy InterventionLearning and Literacy Intervention

Joshua L. Brown Joshua L. Brown Fordham UniversityFordham University

Conference on National and International Perspectives on Conference on National and International Perspectives on Place-Based Randomized Trials in EducationPlace-Based Randomized Trials in Education

Institute of Human Development and Social Change Institute of Human Development and Social Change

New York UniversityNew York UniversityOctober 3, 2008October 3, 2008

Principal InvestigatorsPrincipal Investigators

Joshua L. BrownJoshua L. Brown Stephanie M. JonesStephanie M. JonesFordham UniversityFordham University Harvard UniversityHarvard University

J. Lawrence AberJ. Lawrence AberNew York UniversityNew York University

NYC Study of Social and NYC Study of Social and Literacy DevelopmentLiteracy Development

Research TeamResearch Team::Genevieve Okada, Site CoordinatorGenevieve Okada, Site Coordinator

Suzanne Elgendy, Vanessa Lyles, Emily Pressler, RAsSuzanne Elgendy, Vanessa Lyles, Emily Pressler, RAsWendy Hoglund, Postdoctoral FellowWendy Hoglund, Postdoctoral FellowMaria LaRusso, Postdoctoral FellowMaria LaRusso, Postdoctoral Fellow

Juliette Berg, Catalina Torrente, GRAsJuliette Berg, Catalina Torrente, GRAs

Program PartnersProgram Partners:: Tom RoderickTom RoderickAudrey MajorAudrey Major

Morningside Center for the Teaching of Social ResponsibilityMorningside Center for the Teaching of Social Responsibility

FundersFunders::Institute for Education Sciences, DOEInstitute for Education Sciences, DOE

National Center for Injury Prevention and Control, CDCNational Center for Injury Prevention and Control, CDCWilliam T. Grant FoundationWilliam T. Grant Foundation

National Institute of Mental HealthNational Institute of Mental Health

AcknowledgementsAcknowledgements

OutlineOutline

1.1. Background/Context of StudyBackground/Context of Study

2.2. Program and Study DesignProgram and Study Design

3.3. Pre-Randomization ActivitiesPre-Randomization Activities

4.4. Matching and RandomizationMatching and Randomization

5.5. Implications of ApproachImplications of Approach

6.6. Current Preliminary Findings (Y1, Y1-2)Current Preliminary Findings (Y1, Y1-2)

7.7. Conclusions and Future ChallengesConclusions and Future Challenges

BackgroundBackground (1) (1)

Co-occurrence of social-emotional and Co-occurrence of social-emotional and behavioral problems with low academic behavioral problems with low academic achievement.achievement.

Theoretical and initial empirical links between Theoretical and initial empirical links between self-regulation and reading/math.self-regulation and reading/math.

Emphases on standardized testing and Emphases on standardized testing and instructional improvement have crowded out instructional improvement have crowded out attention to social-emotional-character attention to social-emotional-character development (among other things).development (among other things).

BackgroundBackground (2) (2)

Early efforts at whole school strategies to Early efforts at whole school strategies to prevent behavior problems, violence, and prevent behavior problems, violence, and substance use plagued by intervention design substance use plagued by intervention design and implementation fidelity problems.and implementation fidelity problems.

Early research on whole school strategies Early research on whole school strategies plagued by low power, and inappropriate plagued by low power, and inappropriate statistical analyses.statistical analyses.

Need to rigorously test promising but Need to rigorously test promising but unproven approaches to SEL/SACD.unproven approaches to SEL/SACD.

BackgroundBackground (3) (3)

Birth of “Social and Character Development” Birth of “Social and Character Development” Research Network.Research Network.

7 different interventions in 7 different sites.7 different interventions in 7 different sites. 7 Local Evaluations and 1 National Evaluation 7 Local Evaluations and 1 National Evaluation

(Mathematica Policy Research).(Mathematica Policy Research).

Reading, Writing, Respect and Resolution (4Rs)Reading, Writing, Respect and Resolution (4Rs)Program and Study DesignProgram and Study Design

The 4Rs ProgramThe 4Rs Program Universal, school-based intervention in literacy development, Universal, school-based intervention in literacy development,

conflict resolution, and intergroup understanding.conflict resolution, and intergroup understanding. 3 Primary components:3 Primary components:

7-unit literacy-based curriculum in conflict resolution and social-7-unit literacy-based curriculum in conflict resolution and social-emotional learning. emotional learning.

Each unit organized around grade-appropriate book, includes 2 Each unit organized around grade-appropriate book, includes 2 literacy activities, and 3-5 SEL lessons (21-35 total lessons). literacy activities, and 3-5 SEL lessons (21-35 total lessons).

Total possible activities per unit = 5-7Total possible activities per unit = 5-7 Total possible activities per year = 35-49Total possible activities per year = 35-49

Training and ongoing coaching of teachers in the delivery of the Training and ongoing coaching of teachers in the delivery of the 4Rs curriculum.4Rs curriculum.

25 hours introductory training25 hours introductory training Ongoing classroom coaching, minimum 12 contactsOngoing classroom coaching, minimum 12 contacts Learning kitLearning kit

Family ConnectionsFamily Connections 1 parent-child “homework” per unit1 parent-child “homework” per unit

Heuristic Model: 4Rs Child-Level StudyHeuristic Model: 4Rs Child-Level Study

4RsExperimental (classroom

and parent) vs.Control

TeacherDevelopment

Extended Opportunities &

Supports

Social-EmotionalSkills &

Behaviors

Literacy Skills &Academic

Achievement

ChildDevelopmentalOutcomes: SEL &Academic Achievement

4Rs:Instruction,Teacher Training & Coaching

Child Behavioral Dispositions & Normative Beliefs

The Classroom System:Culture and Climate

School Culture and Climate

Teacher-Child Relationships

Teacher Affective& PedagogicalProcesses &Practices

ClassroomEmotional, Instruct. &Org. Climate

Heuristic Model: 4Rs Setting-Level StudyHeuristic Model: 4Rs Setting-Level Study

Overall Study DesignOverall Study Design

3-year, 6 wave longitudinal experimental design3-year, 6 wave longitudinal experimental design 18 NYC elementary schools matched and 18 NYC elementary schools matched and

randomly assigned to 4Rs and control group (9 randomly assigned to 4Rs and control group (9 assigned to each group)assigned to each group)

Intervention is implemented school-wide, grades Intervention is implemented school-wide, grades K-6 for 3 yearsK-6 for 3 years

All 3All 3rdrd grade children in each school followed over grade children in each school followed over three years through 5three years through 5thth grade grade

Schools represent demographic character of NYC Schools represent demographic character of NYC public elementary schoolspublic elementary schools Racially/ethnically diverse; School lunch receipt ~70%; Racially/ethnically diverse; School lunch receipt ~70%;

Mobility/Stability = ~18%/60%; Suspensions = 23%Mobility/Stability = ~18%/60%; Suspensions = 23%

Pre-Randomization ActivitiesPre-Randomization Activities

Identifying Candidate SchoolsIdentifying Candidate Schools

Planning Year (January-March, 2004)Planning Year (January-March, 2004) History of practitioner’s work in NYC led to letters of History of practitioner’s work in NYC led to letters of

support from Regional Superintendentssupport from Regional Superintendents Facilitated direct contact with Local Instructional Facilitated direct contact with Local Instructional

Superintendents who recommended schools based on:Superintendents who recommended schools based on: no prior history of implementing 4Rsno prior history of implementing 4Rs willingness to implement program (and research) activities:willingness to implement program (and research) activities:

all Teachers participate in Intro. Training, teach curriculum ~1 all Teachers participate in Intro. Training, teach curriculum ~1 lesson/week, and receive ongoing coaching from 4Rs Staff Dev’slesson/week, and receive ongoing coaching from 4Rs Staff Dev’s

Principals attend 2-3 workshops/year and appoint “4Rs Liaison”Principals attend 2-3 workshops/year and appoint “4Rs Liaison” School administration and teachers cooperate with data collectionSchool administration and teachers cooperate with data collection

Resulted in LIS recommendations of 41 schoolsResulted in LIS recommendations of 41 schools

Planning Year (March-June, 2004)Planning Year (March-June, 2004) Goal: assess/recruit “viable” schools for program Goal: assess/recruit “viable” schools for program

implementation in context of research study, (i.e., implementation in context of research study, (i.e., capacity for sustained, high-quality implementation, but capacity for sustained, high-quality implementation, but room to improve; willing to be randomly assigned)room to improve; willing to be randomly assigned)

Process: Meetings and “walk-throughs” of all 41 Process: Meetings and “walk-throughs” of all 41 schools:schools: Individual meeting with Principals to present program model Individual meeting with Principals to present program model

and overall research designand overall research design Visits to classroomsVisits to classrooms ““Organizational Readiness” assessment completed by Organizational Readiness” assessment completed by

practitioners (co-developed with research team)practitioners (co-developed with research team)

Assessing Candidate SchoolsAssessing Candidate Schools

Principal LeadershipPrincipal Leadership Organizational skillsOrganizational skills Enthusiasm for/compatibility with 4Rs Enthusiasm for/compatibility with 4Rs Rapport with Students & StaffRapport with Students & Staff

Teachers & School Leadership TeamTeachers & School Leadership Team Relationship with PrincipalRelationship with Principal Enthusiasm for/quality of questioning about 4RsEnthusiasm for/quality of questioning about 4Rs Stress, morale & attitudes toward childrenStress, morale & attitudes toward children

School EnvironmentSchool Environment Tone of adult-child interactionTone of adult-child interaction Engagement and behavior of studentsEngagement and behavior of students Physical environment (e.g., use of bulletin boards, etc.)Physical environment (e.g., use of bulletin boards, etc.)

Organizational ReadinessOrganizational Readiness

17 schools eliminated from initial pool:17 schools eliminated from initial pool: Grade structure other than K-5 (e.g., no 5Grade structure other than K-5 (e.g., no 5thth grade, K-3) grade, K-3) Lack of Principal and/or teacher interest in programLack of Principal and/or teacher interest in program

E.g., can’t support school-wide implementation requirement and/or E.g., can’t support school-wide implementation requirement and/or balance competing academic demandsbalance competing academic demands

Highly chaotic environments (e.g., adult-adult/adult-child Highly chaotic environments (e.g., adult-adult/adult-child yelling, extensive behavior problems, frequent crises)yelling, extensive behavior problems, frequent crises)

Unwilling to risk random-assignment to Control condition Unwilling to risk random-assignment to Control condition (one preferred to purchase program)(one preferred to purchase program)

24 schools held staff vote, signed letters of agreement 24 schools held staff vote, signed letters of agreement for random assignmentfor random assignment

Recruiting/Selecting SchoolsRecruiting/Selecting Schools

Given potential for “bad draw” with small number of Given potential for “bad draw” with small number of schools, IES grantees agreed to pairwise match schools to schools, IES grantees agreed to pairwise match schools to ensure balance on key variables and increase precisionensure balance on key variables and increase precision

24 schools pairwise matched and rank-ordered based on 24 schools pairwise matched and rank-ordered based on “distance” of each school from every other eligible school “distance” of each school from every other eligible school across 20 key school characteristics across 20 key school characteristics 12 pairs 12 pairsSchool Characteristics Include:School Characteristics Include: Size (total N)Size (total N) Race/ethnic and gender compositionRace/ethnic and gender composition School lunch receiptSchool lunch receipt Attendance (Students and Teachers)Attendance (Students and Teachers) Reading achievement (% of students at or above proficiency on ELA test)Reading achievement (% of students at or above proficiency on ELA test) Within year student mobility/two-year stabilityWithin year student mobility/two-year stability Teacher full licensure and years of experienceTeacher full licensure and years of experience Expenditures Expenditures Organizational Readiness (Overall)Organizational Readiness (Overall)

Pairwise Matching of SchoolsPairwise Matching of Schools

Funding for 18-schools, kept 9 best matching pairs, but Funding for 18-schools, kept 9 best matching pairs, but maintained 3 back-up pairs during lead-up to program maintained 3 back-up pairs during lead-up to program implementation/data collectionimplementation/data collection

Random numbers generator used to assign 1 school in Random numbers generator used to assign 1 school in each pair to intervention and 1 to control conditionseach pair to intervention and 1 to control conditions

Post-random assignment, 2 schools and their respective Post-random assignment, 2 schools and their respective matches were dropped and 2 back-ups engagedmatches were dropped and 2 back-ups engaged Principal of Tx school had been previously trained in RCCPPrincipal of Tx school had been previously trained in RCCP LIS ultimately did not condone RCT design for her schoolsLIS ultimately did not condone RCT design for her schools

Note, pairwise matching can protect the experimental Note, pairwise matching can protect the experimental design (from selection bias) in case of schools dropping design (from selection bias) in case of schools dropping out after start of study (King et al., 2007)out after start of study (King et al., 2007)

Matching Matching Random Assignment Random Assignment

Implications of Matching for AnalysesImplications of Matching for Analyses

Should “blocks” (e.g., matched pairs) be regarded as fixed or Should “blocks” (e.g., matched pairs) be regarded as fixed or random effects? random effects?

Current debate in field, depends on:Current debate in field, depends on: Number of units per block (when only 2, need RE model)Number of units per block (when only 2, need RE model) Treatment effect heterogeneity (i.e, across matched pairs)Treatment effect heterogeneity (i.e, across matched pairs)

If large, RE model allows heterogeneity of Tx effect to contribute to If large, RE model allows heterogeneity of Tx effect to contribute to standard errors and tests for the average effect of Tx.standard errors and tests for the average effect of Tx.

Interest in generalizabilityInterest in generalizability In FE model, the blocks constitute the population or universe of In FE model, the blocks constitute the population or universe of

generalization generalization In RE model the blocks are seen as representing a larger universe of In RE model the blocks are seen as representing a larger universe of

possible blocks (or settings) in which Tx might be implemented.possible blocks (or settings) in which Tx might be implemented. See Schochet, 2004; Raudenbush, 2004; and Bloom, 2005; See Schochet, 2004; Raudenbush, 2004; and Bloom, 2005;

all SACD internal network communicationsall SACD internal network communications We estimate blocks as random effects at the school-level – We estimate blocks as random effects at the school-level –

most conservativemost conservative

Implications of ApproachImplications of Approach Evidence of effective matching process -- no Tx/Control Evidence of effective matching process -- no Tx/Control

differences in 20 initial matching variables, or baseline differences in 20 initial matching variables, or baseline constructs assessed via child, teacher, and parent-reportsconstructs assessed via child, teacher, and parent-reports

Focus on initial identification and subsequent selection of Focus on initial identification and subsequent selection of “viable” schools limits generalizability to district- or “viable” schools limits generalizability to district- or citywide populationcitywide population

However, features of school organizational capacity and However, features of school organizational capacity and support have been clearly linked to schools’ ability for support have been clearly linked to schools’ ability for quality program implementation (Payne, Gottfredson & quality program implementation (Payne, Gottfredson & Gottfredson, 2006)Gottfredson, 2006)

A fully representative sample of NYC elementary schools A fully representative sample of NYC elementary schools would yield many schools with weak engagement and early would yield many schools with weak engagement and early withdrawal from studywithdrawal from study

Although generalizable only to “willing” implementers, we Although generalizable only to “willing” implementers, we see preliminary evidence of Tx impactssee preliminary evidence of Tx impacts

Year 1 Child Impacts Year 1 Child Impacts (Jones et al, under review)(Jones et al, under review)

Main effects of Tx on 2 of 9 child outcomes (2L HLM)Main effects of Tx on 2 of 9 child outcomes (2L HLM) Controlling for baseline levels, children in the Tx group had Controlling for baseline levels, children in the Tx group had

lower mean levels of lower mean levels of Hostile Attribution BiasHostile Attribution Bias and and DepressionDepression than those in the control group at the end of Y1than those in the control group at the end of Y1

Tx by baseline covariate interactions for 5 of 9 outcomesTx by baseline covariate interactions for 5 of 9 outcomes E.g., Tx by baseline Behavioral Risk (“elevated” on teacher-E.g., Tx by baseline Behavioral Risk (“elevated” on teacher-

report aggression and/or conduct problems at baseline)report aggression and/or conduct problems at baseline) Children with the highest level of baseline behavioral risk Children with the highest level of baseline behavioral risk

show the greatest positive difference in show the greatest positive difference in Aggressive FantasiesAggressive Fantasies, , Teacher-Report of Academic SkillsTeacher-Report of Academic Skills, , Reading Scale ScoreReading Scale Score, and , and AttendanceAttendance between the intervention and control groups between the intervention and control groups

0

0.2

0.4

0.6

0.8

1

OverallClassroomQuality

Emotional Instructional Organizational

Year 1 Classroom-Level ImpactsYear 1 Classroom-Level Impacts (in Effect Sizes)(in Effect Sizes)

Classrooms in the Tx group had higher mean Classrooms in the Tx group had higher mean Overall Classroom Overall Classroom Quality scores, Quality scores, accounted for by higher meanaccounted for by higher mean Emotional Support Emotional Support

and and Instructional SupportInstructional Support scores, than the control group scores, than the control group

*

*

*

n.s.

Preliminary Year 1-2 Program ImpactsPreliminary Year 1-2 Program ImpactsChild-LevelChild-Level

Results: Child-LevelResults: Child-LevelTX Main EffectsTX Main Effects

Local: significant impacts for 2 of 6 constructsLocal: significant impacts for 2 of 6 constructs Child Self-Report Hostile Attributional Biases Child Self-Report Hostile Attributional Biases Child Self-Report Depression Child Self-Report Depression

Multisite: significant impacts for 3 constructsMultisite: significant impacts for 3 constructs Teacher-Report of AggressionTeacher-Report of Aggression Teacher-Report Social CompetenceTeacher-Report Social Competence Teacher-Report ADHD SymptomsTeacher-Report ADHD Symptoms

Y1-Y2 TX Main Effects: SummaryY1-Y2 TX Main Effects: SummaryIntercept

Unstand. Est. (SE)Slope

Unstand. Est. (SE)

Hostile AttributionalBias

-.034 (.029) -.043 (.023) t

Depression .013 (.034) -.065 (.023) *

TR: Social Competence (Emotion Regulation; Prosocial Behavior)

-.009 (.100) .137 (.077) *

TR: ADHD (Hyperactivity; Inattention)

.030 (.086) -.084 (.053) t

TR: Child Aggression .027 (.031) -.047 (.017) *

0 0.75 1.50 2.25 3.000.33

0.39

0.44

0.49

0.54

Time

Ho

sti

le A

ttri

bu

tio

na

l Bia

s (

0-1

)

Control

Treatment

Treatment

Control

TX on HAB SlopeTX on HAB Slope

Conclusions and Future ChallengesConclusions and Future Challenges Trade-offs between generalizability and design feasibility Trade-offs between generalizability and design feasibility

in school selection process for school RCTs.in school selection process for school RCTs. We opted for selectivity based on:We opted for selectivity based on:

LIS perspective of school need/capacityLIS perspective of school need/capacity Pre-randomization assessment by program practitioners of Pre-randomization assessment by program practitioners of

school organizational readiness, used to identify final sampleschool organizational readiness, used to identify final sample

Need rigorous and field efficient assessment tools that tap Need rigorous and field efficient assessment tools that tap multiple dimensions of school organizational capacitymultiple dimensions of school organizational capacity

How might RCT design innovations enable the inclusion How might RCT design innovations enable the inclusion of disorganized and at-risk schools most in need of of disorganized and at-risk schools most in need of intervention? intervention?