of PtiP revention and It tiI ntervention rograms on and ...€¦ · Inclusion Criteria •...
Transcript of of PtiP revention and It tiI ntervention rograms on and ...€¦ · Inclusion Criteria •...
Eff t f P ti d I t ti PEffects of Prevention and Intervention Programs on School Completion and Dropout
Results from a Systematic ReviewResults from a Systematic ReviewSandra Jo Wilson
b d hPeabody Research Institute
The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org
Acknowledgments
• Funding provided by the Campbell Collaboration and the Peabody Research Institute
• My co‐authors:– Mark W. LipseyMark W. Lipsey– Emily E. Tanner‐SmithChiungjung Huang– Chiungjung Huang
– Katarzyna Steinka‐Fry– Jan Morrison– Nianbo Dong
The Campbell Collaboration www.campbellcollaboration.org
Overview
• Discussion of different kinds of interventions that target school dropout.
• Systematic review methodology.• Preliminary results from the systematic review• Preliminary results from the systematic review.
The Campbell Collaboration www.campbellcollaboration.org
Risk‐oriented Programming
• There are a great variety of strategies for preventing school dropout.
• The different strategies are generally developed out of different theories about the causes of dropout.p
• The biggest risk factors for school dropout are pregnancy low school attendance and poor academicpregnancy, low school attendance, and poor academic performance.
The Campbell Collaboration www.campbellcollaboration.org
Risk‐oriented Programming
Risk Factor for Dropout Targeted Program or StrategyTeenage pregnancy Provide child care services or financial
assistance to young mothers.Poor attendance Monitor and reward attendance.P d i f P id l t l d i iPoor academic performance Provide supplemental academic services.
Lack of support for higher education Provide college‐oriented curricula and advising for students.g
Family and personal problems Connect students with appropriateservices through case management, provide mentors or counselorsprovide mentors or counselors.
Traditional school structure doesn’t work for some students
Restructure or reorganize the school day, curriculum, etc. to better fit the needs of different types of students.
Students lack purpose, goals, understanding of the purposes of
Create career/work oriented courses;involve students in community.
The Campbell Collaboration www.campbellcollaboration.org
g p peducation
y
Objectives of the ReviewObjectives of the Review
• Summarize the available evidence on the effects of prevention and intervention programs aimed at increasing school completion or reducing dropout.c eas g sc oo co p et o o educ g d opout
• Primary emphasis on the comparative effectiveness of different types of programs in the context of differentdifferent types of programs, in the context of different methodologies, different levels of implementation quality and different configurations of studentquality, and different configurations of student characteristics.
The Campbell Collaboration www.campbellcollaboration.org
Inclusion Criteria
• Interventions are school‐based, school‐affiliated, or community‐based programs.
• Outcomes must be dropout, graduation, or attendance.p g• Students are between the ages of 4‐18.
Recent dropouts (18 22) are eligible if school completion is– Recent dropouts (18‐22) are eligible if school completion is explicit goal.For students under the age of 12 dropout must be assessed– For students under the age of 12, dropout must be assessed.
• Experimental and quasi‐experimental designs are l d dincluded.
• 1985 and later, any language, any publication type.
The Campbell Collaboration www.campbellcollaboration.org
Literature Search• Comprehensive literature search of major educational databases, including grey literature.databases, including grey literature.
• Search of international databases not yet completed.Id ifi d 16 962• Identified 16,962 reports– 2,627 reports were deemed potentially relevant and retrieved
– 878 reports were determined to be eligible878 reports were determined to be eligible• 416 studies (188 with dropout; 228 with attendance)
– 164 coded studies (so far)164 coded studies (so far)– 450 dropout effect sizes from 302 independent study samples
The Campbell Collaboration www.campbellcollaboration.org
Study CodingStudy Coding
• Study coding included information about the programs• Study coding included information about the programs under evaluation, the students in those programs, and the methods used to evaluate the programs.methods used to evaluate the programs.
• Outcomes indexed using odds ratios.Ratio of the odds of completing school for treatment– Ratio of the odds of completing school for treatment participants to the odds of completing school for control participantsparticipants
– Odds ratio < 1 means control groups are favored; Odds ratio > 1 means treatment groups are favored; Oddsratio > 1 means treatment groups are favored; Odds ratio = 1 means odds are equal for the two groups.
The Campbell Collaboration www.campbellcollaboration.org
Treatment Typesyp
• The 302 independent samples were divided into twoThe 302 independent samples were divided into two groups
269 general programs– 269 general programs• 299 effect sizes measuring dropout, graduation, GED, or enrollment for general population & at‐risk samples.e o e t o ge e a popu at o & at s sa p es
– 33 teen parent (mostly teen mothers) programs• 150 effect sizes measuring dropout, graduation, GED, or150 effect sizes measuring dropout, graduation, GED, or enrollment for teen parents.
– NOTE: enrollment defined as not dropped out, not yet pp , ygraduated, and still attending school.
The Campbell Collaboration www.campbellcollaboration.org
How effective were dropout programs overall?
Mean ORLower CI
Upper CI nes Ksamples Q tau2
G l T 1 68 1 52 1 88 300 269 1712 3* 23General Tx 1.68 1.52 1.88 300 269 1712.3* .23Teen Parent Tx 1.75 1.41 2.17 150 33 191.3* .37Random effects analysis
Mean Odds Ratios Translated to PercentagesMean Odds Ratios Translated to Percentages
TX % CT %
General Tx Programs (Percent dropped out) 33% 45%Teen Parent Tx Programs (Percent enrolled) 41% 28%
The Campbell Collaboration www.campbellcollaboration.org
How effective are dropout programs overall?• After treatment, the odds of completing school were significantly better for students in preventionsignificantly better for students in prevention programs than for students who received no special programmingprogramming.– Treated students were less likely to drop out, and more likely to graduate obtain a GED or stay enrolled inlikely to graduate, obtain a GED, or stay enrolled in school.
Heterogeneity was evident for both the general• Heterogeneity was evident for both the general programs and for the teen parent programs.
• What study characteristics are associated with that heterogeneity?
The Campbell Collaboration www.campbellcollaboration.org
Characteristics of the Research
• Literature on dropout largely unpublished; 71% of the studies were technical reports (66%) or dissertations (5%).
• Evaluator role & routine practicep– Some evaluators were closely involved in the research; other programs had independent evaluators.other programs had independent evaluators.
– About 50% were routine practice programs.55% of the studies used random assignment• 55% of the studies used random assignment.
The Campbell Collaboration www.campbellcollaboration.org
Characteristics of the Students
• 88% were high school aged samples; 12% were middle school or younger samples.
• Mixed ethnicity groups of students were common. y g p75% of the student samples were mostly minority students.
• With the exception of specialized programs for teenage mothers most programs were delivered toteenage mothers, most programs were delivered to mixed gender groups.N l ll t d t t i k f d i t d• Nearly all students were at risk for dropping out and were from low socioeconomic status families.
The Campbell Collaboration www.campbellcollaboration.org
Characteristics of the Dropout ProgramsCharacteristics of the Dropout Programs
• Community‐based programs make up 15% of the sample; the remaining 85% are school‐based or school‐affiliated.
• Duration averaged 90 weeks (typical school year was coded as 38 weeks).)
• Frequency of treatment varied, but over half of the programs involved daily contactprograms involved daily contact.
• Implementation quality also varied: 32% experienced l bl 15% ll d d t ibl blclear problems, 15% alluded to possible problems, and 53% experienced no problems or mentioned no
blThe Campbell Collaboration www.campbellcollaboration.org
problems.
Treatment Types: General ProgramsTreatment Types: General ProgramsTreatment Type Description N
School or class restructuring Small learning communities block schedules 109School or class restructuring Small learning communities, block schedules, career academies, small class size.
109
Vocational training Coursework, internships, or employment oriented d k i
41toward work or career interests.
Supplemental academicservices
Remedial education, tutoring, homework assistance, etc.
25
Community service Programs involved planning and carrying out a community service project (commonly coupled with a weekly life skills curriculum).
24
y )Attendance monitoring & contingencies
Monitoring and services to increase attendance; some offer financial incentives.
17
C ll i t d C ll t i l ll i t d 12College‐oriented programming
College preparatory curriculum, college‐oriented academic advising.
12
Case management Programs revolved around connecting students & 11
The Campbell Collaboration www.campbellcollaboration.org
families with appropriate services.
Treatment Types: General ProgramsT T D i iTreatment Type Description nMentoring, counseling Programs provided adult mentors or trained
counselors for students. Though mentors focused 11
more on career/work, both mentors and counselors dealt with students’ personal issues.
Skills training, including CBT Generally oriented toward improving self‐esteem 6g, g y p gor attitudes about school, or preventing drug use.
Multi‐service package Large, comprehensive programs; generally included academic vocational and case
6included academic, vocational, and case management services (e.g., Job Corps)
Family outreach School contact with parents and families 3
Other Recreational, residential services for homeless, etc.
4
The Campbell Collaboration www.campbellcollaboration.org
Treatment Types for Teen ParentsT T D i iTreatment Type Description nAttendance monitoring & contingencies
Monitoring and services to increase attendance; most offer financial incentives and/or
22
contingencies tied to welfare.Case management Programs revolved around connecting teen
parents with appropriate services for parents and 6
p pp p pchildren.
Parenting skills training Parent skills training (and often child care services) for teen mothers
3services) for teen mothers.
School or class restructuring Teenage Pregnancy & Parenting Program 1
l k l hMulti‐service package Family Growth Center 1
The Campbell Collaboration www.campbellcollaboration.org
Analytic PlanAnalytic Plan• Meta‐regression models with robust standard errors used to examine influence of study characteristics on treatment effects.
• Regression models were then used to calculate covariate adjusted posttest effect sizes for eachcovariate adjusted posttest effect sizes for each treatment type, adjusting for method and subject h t i ti th t f dcharacteristics that may confound group differences.
• Models run separately by program groups (general vs. teen parent)
The Campbell Collaboration www.campbellcollaboration.org
(g p )
Meta‐regression Results for General ProgramsGeneral Programs b seMethod VariablesRandom assignment design ‐.159 .157Matched groups design ‐.051 .177
( )Attrition (%) ‐.041 .400Data in OR adjusted ‐.300 * .113Subject CharacteristicsSubject CharacteristicsGender mix .049 .045Ethnicity mostly white .059 .123Treatment CharacteristicsRole of evaluator (higher=more independent) ‐.144 † .083Implementation quality (higher better) 173 * 065Implementation quality (higher=better) .173 * .065Treatment duration (weeks) ‐.000 .001Frequency of service per week (hours) ‐.004 .004
The Campbell Collaboration www.campbellcollaboration.org
00 00nes=299; k=268
Study Characteristics associated with Treatment yEffectiveness: General Programs
• Effect sizes calculated with covariate‐adjusted data tended to be smallertended to be smaller.
• Study design and attrition not significantly associated with treatment effects.
• Gender mix and ethnicity not associated with differential effects.
The Campbell Collaboration www.campbellcollaboration.org
Study Characteristics associated with Treatment yEffectiveness: General Programs
• Studies in which evaluators were closely involved produced larger effectsproduced larger effects.
• Implementation quality resulted in better outcomes.• Treatment length and treatment frequency not associated with effects.
The Campbell Collaboration www.campbellcollaboration.org
Treatment
Family Outreach
Type
3
n
3
k
0.88 (0.68, 1.14)
OR (95% CI)
0.88 (0.68, 1.14)
OR (95% CI)
Monitoring & Contingencies
Multi‐service Package
Case Management
21
8
18
17
6
11
1.09 (0.93, 1.27)
1.17 (0.67, 2.04)
1.21 (0.79, 1.87)
1.09 (0.93, 1.27)
1.17 (0.67, 2.04)
1.21 (0.79, 1.87)Case Management
Skills Training
Counseling, Mentoring
S l A d i T i i
18
5
13
27
11
5
11
25
( , )
1.34 (1.07, 1.70)
1.43 (0.98, 2.09)
1 52 (1 13 2 05)
( , )
1.34 (1.07, 1.70)
1.43 (0.98, 2.09)
1 52 (1 13 2 05)Suppl. Academic Training
School or Class Restructuring
Other Treatments
27
122
5
25
109
4
1.52 (1.13, 2.05)
1.78 (1.54, 2.06)
2.23 (1.31, 3.81)
1.52 (1.13, 2.05)
1.78 (1.54, 2.06)
2.23 (1.31, 3.81)
Vocational Training
Community Service
College‐oriented Program
41
24
12
41
24
12
2.32 (1.91, 2.82)
2.57 (1.48, 4.47)
2.85 (1.69, 4.81)
2.32 (1.91, 2.82)
2.57 (1.48, 4.47)
2.85 (1.69, 4.81)
Favors CT Favors TX
1.5 1 1.5 2 2.5 3 5.1
The Campbell Collaboration www.campbellcollaboration.org
Favors CT Favors TX
Mean Adjusted Odds Ratios for General Programs
Summary for General Programs• All general programs except family outreach had positive effects on dropout, graduation, or continued enrollment.– Attendance monitoring, case management, multi‐service g, g ,packages, and counseling/mentoring programs not significant.
– College‐oriented programming, community service, vocational training, supplemental academic training, and g gschool restructuring programs were the most effective, net of study characteristics in the regression models.
– Overlapping confidence intervals for program types indicates that programs were not significantly different
The Campbell Collaboration www.campbellcollaboration.org
from each other.
Translating Adjusted Means
• Recall that our overall average odds ratio = 1.68– Translation: 45% of control students drop out vs. 33% of treated students
• Adjusted OR for school restructuring programs = 1.78– Translation: 45% of control students drop out vs. 32% ofTranslation: 45% of control students drop out vs. 32% of treatment students
• Adjusted OR for vocational training = 2 32• Adjusted OR for vocational training = 2.32– Translation: 45% of control students drop out vs. 26% of treatment studentstreatment students
The Campbell Collaboration www.campbellcollaboration.org
Meta‐regression Results for Teen Parent Programs
Teen Parent Programs b seM th d V i blMethod VariablesRandom assignment design ‐.596 .391Matched groups design ‐2.051 * .490g p g 2.051 .490Attrition (%) 1.188 * .520Data in OR adjusted .534 * .234Subject CharacteristicsEthnicity mostly white .043 .138Treatment CharacteristicsTreatment CharacteristicsRole of evaluator (higher=more independent) ‐.459 * .178Implementation quality (higher=better) .979 * .287Treatment duration (weeks) ‐.014 * .005Frequency of service per week (hours) ‐.005 .019
k
The Campbell Collaboration www.campbellcollaboration.org
nes=150; k=33
Study Characteristics associated with Treatment Effectiveness: Teen Parent Programs
• Effect sizes calculated with covariate‐adjusted data tended to be smallertended to be smaller.
• Study design and attrition both significantly associated ith t t t ff twith treatment effects.
– Random assignment and matched designs produced ll dd h d h dsmaller odds ratios than non‐random, non‐matched
designs.• Ethnicity not associated with differential effects.
The Campbell Collaboration www.campbellcollaboration.org
Study Characteristics associated with Treatment Effectiveness: Teen Parent Programs
• Studies in which evaluators were closely involved produced larger effectsproduced larger effects.
• Implementation quality resulted in better outcomes.• Shorter treatments were associated with larger treatment effects.
The Campbell Collaboration www.campbellcollaboration.org
Treatment
Type n k OR (95% CI)OR (95% CI)
School or Class Restructuring
Parent Training & Child Care
4
4
3
3
1.59 (1.31, 1.93)
1.59 (1.31, 1.93)
1.59 (1.31, 1.93)
1.59 (1.31, 1.93)g
Case Management 15 6
( , )
1.62 (1.33, 1.98)
( , )
1.62 (1.33, 1.98)
Monitoring & Contingencies 125 22 1.74 (1.56, 1.94)1.74 (1.56, 1.94)
Favors CT Favors TX 1.5 1 1.5 2 2.5 3
d d dd f
The Campbell Collaboration www.campbellcollaboration.org
Mean Adjusted Odds Ratios for Teen Parent Programs
Summary for Teen Parent Programs
• All four treatment strategies for teen parents were significantly better than controls for keeping students enrolled in school.
• The most common strategy (attendance monitoring gy ( gand financial incentives) had the largest adjusted odds ratio, though no strategy was significantly better thanratio, though no strategy was significantly better than another.
The Campbell Collaboration www.campbellcollaboration.org
Final CommentsFinal Comments
Dropout programs are generally effective with• Dropout programs are generally effective, with negligible differences between different strategies h t d h t i ti t ll dwhen study characteristics are controlled.
• Study methods are influential.• Implementation quality is strongly associated with better treatment outcomes.
• Studies in which evaluators were closely involved with treatment delivery tended to produce bettertreatment delivery tended to produce better outcomes.
The Campbell Collaboration www.campbellcollaboration.org
Sandra Jo Wilsonemail: [email protected]
The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org