Implementing School-Wide Positive Behavior Supports with Fidelity: Exploring the
Evidence
Nicholas A. Gage, Ph.D.University of Florida
A little about me…
Goals• We will:
• Provide an overview of School-wide Positive Behavior Interventions and Supports
• Discuss the evidence-base supporting the implementation of SWPBIS• Describe a series of state-level quasi-experimental design studies evaluating
the effect of SWPBIS on disciplinary exclusions
• We will: • Be moderately entertaining• Not talk too fast• Not hop too much
What are School-Wide Positive Behavior Interventions and Supports?
Continuum of Multi-Tiered Systems of Support
Primary Prevention:School-/Classroom-Wide Systems for
All Students,Staff, & Settings
Secondary Prevention:Specialized Group
Systems for Students with At-Risk
Tertiary Prevention:Specialized
IndividualizedSystems for Students
with High-Risk
~80% of Students
~15%
~5%
IMPLEMENTATION W/ FIDELITY
CONTINUUM OF EVIDENCE-BASEDINTERVENTIONS
CONTENT EXPERTISE &
FLUENCY
TEAM-BASEDIMPLEMENTATION
CONTINUOUSPROGRESS
MONITORING
UNIVERSAL SCREENING
DATA-BASEDDECISION MAKING
& PROBLEM SOLVING
CORE FEATURESSWPBIS
Universal Prevention
1. Leadership team2. Behavior purpose statement3. Set of positive expectations & behaviors4. Procedures for teaching SW & classroom-wide
expected behavior5. Continuum of procedures for encouraging expected
behavior6. Continuum of procedures for discouraging rule
violations7. Procedures for on-going data-based monitoring &
evaluation
School-Wide
Implementation Fidelity!
Why is Fidelity of Implementation Important?
• Fidelity data is necessary to make valid conclusions about outcomes.• Implementation acts as a potential moderator of the relationship
between interventions [programs] and their intended outcomes• A practical example: an antibiotic
What Fidelity of Implementation Measures Are Available for SWPBS?
Tier 1 Fidelity Measures Multiple Tiers Fidelity Measures• School-wide Evaluation Tool
(SET)• Benchmarks of Quality (BoQ)• Team Implementation
Checklist (TIC)• PBIS Self-Assessment Survey
(SAS)
• Monitoring Advanced Tiers Tool• Benchmarks for Advanced Tiers
(BAT)• PBS Implementation Checklist for
Schools• Individual Student Systems
Evaluation Tool (ISSET)• SWPBS Tiered Fidelity Inventory
(TFI)
Evidence Supporting the Effectiveness of SWPBIS
• Horner, Sugai, & Anderson (2010)• Narrative review found positive effect on school organizational health,
student perceptions of school safety, and problem behavior • Problem: not systematic and unclear how large an effect
• Solomon et al., (2012)• Meta-analysis of 20 single-case design studies that found small effects on
ODR and problem behavior• Problem: combined school and student outcomes
Evidence Supporting SWPBS
• Gage, Whitford, & Katsiyannis (2018)• Conducted a What Works Clearinghouse Review of SWPBIS effects on
disciplinary exclusion
Correlation n k g LL UL QOverall 90 4 -0.61 -1.27 0.05 8.69*Disciplinary Exclusion Type
ODR 53 3 -0.62 -1.29 0.06 8.43*
Suspension 72 2 -0.86* -1.68 -0.44 0.34WWC Standards
Meets Without Reservations 43 2 1.49 -0.24 3.21 1.92
Does Not Meet Standards 47 2 -0.97 -1.68 -0.26 0.10
Table 2.Fixed-Effect Meta-Analysis Results for SWPBS Effect on Disciplinary Exclusion
Limitations of the Literature
• Only four experimental studies with a treatment group and control group conducted to date at the school-level
• Those studies have only include 90 schools • Limited data on outcomes directly related to problem behavior
Solution!State-level analyses comparing schools implementing SWPBIS (with
fidelity) to schools that have never received training
***
Study #1: Kentucky
• Research Question:• R1: Are there differences in the number of behavioral events, and in- and out-
of-school suspensions between schools that have received SWPBIS training (Trained) and matched comparison schools (ITT models)?
• R2: Are there differences in the number of behavioral events, and in- and out-of-school suspensions between schools that have received SWPBIS training and implemented the behavior support practices as prescribed (Fidelity Behavior) and matched comparison schools (TOT models)?
Sample and Measures• Data from all public schools in the state of Kentucky (n =
1,171 schools)• 29 received SWPBIS training between the fall of 2013 and
spring of 2016, including two high schools, five middle schools, and 22 elementary schools
• School demographics and disciplinary exclusions (i.e., behavioral incidents and suspensions)
Research Design• We took two approaches in the analysis:
• an intent-to-treat model treating all schools ever receiving SWPBIS training, and
• a treatment-on-the-treated focusing on fidelity of implementation of the SWPBIS model.
• Missing data: < 10% missingness, thus used multiple imputation• Propensity Score matched Treatment schools to Control schools• Treatment Effect: Poisson Regression• Effect Sizes: WWC procedures for dichotomous outcomes by
calculating the Cox index based on the log odds ratio
Results
• Equivalence• Treatment Effect• Effect Sizes: -0.41 for ABRI schools, - .60 ABRI schools implmeneting
with fidelity for in-school suspensions and -0.33 for out-of-school suspensions for schools implementing with fidelity.
All Comparison Schools
(n = 1,138)
PSM Comparison Schools(n = 29)
Trained(n = 29)
Fidelity(n = 13) Equivalence
Covariate M SD M SD M SD M SD g1 g2Enrollment 545.4 291.4 419.2 223.6 450.2 112.7 464.9 134.0 0.18 0.24% Male 51.4 51.7 52.0 51.8 0.01 0.00% Female 48.7 48.5 48.2 48.3 -0.01 0.00% White 80.8 75.4 71.0 74.7 -0.14 -0.11% Black 8.8 12.3 15.2 13.8 0.15 0.07% Hispanic 5.7 6.3 7.5 6.5 0.11 0.09% Asian 1.3 0.7 1.0 1.2 0.00 -0.13% Native American 0.1 0.1 0.1 0.1 0.00 0.13% Hawaiian 0.1 0.1 0.1 0.0 0.00 0.00% Multi-Racial 3.3 5.2 5.5 3.9 0.02 0.21% Free/Reduced Lunch 63.8 68.3 69.1 63.4 0.02 0.15
Attendance Rate 94.8 95.1 95.0 95.3 -0.01 -0.04# of Teachers 35.9 16.9 29.3 13.8 31.8 6.5 30.6 7.4 0.23 0.12% Male Teachers 0.22 0.19 0.17 0.17 -0.08 -0.11% Female Teachers 0.78 0.81 0.83 0.83 0.08 0.11% White Teachers 0.96 0.93 0.95 0.94 0.21 0.05% Black Teachers 0.03 0.06 0.04 0.05 -0.14 -0.12% Hispanic Teachers 0.01 0.00 0.00 0.01 0.15 0.02Average Years of Experience
11.9 2.6 10.8 2.7 10.5 2.6 10.7 2.8 -0.09 -0.02
% of Teachers with BA 23.7 25.4 23.9 20.9 -0.05 -0.15
% of Teachers with MA 46.8 47.0 51.1 53.8 0.10 0.16
% of Teachers Rank 1 28.6 26.1 24.0 24.2 -0.07 -0.06% of Specialists 0.7 0.7 0.6 0.8 -0.04 0.09% of Proficient or Above: Language Mechanics
50.1 36.8 39.9 46.1 0.08 0.23
% of Proficient or Above: Math
49.0 41.2 42.8 48.1 0.04 0.17
% of Proficient or Above: Reading
56.4 46.0 46.9 51.2 0.02 0.12
Trained Schools Fidelity Schoolsβ SE 1-(exp(β)) β SE 1-(exp(β))
Total Suspensions -0.37*** 0.03 31% -0.40*** 0.03 33%
In-School Suspensions -0.36*** 0.03 30% -0.13** 0.04 12%
Out-of-School Suspension
-0.37*** 0.04 31% -0.84*** 0.06 57%
Behavioral Events -0.05* 0.03 5% 0.02 0.03 <1%
Study #2: Georgia
• Research Question #1: Is there a statistically significant difference in the number of disciplinary incidents and in-school and out-of-school suspensions for schools implementing SWPBIS with fidelity compared to matched comparison schools?
• Research Question #2: Is there a statistically significant difference in the number of disciplinary incidents and in-school and out-of-school suspensions for schools implementing SWPBIS at different fidelity levels compared to matched control schools?
Sample and Measures
• Data from all public schools in the state of Georgia during the 2015-2016 school year were collected from the Georgia Department of Education (GaDOE) website (n = 1,755)
• A total 119 schools implemented SWPBIS with fidelity during the 2015-2016 school year.
• BoQ, demographics, behavioral incidents, school suspensions
Research Design• Three levels of fidelity of implementation: Installing, Emerging,
Operational• We focused exclusively on schools implementing SWPBIS with fidelity
(Emerging and Operational), and excluded schools (n =218) that were “Installing”.
• Used Propensity Score Matching• Conducted ToT models using Poisson Regression• Calculated effect sizes as described above
ResultsOut-of-School Suspension In-School Suspension Disciplinary Incidents
Parameter β s.e. β s.e. β s.e.Intercept 3.79*** 0.259 2.86*** 0.221 3.74*** 0.149
SWPBIS -0.80*** 0.020 -1.60*** 0.021 -0.89*** 0.011
% White 0.00 0.002 0.01*** 0.002 0.01*** 0.001
% Black 0.01*** 0.003 0.01*** 0.002 0.02*** 0.001
% Hispanic 0.00 0.003 0.01* 0.002 0.01*** 0.002
% SPED 0.05*** 0.003 0.09*** 0.002 0.05*** 0.002
% Proficient: Math -0.05*** 0.002 -0.12*** 0.002 -0.07*** 0.001
% Proficient: Language Arts
0.03*** 0.002 0.08*** 0.002 0.04*** 0.001
% Proficient: Science 0.01*** 0.002 0.04*** 0.001 0.03*** 0.000
Out-of-School Suspension In-School Suspension Disciplinary IncidentsParameter β s.e. β s.e. β s.e.
Intercept 3.51*** 0.263 2.87*** 0.222 3.53*** 0.150
SWPBIS: Emerging -0.63*** 0.028 -1.62*** 0.033 -0.75*** 0.016
SWPBIS: Operational
-0.92*** 0.025 -1.59*** 0.024 -0.98*** 0.014
% White 0.00 0.002 0.01*** 0.002 0.01*** 0.001
% Black 0.02*** 0.003 0.01*** 0.002 0.02*** 0.001
% Hispanic 0.00 0.003 0.01* 0.002 0.01*** 0.002
% SPED 0.05*** 0.003 0.09*** 0.002 0.05*** 0.002
% Proficient: Math -0.05*** 0.002 -0.12*** 0.002 -0.07*** 0.001
% Proficient: Language Arts
0.03*** 0.002 0.08*** 0.002 0.04*** 0.001
% Proficient: Science 0.01*** 0.002 0.04*** 0.001 0.03*** 0.001
SWPBIS Schools PSM Comparison Schools
Outcome M SD M SD gOSS 29.9 34.2 69.2 95.8 -0.54ISS 24.6 55.7 123.1 188.2 -0.71Disciplinary Incident
92.1 97.5 232.6 291.2 -0.64
Study #3: Florida• Primary research questions.• 1. Is there a statistically significant difference in the number of
disciplinary exclusions between schools implementing SWPBIS with fidelity and propensity score matched comparison schools?
• Exploratory research questions• 2. Are there differences in disciplinary exclusions for subgroups of
students in schools implementing SWPBIS with fidelity compared with propensity score matched comparison schools?
Sample and Outcomes
• Data from all Florida public schools for all available discipline outcomes for the 2013-2014 school year (n = 3,513)
• During that year, 1,129 regular schools were trained to implement SWPBIS. Of those, 593 implemented SWPBIS with fidelity and were retained, while the remaining 536 schools were removed from the dataset
Research Design• Variables: Demographics, and disciplinary actions: (a) corporal
punishment, (b) in-school suspension (ISS), (c) out-of-school suspension (OSS), (d) expelled, and (e) contact with law enforcement.
• Propensity score matching of schools implementing with fidelity• Zero-predicted Poisson Regression Models• Effect sizes as described above
ResultsISS OSS Corporal Punishment
Estimate OR se Estimate OR se Estimate OR seIntercept -0.85*** 0.113 0.27 0.291 2.11*** 0.330SWPBIS 0.06 1.06 0.147 -0.99*** 0.37 0.243 -0.11 0.90 0.283Middle School -0.77* 0.46 0.302 -0.30 0.74 0.610 -0.26 0.77 0.460High School -0.65 0.52 0.426 0.71 2.04 0.784 -0.95 0.39 0.712Other -14.02 0.00 566.001 1.71 5.53 1.392 -2.60*** 0.07 0.617Total # of Students 0.00* 1.00 0.000 0.00*** 1.00 0.000OSS 2011 0.00*** 1.00 0.001 -0.07*** 0.93 0.010 0.00 1.00 0.002
Referral to Law Enforcement School-Related Arrest ExpulsionEstimate OR se Estimate OR se Estimate OR se
Intercept 2.17*** 0.202 3.64*** 0.390 6.85*** 1.078SWPBIS -0.18 0.84 0.146 0.27 1.31 0.234 -0.16 0.86 0.418Middle School -1.62*** 0.20 0.238 -1.88*** 0.15 0.314 -2.97** 0.05 1.102High School -1.09* 0.34 0.434 -1.90*** 0.15 0.488 -3.49** 0.03 1.211Other -1.90** 0.15 0.687 -0.82 0.44 1.192 -3.97** 0.02 1.331Total # of Students 0.00*** 0.00 1.00 0.000 0.00 1.00 0.000 0.00 1.00OSS 2011 0.00*** 1.00 0.001 0.00 1.00 0.001 0.00* 1.00 0.001
ResultsSample Discipline Outcome OR dAll Students
OSS 0.37 -0.55***ISS 1.06 0.03Corporal Punishment 0.9 -0.06Referral to Law Enforcement 0.84 -0.10
School-Related Arrest 1.31 0.15Expulsion 0.86 -0.08
SWDOSS 0.36 -0.56***one OSS 0.43 -0.46***more than one OSS 0.40 -0.50***
BlackOSS 0.57 -0.31**one OSS 0.63 -0.26**more than one OSS 0.62 -0.27**
ResultsOSS Only One OSS More than One OSS
Estimate OR se Estimate OR se Estimate OR se
Intercept 0.32 0.29 2.05 0.28 0.37 0.29
1-2 Years -0.82 0.44 0.44 -0.75** 1.10 0.20 -0.63 0.53 0.39
3-5 Years -1.53*** 0.22 0.37 -1.00*** 1.21 0.25 -1.33*** 0.26 0.33
6-10 Years-0.36 0.69 0.35 -0.63* 1.22 0.28 -0.38 0.68 0.34
Middle School
-0.32 0.73 0.61 0.34 2.50 0.36 -0.44 0.64 0.60
High School 0.78 2.19 0.79 1.14 1.71 0.61 0.62 1.85 0.78
Other 1.93 6.90 1.34 0.80 1.77 0.92 1.62 5.04 1.32
Total # of Students
0.00* 1.00 0.00 0.00 1.00 0.00 0.00* 1.00 0.00
OSS 2011 -0.07*** 0.93 0.01 -0.04*** 1.00 0.00 -0.07*** 0.94 0.01
Academic Achievement and SWPBS in FL
• Gage, N. A., Leite, W. L., Childs, K., & Kincaid, D. (2018). Average treatment effect of school-wide positive behavior supports (SWPBIS) on school-level academic achievement in Florida. Journal of Positive Behavior Interventions.
FCAT Mathematics FCAT ReadingEmpty Model Full Model Empty Model Full Model
Estimate s.e. Estimate s.e. Estimate s.e. Estimate s.e.Fixed Effects:
Intercept 57.07*** 1.89 64.00*** 1.94 58.43*** 1.87 62.37*** 1.76Treatment 1.65*** 0.33 1.94*** 0.39Years SWPBIS -0.50*** 0.01 -0.59*** 0.03Typical School 33.30*** 0.79 30.83*** 0.67SPED School -7.15*** 1.13 -9.45*** 0.95Title I -3.32*** 1.71 -3.50*** 0.16City -0.66*** 0.19 0.01 0.00Suburban -0.06 0.17 0.00 0.01Town -0.31 0.29 -1.29*** 0.25Grade 4 -2.39*** 0.13 -0.04 0.11Grade 5 -11.4*** 0.13 -1.21*** 0.11Total # of Students 0.02*** 0.01 -3.50*** 0.15# Free/Reduced Lunch -0.39*** 0.00 -0.04*** 0.00# White Students 0.01*** 0.00 0.01*** 0.00# Black Students -0.01*** 0.00 -0.01*** 0.00#Hispanic Students 0.01*** 0.00 0.01*** 0.00
Teacher/Student Ratio -0.13*** 0.01 -0.08*** 0.00Random Effects:
Time 79.92 33.34 100.49 28.41Treatment 8.33 0.33 11.03 0.94District 83.53 28.33 92.10 14.63
Residual 290.69 153.76 241.28 113.95
Fit:ICC (Within) 0.64 0.71 0.56 0.73ICC (Time) 0.18 0.15 0.23 0.18ICC (District) 0.19 0.13 0.21 0.09AIC 498,815 461,490 488,171 444,143BIC 498,869 461,687 488,225 444,340Deviance 498,803 461,446 488,159 444,099
Table 3. Mixed-Effects Regression Model Results for the Effect of SWPBIS with Fidelity on FACT Mathematics and Reading Tests
Study 4: SWPBIS in California
Primary research questions.1. Is there a statistically significant difference in the number of disciplinary exclusions between schools implementing SWPBIS with fidelity and propensity score matched comparison schools? Exploratory research questions2. Are there differences in disciplinary exclusions for subgroups of students in schools implementing SWPBIS with fidelity compared with propensity score matched comparison schools?
Sample and Outcomes
• Data from all California public schools for all available discipline outcomes for the 2015-2016 school year (n = 7,775)
• During that year, 1,384 regular schools were trained to implement SWPBIS. Of those, 544 implemented tier 1 SWPBIS with fidelity based on the TFI and were retained.
• Overall, there were 396,194 students in SWPBIS schools implementing with fidelity.
Research Design• Variables: Demographics, and disciplinary actions: (a) corporal
punishment, (b) in-school suspension (ISS), (c) out-of-school suspension (OSS), (d) expelled, (e) contact with law enforcement and (f) removal to alternative school for behavior.
• Propensity score matching of schools implementing with fidelity• Zero-predicted Poisson Regression Models• Effect sizes as described above
Demographic Characteristics
Treatment Effect Estimation
• Zero-inflated Poisson (ZIP) regressionoAccounts for data scaled as frequency counts and excessive zeros in outcomes
• Odds ratioso Likelihood that an event (e.g., suspension) will occur
• Standardized mean differences (g)o Size of difference between treatment and control schools
Results
Sample Discipline Outcome OR gAll students
ISS 1.05 0.03One OSS 0.66 -0.23More than one OSS 0.78 -0.14All OSS 0.63 -0.25OSS Incidents 0.69 -0.21Days missed due to OSS 0.65 -0.24Expulsions 1.05 0.03Referrals to alternative schools for disciplinary reasons 1.12 0.06Referrals to law enforcement 0.94 -0.03School-related arrests 1.05 0.03
SWDReferrals to alternative schools for disciplinary reasons 0.31 -0.65
HispanicOne OSS 0.67 -0.22
BlackMore than one OSS 0.72 -0.18
ParameterOne OSS All OSS Days Missed due to OSS
Estimate OR SE Estimat
e OR SE Estimate OR SE
Intercept -2.00 1.08 -2.12 1.16 -2.32* 1.16SWPBIS -0.42* 0.66 0.20 -0.45* 0.63 0.22 -0.43* 0.65 0.22Suburb 0.26 1.29 0.21 0.36 1.43 0.23 0.33 1.39 0.23Town -1.52** 0.22 0.55 -1.36* 0.26 0.56 -1.34* 0.26 0.56Rural 0.15 1.16 0.45 0.05 1.05 0.49 0.07 1.07 0.49Title I ineligible
1.25*** 3.48 0.35 1.12** 3.07 0.38 1.23** 3.43 0.38Middle school -0.41 0.66 0.36 -0.28 0.76 0.38 -0.26* 0.77 0.38High school -0.24 0.79 0.68 -0.04 0.96 0.70 -0.02 0.98 0.70Other configuration 2.70* 14.93 1.07 2.88** 17.84 1.11 2.86 17.39 1.12Total enrollment 0.00 1.00 0.00 0.00 1.00 0.00 0.00 1.00 0.00FTE Teachers -0.01 0.99 0.04 -0.01 0.99 0.04 -0.02 0.98 0.04Meet or exceed ELA standards in 2015-16 0.03*** 1.03 0.01 0.03** 1.03 0.01 0.03* 1.03 0.01FRL 1.39 4.00 0.80 1.61 4.98 0.85 1.81 6.12 0.86SWD -2.14 0.12 2.23 -1.85 0.16 2.36 -1.58 0.21 2.37LEP 0.46 1.59 0.77 -0.02 0.98 0.83 -0.01 0.99 0.83Black students -2.67* 0.07 1.30 -3.20* 0.04 1.47 -3.21 0.04 1.47White students -0.10 0.90 0.74 -0.11 0.90 0.80 -0.08 0.92 0.80
Does it really matter though?
What does it all mean
• SWPBIS is an effective approach for addressing disciplinary exclusions• Next Steps: MORE STUDIES
Thank You!
Top Related