Evaluation of the Discipline Foundation Policy October 24, 2011

8/3/2019 Evaluation of the Discipline Foundation Policy October 24, 2011

1/47

LAUSD Discipline Foundation Policy

Evaluation of the Relationship between School-Wide Positive Behavior Interventions and

Supports (SW-PBIS) Implementation and Outcomes

10/24/2011

Jeffrey Sprague, Ph.D.

Claudia Vincent, Ph.D.

University of Oregon

Institute of Violence and Destructive Behavior (IVDB)

1265 University of Oregon

Eugene, Oregon 97403-1265

Tel: 541 346 3592

Fax: 541 346 2594


2/47

EXECUTIVE SUMMARY

BACKGROUND

On March 27, 2007, the Los Angeles Unified School District (LAUSD) adopted the

Discipline Foundation Policy: School-Wide Positive Behavior Support (BUL-3638.0) which

mandates the development of school-wide positive behavior support and discipline plans atevery school in the District. The policy states that, Every student, pre-school through adult, has

the right to be educated in a safe, respectful and welcoming environment. Every educator has

the right to teach in an atmosphere free from disruption and obstacles that impede learning.

This will be achieved through the adoption and implementation of a consistent school-wide

positive behavior support and discipline plan for every school in LAUSD.

Extensive work has been done since the adoption of the Discipline Foundation Policy to

facilitate the implementation of this important, evidence-based policy. The policy also calls for

ongoing oversight, systematic review and evaluation which includes an analysis of:

1. Policy implementation

2. Communication mechanisms

3. Any adjustments or changes in school practices (determined through data collection)

to ensure that school practices are strengthened and aligned with District policy

The University of Oregon Institute on Violence and Destructive Behavior collaborated

with LAUSD leadership personnel to conduct an independent and impartial evaluation of the

implementation of the Discipline Foundation Policy. The results document policy processes and

outcomes. Recommendations will be used to guide development of a three to five year plan to

enhance and improve districtwide policy implementation including specification of

intervention fidelity measures, specifically assisting in the development of data-based decision

rules for continuous program improvement.

EVALUATION QUESTIONS

The goal for the evaluation was to describe how well the Discipline Foundation Policy is

being implemented and how closely the implementation matches the policys goals. The

findings will be used to recommend a three to five year future plan for implementation of the

policy. This evaluation focused on the following questions:

1. What is the level of implementation of School-Wide Positive Behavior Interventions andSupports (SWPBIS) at the local school and local district level based on the Discipline

Foundation Policy criteria?2. Are students, families and school/district personnel aware of and involved with the

policy and its elements (the eight key features of SWPBIS)?

3. What impact does implementation of the Discipline Foundation Policy and SWPBISpractices have on LAUSD schools and the students and families they serve?


3/47

THEORY OF ACTION

If schools implement the Discipline Foundation Policy: School-Wide Positive Behavior

Support with high levels of fidelity then higher levels of academic achievement and attendance

will be observed along with lower levels of out of school suspensions and opportunity transfers.

METHODOLOGYGuided by a comprehensive evaluation table of specifications per commonly

recognized standards for professional program evaluation, we conducted an analysis of the

inputs, outputs (processes), outcomes and impacts of the implementation of the Discipline

Foundation Policy.

Sample

A random sample of 142 schools (84 elementary, 28 middle, and 30 high) was selected

to participate in the evaluation. These schools are from the eight Local Districts and T schools.

In those 142 schools, a total of 156,595 students were enrolled. The average enrollment at the

elementary level was 637.6 (SD = 264.14), at the middle school level 1643.0 (SD = 490.43) and

at the high school level 1901.1 (SD = 1151.68).

Evaluation Measures and Data Analysis

The evaluation included a review of the Discipline Foundation Policy, collection of

fidelity of implementation data from two different sources, and surveys for parents and staff

members. For the schools sampled, we also received archival data from the 09-10 school year.

We used evidence-based tools to measure implementation fidelity, as well as generally

recognized instruments and practices to assess the characteristics and needs of schools,

evaluation participants, and program outcomes. Each instrument was matched to the needs

and characteristics of individual evaluation components.

Simple logic and research would suggest that higher quality SWPBIS practices in schoolswould be related to improved academic outcomes and reduced use of out of school

suspensions and opportunity transfers.

Fidelity of Implementation

The level of SWPBIS implementation was measured with the Team Implementation

Checklist (TIC) and the Rubric of Implementation (RoI). The TIC originally developed by Horner,

Sugai and Lewis-Palmer (2002) has been used to monitor positive behavior intervention and

support implementation in schools throughout the Nation. LAUSD was given permission to

modify the original version to a total of 18 self-assessment questions that measure the levels of

commitment and teaming, as well as the established school-wide expectations and the capacityfor function-based behavioral support. Possible responses to each question are Achieved, In-

Place, or Not Started. The RoI is a District developed measure that reflects the 8 Key-

Features of SWPBIS practices and is completed by Local District staff persons from outside the

school-site. Scores of 1 (Little to no evidence) to 4 (Strong evidence) are given for each feature.

Staff Member and Parent Survey Data

LAUSD personnel prepared an anonymous electronic survey that was distributed to


4/47

staff members and families at the schools selected for the evaluation. Questions were derived

from a variety of sources, and were intended to capture information about stakeholder

awareness of, and satisfaction with features of the Discipline Foundation Policy. The staff

member and parent survey data were analyzed to determine levels of awareness, satisfaction

with policy implementation, gather information for improvement, and to discover needs or

changes in services, training and curriculum.

Archival Data

We received an archival data set on the sampled schools which consisted of attendance,

academic achievement, Free and reduced lunch, Opportunity Transfers (OT), out of school

suspensions and suspension rates for the 09-10 school year.

RESULTS

QUESTION 1: What is the level of implementation of School-Wide Positive Behavior

Interventions and Supports (SWPBIS) at the local school and local district level based on the

Discipline Foundation Policy criteria?

Recognizing that the TIC is a self-evaluation instrument and the RoI is completed by

Local District personnel, we correlated TIC and RoI subscale scores and total scores to evaluate

the extent to which the outcomes of the two measures yield similar outcomes. Elementary and

middle schools appeared to be the strongest implementers, with middle schools reporting the

most fully implementing on the TIC. Based on RoI scores, elementary schools were the

strongest implementers overall, followed by middle schools and high schools. However, middle

school scores were highest in Administrative Leadership, Team-Based Implementation,

Monitoring and Correcting, and Data-Based Decision-Making.

QUESTION 2: Are students, families and school/district personnel aware of and involved withthe policy and its elements (the eight key features of SWPBIS)?

The responses that had the highest percentage of agreement in the staff member

survey were: I think having the same 3-5 behavior expectations/rules throughout our school

and in every classroom helps students follow the behavior expectations/rules (90%) and Most

teachers help students to know the behavior expectations/rules for our school (87%). The

responses that had the highest percentage of disagreement in the staff survey were: Updates

on the School-Wide Positive Behavior Support efforts are communicated through newsletters,

brochures, open house, parent meetings, etc. at least 5 times per school year (47%) and Our

school uses behavior data to make decisions in designing, implementing, and revising school-

wide efforts at least 2 times per school year (43%) (see table 1 below).The responses that had the highest percentage of agreement in the parent survey were:

My child knows the behavior expectations/rules the school has for them (91%) and I know

the behavior expectations/rules at my childs school (90%). The responses that had the highest

percentage of disagreement in the parent survey were: I have been invited to provide my

opinions regarding what the school behavior expectations/rules should be at my childs school

(42%) and My child has received an award/recognition from the school for meeting

expectations school behavior expectations/rules (37%) (see table 2 below).


5/47

Table 1: Responses that had the highest percentage of agreement and disagreement (Staff Member

Survey)

Statement % agree

I think having the same 3-5 behavior expectations/rules throughout our

school and in every classroom helps students follow the behaviorexpectations/rules.

90%

Most teachers help students to know the behavior expectations/rules for our

school.

87%

Statement % disagree

Updates on the School-Wide Positive Behavior Support efforts are

communicated through newsletters, brochures, open house, parent

meetings, etc. at least 5 times per school year.

47%

Our school uses behavior data to make decisions in designing, implementing,

and revising school-wide efforts at least 2 times per school year.

43%

Table 2: Responses that had the highest percentage of agreement and disagreement (Parent Survey)

Statement % agree

My child knows the behavior expectations/rules the school has for them. 91%

I know the behavior expectations/rules at my childs school. 90%


I have been invited to provide my opinions regarding what the school

behavior expectations/rules should be at my childs school.

42%

My child has received an award/recognition from the school for meeting

expectations school behavior expectations/rules.

37%

QUESTION 3: What impact does implementation of the Discipline Foundation Policy and

SWPBIS practices have on LAUSD schools and the students and families they serve?

There isa general pattern of positive relationship between SWPBIS implementation and

higher academic outcomes, as well as a negative relationship between SWPBIS implementation

and suspension and opportunity transfers. In the elementary school sample, RoI scores were

generally associated with lower suspension rates, and higher ELA and Math proficiency rates. In

middle schools, higher RoI scores were associated with higher in seat attendance, lower

suspension rates, and higher ELA and Math proficiency rates. Higher in seat attendance was

again correlated with the statistically significantly lower suspension rates, lower opportunity

transfer rates, and higher academic achievement in both ELA and Math. The high school sampleyielded similar patterns. RoI scores were associated with lower suspension and opportunity

transfer rates, and higher proficiency rates in ELA and Math. These findings support the

predicted association between high fidelity implementation of SWPBIS and lower behavioral

incidents as well as increases in academic achievement.


6/47

Summary of Recommendations

In order to increase and enhance district-wide implementation of the Discipline

Foundation Policy and fidelity of practice to its components, it is recommended that the District

should:

Use culturally relevant practices to actively engage family partnerships in developing

understanding and implementation of School Wide Positive Behavior Interventions and

Supports. The policy has roles and responsibilities defined for each stakeholder group; We

recommend that the district use that guidance along with culturally relevant practices to share

the core messages of safe, responsible and respectful behavior to engage everyone in

educating LAUSD children and youth.

Link advocated practices in the policy to valued outcomes in order to revitalize school

staff support for full, meaningful implementation (social marketing). LAUSD is very large and

has multiple school organizations including district operated schools, Charters and Partnership

schools. Some mechanisms to link the positive actions and outcomes among all groups and

community members include:

Actively and continuously inform stakeholders through multiple communicationchannels

Enhancing methods to inform and invite students, parents, and staff to celebratesafe and healthy schools across all school organizations

Conducting other studies, using a larger sample (especially middle and high schools)to better understand the relationship between SWPBIS fidelity and valued school

outcomes

Ensure that high-quality leadership and support is provided by a principal or other

administrator. Continue to provide training and support for school leadership personnel.

Leaders should be held accountable for implementation of the Discipline Foundation Policy.

Training with accountability features must be systemic for all staff members involved in

discipline. (e.g., Deans training that is a prerequisite for applying for a position as dean)

Provide just in time consultation and technical assistance in establishing appropriate

systems and infrastructure. Research indicates that active coaching and support is a critical

element for gaining implementation and maintenance of SWPBIS. If the Discipline Foundation

Policy is to continue its impact and expand, the District should consider the systematic use of a

research-based coaching model utilizing existing expertise in the District. Coaching can becarried out in multiple forms. In order to provide timely and accurate support, the District

should utilize multiple data sources. (i.e., TIC, RoI, Office Discipline Referral (ODR), attendance,

suspension) to allocate district-level assistance.

Develop and standardize a system of performance-based feedback for implementers.

Use of data, and providing staff member feedback on SWPBIS outcomes was among the lowest

rated items on the survey. Staff development should be designed to specifically address this


7/47

finding. We also recommend that LAUSD repeat the fidelity assessments annually and give

feedback to staff members and families. Data are now available through systems such as My

Data which report on the office discipline referrals as well as suspensions. Linking the data

beyond discipline and connecting these outcomes to other data such as achievement and

attendance must be the expectation. The appropriate use of the RoI by individuals outside of

the school staff could also be a meaningful tool for providing feedback to improve and enhancethe implementation of SW-PBIS. This must be augmented with multiple levels of support in the

regular use of data to improve implementation of the Discipline Foundation Policy at each level,

therefore improving other outcomes.

Continue the current Central and Local District support infrastructure (Task Force,

Central Steering Committee, LD Implementation Teams) to increase the dissemination of

evidence-based best practices, research and opportunities for collaboration. While this

recommendation is implied in the earlier recommendations, it is important to explicitly

recommend that the District continue to support and invest in systemic and systematic staff

development activities. This would include continued staff development and coaching in order

to maintain implementation fidelity. The tools in the policy itself should be used to build

understanding around suspension alternatives, evidence based practices for reinforcing and

correcting student behavior, progressive discipline and prevention.

Commit adequate financial resources for adoption, implementation, and maintenance

of the policy. Any staff development and information campaign requires adequate fiscal

support. We recommend that the district identify and commit adequate financial resources to

further support this initiative. The District should design and implement low-cost/no-cost

measures that are essential such as connect and integrate the components of the Discipline

Foundation Policy into District priorities and initiatives: Teaching and Learning, Leadership

Framework, Quality 1st Instruction, and K-12 Common Core standards, attendance and dropoutprevention, crisis prevention.


8/47

LAUSD Discipline Foundation Policy

Evaluation of the Relationship between School-Wide Positive Behavior Interventions and

Supports (SW-PBIS) Implementation and Outcomes

FULL REPORT

10/24/2011

Jeffrey Sprague, Ph.D.

Claudia Vincent, Ph.D.

University of Oregon

Institute of Violence and Destructive Behavior (IVDB)

1265 University of Oregon

Eugene, Oregon 97403-1265

Tel: 541 346 3592

Fax: 541 346 2594


9/47

Background Research on SWPBIS and Implementation Fidelity

School Wide Positive Behavior Interventions and Supports

To prevent minor, as well as serious antisocial behavior, thousands of schools areturning to a comprehensive approach to discipline commonly referred to as School Wide

Positive Behavior Interventions and Supports (SW-PBIS) (F. M. Gresham, 2004; Scott & Barrett,

2004; J. Sprague & Golly, 2004; Sugai & Horner, 2002). SW-PBIS is based on the assumption that

when all adults in a school actively teach and acknowledge expected behavior, the proportion

of students with serious behavior problems will be reduced and the schools overall educational

climate will improve. The U.S Department of Education funds a National Technical Assistance

Center on PBIS at the University of Oregon (www.pbis.org) and the website provides a rich

source of information and resources regarding SW-PBIS practices and research. LAUSD also has

an excellent district website providing access to the Discipline Foundation Policy and multiple

staff development materials and assessment toolshttp://notebook.lausd.net/portal/page?_pageid=33,911578&_dad=ptl&_schema=PTL_EP.

SW-PBIS practices are organized around three main themes: Prevention; Multi-Tiered

Support; and Data-Based Decision Making. Preventing problem behavior involves (a) defining

and teaching 3-6 core behavioral expectations (e.g., be safe, be respectful, be responsible), (b)

acknowledging and rewarding appropriate behavior (e.g., compliance to school rules, safe and

respectful peer to peer interactions, academic effort/engagement), and (c) establishing a

consistent continuum of consequences for problem behavior. The goal of SW-PBIS is to

establish a positive social climate in the school in which behavioral expectations for students

and teachers are highly predictable, directly taught, consistently acknowledged, and actively

monitored.

Data-based decision-making is a theme that is interwoven throughout SW-PBIS, and

builds on the assumption that the faculty members, staff members, family members, and

students will be most successful in the design and implementation of interventions if they have

access to regular, accurate information about the behavior of students and the quality and

consistency of their own behavior. The SW-PBIS approach includes adoption of practical

strategies for collecting, summarizing, reporting, and using behavioral and intervention fidelity

data on regular cycles.

Evidence suggests that sustained use of SW-PBIS practices can alter the trajectory of at-

risk children toward destructive outcomes and prevent the onset of risk behavior in typically

developing children. It is expected that effective and sustained implementation of SW-PBIS will

create a more responsive school climate that supports the goal of schooling for all children(Bradshaw, 2008; R. Horner et al., 2009; Walker, 1996). The success of SW-PBIS implementation

depends on consistent and sustained implementation of the systems and practices. This is

referred to as intervention fidelity.

Intervention fidelity, sometimes referred to as treatment fidelity in the clinical

psychology literature or procedural reliability in the applied behavior analysis literature refers

to the extent to which an intervention is implemented as designed(Fixsen, Naoom, Blase,

Friedman, & Wallace, 2005; F. Gresham, 2004). Research has demonstrated that poor
http://www.pbis.org/http://www.pbis.org/http://www.pbis.org/http://notebook.lausd.net/portal/page?_pageid=33,911578&_dad=ptl&_schema=PTL_EPhttp://notebook.lausd.net/portal/page?_pageid=33,911578&_dad=ptl&_schema=PTL_EPhttp://notebook.lausd.net/portal/page?_pageid=33,911578&_dad=ptl&_schema=PTL_EPhttp://www.pbis.org/


10/47

treatment integrity is a variable that often undermines the efficacy of interventions delivered in

educational settings. As a result, when a particular intervention results in poor outcomes, the

first question that the school team must address before any other conclusion can be reached is,

Was the intervention implemented with integrity?

There are five major methods for assessing treatment fidelity: direct observation,

feedback from consultants, self-monitoring and reporting from teachers, review of permanentproducts, and treatment manualization. Direct observation intervention fidelity is perhaps the

most accurate assessment, although it is also the most expensive. The ability to substitute a

less-expensive assessment for direct observation and still retain a tight management of fidelity

would be desirable. Intervention fidelity data represent a vital ingredient in any decision

making system (Gilbert, 1996). Failure to document whether positive behavior supports were

implemented with fidelity in a SW-PBIS implementation poses serious limitations in the ability

to draw valid conclusions about a single students or group of students responses to the

intended intervention. That is, without data on the fidelity of implementation, researchers and

educators will be unable to determine whether students failed to respond well to the

intervention because it was implemented inaccurately or inconsistently or whether the

students actually resisted an otherwise effective, well implemented intervention (J. R. Sprague,

Cook, Wright, & Sadler, 2008).

Evaluation Questions

The University of Oregon Institute on Violence and Destructive behavior collaborated

with LAUSD leadership personnel to conduct an evaluation of the implementation of the

Discipline Foundation Policy for the Los Angeles Unified School District (LAUSD). The results are

intended to be used to document program processes and outcomes, and to develop a three to

five year plan of districtwide implementation including specification of intervention fidelity

measures, specifically assisting in the development of data-based decision rules for continuousprogram improvement.

The goal for the evaluation was to describe how well the Discipline Foundation Policy is



policy.

Within the timeline and resources available, we focused on the following evaluation questions:

1. What is the level of implementation of School-Wide Positive Behavior Interventionsand Supports (SWPBIS) at the local school and local district level based on theDiscipline Foundation Policy criteria?

2. Are students, families and school/district personnel aware of and involved with thepolicy and its elements (the eight key features of SWPBIS)?

3. What results does implementation of the Discipline Foundation Policy and SWPBISpractices have on LAUSD schools and the students and families they serve?


11/47

THEORY OF ACTION

If schools implement the Discipline Foundation Policy: School-Wide Positive Behavior

Support with high levels of fidelity then higher levels of academic achievement and attendance

will be observed along with lower levels of suspension and opportunity transfers.

METHODOLOGY

Guided by a comprehensive evaluation table of specifications per commonly recognized

standards for professional program evaluation, we conducted a limited analysis of the outputs

(processes), outcomes and impacts of the implementation of the Discipline Foundation Policy

This table of specifications is included as Attachment A.

Stakeholders participated in the following types of evaluation:

Formative Evaluation: Formative evaluation focuses on assessing fidelity ofimplementation and tracking short term outcomes (e.g., academic and behavioral

indicators).

Summative Evaluation: School staff members and parents were asked to makejudgments about the overall implementation of the Discipline Foundation Policy

processes, outcomes and impacts. We also assessed their awareness of the features of

SWPBIS practice.

Sample and Data Analysis

Sample Overview. LAUSD personnel selected at random a total of 142 schools (84

elementary, 28 middle, and 30 high) to participate in the evaluation. Figure 1 (below)

summarizes selected schools by level and local district. In those 142 schools, a total of 156,595students were enrolled. The average enrollment at the elementary level was 637.6 (SD =

264.14), at the middle school level 1643.0 (SD = 490.43) and at the high school level 1901.1 (SD

= 1151.68). Enrollment by race/ethnicity and school level is summarized in Table 1 and Figure 2

below:


12/47

Figure 1: Schools by Local District and Level:

Table 1: Enrollment by race/ethnicity across school levels:Elem (n = 84) Mid (n = 28) High (n = 30) Overall (n = 142)

Avg % SD Avg % SD Avg % SD Avg % SDNat Am 0.35 0.41 0.36 0.29 0.23 0.2 0.33 0.35

Asian 5.05 10.62 2.53 3.24 3.84 5.85 4.3 8.74

Afr Am 12.08 20.52 13.38 16.7 8.67 15.64 11.62 18.82

Hisp 70.74 28.67 73.98 20.89 78.12 22.33 72.94 26.05

White 9.09 15.09 5.62 8 6.16 12.01 7.78 13.36

Filipino 2.17 3.23 3.56 5.05 2.71 3.85 2.56 3.79

PacIs 0.53 1.44 0.57 1.46 0.26 0.39 0.48 1.29

Across all school levels, Hispanic students comprised the largest student population, followed

by African-American, White, Asian, Filipino, Pacific Islander, and Native American students.

Within the entire sample, almost 75% of students were from Hispanic background. Large

standard deviations indicate, however, that a fair amount of variability existed in the

racial/ethnic composition of student populations across individual schools.

15

108

17

117

59

2

1

2 2

7

3

2

1

6

3

3

3

2

9

5

2

3

1

2

LD 1 LD 2 LD 3 LD 4 LD 5 LD 6 LD 7 LD 8 LD T

Elementary Middle School High School

Randomly Selected Schools by Local District and Level

Local Districts

r

fSc

ls


13/47

0

10

20

30

40

50

60

70

80

90

100

Elem (n = 84) Mid (n = 28) High (n = 30) Overall (n = 142)

Percent

Enrollment by race/ethnicity

NatAm

Asian

AfrAm

Hisp

White

Filipino

PacIs

Figure 2: Enrollment by race/ethnicity across school levels.

Table 2 and Figure 3 provide additional information about student demographics, including the

percent of students identified on Free and Reduced Lunch (FRL), Socially and Economically

Disadvantaged (SED), students receiving special education services, migrant families, and

Limited English Proficient (LEP). On average, approximately 80% of students were on FRL across

all school levels. The number of students identified as LEP declined from the elementary to the

secondary (middle/high) level, while the number of students receiving special education

services increased from elementary to middle to high school. Only a very small number of

students were from migrant families.

Table 2: Student demographics


Avg % SD Avg % SD Avg % SD Avg % SD

FRL 80.58 21.36 80.69 17.25 75.35 15.47 79.5 19.48

SED 81.69 21.4 83 16.62 79.33 15.2 81.45 19.27

SpecEd 10.88 10.07 11.69 2.25 13.52 16.38 11.6 10.81

Migrant 0.29 0.48 0.19 0.27 0.18 0.27 0.25 0.41

LEP 38.88 19.74 24.6 10.99 28.58 15.81 33.88 18.49


14/47

Figure 3: Student demographics

Evaluation Measures and Data Analysis

The evaluation design included a review of the Discipline Foundation Policy, fidelity of

implementation data from two different sources, and surveys to parents and staff members.

We also received archival data from the 09-10 school year for the school sample. We used

evidence-based tools to measure implementation fidelity, as well as generally recognized

instruments and practices to assess the characteristics and needs of schools, evaluation

participants, and program outcomes. Each instrument was matched to the needs and

characteristics of individual evaluation components.

A simple logic and research would suggest that higher quality SW-PBIS practices in

schools would be related to improved academic outcomes and reduced use of out of school

suspension and opportunity transfers.

Fidelity of Implementation. The level of SWPBIS implementation was measured with

the Team Implementation Checklist (TIC) and the Rubric of Implementation (RoI). The TIC was

developed at the University of Oregon and is widely used (www.pbis.org) to assess PBIS

implementation and school functioning. The TIC assesses the school level PBIS implementation

team process and SW-PBIS practices. An ideal TIC score is 80% of the items rated as fully

implementing. The ROI is a locally developed measure that reflects commonly referenced SW-

PBIS practices. The TIC is included as Attachment B and the RoI can be downloaded from

http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R

%201.16.09.PDF.

Staff member and parent survey data. LAUSD personnel prepared an electronic survey

that was distributed from the district office. Questions were derived from a variety of sources,

and were intended to capture information about stakeholder awareness of and satisfaction

with features of the Discipline Foundation Policy. The staff member and parent survey data

were analyzed to determine levels of satisfaction with policy implementation, gain suggestions

0

10

20

30

40

50

60

70

80

90

100


Percent

Student demographics

FRL

SED

SpecEd

Migrant

LEP
http://www.pbis.org/http://www.pbis.org/http://www.pbis.org/http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/STUDENT_HEALTH_HUMAN_SERVICES/SHHS/DISCIPLINE_POLICY/RUBRIC%20OF%20IMPLEMENTATION%20R%201.16.09.PDFhttp://www.pbis.org/


15/47

for improvement, discover needs or changes in services, training and curriculum.

Archival Data. We received an archival data set on the sampled schools including

number of disciplinary hearings, disciplinary referrals for policy violations, and Opportunity

Transfers (OT), out of school suspensions and expulsions, and the risk ratio ofsuspensions

and expulsions for students with disabilities and African-American students. We also gathered

information on attendance and academic achievement.

Results

The information collected, analyzed, and reported is intended to be used to identify

implementation progress, including what is working at the school and local district levels and

what needs to be improved or expanded. The evaluation activities can guide program

improvement and assess the impact of each service/activity/program/curriculum on the

achievement of the policy objectives. The evaluation results will be used to develop a plan of

action for LAUSD over the next 3 to 5 years.

Descriptive Summary of Outcome Data for 09-10 School Year. Discipline and academic

outcomes by school level are summarized in Table 3 and Figure 4. Later in this report we will

illustrate simple correlations between measures of SW-PBIS implementation and these archival

data.

Table 3: Discipline and Academic Outcome Data 09-10


Avg SD Avg SD Avg SD Avg SD

Pct Attendance 95.42 1.45 94.77 1.33 92.17 3.02 94.61 2.26

Pct ELAProf 46.59 14.74 35.21 13.29 31.59 16.73 41.17 16.21

Pct MathProf 57.37 13.53 32.14 12.21 13.45 13.19 43.12 22.4

# Susp/100St 2.1 2.65 16.45 13.98 7.51 6.37 6.07 9.01

# OppTrans/100St 0 0 0.85 0.54 0.38 0.46 0.25 0.46

# GrowthPoints 10.27 24.48 15.43 16.38 22.21 14.9 13.8 21.72

The percentage rate of attendance declined slightly from elementary to middle to high

school, but was overall at about 95%. The percent of students at proficiency in English Language

Arts (ELA) and Math also declined from elementary to middle to high school. In elementary

schools, the percent of students proficient in ELA ranged from a minimum of 0% to a maximum

of 93%, in middle school it ranged from 15% to 70%, and in high school it ranged from 0% to

81%. The percent of students proficient in Math ranged in elementary schools from 0% to 93%,

in middle schools from 10% to 64%, and in high schools from 0% to 59%.

The number of suspensions was recorded for each school. To make these numberscomparable across schools with varying enrollments, the rate of suspensions per 100 students

was calculated as follows: (number of suspension/total enrollment) * 100. The same procedure

was followed for opportunity transfers. Suspensions increased dramatically from elementary to

middle school, and then declined in high school. Opportunity transfers occurred only in middle

and high school. The average number of API growth points from 2009 to 2010 increased from

elementary to middle to high school.


16/47

0

10

20

30

40

50

60

70

80

90

100

Percent

Discipline & Academic Outcomes

Elem (n = 84)

Mid (n = 28)

High (n = 30)

Overall (n = 142)

Figure 4: Discipline and Academic Outcome Data 09-10

The following section details the results of our analyses. The evaluation question is

listed as a header, followed by our findings.

1) What is the level of implementation of School-Wide Positive Behavior Interventions andSupports (SWPBIS) at the local school and local district level based on the Foundation

Discipline Policy criteria?

Evidence suggests that SW-PBIS practices can reduce challenging behavior and negativepeer processes in schools (J. Sprague & Golly, 2004; Sugai, Horner, & Gresham, 2002). Evidence

also suggests a relationship between implementation fidelity and outcomeS in SW-PBIS

research (R. H. Horner et al., 2004).

Team Implementation Checklist. The number and percent of schools meeting minimal,

partial, and full implementation criteria on the TIC is provided in Table 4 and Figure 5. A total of

80 elementary schools, 21 middle schools and 17 high schools provided data on all TIC items.

Elementary and middle schools appeared to be the strongest implementers, with middle

schools reporting the most fully implementing.

Table 4: TIC summary scores by school level

Minimally Partially Fullynumber percent number percent number percent

Elem (n = 80) 22 27.5 34 42.5 24 30

Middle (n = 21) 6 28.57 7 33.33 8 38.09

High (n = 17) 6 35.29 8 47.05 3 17.65

Figure 5: TIC Summary Scores by School Level


17/47

Table 5 and Figure 6 provide a more fine-grained overview of the average percent of

item scores as achieved per subscale across school levels. Although elementary schools were

the overall strongest implementers, high schools rated themselves as having achieved 60% of

items on Establishing Information System and 69.23% of items on Capacity for Function-

Based Support.

Table 5: Mean Percent of Items Rated Achieved by TIC Subscale

Elem Middle High

Mean Pct SD Mean Pct SD Mean Pct SD

TIC EstabComm 71.43 36.64 48.21 39.63 56.67 34.07

TIC EstabTeam 55.56 33.27 41.67 40.19 51.11 33.60

TIC SelfAssess 48.41 38.86 41.67 44.10 46.67 41.61

TIC Estab Exp 71.23 31.93 53.57 39.90 45.00 34.23

TIC Estab InfoSys 61.90 48.85 57.14 50.40 60.00 49.83

TIC Sys Info Fam 37.04 48.59 54.55 50.96 42.31 50.38

TIC Cap FB Supp 63.58 37.09 54.55 40.56 69.23 28.55

TIC Total 62.74 26.91 55.32 35.23 55.14 26.42

0

5

10

15

20

25

30

3540

45

50

Elem (n = 80) Middle (n = 21) High (n = 17)

Percent

TIC--Implementation Level

Minimally

Partially

Fully


18/47

Figure 6: Percent of Items Achieved by TIC Subscales and School Level

Rubric of Implementation (RoI). The RoI provided scores on 8 key features of SWPBIS

implementation: (1) administrative leadership and support, (2) team-based implementation, (3)

behavioral expectations defined, (4) behavioral expectations taught, (5) appropriate behaviors

acknowledged and reinforced, (6) monitoring and correcting of behaviors, (7) data-based

decision making, and (8) family and community collaboration. Scores of 3 and 4 on the RoI

indicate that strong evidence of implementation is available; scores of 1 and 2 indicate that

more support is needed to achieve full implementation. To compare RoI scores across subscales

and school levels, we computed the percent of points earned on each subscale. For example, a

school scoring 3 on Administrative Leadership earned 75% of points. Table 6 summarizes the

average percentages of points across subscales and school levels and Figure 7 provides a

graphic illustration.

Table 6: Mean Percent of Points Earned by RoI Subscale and School Level

Elem Middle High

Mean Pct SD Mean Pct SD Mean Pct SD

RoI Admin Lead 76.51 12.56 77.88 12.90 71.55 18.57

RoI Team Based Imp 71.99 16.29 73.08 22.27 63.79 23.70

RoI Beh Exp Defined 73.19 18.64 64.42 25.66 57.76 27.63

RoI Beh Exp Taught 78.92 16.79 73.08 17.21 64.66 26.32

RoI Appr Beh Reinforced 83.43 16.69 64.42 23.64 56.90 28.27

RoI Monitor Correct 77.41 17.73 77.88 16.32 68.97 20.76

RoI Databased Dec 64.76 16.58 70.19 18.73 62.93 20.72

RoI Fam Comm Coll 62.35 20.80 57.69 20.94 57.76 25.09

RoI Total 73.57 11.14 69.83 15.18 63.04 17.91

10

20

30

40

50

60

70

8090

100

AveragePercent

TIC Subscales by School Level

Elem

Middle

High


19/47

Figure 7: RoI Subscales by School Level

Based on RoI scores, elementary schools were the strongest implementers overall,

followed by middle schools and high schools. However, middle schools scores highest in

Administrative Leadership Team-Based Implementation Monitoring and Correcting and

Data-based Decision-Making.

TIC and RoI Correlations

While the TIC is a self-assessment completed by school personnel based on self-ratings,

the RoI is completed by outside of school support staff members gathering evidence of

implementation based on review of permanent products, observations of school environments,

staff interviews and student interviews.

We correlated TIC and RoI subscale scores with total scores to evaluate the extent to

which the outcomes of the two measures yield similar outcomes. If the TIC and RoI performed

similarly, one would expect high correlations between TIC subscale 1 (Establish Commitment)

and RoI subscale 1 (Administrative Leadership), TIC subscale 2 (Establish and Maintain Team)

and RoI subscale 2 (Team-based Implementation), TIC subscales 3 and 5 (Self-Assessment and

Establishing Information System) and RoI subscale 7 (Data Based Decision Making), TIC subscale4 (Establish School-wide Expectations) and RoI subscales 3, 4, 5, and 6 (Behavioral Expectations

Defined, Behavioral Expectations Taught, Appropriate Behavior Reinforced, and

Monitoring/Correcting), TIC subscale 6 (System for Inviting and Informing Families) and RoI

subscale 8 (Family/Community Collaboration) and the TIC total and RoI total scores.

Below we summarize the correlation outcomes by school level. Each cell shows the

correlation coefficient r, the p-value indicating statistical significance, and the number of cases

on which the correlation was based. Table 7 summarizes the expected correlation patterns if

10

20

30

40

50

60

7080

90

AveragePercent

RoI Subscales by School Level

Elem

Middle

High


20/47

the TIC and RoI were to perform similarly. Table 8 shows outcomes for elementary schools,

Table 9 for middle schools, and Table 10 for high schools. Blue shaded cells indicate statistically

correlation between TIC and RoI scales. The blue shaded cells would show high correlations.

Table 7: Expected Correlation Patterns of TIC and RoI

RoI1 2 3 4 5 6 7 8 Tot

TIC

1

2

3

4

5

6

7

Tot

Below we summarize the obtained correlation outcomes by school level. Each cell

shows (a) the correlation coefficient r, the p-value (Sig) indicating statistical significance, and

the number of cases on which the correlation was based. Table 8 shows outcomes for

elementary schools, Table 9 for middle schools, and Table 10 for high schools. A p-value of .05

and smaller indicates statistical significance.

It is very important to note that statistical significance may not be the most important

criteria for evaluating these relationships. The small sample sizes (school-level), particularly at

the middle and high school level would require very strong correlations to be statistically

significant (this refers to statistical power). Roughly, any correlation above r=.2 suggests a

relationship between the two scales.


21/47

Table 8: TIC and RoI correlations for elementary schools

RoI Admin

Lead Pct

RoI

Team

Based

Imp Pct

RoI Beh Exp Defined

Pct

RoI Beh Exp Taught

Pct

RoI Appr Beh

Reinforce Pct

RoI

Monitor

Correct

Pct

RoI

Databased

Dec Pct

RoI Fam

Comm

Collab

Pct

ROI

Total

Pct

TIC Estab

Commitmt Pct

r .195 .246 .129 .081 .140 .249 .310 .055 .261

Sig. .078 .025 .245 .467 .207 .023 .004 .619 .017

N 83 83 83 83 83 83 83 83 83

TIC Estab Maint

Team Pct

r .211 .092 .151 .149 .181 .253 .491 .146 .316

Sig. .056 .406 .172 .180 .102 .021 .000 .187 .004

N 83 83 83 83 83 83 83 83 83

TIC Self

Assessmt Pct

r .160 .060 .068 .092 .076 .270 .395 .060 .220

Sig. .149 .592 .542 .408 .494 .014 .000 .592 .045

N 83 83 83 83 83 83 83 83 83

TIC Estab SW

Expect Pct

r .213 .303 .192 .221 .116 .261 .345 .146 .339

Sig. .054 .005 .082 .044 .295 .017 .001 .186 .002

N 83 83 83 83 83 83 83 83 83

TIC Estab Info

Syst Pct

r .243 .049 -.008 -.005 .167 .247 .238 .250 .224

Sig. .027 .662 .942 .961 .131 .024 .030 .022 .042

N 83 83 83 83 83 83 83 83 83

TIC Syst Inv

Inform Fam Pct

r .145 .198 .052 .083 .038 .133 .339 .014 .183

Sig. .198 .076 .648 .463 .734 .235 .002 .902 .102

N 81 81 81 81 81 81 81 81 81

TIC Cap Funct

BasSupp Pct

r .183 .109 .116 .061 .167 .132 .273 .033 .198

Sig. .102 .332 .303 .587 .135 .239 .014 .773 .076

N 81 81 81 81 81 81 81 81 81

TIC Total Pct r .214 .203 .139 .163 .151 .317 .463 .134 .337

Sig. .054 .067 .214 .143 .175 .004 .000 .229 .002

N 82 82 82 82 82 82 82 82 82


22/47

Table 9: TIC and RoI correlations for middle schools

RoI Admin

Lead Pct

RoI Team

Based Imp

Pct

RoI Beh Exp

Defined Pct

RoI Beh Exp

Taught

Pct

RoI Appr Beh

Reinforced Pct

RoI Monitor

Correct Pct

RoI

Databased

Dec Pct

RoI Fam Comm

Collab Pct ROI Total Pct

TIC Estab

Commitmt Pct

r .289 .237 .172 .381 .187 .228 .151 .104 .273

Sig. .152 .245 .400 .055 .360 .262 .461 .612 .177

N 26 26 26 26 26 26 26 26 26

TIC Estab Maint

Team Pct

r .062 .026 .254 .373 .346 .305 .033 .209 .267

Sig. .763 .900 .211 .060 .083 .130 .875 .307 .187N 26 26 26 26 26 26 26 26 26

TIC Self Assessmt

Pct

r .056 .057 .230 .557 .313 .414 .070 .225 .310

Sig. .785 .781 .259 .003 .119 .035 .735 .270 .123

N 26 26 26 26 26 26 26 26 26

TIC Estab SW

Expect Pct

r .290 .135 .272 .478 .314 .336 .216 .193 .354

Sig. .150 .511 .179 .014 .118 .093 .290 .345 .076

N 26 26 26 26 26 26 26 26 26

TIC Estab Info

Syst Pct

r .180 .111 .139 .378 .151 .266 .116 .104 .223

Sig. .378 .588 .498 .057 .462 .189 .573 .614 .273

N 26 26 26 26 26 26 26 26 26

TIC Syst Inv

Inform Fam Pct

r .153 .282 .034 .271 .081 .115 .197 .142 .200

Sig. .498 .204 .881 .223 .721 .610 .378 .530 .373

N 22 22 22 22 22 22 22 22 22

TIC Cap Funct

BasSupp Pct

r .192 .285 .101 .340 .164 .145 .172 .253 .263

Sig. .393 .199 .654 .121 .467 .521 .443 .256 .237

N 22 22 22 22 22 22 22 22 22

TIC Total Pct r .082 .130 .252 .419 .167 .268 .061 .285 .273

Sig. .709 .555 .245 .046 .447 .216 .783 .188 .208

N 23 23 23 23 23 23 23 23 23


23/47

Table 10: TIC and RoI correlations for high schools

RoI Pct

RoI Team Based

Imp Pct

RoI Beh Exp

Defined Pct

RoI Beh Exp

Taught

Pct

RoI Appr Beh

Reinforced Pct

RoI Monitor

Correct Pct

RoI Databased

Dec Pct

RoI Fam Comm

Collab Pct ROI Total Pct

TIC Estab

Commitmt Pct

r .315 .187 .339 .214 .196 .238 .221 .426 .3

Sig. .096 .331 .072 .266 .309 .214 .250 .021 .0

N 29 29 29 29 29 29 29 29

TIC Estab Maint

Team Pct

r .435 .431 .284 .130 .068 .278 .383 .240 .3

Sig. .018 .020 .136 .500 .724 .144 .041 .210 .0

N 29 29 29 29 29 29 29 29

TIC Self Assessmt

Pct

r .405 .443 .443 .306 .134 .398 .274 .401 .4

Sig. .029 .016 .016 .106 .487 .032 .151 .031 .0

N 29 29 29 29 29 29 29 29

TIC Estab SW

Expect Pct

r .384 .192 .461 .444 .307 .352 .279 .595 .5

Sig. .040 .319 .012 .016 .106 .061 .143 .001 .0

N 29 29 29 29 29 29 29 29

TIC Estab Info Syst

Pct

r .242 .234 .289 .374 .258 .204 .060 .390 .3

Sig. .206 .222 .129 .045 .176 .288 .757 .036 .0

N 29 29 29 29 29 29 29 29

TIC Syst Inv

Inform Fam Pct

r .297 .007 .316 -.037 .036 .036 .141 .014 .1

Sig. .149 .972 .124 .861 .866 .863 .501 .948 .5

N 25 25 25 25 25 25 25 25

TIC Cap Funct

BasSupp Pct

r .363 .227 .373 .486 .313 .318 .177 .421 .4

Sig. .074 .275 .066 .014 .128 .121 .398 .036 .0

N 25 25 25 25 25 25 25 25

TIC Total Pct r .336 .156 .329 .273 .105 .196 .160 .352 .3

Sig. .093 .447 .101 .177 .611 .337 .434 .077 .0

N 26 26 26 26 26 26 26 26


24/47

The strongest correlations between TIC and RoI subscales existed in the elementary

school sample, followed by high schools and then middle schools. We would not expect perfect

correlation between the two instruments as their content does not completely align.

2) Are students, families and school/district personnel aware of and involved with the policyand its elements (the eight key features of SWPBIS)?

What is their view of the effectiveness? What is their view of the feasibility of implementation? What is their level of satisfaction with the Discipline Foundation Policy

implementation?

Staff Member Survey. The staff member survey consisting of 21 questions was

administered to all schools. A total of 1643 respondents completed the survey. Question 1

asked respondents to list the name of their school. All responses were recoded into

elementary, middle, and high school based on a list of participating schools. This recodingresulted in 550 elementary school responses, 633 middle, and 429 high. 31 responses could not

be matched to any school name on the list provided. The results follow a similar pattern as the

TIC and RoI fidelity outcomes: Ratings generally decline from elementary to middle to high

School. In all settings, most responses are distributed toward the agree category with data

use, SWPBIS updates, and Resources receiving the most disagree responses.

Outcomes are summarized in figures 8-10 below.

Figure 8: Elementary Staff Member PBIS Survey Results by Item

.05.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Studentsunderstand

Studentsunderstand

Studentsunderstand

Studentsare

Teachershelpstudents

Behexpareconsistent

Fairconsequencesfor

Adminsupportsbehexp

Behexparetaught

Parentsknow

behexp

Myinputonbehexpwas

Studentsarereminded

Consistentexphelp

Moststudentsfollow

Ourteamrepresentsall

Behdataare

Dataareusedfor

UpdatestoSWPBISare

Ihaveresourcesto

PercentofResponses

Staff Survey Elementary Schools

Str Disagree

Disagree

Smwh Disagree

Smwh Agree

Agree

Str Agree


25/47

Figure 9: Middle School Staff Member PBIS Survey Results by Item

Figure 10: High School Staff Member PBIS Survey Results by Item

.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.045.0

50.0

Studentsundersta

ndsafety

Studentsun

derstand

Studentsun

derstand

Studentsareacknowledged

Teachershelp

students

Behexparec

onsistent

Fairconsequ

encesfor

Adminsupport

sbehexp

Behexpa

retaught

Parentsknow

behexp

Myinputonbehexpwas

Studentsarerem

indedof

Consistentexphelp

Moststudentsfollowbeh

Ourteamrepr

esentsall

Beh

dataare

Dataareusedfor

decisions

UpdatestoSW

PBISare

Ihaveres

ourcesto

PercentofResponses

Staff Survey Middle Schools

Str Disagree

Disagree

Smwh Disagree

Smwh Agree

Agree

Str Agree

.05.0

10.015.020.025.030.035.040.045.050.0

Studentsunderstand

Studentsunderstand

Studentsunderstand

Studentsare

Teachershelpstudents

Behexpareconsistent

Fairconsequencesfor

Adminsupportsbehexp

Behexparetaught

Parentsknowbehexp

Myinputonbehexpwas

Studentsareremindedof

Consistentex

phelp

Moststudentsfollo

wbeh

Ourteamreprese

ntsall

Behda

taare

Dataareusedfordecisions

UpdatestoSWPBISare

Ihaveresourcesto

PercentofResponses

Staff Survey High Schools

Str Disagree

Disagree

Smwh Disagree

Smwh Agree

Agree

Str Agree


26/47

Parent Survey. A parent survey consisting of 16 questions was administered to all

participating schools. A total of 728 respondents completed the survey. Question 1 asked

respondents to list the name of their school. All responses were recoded into elementary,

middle, and high school based on a list of participating schools. This recoding resulted in 490

elementary, 167 middle and 44 high school responses. There were 27 responses that could notbe matched to any school name on the list provided.

The level of agreement with the statements tended to decline from elementary through

high school as observed in the previous measures. Lowest rated items included those

addressing parent input, information about PBIS is shared, and child/student recognition.

Survey outcomes by question are summarized in figures 11-13 below:

Figure 11: Elementary School Parent PBIS Survey Results by Item

.0

10.0

20.0

30.0

40.0

50.0

60.0

PercentofResponses

Parent Survey Elementary Schools

Str Disagree

Disagree

Smwh Disagree

Smwh Agree

Agree

Str Agree


27/47

Figure 12: Middle School Parent PBIS Survey Results by Item

Figure 13: High School Parent PBIS Survey Results by Item

.0

10.0

20.0

30.0

40.0

50.0

60.0

PercentofResponses

Parent Survey Middle Schools

Str Disagree

Disagree

Smwh Disagree

Smwh Agree

Agree

Str Agree

.0

10.0

20.0

30.0

40.0

50.0

60.0

PercentofResponses

Parent Survey High Schools

Str Disagree

Disagree

Smwh Disagree

Smwh Agree

Agree

Str Agree


28/47

A summary of the survey data shows that the responses that had the highest

percentage of agreement in the staff survey were: I think having the same 3-5 behavior

expectations/rules throughout our school and in every classroom helps students follow the

behavior expectations/rules (90%) and Most teachers help students to know the behavior

expectations/rules for our school (87%) (see table 10). The majority of Staff responses are

distributed toward the agree category with allstatements which include Most students atmy school understand what it means to be safe (85%); If a student is not following the

behavior expectation/rules, they are reminded of the behavior expectations/rules before being

assigned other consequences (84%); most students at my school follow the behavior

expectations/rules (72%). The responses that had the highest percentage of disagreement in

the staff survey were: Updates on the School-Wide Positive Behavior Support efforts are

communicated through newsletters, brochures, open house, parent meetings, etc. at least 5

times per school year (47%) and Our school uses behavior data to make decisions in

designing, implementing, and revising school-wide efforts at least 2 times per school year

(43%) (see table 11). It is important to note that staff responses were more varied with level of

agreement on statements about Data use, SWPBIS updates, and Resources and Support.

The responses that had the highest percentage of agreement in the parent survey were:

My child knows the behavior expectations/rules the school has for them(91%) and I know

the behavior expectations/rules at my childs school (90%) (see table 12). A majority of parents

responded that they agreed or strongly agreed with the statements The students at my childs

school are: safe (87%), respectful (80%), and responsible (84%). Parents also agreed or

strongly agreed with the statement that teachers help their child know the behavior

expectations/rules (89%). The responses that had the highest percentage of disagreement in

the parent survey were: I have been invited to provide my opinions regarding what the school

behavior expectations/rules should be at my childs school (42%) and My child has received

an award/recognition from the school for meeting expectations school behavior

expectations/rules (37%) (see table 12).

Table 11: Responses that had the Highest Percentage of Agreement and Disagreement (Staff Survey)

Statement % agree

I think having the same 3-5 behavior expectations/rules throughout our

school and in every classroom helps students follow the behavior

expectations/rules.

90%

Most teachers help students to know the behavior expectations/rules for our

school.

87%


Updates on the School-Wide Positive Behavior Support efforts arecommunicated through newsletters, brochures, open house, parent

meetings, etc. at least 5 times per school year.

47%

Our school uses behavior data to make decisions in designing, implementing,

and revising school-wide efforts at least 2 times per school year.

43%


29/47

Table 12: Responses that had the Highest Percentage of Agreement and Disagreement (Parent Survey)

Statement % agree

My child knows the behavior expectations/rules the school has for them. 91%

I know the behavior expectations/rules at my childs school. 90%

Statement %disagree

I have been invited to provide my opinions regarding what the schoolbehavior expectations/rules should be at my childs school.

42%

My child has received an award/recognition from the school for meeting

expectations school behavior expectations/rules.

37%

3) What results does implementation of the Discipline Foundation Policy and SWPBISpractices have on LAUSD schools and the students and families they serve?

ODR, suspension, expulsion, Opportunity Transfers (OT) Disproportionality/risk ratios for suspensions Attendance Achievement

To evaluate the relationship between LAUSDs SWPBIS implementation and student

outcomes, we correlated both TIC and RoI subscales with the following key student outcomes:

(a) in seat attendance, (b) suspension rates per 100 students, (c) opportunity transfer rates per

100 students, (d) percent of students ELA proficient, and (e) percent of students Math

proficient. We expected greater SWPBIS implementation to be related to higher in seat

attendance, lower rates of suspension and opportunity transfers, and higher percentages of

students meeting proficiency criteria in ELA and Math. Below we summarize the correlation

outcomes for elementary, middle, and high schools. Shaded cells indicate statistically significant

correlations (Tables 13-15).

Given the small sample size, statistical significance may not be the best indicator of

relationship strength. We recommend looking at positive correlations (r scores) of .2 or higher

for desired outcomes such as in seat or achievement) and negative correlations of -.2 or

lower for outcomes targeted for reduction such as suspensions or opportunity transfers.

We see a general pattern of positive correlation between SW-PBIS implementation and

academic outcomes, and a negative correlation between SW-PBIS implementation and

discipline outcomes. This finding suggests support for the hypothesized relation between

implementation fidelity, behavior and academic outcomes.


30/47

Table 13: Correlation of TIC Scores with Student Outcomes in Elementary Schools

In Seat

attendan

ce Rate

SuspPer100Stu

dents

OppTransfPer100Stude

nts

ELA Proficient

Pct

Math Proficient

Pct

TIC Estab

Commitmt

Pct

r -.179 -.045 -.002 -.002

Sig. .103 .684 N/A .985 .987

N 84 84 82 82

TIC Estab

Maint

Team Pct

r -.209 -.033 .051 .045

Sig. .056 .768 N/A .647 .687

N 84 84 82 82

TIC Self

Assessmt

Pct

r -.228 .001 .174 .145

Sig. .037 .991 N/A .117 .194

N 84 84 82 82

TIC Estab

SW Expect

Pct

r -.105 -.087 -.023 -.032

Sig. .343 .432 N/A .836 .773

N 84 84 82 82

TIC Estab

Info Syst

Pct

r -.102 -.044 .160 .113

Sig. .356 .691 N/A .151 .313

N 84 84 82 82

TIC Syst Inv

Inform

Fam Pct

r -.136 .016 .041 .002

Sig. .226 .885 N/A .719 .989

N 81 81 79 79

TIC Cap

Funct Bas

Supp Pct

r -.069 .054 .117 .057

Sig. .540 .631 N/A .303 .615

N 81 81 79 79

TIC Total

Pct

r -.180 .034 .163 .080

Sig. .105 .762 N/A .148 .480

N 82 82 80 80

In Seat

attendanc

e Rate

r -.114 .352 .463

Sig. .301 N/A .001 .000

N 84 82 82

Susp Per

100

Students

r -.123 -.200

Sig. N/A .269 .072

N 82 82

OppTransf

Per 100

Students

r .a .a

Sig. N/A . .

N 82 82

ELA

Proficient

Pct

r .914

Sig. N/A .000

N 82


31/47

Table 14: Correlation of TIC Scores with Student Outcomes in Middle SchoolsIn Seat

attendance

Rate SuspPer100Students OppTransfPer100Students

ELA

Proficient

Pct

Math

Proficient

Pct

TIC Estab Commitmt Pct r .171 -.211 .141 .398 .303

Sig. .385 .281 .473 .036 .117

N 28 28 28 28 28TIC Estab Maint Team Pct r .102 -.147 .028 .195 .038

Sig. .607 .455 .889 .319 .849

N 28 28 28 28 28

TIC Self Assessmt Pct r .183 -.299 .083 .258 .204

Sig. .351 .122 .676 .185 .297

N 28 28 28 28 28

TICEstabSWExpectPct r .209 -.228 .096 .365 .240

Sig. .287 .244 .628 .056 .219

N 28 28 28 28 28

TICEstabInfoSystPct r .202 -.230 .145 .197 .179

Sig. .303 .239 .462 .316 .362

N 28 28 28 28 28

TICSystInvInformFamPct r .192 -.284 .198 .051 .080

Sig. .391 .201 .378 .822 .724

N 22 22 22 22 22

TICCapFunctBasSuppPct r .293 -.308 .205 .073 .058

Sig. .185 .163 .360 .746 .798

N 22 22 22 22 22

TIC Total Pct r .147 -.204 .017 .226 .088

Sig. .492 .340 .938 .288 .681

N 24 24 24 24 24

In Seat attendance Rate r -.855 -.419 .527 .633

Sig. .000 .026 .004 .000

N 28 28 28 28

SuspPer100Students r .473 -.518 -.587

Sig. .011 .005 .001

N 28 28 28

OppTransfPer100Students r -.394 -.425

Sig. .038 .024

N 28 28

Growth Points r -.094 -.085

Sig. .636 .667

N 28 28

ELA Proficient Pct r .910

Sig. .000

N 28


32/47

Table 15: Correlation of TIC Scores with Student Outcomes in High SchoolsIn Seat

attendance

Rate

SuspPer100Stud

ents

OppTransfPer10

0Students

ELA Proficient

Pct

Math

Proficient Pct

TIC Estab Commitmt

Pct

r .314 -.256 -.077 .159 .353

Sig. .091 .172 .684 .409 .061

N 30 30 30 29 29TICEstabMaintTeamPct r .239 -.113 -.007 -.032 .130

Sig. .204 .552 .972 .868 .503

N 30 30 30 29 29

TICSelfAssessmtPct r .324 -.312 -.183 .286 .319

Sig. .080 .094 .332 .133 .092

N 30 30 30 29 29

TIC Estab SW Expect

Pct

r .451 -.198 -.114 .279 .512

Sig. .012 .294 .549 .142 .005

N 30 30 30 29 29

TICEstabInfoSystPct r .258 -.159 -.041 .175 .305

Sig. .168 .401 .831 .365 .108

N 30 30 30 29 29

TICSystInvInformFamPc

t

r .309 -.192 -.280 .018 .143

Sig. .124 .346 .165 .932 .495

N 26 26 26 25 25

TICCapFunctBasSuppPc

t

r -.015 -.309 -.143 -.142 -.018

Sig. .943 .125 .486 .499 .931

N 26 26 26 25 25

TICTotalPct r .364 -.228 -.249 .113 .329

Sig. .062 .253 .210 .582 .101

N 27 27 27 26 26

In Seat attendance

Rate

r -.497 -.522 .408 .616

Sig. .005 .003 .028 .000

N 30 30 29 29

SuspPer100Students r .718 -.127 -.173

Sig. .000 .512 .369

N 30 29 29

OppTransfPer100Stude

nts

r -.204 -.247

Sig. .289 .196

N 29 29

Growth Points r .045 -.179

Sig. .819 .362

N 28 28


Sig. .000

N 29

Although correlations did not always reach statistical significance, it was encouraging to

see that TIC scores were generally associated with lower suspension rates, and higher ELA andMath proficiency rates in elementary schools. In middle schools, higher TIC scores were

associated with higher in seat attendance, lower suspension rates, and higher ELA and Math

proficiency rates. Particularly encouraging in the middle school sample was the overall

association between TIC scores and higher in seat attendance paired with the statistically

significant correlations between in seat attendance and lower suspension rates, lower

opportunity transfer rates, and higher academic achievement in both ELA and Math. This seems


33/47

to indicate that high fidelity implementation of behavioral support serves to lower behavioral

incidents and increase academic achievement. The same pattern emerged more powerfully in

the high school sample. TIC scores were clearly associated with increased in seat attendance,

which in turn was statistically significantly associated with lower suspension and opportunity

transfer rates, and higher proficiency rates in ELA and Math. A primary focus of SWPBIS is to

reduce the use of office referrals, suspensions and expulsions. We see strong emergingevidence of the relationship between reduced use of these types of sanctions and increased

attendance and achievement.

Rubric of Implementation. We conducted the same correlational analysis to evaluate

relationships between RoI scores and student outcomes. The tables below (Tables 16-18)

summarize those outcomes for elementary, middle, and high schools.

Table 16: Correlation of RoI Scores with Student Outcomes in Elementary SchoolsIn Seat

attendance

Rate SuspPer100Students

Opp Transf Per

100 Students

ELA Proficient

Pct

Math

Proficien

t PctRoI Admin Lead Pct r -.236 -.026 -.085 -.091

Sig. .031 .818 N/A .453 .418

N 83 83 81 81

RoI Team Based Imp Pct r -.176 -.048 -.149 -.137

Sig. .112 .665 N/A .185 .222

N 83 83 81 81

RoI Beh Exp Defined Pct r .029 -.082 .091 .049

Sig. .795 .461 N/A .420 .665

N 83 83 81 81

RoI Beh Exp Taught Pct r -.057 -.119 .116 .088

Sig. .610 .283 N/A .302 .432

N 83 83 81 81

RoI Appr Beh ReinforcedPct

r -.070 -.169 .036 .083Sig. .528 .126 N/A .749 .464

N 83 83 81 81

RoI Monitor Correct Pct r -.110 -.170 .177 .196

Sig. .322 .125 N/A .113 .080

N 83 83 81 81

RoI Databased Dec Pct r -.197 .116 .074 .038

Sig. .074 .298 N/A .509 .735

N 83 83 81 81

RoI Fam Comm

Collaboration

r -.062 -.067 .172 .143

Sig. .577 .546 N/A .126 .204

N 83 83 81 81

RoI Fam Comm Collab Pct r -.062 -.067 .172 .143

Sig. .577 .546 N/A .126 .204

N 83 83 81 81

ROITotalPct r -.156 -.112 .097 .084

Sig. .158 .315 N/A .387 .459

N 83 83 81 81

In Seat attendance Rate r -.114 .352 .463

Sig. .301 N/A .001 .000

N 84 82 82

SuspPer100Students r -.123 -.200

Sig. N/A .269 .072


34/47

In Seat

attendance

Rate SuspPer100Students

Opp Transf Per

100 Students

ELA Proficient

Pct

Math

Proficien

t Pct

N 82 82

OppTransfPer100Students r

Sig. N/A N/A N/AN

Growth Points r .016 .055

Sig. N/A .890 .621

N 82 82


Sig. N/A .000

N 82

Table 17: Correlation of RoI Scores with Student Outcomes in Middle SchoolsIn Seat

attendance


ELA

Proficient

Pct

Math

Proficient

Pct

RoIAdminLeadPct r .365 -.215 .127 .241 .226

Sig. .067 .292 .535 .236 .267N 26 26 26 26 26

RoITeamBasedImpPct r .485 -.409 -.042 .207 .227

Sig. .012 .038 .839 .309 .265

N 26 26 26 26 26

RoIBehExpDefinedPct r .293 -.183 -.238 .361 .334

Sig. .146 .371 .241 .070 .095

N 26 26 26 26 26

RoIBehExpTaughtPct r .202 -.249 .039 .151 .052

Sig. .324 .220 .851 .460 .802

N 26 26 26 26 26

RoIApprBehReinforcedPct r .206 -.184 -.098 .284 .220

Sig. .313 .368 .634 .159 .280

N 26 26 26 26 26

RoIMonitorCorrectPct r .057 -.123 .182 .181 .007

Sig. .780 .549 .374 .376 .974

N 26 26 26 26 26

RoI Databased Dec Pct r .187 -.155 -.037 .057 .123

Sig. .362 .450 .856 .782 .550

N 26 26 26 26 26

RoI Fam Comm

Collaboration

r .266 -.208 -.007 .065 .003

Sig. .190 .308 .975 .752 .990

N 26 26 26 26 26

RoI Fam Comm Collab Pct r .266 -.208 -.007 .065 .003

Sig. .190 .308 .975 .752 .990

N 26 26 26 26 26

ROI Total Pct r .341 -.284 -.041 .261 .207

Sig. .089 .160 .844 .198 .311

N 26 26 26 26 26


Sig. .000 .026 .004 .000

N 28 28 28 28

Susp Per 100 Students r .473 -.518 -.587

Sig. .011 .005 .001

N 28 28 28

Opp Transf Per 100 r -.394 -.425


35/47

In Seat

attendance


ELA

Proficient

Pct

Math

Proficient

Pct

Students Sig. .038 .024

N 28 28

Growth Points r -.094 -.085

Sig. .636 .667

N 28 28


Sig. .000

N 28

Table 18: Correlation of RoI Scores with Student Outcomes in High SchoolsIn Seat

attendance

Rate

Susp Per 100

Students

OppTransfPer100Stu

dents

ELA

Proficient Pct

Math

Proficient Pct

RoI Admin Lead Pct r -.001 .135 -.073 .440 .413

Sig. .994 .484 .705 .019 .029

N 29 29 29 28 28

RoITeamBasedImpPct r -.142 .132 .139 .322 .092Sig. .461 .495 .472 .095 .643

N 29 29 29 28 28

RoIBehExpDefinedPct r .199 -.383 -.402 .378 .250

Sig. .300 .040 .031 .047 .199

N 29 29 29 28 28

RoIBehExpTaughtPct r .061 -.237 -.289 .148 .238

Sig. .753 .216 .129 .454 .223

N 29 29 29 28 28

RoIApprBehReinforcedPct r -.055 -.156 -.202 .092 .075

Sig. .776 .420 .294 .640 .703

N 29 29 29 28 28

RoIMonitorCorrectPct r -.085 .085 -.036 .491 .385

Sig. .662 .660 .854 .008 .043N 29 29 29 28 28

RoIDatabasedDecPct r .165 -.165 -.280 .161 .147

Sig. .392 .393 .142 .414 .456

N 29 29 29 28 28

RoIFamCommCollaboration r .142 -.204 -.058 .342 .508

Sig. .463 .288 .766 .075 .006

N 29 29 29 28 28

RoIFamCommCollabPct r .142 -.204 -.058 .342 .508

Sig. .463 .288 .766 .075 .006

N 29 29 29 28 28

ROI Total Pct r .051 -.156 -.213 .388 .345

Sig. .791 .419 .268 .041 .072

N 29 29 29 28 28


Sig. .005 .003 .028 .000

N 30 30 29 29

Susp Per 100 Students r .718 -.127 -.173

Sig. .000 .512 .369

N 30 29 29

Opp Transf Per 100 Students r -.204 -.247

Sig. .289 .196

N 29 29


36/47

In Seat

attendance

Rate

Susp Per 100

Students

OppTransfPer100Stu

dents

ELA

Proficient Pct

Math

Proficient Pct

Growth Points r .045 -.179

Sig. .819 .362

N 28 28


Sig. .000

N 29

Correlations between RoI scores and student outcomes are similar to those between TIC

scores and student outcomes. In the elementary school sample, RoI scores were generally

associated with lower suspension rates, and higher ELA and Math proficiency rates. In middle

schools, higher RoI scores were associated with higher in seat attendance, lower suspension

rates, and higher ELA and Math proficiency rates. Higher in seat attendance was again paired

with the statistically significantly lower suspension rates, lower opportunity transfer rates, and

higher academic achievement in both ELA and Math. This further supports the association

between high fidelity implementation of SWPBIS and lower behavioral incidents as well asincreases in academic achievement. The high school sample yielded similar patterns. RoI scores

were associated with lower suspension and opportunity transfer rates, and higher proficiency

rates in ELA and Math.

The figures below show correlations between RoI total scores and behavioral outcomes

(attendance, percent suspensions, percent of opportunity transfers) as well as correlations

between RoI total scores and academic outcomes (percent of student proficient in ELA, percent

of student proficient in math) by district. Districts were numbered 1-8; 9 schools were not

coded with any district number, and 5 schools had a district coding of T. The number of schools

included in each district is indicated in the x-axis labels of the figures (e.g. n = 16).

Correlations can take any value between -1 and +1. Positive values indicate that high RoI

scores are associated with high student outcomes, and low RoI scores are associated with low

student outcomes. Negative values indicate that high RoI scores are associated with low

student outcomes, and low RoI scores are associated with high student outcomes.

For the correlations between RoI total score and behavioral outcomes, we would expect high

implementation to be associated with high attendance and low suspensions/opportunity

transfers. As Figure 14 illustrates, these expected associations were strongest in districts 3 and

T, followed by districts 7, 8, 4, and 1.


37/47

Figure 14: Correlations between RoI Total Score and Behavioral Outcomes by District

For the correlations between RoI total score and academic outcomes, we would expect high

implementation to be associated with high percentages of students to be proficient in ELA and

Math. As Figure 15 illustrates, these expected associations were strongest in districts T and 6,

followed by districts 3, 8, 4, 1, and 2.

Figure 15: Correlations between RoI Total Score and Academic Outcomes by District

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.60.8

1

Correlation

Coefficientr

Correlations between RoI Total Score and Behavioral Outcomes by

District

RoI/Attendance

RoI/SuspRate

RoI/OppTransRate

-1

-0.5

0

0.5

1

Correl

ation

Coefficientr

Correlations between RoI Total Score and Academic Outcomes by

District

RoI/PctELAProficient

RoI/PctMathProficient


38/47

Figures 16 and 17 provide an overview of the mean percent of points on the RoI in relation to

student behavioral outcomes (percent attendance, number of students suspended per 100

students, number of opportunity transfer per 100 students), and in relation to student

academic outcomes (percent of students proficient in ELA and percent of students proficient in

math) by district.

Figure 16: Mean Percent of RoI Total Scores and Behavioral Outcomes

Figure 17: Mean Percent of RoI Total Scores and Academic Outcomes

0

10

2030

40

50

60

70

80

90

100

Percent

Means of SWPBS implementation and behavioral outcomes by district

RoI Pct of Total

Attendance

SuspRate/100 Stu

OppTransRate/100 Stu

010

20

30

40

50

60

70

80

90

100

Percent

Means of SWPBS implementation and academic outcomes by district

RoI Pct of Total

PctELAProficient

PctMathProficient


39/47

Summary

Elementary and middle schools appeared to be the strongest implementers. Based on

RoI scores, elementary schools were the strongest implementers overall, followed by middle

schools and high schools. However, middle schools scores highest in Administrative

Leadership, Team-Based Implementation, Monitoring and Correcting, and Data-BasedDecision-Making.

We found a mild to moderate correlation between high SW-PBIS implementation and in

seat attendance, lower suspensions and opportunity transfers and higher ELA and Math

Proficiency. This is impressive given the relatively small sample of schools at the middle and

high school level and suggests positive benefits from the implementation of the FDP.

Recommendations

The goal for this evaluation was to describe how well the Discipline Foundation Policy is



policy.

Establishment of the Discipline Foundation Policy and the accompanying marketing and

staff development program in LAUSD represent a major accomplishment for such a large

organization. The data from this report suggest that policy implementation and related

activities are having a positive and measurable impact on practice and outcomes in schools.

What needs to be in place to adopt, implement and maintain the Discipline Foundation Policy?

Schools that have high levels of collegiality, job control, and teacher participation in

decision-making perform well in numerous respects. These conditions may lead to betteradoption, implementation, and maintenance of SW-PBIS, provided the school embraces the

goal of adopting these practices.

Bryk and Driscoll (1988) identified five features of the school community that appear

to be associated with the general effectiveness of schools: (a) a system of shared values about

the purpose of the school, (b) clear expectations for students and staff, (c) high expectations for

student learning, (d) a common agenda of activities designed to foster meaningful social

interactions among school members, and (e) social relations marked by caring. Other studies

have also shown greater gains in academic achievement in schools where faculty, students, and

parents have participated in determining the goals and practices of the school. Such

cooperative school environments are associated with higher levels of teachers sense of efficacyand satisfaction as well. This is a core goal of SW-PBIS practices.

Additionally, organizational effectiveness depends on (a) high-quality leadership and

support provided by a principal or other administrator, (b) an internal champion for a

program, (c) access to formal training and technical assistance, and (d) adequate financial

resources for adoption, implementation, and maintenance of the program. Successful

organization, implementation and maintenance of prevention programs, in particular, have

been shown to be related to principals ability to (a) initiate and sustain innovation (b) provide


40/47

leadership in maintaining good relations with teachers, parents, school boards, site councils,

and students (Gottfredson et al., 2000); and (c) provide support and encouragement to

teachers delivering the program. Collectively, research studies support the importance of

strong and consistent principal leadership as a critical component of getting SWPBIS practices

off the ground and sustaining them once they are up and running.

The availability of a strong program coordinator, or influential program champion, evenwithout the support of a principal or agency director, also appears to be instrumental in

determining whether an organization adopts, implements, and maintains an evidence-based

prevention program. To increase the likelihood of success, there should be a program champion

in every school where you are attempting to implement a SW-PBIS model for behavior.

Practices are more likely to be successfully implemented when teaching staff and

administrators believe that the practices are effective, address a real problem, are worth the

effort needed to implement, and will present no adverse side effects in the teaching

environment. Mihalic and Irwin (Mihalic & Irwin, 2003) found that program features were the

most important factors influencing the delivery and maintenance of eight different evidence-

based programs; specifically, complex and/or poorly structured programs were subject to poor

implementation practices.

The following recommendations are related to the findings, and research on what is

known about adoption, implementation, and maintenance of evidence-based behavior support

practices in schools.

In summary, the district should:

Use culturally relevant practices to actively engage family partnerships in developing

understanding and implementation of School Wide Positive Behavior Interventions and

Support. Utilize the shared valued outcomes (school-site, parent and community) to enhance

the knowledge and skills of all stakeholder groups. The policy itself has roles and

responsibilities outlined for each group. Use that guidance to share the core messages of safe,responsible and respectful behavior, along with culturally relevant practices to engage in

educating LAUSD children and youth.

Link advocated practices in the policy to valued outcomes to revitalize school staff

support for full, meaningful implementation (Social marketing). Rogers work on the diffusion

of innovations (Rogers, 1995) suggests that the decision to adopt a program is influenced by

program characteristics such as the relative advantage of the proposed program to existing

programs; its compatibility with existing values, previous experience, and current needs; the

extent to which the program may be tested prior to adoption; and the extent to which the

results of the program are publicly observable. Consistent with social marketing and DOItheory, information about the practice to be adopted must be persuasive and must link the

Evaluation of the Discipline Foundation Policy October 24, 2011

Documents

Transcript of Evaluation of the Discipline Foundation Policy October 24, 2011