Implementing an Evaluation System for SWPBS

Bob AlgozzineRob Horner

National PBIS Leadership ForumChicago Hyatt Regency O’Hare

October 8, 2009

http://www.dpi.state.nc.us/positivebehavior/

IMPLEMENTING AN EVALUATION SYSTEM FOR SWPBS



Objectives

To provide brief overview of evaluation field To describe NC PBS evaluation model To illustrate North Carolina’s evaluation efforts

oContexto Inputo Fidelityo Impact

Evaluation Collecting and Reporting Information

Who Where When Why What How

“Make a Case” What do we know? What do we need to know? How good is acceptable? How bad is unacceptable?

EvaluationCollecting and Reporting Information for Decision-Makers

Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey Bass.

EvaluationCollection and Reporting of Information for Decision-Makers

Defining

Obtaining

Applying

Analyzing Project

Context

InputProcess

Product

WhatHow

Who Where When Why

What difference did it make?

Are we doing it as planned?

Outcome

What should we do?

How should we do it?

SWPBS/NC EvaluationCollection and Reporting of Information for Decision-Makers

Plan

Perform

Measure

Compare

School-Wide Positive Behavior Support

Context

InputFidelity

Impact

WhatHow

Who Where When Why


NC PBS EvaluationCollection and Reporting of Information for Decision-Makers

The North Carolina Positive Behavior Support Initiative is part of the North Carolina State Improvement Program funded through IDEA.

VISION[IMPACT]

All schools in North Carolina will implement Positive Behavior Support as an effective and proactive process for improving social competence

and academic achievement for all students.

MISSION[CONTEXT, INPUT, FIDELITY]

Provide leadership, professional development, resources, and on-going assistance in order for schools to successfully implement Positive

Behavior Support.


Context

Districts and schools participating in the North Carolina Positive Behavior Support Initiative regularly document the

goals, objectives, and activities of their programs to establish the extent to which systems and practices that

should be available are available.


Context

Who Where When Why

STATEWIDE LEADERSHIP AND COORDINATION

One full-time Consultant and eight part-time Regional Coordinators support implementation of PBS in NC. The

Consultant is in the Behavior Support and Special Programs Section of the Exceptional Children Division at the NC

Department of Public Instruction. The Regional Coordinators are hosted by LEAs or another agency in their region and

spend 1/3 of their time working with the PBS implementation in that school system or Charter School. The primary role of

these professionals is to coordinate training, support trainers/coaches/coordinators in LEAs, and facilitate the

evaluation of the statewide initiative.http://www.ncpublicschools.org/positivebehavior/coordinator/

http://www.ncpublicschools.org/positivebehavior/coordinator/


Context

Who Where When Why

Eighty-four of the 100 counties in the state have at least

one school participating in the

North Carolina Positive Behavior Support Initiative.


Context

Who Where When Why

Currently, 104 (90%) of the 115

counties and local education agencies in the state have at

least one school participating in the

North Carolina Positive Behavior Support Initiative.


Context

Who Where When Why

Steady growth has been evident in the number of

schools that have

implemented PBS and current

estimates suggest that

about 85% are still

implementing.

Type ECH/Preschool Elem Mid/Jr. High K (8-12) Alt./Ctr. NC Totaln 3 488 175 65 12 34 777

Percent of NC Total 0.39% 62.81% 22.52% 8.37% 1.54% 4.38% 100.00%


Context

Who Where When Why


Input

The North Carolina Positive Behavior Support Initiative includes critical features of SWPBS.

Districts and schools participating in the North Carolina Positive Behavior Support Initiative regularly assess the extent to which key features are being implemented and

they use this information to develop and refine action plans for sustaining the effort.


The North Carolina Positive Behavior Support Initiative includes critical features of SWPBS.

Leadership Team Representative stakeholders meets regularly to support effort.

Coordination State and local coordinators manage day-to-day operations.

Funding Implementation supported with targeted state-approved funds.

Visibility Ongoing local, state, and national dissemination efforts.

Political Support Leadership team reports annually on activities and outcomes.

Training Capacity Team of trainers is available to build and sustain practices.

Coaching CapacityIndividuals are available to support emerging practices.

Demonstrations Large number of schools available for model demonstrations.

Evaluation Process in place for collecting and reporting information on

Implementation of action plan.

Use of SWPBS.

Impact of SWPBS on student outcomes.Input

What

http://www.dpi.state.nc.us/docs/positivebehavior/implementation/brochure/brochure.pdf


The North Carolina Positive Behavior Support Initiative focuses on four key elements and schools participating in the program intervene at three distinct, but related levels.

Universal Interventions School-wide Rules & Procedures Systematic Reinforcement Social Skills Instruction Culturally Responsive Practices Data-Based Decision-making Parent & Community Partnerships

Targeted Group Interventions Social Skills Instruction Reinforcement of Specific Skills Group Behavior Strategies Classroom Coaching

Intensive, Individual Interventions Functional Behavior Assessment Behavior Intervention Planning

Input

What

http://www.dpi.state.nc.us/docs/positivebehavior/implementation/brochure/brochure.pdf

PRACTICES

SupportingStaff Behavior

SupportingDecisionMaking

SupportingStudent Behavior

PositiveBehaviorSupport OUTCOMES

Social Competence &Academic Achievement


Schools implementing PBS regularly assess the extent to which key features of PBS are being implemented and they use this information to develop action plans for refining and sustaining the effort.

Input

What


Fidelity

Schools participating in the North Carolina Positive Behavior Support Initiative regularly document the

implementation levels of the support being provided and they use this information to refresh, redirect,

and sustain their efforts.

“Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data

can determine the effectiveness of a program, process data determine whether a program exists in the first place” (Gilliam, Ripple, Zigler, &Leiter, 2000, p. 56).

Gilliam, W. S., Ripple, C. H., Zigler, E. F., & Leiter, V. (2000). Evaluating child and family demonstration initiatives: Lessons from the Comprehensive Child Development Program. Early Childhood Research Quarterly,15(1), 41-59.


Average [08-09] scores on SET

subscales varied across levels of schools (N=336)

and were generally higher for elementary (n=226) than

middle (n=68), high (n=24), or

combined (n=18) enrollment schools.

Fidelity

How


Fidelity

How

More schools reported SET scores

and they were generally higher for 2008-2009 than for

previous school years.


Impact

The North Carolina Positive Behavior Support Initiative is focused on improving social competence and

academic achievement for all students.

Information from impact evaluation indicators reflects the extent to which targeted outcomes are being and/or

likely to be achieved.


Office discipline referral data

(majors) from schools

implementing PBS in North

Carolina [07-08] compare

favorably with national

averages.

Impact



Impact


There has been a consistent decrease in

reported suspensions

across schools implementing PBS over the

past five years.


Impact


Levels of behavior risk in schools

implementing PBS were comparable to

widely-accepted expectations and

better than those in comparison schools not systematically

implementing PBS.


Impact


Rates of office discipline referrals

in schools that met SET

expectations were lower than in

schools that did not meet them.


Impact


While achievement differences were not statistically

significant across schools with

different levels of implementation,

effect sizes reflecting practical differences were

large (.55 for reading and .98 for

mathematics).


Impact


[A]chievement causes

[B]ehavior?

[B]ehavior causes

[A]chievement?

[C]ontext causes

[A]chievement and [B]ehavior?


Sustainability

Continuing to Build Capacityo stateo districto school

Expanding Role of Local Coaches Expanding Efforts to Make Evaluation Easier

o report and presentation templateso external support

o www. pbsurveys.orgo www. pbseval.orgo www. pbsassessment.org

Web site


Annual Performance Reports Available

http://www.dpi.state.nc.us/positivebehavior/data/evaluation/



http://www.dpi.state.nc.us/positivebehavior/data/evaluation/

Implementing an Evaluation System for SWPBS

Documents

Transcript of Implementing an Evaluation System for SWPBS