Finn-Aage Esbensen, Ph.D.University of Missouri-St. Louis

Law Enforcement Planners MeetingSt. Louis, MO

September 2009This research is funded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice, Award # 2003-JN-FX-0003 (October 2003 – June 2009) and Award # 2006-JV-FX-0011 (July 1, 2006 – December 2012). The opinions, findings, and conclusions or recommendations expressed in this presentation are those of the author and do not necessarily reflect the views of the Department of Justice.

OverviewComprehensive process evaluations are essential

components of outcome evaluations.

How and why to assess fidelity of program implementation. 

Results from two NIJ funded evaluations will be described.

Both programs are law enforcement, in-school prevention programs.

Program Implementation Program Fidelity

(1) Dosage- is enough of the program actually implemented to expect the anticipated effect.

(2) Program Adherence- to what extent are program components actually taught or delivered.

(3) Quality of Delivery- if delivered, what is the quality of the delivery?

Why Document Program Fidelity?It Compliments Outcome Analyses:

One can confidently tie outcome effects to the program/policy under study.

- Or –When no effect is found

(1) the program, as designed and implemented, has no effect Often Assumed

(2) the program, as designed, is not well suited for the given setting

(3) implementation failure

Research on Program Fidelity

Research to date: “about half of school-based prevention activities are of such poor quality that they cannot reasonably be expected to make a difference…”

(Gottfredson et al., 2000: 7).

Low levels of program effect …are more likely attributable to improper implementation practices than inept program content (DuBois et al. 2002…)

Common Hurdles to Program Fidelity

The idea that “something is better than nothing.”Reality: Half-hearted implementation of programs

designed for high-risk youth fair no better than no implementation at all (Kovaleski et al., 1999).

Insufficient “buy in” on the part of providers.Effect: Program providers deviate from the

program, and administer the lesson(s) differently than intended (Ruiz-Primo, 2005).

Case StudiesTeens, Crime, and the Community and

Community Works (CW)National Crime Prevention CouncilVictimization reduction program

G.R.E.A.T. – Gang Resistance Education and TrainingBureau of Justice Assistance Gang prevention program

Common Program CharacteristicsUtilize law enforcement officers (or probation

officers)School-based and general prevention modelIncorporate interactive and active learning

strategiesEmphasize skills buildingRequire implementers to be trained

CW is a two-day sessionG.R.E.A.T. is one week for experienced officers or

two weeks for officers who have not previously taught

Program componentsSkill-based curriculum consisting of:

A 31-Lesson CurriculumCore program consists of 11 lessons that, if

taught with fidelity, requires approximately 30 hours of class time

The use of community resource people (CRP) to deliver the program

A group service learning project, referred to as Action Projects (AP)

Taught by teacher &/or SRO/JPO

CW Program GoalsIncrease teen awareness of their vulnerability to

crimeEducate teens on how to protect themselves and

their communitiesMotivate teens to take action in their communitiesHelp teens understand the costs of crime and what

can be done to prevent itFoster community relations by bringing community

resource people into the sessionsBond youth to the community through increased

self-esteemIncrease teen empathy toward crime victims

G.R.E.A.T. Program

Taught by uniformed law enforcement officerSkills building program13 lessons that take 13-14 hours to complete

Goal-setting; Decision-making; Communication skills; Conflict resolution

Student workbookG.R.E.A.T. project

G.R.E.A.T. Program Goals

Prevent youth from becoming involved in gangs, violence, and delinquency

Develop positive relationships between youth and law enforcement

Evaluation DesignsSimilar design for evaluation of both projectsPurposive selection of cities (G.R.E.A.T.) or

schools (CW)Observation of trainings

Observation of program delivery

Surveys of teachers, officers, &/or implementers

Outcome component (longitudinal survey of students)

CW Site Selection15 middle school programs in 9 cities/towns in 4

states

98 classes – 49 CW & 49 Comparison

2 – 10 classes per school (half CW, half comparison)

1,686 students34 to 229 students per school6th – 9th grade

G.R.E.A.T. Site Selection31 schools –

4 to 6 in each of 7 cities (representative of districts)195 classes –

102 G.R.E.A.T. & 93 Control3 - 12 classes in each school (half receive

G.R.E.A.T.) 24 - 35 classes per city

3,820 StudentsBetween 67 and 186 students per school6th or 7th (5 schools) grade

Training: Research QuestionsAre trainees adequately prepared to teach

the program after completion of training?Are trainees provided the opportunity to

practice program delivery?Are teaching strategies modeled during

training?Are trainers professional and knowledgeable?Do trainers adhere to suggested time frames?Is the time allotted to training adequate?

Community Works TrainingObserved 7 training sessions (6 two-day, 1

one-day refresher training).

Typical Training Scenario3 trainers (1 primary, 2 assistants)25-30 traineesTrainees represented a mixture of professions

(SROs, teachers, juvenile justice workers, …)

Community Works TrainingTrainers

Appeared comfortable when speaking from a script.

When asked questions, trainers sometimes provided vague or inaccurate information e.g., misdemeanor vs. felony

Emphasized the program was “adaptable and flexible” Is it a program or a resource?

Community Works TrainingTeaching Strategies

Training staff made use of many group activities and teaching strategies. e.g., icebreakers were modeled frequently

Activities were seldom linked to program content, even though a clear link exists.

Community Works TrainingProgram Content

Training provided attendees with an overview of the program, as well as key components of the program

More time is needed to allow for more modeling of program delivery.

More time needs to be devoted to explaining the program as a whole, not just individual lessons (e.g., skills building approach).

Little time was devoted to explaining the substantive content of the lessons. (i.e., not all in attendance had a law background)

Community Works TrainingProgram Content (Continued)

Trainers did not seem to alter trainings according to their audience. (e.g., English teachers may need more attention paid

to substantive knowledge, while SROs may need more time devoted to teaching strategies.)

Trainers did not provide remedies to common problems faced when implementing the program (e.g., substitutes, short classes).

Community Works TrainingTime Management

Posted timelines for trainings were seldom followed (e.g., long lunch, leave early).

Demonstration of a 45 minute lesson lasted 90 minutes. Coincidently, this lesson was never implemented

correctly in the classroom, per our observations.When trainees voiced concern over the time

needed to implement the program, trainers stated that one can either “use” the curriculum or “implement” the program; both could be used effectively (Flexibility).

Community Works TrainingBased on observation of training, what are your

expectations for program implementation fidelity?

G.R.E.A.T. TrainingObserved 7 G.R.E.A.T. Officer Trainings (GOTs)

(5 one week sessions and 2 two week session)

Typical Training Scenario (8+ trainers)Seminar Supervisor & logistical support from

Regional Administrator and staff6 trainers (Team Leaders)Gang Specialist and Education Specialist30 - 36 traineesTrainees primarily local law enforcement officers

G.R.E.A.T. TrainingTraining consists of multiple teaching blocks,

including:Transition from law enforcement to preventionGangsAdolescent development and instructional

issuesIssues of G.R.E.A.T. concern

Team leader models each lesson (although a few lessons are taught as an overview)

G.R.E.A.T. Training Team leaders typically adhere to training

rules and structure (e.g., time frames)Trainers stress importance of program

adherence, including historical development and program revision, skills-based program in which each lesson builds on prior lessons

Trainers model lessons so that trainees have experience of observing different teaching styles and different personal approaches to teaching standard curriculum

G.R.E.A.T. TrainingTeam leaders are “debriefed” by Supervisor

after each modeling sessionGroup discussion of lesson content and/or

delivery issuesTrainers utilize multiple teaching strategiesTeam leaders meet with team of 5 – 6 officers

for more in-depth discussion of program

G.R.E.A.T. TrainingTrainees (officers) are required to develop

one lesson overview and present to groupTrainees are subsequently required to model

their lesson to their teamEach trainee has experience with one lesson

and observes five others modeled by their team members

Trainees have considerable exposure to program content in this manner

G.R.E.A.T. Training ConcernsTrainings sometimes adjust schedule to

accommodate gang or education specialists – suggest consistency to enhance “flow”

Consider more emphasis on classroom management issues

Gang block has tendency to regress to discussion of stereotypical media images of gangs

Given observations of training, what are expectations for program implementation fidelity?

Observation of Program Delivery

Observed more than 100 CW classrooms

Observed more than 500 G.R.E.A.T. classrooms

SESSION 1 CHECKLISTCreating a Community Vision

Part 1 (55 minute estimate)

Instructions: Please place a check mark after each area that is covered in the session. Leave the line blank when certain areas are not covered. Also, please note the approximate time spent on each part and steps within each part. After the checklist is completed, please complete the subsequent qualitative session evaluation.

Start Time: ________

Step A: Warm up (15 minute estimate)

1. Introduction and welcome _______a. teens introduce themselves and state their favorite activity ______b. name tags distributed and used _______

2. Icebreaker: Shapes (kids with same shapes get together and form shape) ________3. Overview of program and today's session _________

a. teens put together puzzle messages (small group activity) _______

Actual time spent on Step A: ______

Step B: Creating a Community Vision (30 minute estimate) 1. Brainstorm what does it mean to be safe? _________2. Defining community ________a. help teens to come up with a definition similar to this: "a group of people that have something important in common" ______

b. teens consider what makes a school a community ______c. ask teens what things we have in common to make school a community _____

3. Think about what makes a community safe and secure _______

CW Observation Results

Overall, out of the 110 class periods observed, only 18 (16%) lessons were found to be delivered in the manner intended by program developers.

Common Deviations Skipping Steps in the LessonPoor Time Management Covering Multiple Parts in One Class Period

CW Observation ResultsThe program, as designed, is not well suited for

implementation in the given setting.

One school classified as high fidelity.

Three schools classified as medium fidelity. Could conceivably link program outcome to program

delivery

Eight schools classified as low fidelity. If program effect found, it could not be attributed to

program delivery.

G.R.E.A.T. Classroom ObservationsPurpose: determine extent to which program

is delivered as intended with regard to: Dosage Adherence to intended lessons Quality of instruction

518 observations of G.R.E.A.T. program delivery

33 different officers

Each lesson observed at least 30 times

G.R.E.A.T. ObservationsTime frame adherence

Did not appear to be a problem if there were no behavioral problems

When problems occurred, generally due to external factors (shortened day, substitute teacher, fire drill)

Topical areaOverall, officers covered all lessons

G.R.E.A.T. ObservationsLesson Adherence

Overall close adherenceSome officers skipped around, teaching

components out of orderOverall quality

Generally good qualityExternal factors most notable in terms of

affecting lesson quality (time of day, substitute teacher, some particularly “bad” classrooms)

G.R.E.A.T. Overall Program Fidelity27 officers (80 classes) implemented with

good fidelity (adhered to lesson components, topical areas, time frame, and “quality” of instruction)

3 officers (11 classes) had low fidelity (implemented much of lesson but with low quality)

3 officers (11 classes) failed to implement (did not cover lesson components and poor quality)

Implementer SurveysA complimentary technique to address program

fidelity is through implementer surveys.

Allows us to document lessons taught, CRP used, and how Action Projects were implemented.

Note: Research shows that this technique tends to overestimate degree of program fidelity.

CW Implementer Surveys

Impressions of the CurriculumOverall positive assessment.82% agreed that the program raised awareness of

programs that assist victims of crime.Most agreed that the lessons were relevant.59% felt the materials were appealing to students.

This may signal that materials should be modified to make them more appealing.

CW Implementer Surveys Program Fidelity

Programs were categorized as implementers and non-implementers based on the following standards:1) At least 70% of the “core 11” lessons taught2) At least 20 hours of program delivery time3) CRP had to be used4) AP had to be at least planned and initiated

Based upon the implementer surveys, 6 of the 14 schools could have been classified as having implemented the program.

G.R.E.A.T. Officer SurveySurvey of G.R.E.A.T. officers in 7 cities

(anonymous questionnaires)

137/205 = 67% response rate

G.R.E.A.T. Officer AttitudesProgram Goals (% agreeing or strongly agreeing)

G.R.E.A.T. teaches skills to avoid gangs and violence (80%)

G.R.E.A.T. has improved police/youth relations (90%)

G.R.E.A.T. strengthened police/community relations (90%)

G.R.E.A.T. has reduced community gang problem (29%)

G.R.E.A.T. has reduced community crime problem (29%)

G.R.E.A.T. Officer AttitudesProgram

G.R.E.A.T. is appropriate for age and comprehension levels (83%)

G.R.E.A.T. lessons address risk factors for gangs and delinquency (71%)

Enough time to cover important topics (72%)

G.R.E.A.T. School Personnel SurveyPurpose: obtain opinions about school-

based prevention programs in general and G.R.E.A.T. in particular.

All G.R.E.A.T. grade-level teachers and administrators at 31 participating schools (n=231/373, 62%).

G.R.E.A.T. School PersonnelProgram Assessment (% who agreed or strongly

agreed)

88% in favor of having G.R.E.A.T. in their school

90% - curriculum age appropriate

82% - gives students skills to avoid gangs and violence

86% - improved students attitudes about police

52% - reduced gang participation at school

46% - reduced gang participation in community

G.R.E.A.T. School PersonnelG.R.E.A.T. Classroom Teachers

80% - officer adequately trained in content

70% - officer adequately trained in classroom management

81% - students responsive to officer

78% - officer punctual (or rescheduled if needed

48% - officer incorporated teacher into lesson

14% - officer strayed from lesson plan

16% - officer had difficulty controlling class

G.R.E.A.T. Bottom LineGenerally favorable opinions expressed by

officers and school personnel

However, it is noteworthy that both officers and teachers more positive about mediating effects than on program objectives

Summary CommentsEvidence-Based Prevention Programming has

become a reality.Funding, Adoption in Schools/Communities

Attention to program fidelity is necessary.Tale of Two Programs

CW- Implementation Failure and No Program Effect

Is this a bad program, or one that was not implemented correctly?

With G.R.E.A.T., we have program fidelity and preliminary analyses indicate positive program effect.

Thank you for listening.