2

Factors Contributing to Successful Implementation of EBPs

The success of the intervention strategies themselves and

The success of the implementation

processes

Too often, evaluations focus more on changes in client outcomes without due consideration of fidelity to the intervention model or the effectiveness of the implementation process itself.

3

Implementation Defined

Implementation is a specified set of activities designed to put into practice an activity or program of known dimensions.

The National Implementation Research Network at the University of Southern Florida conducted an exhaustive review of research on implementation and is the primary source for this presentation.

4

A Conceptual Framework for Understanding Implementation of Evidence-Based Practices

The essential implementation outcomes are:

Changes in adult professional behavior

Changes in organizational structures and cultures

Changes in relationships to consumers, stakeholders, and system partners

5

Implementation Framework

6

Stages of Implementation

Exploration & Adoption

Program Installation

Initial Implementation

Full Operation

Innovation

Sustainability

7

Stages of Implementation

Exploration & Adoption

Identify the need for an intervention considering existing conditions

Assess the fit between the intervention and program and community needs

Prepare organization, staff, and resources for mobilizing information and support.This was essentially accomplished in the work of the Work Groups.

8

Program Installation

Program installation begins after an adoption has occurred and involves:

Ensuring availability of funding

Human resource strategies (recruitment, hiring, training, etc.)

Policy development to support change (e.g. referral mechanisms, reporting frameworks and outcome expectations)

Facility requirements

9

Initial Implementation

Initial implementation begins once conditions identified in the installation phase have been sufficiently completed.

Initial implementation must NOT be confused with full operation, as during initial implementation, many factors contributing to fidelity to the model may not be fully or even partially in place.

10

Barriers to Full Operation

Organizational Barriers

Personnel rules

Social stressors

Union stewards

Anxious administrators

Political pressures

Interpersonal rivalries

Staff turnover

11

Barriers to Full Operation

Human Barriers

Fear of change

Inertia

Investment in the status quo

Inherently difficult work of doing something differently

12

Full Operation

Full Operation ONLY occurs when:

New learning becomes integrated into practitioner, organizational and community practices, policies, and procedures.

Full case loads

Full staffing

Fidelity to the model being implemented

Only once fidelity measures are above criterion levels most of the time, can the effectiveness of an evidence-based practice be evaluated. This is a very important point.

13

Dangers of Premature Evaluation

Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data can determine the effectiveness of a program, process data determine if a program exists at all.

Gilliam, Ripple, Zigler & Leiter (2000).

“

”

14

Innovation or Drift

When evidence-based practice meets new local conditions. Two results can occur:

Innovation: When desirable changes in the standard model are identified.

Drift: Undesirable changes in the standard model.

It is critical to implement the model with fidelity before considering innovation and to make changes only after due consideration of client benefit.

15

Innovation or Drift Illustrated

16

Sustainability

Throughout the two to four-year implementation process, the forces that led to and supported reform will change:

Staff and funding changes New social problems emerge Partnerships and political alliances change Champions and advocates move on to other

causes

Throughout the implementation process, it is essential that leadership maintain focus on sustaining the core components of reform.

17

Factors Contributing to Effective Implementation of EBP

Objective decision-making strategies that involved staff, good information about the reform and organizational leadership support during the exploration stage;

Evidence of a ‘learning culture’ within the organization implementing the reform;

A system in place for monitoring implementation;

Access to technical assistance throughout implementation;

The perceived ability of the organization to manage risks;

Belief in the validity of the reform; and most of all….

Adherence to the core components of BOTH the EBP and effective implementation (discussion follows).

18

Core Intervention Components

An essential first step to implementing EBP or system reform is to identify what are the core components of the intervention itself. Leadership must:

Carefully research alternatives Consider well-evaluated experiential learning from a

number of replications and Achieve a clear understanding of what of the model must

be maintained to achieve fidelity and effectiveness at the consumer level.

In other words, before considering the core components of any intervention, leadership must be clear about what is to be implemented.

19

Core Components for Implementation

The following Implementation Drivers influence staff behavior and organizational culture:

Practitioner Selection Preservice and inservice training A consultant coach Staff & program evaluation Facilitative administration and System Interventions

This is illustrated in the graphic that follows.

20

Implementation Drivers

21

Implementation Drivers

These drivers are integrated in their influence, in that strengths in one area can compensate for weaknesses in other areas.

The relative effectiveness of the implementation drivers is as important as the demonstrated effectiveness of the EBP itself.

22

Practitioner Selection

How and who are selected to carry out the EBP

Who is qualified to carry out the practices and programs?

What are the methods for recruiting & selecting practitioners?

How important are individuals skills, experience and personal characteristics to effective implementation of EBP?

23

Research on Practitioner Selection

There has been relatively little research to isolate the factors involved in selection of practitioners for EBP. Factors identified in effective practitioner selection for a national MST program included:

Responses to behavioral vignettes; Responses to role playing situations related to the

therapeutic; environment for which they were being considered; and

Responses to mini-training that requires behavior change.

Responses are rated with a rubric along several dimensions including: collaborative and strength focused, efforts to overcome barriers, ability to use behavioral language, uses of logical thinking and openness to feedback.

24

Research on Practitioner Selection

Morris & Stuart, 2002 are attempting to identify the generic skills needed by frontline practitioners in a transformed behavioral health field. Among those qualities considered:

Assessment skills; Family and support system involvement; Social and cultural engagement skills; Treatment skills; Methods to optimize recovery and empowerment; Consumer relationship skills; and Community resource and coordination skills.

It is interesting to note here and throughout this research, cultural competence was never discussed as an issue or factor.

25

Research on Staff Selection

A Housing & Urban Development study by Wanberg & Banas (2000) examined practitioner characteristics in the context of organizational change and found that personal resilience and self-efficacy were associated with greater acceptance of change in the work place.

This study would seem particularly important in SMCMH’s system transformation.

26

Research on Staff Selection

From the research we might conclude that as part of the implementation planning process, SMCMH leadership should:

Construct a rubric of ideal staff characteristics particularly as relates to operating in a transformed organization

Design an interview process that includes vignettes and role plays that require candidates

to demonstrate those characteristics

27

Preservice and Inservice Training

Training is an efficient way to provide:

Background knowledge, background information, theory, philosophy, and values;

Demonstration of new skills (through video or role play);

Opportunities to practice new skills through role plays and behavioral rehearsals; and

Feedback in a safe training environment.

28

Preservice and inservice training

Role playing & behavior rehearsals are critical for practicing new skills in training.

Role playing asks you to pretend you are someone else and try this… which builds empathy

Behavioral rehearsals asks you to be in your practitioner role and you are asked to confront a specific situation and perform your practitioner role in reaction to that situation ….which serves as direct preparation for the real thing.

29

Recommendations for Training

Model or demonstrate new skills using role play, behavior rehearsals and video tape.

Emphasize practice and use feedback on practice to teach the finer points of mapping.

Use practice sessions to help trainees integrate thinking and doing.

Provide guidance with respect to the boundaries of using the technique, describing when it may be useful and when it may not be useful.

Provide guidance on the flexible use of the core components.

Encourage peer and administrative support.

30

Limitations of Training

Numerous evaluations on the impact of training in health and human services have found that training alone rarely impacts practice.

The “train-and-hope” approach (Stokes & Baer, 1977) to implementation does not appear to work.

Kelly et al (2000) in a study of HIV service organizations reported the largest increase in adoptions of HIV service guidelines occurred when consultation was added to training.

A meta-analysis (Davis, 1995) found similar results in medicine. Davis concluded that “formal CME conferences and activities without enabling or practice reinforcing strategies, had little impact.” (page 700)

31

Limitations of Training: Spray & Pray

While training may introduce knowledge, philosophy, and new approaches, there is no research study that has demonstrated significant changes in practice resulting from an intervention that provided only training.

The limitations of training are directly related to the way in which adult learning occurs.

32

Stages of Adult Learning

Generally adult learning progresses in stages:

Orientation & new learning

Mechanical use

Routine use

Refinement

Integration

Innovation

33

Challenges of Learning New Behaviors

Based upon decades of research, Joyce & Showers (2002) concluded that

The newly-learned behavior is crude (i.e. mechanical) compared to performance by a master practitioner.

Newly-learned behavior is fragile and needs to be supported in the face of reactions from consumers and others in the service setting.

Newly-learned behavior is incomplete and will need to be shaped to be most functional in a service setting.

34

The Importance of Coaching

While most skills can be introduced in training, newly-learned behaviors are only really learned on the job with help of a consultant or coach precisely because:

The challenges of adopting new behaviors can’t be replicated in a training.

Single-point-in-time training needs to be reinforced continuously in the work setting for behavior changes to be sustained.

35

Consultant or Coach Support

Implementation of EBP requires changes in behavior at the practitioner, supervisory and administrative levels.

Training and coaching are the primary strategies in which behavior change is brought about.

In addition to training, substantial hands-on coaching and practice may be necessary before a counselor feels comfortable with a new strategy.

~ Dansereau & Dess (2002)

“ ”

36

Components of Effective Coaching

Coaching must be work-based, opportunistic, readily available, and reflective (Spouse 2001). Spouse described four roles for the coach:

Supervision

Teaching while engaged in practice activities

Assessment and feedback

Provision of emotional support.

37

Factors that Impact Coaching

Amount of time devoted to coaching (MST coaching occurs once or twice a week for 90 minutes) (Schoenwald et al., 2000)

Includes direct observation of provision of direct services (Smart et al., 1979)

Utilizes coaches who are expert in the content, techniques, and rationales of the program (Denton, Vaughn & Fletcher 2003)

Coaching relationships established during the training experience (Smart et al., 1979)

38

Other Factors Contributing to Effective Coaching

Walker, Koroloff & Schutte (2002) identified four additional factors that accounted for 62% of the variance in the perceived impact of supervision and coaching on practice. The Supervisor (coach):

Taught new skills;

Strengthened confidence;

Offered safety in sessions;

Devoted time to discipline-specific skills

39

Personal Qualities of a Coach

Encouraging & enthusiastic

Supportive

Committed

Sensitive

Flexible

Respectful & diplomatic

Willing to share information, credit and recognition.

40

Barriers to Effective Coaching

Inadequate time allotted to coaching & scheduling conflicts Role confusion due to the dual role of supervisor and coach Feelings of inadequacy on the part of coaches (hence the

importance of selecting skilled coaches) Poor match between coach and practitioner Labor relationships that don’t support observation and

feedback Resistance in the organizational culture Absence of strong leadership & commitment to

implementation Focus on paperwork compliance as opposed to changes in

practitioner behaviors Cost

41

Research on the Impact of Coaching

The study below is based on an meta-analysis of hundreds of studies in education. Note the dramatic differences in impact upon practice.

42

Staff Evaluation

Evaluation provides critical feedback to practitioners, trainers and managers related to:

Fidelity to the model being implemented

Effectiveness of training and coaching strategies

Impact of intervention upon clients

Progress of implementation itself

43

Components of Evaluation

Staff & program evaluation and fidelity seem to consist of some combination of measures of:

Context—prerequisites that must be in place to operate (staffing, qualifications, ratios)

Compliance—extent to which the practitioner uses the core intervention components as prescribed by the EBP and avoids practices proscribed by the EBP

Competence– the level of skill shown by the practitioner in using the core intervention components

44

Effective Staff Evaluation Systems

Huber et al., (2003) described a highly effective hospital management system for staff recruitment and evaluation that included:

Ongoing training and education focusing on specific skills; Cross-training on related roles, and in-services Monthly dinners for discussion; Performance evaluations based on direct observation to assess

practice knowledge, communication skills, and use of time; Prompt verbal feedback followed by a write up with

recommendations; and Quality improvement systems to keep the system on track.

45

Effective Staff Evaluation

In other words, in a highly effective system, staff evaluation is part of a sequence of supports designed to have good people well prepared to do an effective job.

46

Staff Evaluation & Fidelity to a Model

One of the most critical purposes of staff evaluation in the context of implementing EBP is to test practitioner fidelity to the EBP core components.

47

Challenges to Measuring Fidelity

McGrew et al., (1994) noted that development of fidelity measures is hampered by three factors:

Most treatment models are not well defined conceptually making it difficult to identify core intervention components;

When intervention components are identified, they are not operationally defined with agreed-upon criteria; and

Only a few models have been around long enough to study planned and unplanned variations.

48

Staff Evaluation & Fidelity to a Model

Fortunately the SMCMH transformation relies upon implementation of EBPs with effective fidelity instruments already in place.

Multisystemic Treatment (MST) utilizes the Therapist Adherence Measure, a 27-item measure used in phone interviews with parents.

The Wraparound Fidelity Index (WFI) consists of asking facilitators, parents and youth to rate 11 dimensions of services.

Assertive Community Treatment (ACT) utilizes a 73-item tool with 17-item subset used to construct a fidelity index with three subscales: staffing, organization, and service.

49

Fidelity to System Transformation

A major challenge to SMCMH is to go beyond program level (MST, Wraparound, ACT) fidelity and to construct a fidelity index that reflects the qualities of a transformed system as defined and described in our plan.

50

Theory of Change & Fidelity Models

Possible resources to explore for developing a fidelity model for the transformation…

Hodges, Hernandez, Nesman & Lipien (2002) demonstrated how a theory of change exercise can help programs clarify their strategies to develop fidelity measures.

Shern, Trochim & LaComb (1995) used concept mapping to develop fidelity measures for a mental health program.

51

Program Evaluation

Program Evaluation assesses key elements of the overall performance of the organization in facilitating the implementation of the program or practice.

52

Facilitative Administration

Facilitative administration

Provides leadership;

Makes use of data inputs to inform decision making;

Supports overall processes; and

Keeps staff organized and focused on the desired clinical outcomes.

53

Systems Interventions

Systems interventions are strategies to work with external systems to ensure the availability of the financial, organizational, and human resources required to support the work of the practitioners.

54

Relationship between Core Components & External Factors

55

Relationship between Core Components & External Factors

56

Key Elements to Organizational Change

Implementation of EBP almost always requires organizational change. Research points to 8 key elements to organizational change:

Commitment of Leadership to the Implementation Process (Hunter, Hunter & Rogers, 1993). Research indicates that leadership takes many forms, including to:

Initiate and shepherd the organization through the complex change process

Set explicit goals, communicate them clearly throughout the organization, resolve conflicts with other goals, and reinforce persistence

Help create the details of activities, processes, and tasks in order to operationalize implementation policies

Inspire, guide, and provide direction Recruit, select, train, locate, advance, promote, or dismiss

employees to further the aims of implementation policies

57

Other Key Elements to Organizational Change

Involvement of stakeholders in planning and selection of programs to implement; (Bierman et al., 2002)

Creation of an implementation task force made of consumers, stakeholders (including unions), etc. (Joyce & Showers, 2002)

Suggestions for “unfreezing” current organizational practices (Cheung & Cheng, 1997)

Resources for extra costs, effort, equipment, manuals, materials recruiting, access to expertise, re-training for new organizational roles; (Phillips, et al., 1978)

58

Other Key Elements to Organizational Change

Alignment of organizational structures to integrate staff selection, training, performance evaluation, and on-going training (Blasé et al., 1984)

Alignment of organizational structures to achieve horizontal and vertical integration (Unger et al., 2000)

Commitment of on-going resources and support for providing time and scheduling for coaching, participatory planning, exercise of leadership, evolution of teamwork and for generating and using local data (Park & Han, 2002)

59

Considerations for Effective Implementation

At a system level and with staff and consumer input, develop a rubric that depicts what a ‘transformed system’ would look like at the system and program levels.

60

Considerations for Effective Implementation Construct site-based highly structured

exercises for sites working with their consumers to identify how focusing on this rubric would require changes in daily operations.

How would you change staff meetings, staff evaluation, hours of operation, charts, reports, referrals, office operations, schedules, the waiting room?

61

Considerations for Effective Implementation Develop program level rubric that

describe the extent to which a program is achieving fidelity to operating in conformance with the new transformed system.

Build in personnel and program evaluation components that incorporate how staff performance and program structures are supporting the transformation.

62

Considerations for Effective Implementation

At a structural level, consider a wide variety strategies that reinforce fidelity to transformation and the various program EBPs.

Examine what specific practitioner and program changes are important and build a system of reinforcements that extend the impact of training.

63

Considerations for Effective Implementation Be mindful that changing behaviors is

challenging and threatening. Training alone will not generate change in

practitioner behavior or program priorities. Since implementation drivers are

integrative and compensatory, everything you do administratively to facilitate and support changes, will reinforce the other drivers and compensate for where drivers are not as strong.

64

Considerations for Effective Implementation In so many ways, implementation is

about the small things you do: Small changes in intake forms, assessment tools, client

engagement protocols, and client charts, changes that subtly cause practitioners to operate slightly differently

Notes at the top of agenda and specific standing agenda items that focus on transformation issues and allow staff dialogue and input into implementation

E-reminders to managers to observe practitioner groups Changes in personnel review tools to reinforce transformation Use of funds to foster individual staff research and

presentations to teams about issues related to implementing transformation strategies.

65

Considerations for Effective Implementation

In closing, Effective implementation of transformation is about:

Selection of the most effective, high-leverage programs;

Providing training and work-based support to reinforce the values and practices inherent in a transformed system; and

Building structures, procedures and policies that reinforce the priorities of the transformation…

And the devil is in the details. Implementation is All.