EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH: THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS...
-
Upload
ellen-clayton -
Category
Documents
-
view
215 -
download
2
Transcript of EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH: THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS...
EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH: THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS
Maria Scannapieco, Ph.D., Professor & Director
Center for Child Welfare
Mountains & Plains Child Welfare Implementation Center
25 April 2011
Overview
Introduction to Implementation Centers
Basic understanding of Implementation Science
Reflections on lessons learned in systems change and implications for states
Introduction to the Implementation Centers
Implementation Centers and National Resource Centers4
Improve child welfare administration and practice Meet Federal performance standards and
implement effective programs Pursue sustainable and positive systems change Improve outcomes for children, youth & families
Funded by the Children’s Bureau to provide States and Territories, Tribes, and Tribal Consortia individualized training and technical assistance (T/TA) to:
Implementation Centers: Filling the Gap
5
States and Tribes are sometimes without the resources necessary to implement comprehensive strategic plans
National Resource Centers have limited resources to provide intensive, long-term TA
Implementation Centers provide in-depth and long-term consultation and peer networking opportunities to States and Tribes
Project Partner RolesBuilding Capacity to Implement Sustainable Systems
Change
Implementation Center Child Welfare AgencyConduct organizational and readiness assessment to identify potential barriers to implementation
Develop and implement strategies to address organizational barriers and improve readiness for implementation
Guide planning process Develop project plan
Provide and coordinate technical assistance resources to support organizational change and facilitate implementation
Provide and coordinate organizational/ human resources to manage change and support implementation
Provide technical assistance to promote best practices in organizational and child welfare systems change
Utilize best practices to align people, process, structure, measurement/rewards, and technology around the desired systems change
Facilitate implementation Implement plan
Develop methodology (e.g. benchmarks, tools) to monitor implementation progress
Collect project data; use data reports to monitor implementation and inform /refine implementation strategies
Evaluate project outcomes Use findings and lessons learned to inform future implementation efforts
Geographic Assignments
Who are the Implementation Centers? 8
Implementation Center Organization Northeast & Caribbean Implementation Center (NCIC)
University of Southern Maine, Muskie School of Public Service
Atlantic Coast Child Welfare Implementation Center (ACCWIC)
University of Maryland School of Social Work
Midwest Child Welfare Implementation Center (MCWIC)
University of Nebraska-Lincoln, Center on Children, Families & the Law
Mountains & Plains Child Welfare Implementation Center (MPCWIC)
University of Texas at Arlington, partnering with the University of Denver and The Native American Training Institute (NATI)
Western & Pacific Child Welfare Implementation Center (WPIC)
American Institutes for Research, partnering with National Indian Child Welfare Association, Center for the Study of Social Policy, National Technical Assistance Center for Children’s Mental Health at Georgetown University, and the Florida Mental Health Institute at the University of South Florida
Mountains and Plains Child Welfare Implementation Center
MPCWIC
Geographical Service Area
Region 6: Arkansas, Louisiana, New Mexico, Oklahoma, Texas
Region 8: Colorado, Montana, North Dakota, South Dakota, Utah, Wyoming
MPCWIC Projects The Skun-eyah (Garden) Project is a collaborative project to be
implemented by two tribes, Three Affiliated Tribes (Mandan, Hidatsa and Arikara Nations) and Turtle Mountain Band of Chippewa
The Osage Nation implementation project goal is three-fold: develop a business mapping model, develop and articulate a culturally based family centered practice model, and create a decision support data system to help facilitate the first two
Colorado Department of Human Services, Division of Child Welfare– Child Welfare Practice Reform. This three-year project will explore, define and implement a Colorado practice model, inclusive of specific practice strategies, methods and tools to improve outcomes for children and families. Denver, Colorado
Developing a New Mexico Child Welfare Practice Modelwill be focused on the development and implementation of a clearly articulated practice framework, inclusive of vision, mission, values and operating principles, to guide all of the change initiatives underway in the State
Key Elements of Systems Change
Change Management Model
Awareness of the need for change Desire to participate and support
the change Knowledge of how to change Ability to implement the required
skills/behaviors Reinforcement to sustain the
change
Competencies for Leading Change
Creativity and innovation External awareness Flexibility Strategic Thinking Vision
Understanding Implementation Science
Implementation is not a Hobby!
There is a body of research around effective implementation.
Child welfare is slow to apply the science.
Implementation Centers are utilizing methodology to support project work with states and tribes.
What do we know?
Implementation Science National Implementation Research
Network (NIRN) Phases of Change
John Kotter, Harvard Business School Managing Change
ADKAR Model (Awareness, Desire, Knowledge, Ability and Reinforcement)
Implementation Frameworks
Brief overview of the science of implementation
Practice, program and systems change through… Multi-dimensional, fully integrated
use of Implementation Stages Implementation Drivers Implementation Teams
Applying Frameworks
The value of frameworks is To promote the ability to generalize
beyond the immediate project or initiative
To enhance communication among partners (e.g. better understanding of one another)
To more easily share and apply improvements
To increase the relevance of the “lessons learned”
Building the Implementation Platform
Brief overview of the science of implementation
Multi-dimensional, fully integrated Implementation Stages Implementation Drivers Transformation Zones
Science to Service
SCIENCE SERVICEGAPIMPLEMENTATION
What Do We Mean by Implementation?
A specified set of activities designed to put into practice a policy, activity, or program of known dimensions
Implementation processes are purposeful and defined in sufficient detail such that independent observers can detect the presence and strength of these “specified activities”
Overall Challenge Science to Service Gap
What is known often is not what is adopted to help children, families, and caregiversImplementation GapThere are not clear pathways to implementation
What is adopted often is not used with fidelity and good effect
What is implemented often disappears with time and staff turnover
From the Synthesis of the Literature We Know That…
Implementation issues are common across widely diverse domains Human service prevention and
intervention (e.g. child welfare, substance abuse, mental health, violence prevention, education)
Advanced manufacturing technologies
Research-based clinical guidelines Engineering (e.g. bridge
maintenance) Hotel management National franchise operations Cancer prevention and treatment
Positive Intervention Outcomes
≠ ImplementationImplementation has not been achieved by doing more or better research on interventions or on curricula
The usability of a program or practice has nothing to do with the weight of the evidence regarding that program
– “Evidence” on effectiveness helps you select what to implement for whom
– “Evidence” on these outcomes does not help you implement the program or practice
Insufficient Methods
Implementation by laws/ compliance by itself does not work
Implementation by “following the money” by itself does not work
Implementation without changing supporting roles and functions does not work
Fixsen, Naoom, Blase, Friedman, Wallace, 2005
Insufficient Methods
Diffusion/dissemination of information
by itself does not lead to successful
implementation
Training alone, no matter how well
done, does not lead to successful
implementation Fixsen, Naoom, Blase, Friedman, Wallace,
2005
So what does work?
What Works
Effective NOT Effective
Effective
NOT Effective
IMPLEMENTATION – The HOW
INT
ER
VE
NT
ION
Th
e W
HA
T
Performance Implementation (High Fidelity)
Paper ImplementationProcedure Implementation (Low Fidelity)
Getting Here and Staying Here is the Challenge!
Pre-Requisite: Defining “IT”
From what current state to what future state? The “it” must be operationalized
whether it is: An evidence-based practice or program A best practice Initiative A broad systems change initiative
OperationalizePart of Speech: verb Definition: to
define a concept or variable so that it can be measured or expressed quantitativelyWebster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7)Copyright © 2003-2008 Lexico Publishing Group, LLC
Child Welfare Practice Classification System
Well-Supported, Efficacious Practice Evidenced Based Practice (EBP)
Promising and Acceptable Practice Best Practices
Innovative or Novel Practice Concerning Practice
Stages of Implementation
Exploration
Design/Installation
Initial Implementatio
n
Full Implementatio
n
Sustainability
Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network
Stages of ImplementationCore Activities
Exploration
• Conduct organizational assessment
• Garner leadership and stakeholder support
• Review literature
• Solicit multiple perspectives on needs and opportunities
• Establish baseline performance
• Develop intervention model
Design/Installation
• Develop plans:• Implementation
• Communication
• Change management
• Define goals and benchmarks
• Engage stakeholders
• Conduct readiness assessment
• Align organization systems/supports
Initial Implementation
• Build staff competency
• Communicate project status
• Monitor fidelity of implementation
• Monitor implementation climate
• Identify and address barriers to implementation
• Provide performance feedback to staff, management, and stakeholders
Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network
Stages/Activities
Full Implementation•Apply lessons
learned from initial implementation to full operation•Track fidelity through quality assurance and performance evaluation data
•Solicit feedback from multiple stakeholders and consumers•Evaluate impact on child and family outcomes
Innovation
• Adapt or adjust intervention model to reflect lessons learned from stakeholder feedback, tracking, evaluation
• Communicate changes and rationale
• Re-build competency around modified model
Sustainability
• Establish long term funding sources
• Align ongoing quality assurance and performance measurement with model
• Promote visibility of new practice and successful outcomes
Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network
Selection
Training
Coaching
Performance Assessment
(Fidelity)
Systems Intervention
LEADERSHIP
Decision Support Data
System
Integrated &
Compensatory
CO
MPE
TEN
CY
ORG
AN
IZATI
ON
Facilitative Administration
© Fixsen & Blase, 2008
National Implementation Research Network: Implementation Drivers
© Fixsen & Blase, 2008
Performance Assessment
Coaching
Training
Selection
Systems Intervention
Facilitative Administration
Decision Support Data System
Adaptive
Technical
Integrated & CompensatoryC
ompet
ency
Drive
rs
Com
pet
ency
Drive
rs
Org
anization
Drivers
Org
anization
Drivers
LeadershipLeadership
Improved outcomes for children and families
Graphics by Steve Goodman,2009
Implementation Drivers
System Change Demonstrations or “pilots” are a
place to start for innovations (“it’s possible!”)
Don’t usually lead to sustainable service and system change Random acts of innovation Person and passion dependent Can “ghost” system its way to
success Executed by the “extraordinary” No replicable implementation
infrastructure
Transformation Zone
Focus on innovations And implementation
infrastructure development And organizational change And systems change processes
Transformation Zone
Use Innovations
Develop Implementation Infrastructure
Change System
Transformation Zone A “vertical slice” of the service
system (from the front-line to the Capitol) The “slice” is small enough to be
manageable The “slice” is large enough to
include all aspects of the system (court, case work, service system, urban, rural, frontier, diverse communities)
The “slice” is large enough to “disturb the system” – a “ghost” system won’t work.
Transformation Zone
Transformation Zones provide opportunities to… Manage the risks (most innovations
don’t work at first) and experience intended and unintended outcomes
Limit the damage (quick detection, recovery, repair)
Document “what works” Consider the implications of scaling-
up
Process for Implementing Change
TA Outcomes Evaluation Questions:
Research Question Indicator Process Measures
To what extent do implementation project stakeholders view MPCWIC technical assistance as having increased their State/Tribal capacity to meet project goals?
Perceptions of increased state/tribal implementation capacity
Implementation Capacity Assessment
Key Stakeholder Interviews
What components of NIRN are applicable in implementation of child welfare system change projects?
Stages/Drivers alignment with implementation success
Implementation Process Analysis
Global Attainment Scaling Driver Analysis
TA Outcomes Evaluation Questions:
Research Question Indicator Process Measures
To what extent do implementation project stakeholders view MPCWIC technical assistance as helpful in addressing state/tribal systems barriers?
Perceptions of increased state/tribal implementation capacity to address systems barriers
Key Stakeholder Interviews
To what extent do implementation project stakeholders view the NIRN framework as helpful in addressing system barriers?
Perceptions of Stages/Drivers installation related to implementation success
Key Stakeholder Interviews
Implementation Capacity Assessment
Implementation Process Analysis
MPCWIC Outcome Evaluation
Individual Implementation Project Evaluation Do State and Tribal systems improve with respect to the issue
addressed in their individual implementation projects? Do State and Tribal systems make significant advancements
toward the development of successful practice models? How do State and Tribal characteristics or organizational
factors contribute to the development of successful practice models?
What is the impact of the implementation project activities and integrations of the practice model on State or Tribal CSFR and PIP outcomes?
Individually, do State and Tribal partners accomplish the goals set forth in their multi-year strategic plans for sustainable system’s change?
Are State and Tribal partners successfully able to use practice model frameworks with fidelity to guide daily practice?
MPCWIC Outcome Evaluation
Cross- Implementation Project Outcomes As a group, do State and Tribal partners
accomplish goals set forth in their multi-year strategic plans for sustainable systems change?
Do organizations develop successful practice models?
What organizational factors are associated with successful practice models?
Do project type and organizational characteristics differentially affect outcomes?
What is the relationship between TA intensity and type and overall outcomes, particularily those related to culture, climate leadership, supervision and practice?
Summary Stage related activities prepare the
system for a successful change process Competence needs to be developed and
sustained Selection, training, coaching, fidelity measures help
change and support new practitioner behavior and skills
Organizations and systems need to change Data systems need to be used to make decisions Facilitative administrative practices & systems
interventions create hospitable environments Policy enables new practice but practice needs to
inform policy