Measuring Implementation and Intervention Fidelity in Scaling Up: Processes, Tools, and Benefits...
-
Upload
george-wood -
Category
Documents
-
view
222 -
download
0
Transcript of Measuring Implementation and Intervention Fidelity in Scaling Up: Processes, Tools, and Benefits...
Measuring Implementation and Intervention Fidelity in Scaling
Up:Processes, Tools, and Benefits
Carol M. Trivette, Ph.D.
Karen Blase, Ph.D.
Frank Porter Graham Child Development InstituteUNC at Chapel Hill. North Carolina
OSEPProject
Directors ConferenceJuly 2012
Center for Early Literacy Learning Center on Everyday Child Language Learning
Orelena Hawks Puckett InstituteAsheville and Morganton, North Carolina
Through the CELL and SISEP Lens….
GOALS: Describe differences between intervention and implementation processesDescribe the differences between intervention fidelity and implementation fidelityIllustrate how a “cascading logic model” details implementation strategies and outcomes at different levelsConnect implementation fidelity to improved intervention fidelity.
Intervention Practices and Fidelity
Intervention practices include methods and strategies used by intervention agents (teachers, therapists, clinicians, parents, etc.) to affect changes or produce desired outcomes in a targeted population or group of recipients (e.g., children with disabilities).
Therefore, intervention fidelity refers to the degree to which evidence-based intervention practices are used as intended by early childhood practitioners, teachers, parents, or other intervention agents and have expected or intended.
Dunst, C. J., Trivette, C. M., McInerney, M., Holland-Coviello, R., Masiello, T., Helsel, F., & Robyak, A. (2008). Measuring training and practice fidelity in capacity-building scaling-up initiatives. CELLpapers, 3(1), 1-11. Available at http://www.earlyliteracylearning.org/cellpapers/cellpapers_v3_n1.pdf.
Implementation Practices and Fidelity
Implementation practices include methods and procedures used by implementation agents (trainers, coaches, instructors, supervisors, etc.) to promote interventionists’ use of evidence-based intervention practices.
Therefore implementation fidelity refers to the degree to which coaching, in-service training, instruction, or any other kind of evidence-based professional development practice is implemented as intended and has the effect of promoting the adoption and use of evidence-based intervention practices.
Trivette, C. M., & Dunst, C. J. (2011, August). Implementation with fidelity: How to get changes in early childhood classroom practices. Paper presented at the Global Implementation Conference, Washington, DC.
© Fixsen & Blase, 2008
All Students & Families
All Students & Families
School Teachers and
Staff
School Teachers and
Staff
StateDepartment Leadership
StateDepartment Leadership
Implementation-Skilled Workforce
Develop staff with special implementation skills
Re
-Pu
rpo
se
Regional Implementation
Teams
Regional Implementation
Teams
Adult interactions produce Student benefits
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
SISEP Cascading TA Logic Model
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
SISEP Cascading TA Logic Model
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions
Teachers and staff competently and confidently use evidence-based practices
~ Intervention Fidelity~
SISEP Cascading TA Logic Model
Intervention Fidelity: The degree to which evidence-based intervention practices are used as intended by early childhood practitioners, teachers, parents, or other intervention agents and have expected or intended. Dunst, C. J., Trivette, C. M., McInerney, M., Holland-Coviello, R., Masiello, T., Helsel, F., & Robyak, A. (2008).
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions
Teachers and staff competently and confidently use evidence-based practices
~ Intervention Fidelity~
Building and District staff Provision of services from the Regional Implementation Team to LEA (District and School staff)
Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff
~Implementation Fidelity~
SISEP Cascading TA Logic Model
Implementation Fidelity: The degree to which coaching, in-service training,instruction, or any other kind of evidence-based professional development practice is implemented as intended and has the effect of promoting the adoption and use of evidence-based intervention practices. Trivette, C. M., & Dunst, C. J. (2011, August). Implementation with fidelity: How to get changes in early childhood classroom practices. Paper presented at the Global Implementation Conference, Washington, DC.
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions
Teachers and staff competently and confidently use evidence-based practices
~ Intervention Fidelity~
Local and District staff Provision of services from the Regional Implementation Team to LEA (District and School staff)
Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff
~ Implementation Fidelity
Regional Implementation Team Members
Skillful leadership by State Transformation Specialists, supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams.
Timely and skillful provision of services by Regional Implementation Teams to LEA
SISEP Cascading TA Logic Model
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions
Teachers and staff competently and confidently use evidence-based practices
~ Intervention Fidelity~
Building, District, Regional Staff Provision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities
Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff
Regional Implementation Team Members
Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams.
Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities
State Transformation Specialists
State Management Team
1st Generation Regional Implementation Team
Intensive technical assistance by SISEP to
develop implementation capacity at multiple levels
Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams.
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions
Teachers and staff competently and confidently use evidence-based practices
~ Intervention Fidelity~
Building, District, Regional Staff Provision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities
Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff
Regional Implementation Team Members
Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams.
Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities
State Transformation Specialists
State Management Team
1st Generation Regional Implementation Team
Intensive technical assistance by SISEP to
develop implementation capacity at multiple levels
Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams.
Population Intervention
Strategies (WHAT)
Intervention
Outcomes
All students and/or young children
Teachers and staff fully and effectively use evidence-based practices
Improved academic, behavioral outcomes
Populations Implementation Strategies
(HOW)
Implementation Outcomes
Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions
Teachers and staff competently and confidently use evidence-based practices
~ Intervention Fidelity~
Building, District, Regional Staff Provision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities
Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff
Regional Implementation Team Members
Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams.
Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities
State Transformation Specialists
State Management Team
1st Generation Regional Implementation Team
Intensive technical assistance by SISEP to
develop implementation capacity at multiple levels
Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams.
Implementation Drivers Best Practices & DCA
Implementation Drivers Best Practices & DCA
Regional Implementation Team Capacity Assessment (RIT CA)
State Capacity Assessment (SCA & RIT CA)
SCA & RIT CA
All Measures Above
RIT Capacity Assessment
Implementation and Intervention
No intervention practice, no matter its evidence base, is likely to be adopted and used if the implementation methods used to teach or train practitioners to use the practice are not themselves effective.
Therefore concern for the characteristics of implementation practices that are associated with optimal learner and practitioner outcomes should be of paramount importance as part of implementation research.
Dunst, C. J., & Trivette, C. M. (2009). Let's be PALS: An evidence-based approach to professional development. Infants and Young Children, 22(3), 164-175. doi:10.1097/IYC.0b013e3181abe16
Center for Early Literacy LearningCenter on Early Childhood Language Learning
CELL – OSPE TA&D Project – Children 0-3 with disabilities• Developed evidence-based early literacy practices for young
children with disabilities (web site)• Worked with 6 states to scale up these early literacy practices• Worked with state teams to develop training• State teams trained trainers or teachers or practitioners• Teachers and parents provided intervention with children
CECLL – OSEP Model Demonstration – Children 0-3 with disabilities
• Provided training to Part C EI providers in 3 states• Part C provided coaching with parents• Parents provided intervention to children
Tiered Approach to Training to Promote Adoption and Sustained Use of CELL
One
Many
StateResource
(Leadership) Team
Regional Resource(Cohorts) Team
Level I
Local/Regional TrainersLevel II
End Users – Teachers or ParentsInterventions to Children
Research Synthesis of Adult Learning Studiesa
• Research synthesis of studies of accelerated learning, coaching, guided design, and just-in-time training
• 58 randomized control design studies
• 2,095 experimental group participants and 2,213 control or comparison group participants
• Combination of studies in university and nonuniversity settings
• Learner outcomes included learner knowledge, practices, skills, attitudes, and self-efficacy beliefs
• The influence of the adult learning methods on the learner outcomes was estimated by weighted Cohen’s d effect sizes for the differences on the post test scores for the intervention vs. nonintervention group participants
a Dunst, C.J., Trivette, C.M., & Hamby, D.W. (2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. International Journal of Continuing Education and Lifelong Learning, 3(1), 91-112.
Characteristics Used to Code and Evaluate the Implementation Studiesa
Planning
Introduce Engage the learner in a preview of the material, knowledge, or practice that is the focus of instruction or training
Illustrate Demonstrate or illustrate the use or applicability of the material, knowledge, or practice for the learner
Application
Practice Engage the learner in the use of the material, knowledge, or practice
Evaluate Engage the learner in a process of evaluating the consequence or outcome of the application of the material, knowledge, or practice
Deep UnderstandingReflection Engage the learner in self-assessment of his or her acquisition of
knowledge and skills as a basis for identifying “next steps” in the learning process
Mastery Engage the learner in a process of assessing his or her experience in the context of some conceptual or practical model or framework, or some external set of performance standards or criteria
a Donovan, M. et al. (Eds.) (1999). How people learn. Washington, DC: National Academy Press.
Most Effective Adult Learning Method Practices
Characteristic Practice Mean Effect SizeIntroduction Out-of-class learner activities/self-instruction 0.64
Classroom/workshop presentations 0.63
Pre-class learner exercises 0.54
Illustration Trainer role playing/simulations 0.55
Learner informed input 0.53
Practicing Real-life learner application 0.94
Real-life learner application/role playing 0.86
Evaluation Self assessment of strengths/weaknesses 0.94
Reflection Identify performance-improvement goals 1.27
Journaling/behavior suggestions 0.82
Mastery Standards-based assessment 0.86
Cumulative Effects of Different Combinations of the Most Effective Adult Learning Method Practices
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1 2 3 4 5 6
MEA
N E
FFEC
T SI
ZE
NUMBER OF PRACTICES
0 1 2 3 4 5
NUMBER OF ADULT LEARNING METHOD PRACTICES
PALS(Participatory Adult Learning Strategy)
PLAN
APPLICATIONRECYCLE Active
Learner Involvement
Reflection and Mastery
Practice and Evaluate
Introduce and Illustrate
Identify Next Steps in the
Learning Process
INFORMED UNDERSTANDING
CELL Trainer Feedback Form
Content Items1. The importance of literacy-rich environments was well explained by the trainer.2.The key characteristics of responsive teaching were described and illustrated in ways that made the instructional practice easy to understand.Training-Method Items3.The importance of active participation of trainees as a method was made explicitly clear.4.The training method was described in enough detail to understand its key elements.Practice Items5.A sufficient amount of time was devoted to each component of the CELL literacy model.6.The activities for engaging trainees in learning the CELL literacy practices were especially helpful.
Hypothesized RelationshipsAmong the Fidelity Measures
• Variations in implementation fidelity should be related to variations in intervention fidelity. Tests of the hypothesis include evaluation of the relative importance of the quantity and quality of implementation fidelity, and the interactions between the types and elements of fidelity.
• Variations in intervention fidelity should be related to variations in practice outcomes. Tests of the hypothesis include evaluation of the relative importance of the quantity and quality of intervention fidelity, and the interactions between the types and elements of fidelity.
26
Relationship Between Implementation Fidelity and Child Communication Development
PurposeEvaluate the relationship between variations in the fidelity
of parents’ use of the CECLL practices and changes in the children’s early communication development
ParticipantsSixty-three infants and toddlers 8 to 40 months of age and
their parents
Outcome MeasuresFidelity: A weekly parent-completed scale of child participation
in and the characteristics of everyday learning activitiesChild Outcome: Early Communication Development Scale
27
CECLL StaffTraining of
Practitioners
PractitionerImplementation
with Parent
Parent Use of the
Intervention With the Child
Enhanced Language Learning
ImplementationFidelity
Intervention Fidelity
ChildOutcomes
Hypothesized Relationships Between the Implementation and Intervention Fidelity Measures and Child Outcomes
28
Parent-Completed Fidelity Measure
The scale used to assess intervention fidelity captured two aspects of the use of the CECLL practices:
•Quantity of child participation in everyday activities (number of settings, number of activities within settings, frequency of participation)
•Quality of child participation in everyday activities (development-enhancing characteristics of the child learning experiences and opportunities)
29
Influences of Fidelityon Child Communication Development
5
6
7
8
9
10
6 12 18 20 24 30 36
CHIL
D C
OM
MU
NIC
ATI
ON
LO
GIT
SCO
RE
CHILD AGE (Months)
Low Fidelity
High Fidelity
5
6
7
8
9
10
6 12 18 20 24 30 36
CHIL
D C
OM
MU
NIC
ATI
ON
LO
GIT
SCO
RE
CHILD AGE (Months)
Low Fidelity
High Fidelity
Cohen’s d = 0.62 for differences in the slopes
Summary
• Intervention Fidelity Matters to Achieve Outcomes
• Implementation Fidelity Matters to Achieve Intervention Fidelity
• Multiple levels of the system need to be impacted to achieve these outcomes
• Cascading logic models and measures at every level can help develop effective implementation strategies and intervention outcomes.