Expert Panel Discussion: Measuring Fidelity as a Method for Evaluating Technical Assistance...
-
Upload
clarence-golden -
Category
Documents
-
view
221 -
download
1
Transcript of Expert Panel Discussion: Measuring Fidelity as a Method for Evaluating Technical Assistance...
Expert Panel Discussion:Measuring Fidelity as a Method for Evaluating Technical Assistance
Technical Assistance & Dissemination Program Meeting
Jill Lammert & Jill Feldman, Westat
April 28, 20151
Meet the Panelists!
Jill Lammert & Jill Feldman, Westat (Moderators)
Dean Fixsen, State Implementation and Scaling Up of Evidence Based Practices (SISEP)
Lynn Holdheide, CEEDAR Center Rob Horner, Positive Behavioral Interventions &
Supports (PBIS) Amy McCart, Schoolwide Integrated Framework for
Transformation (SWIFT) Kristin Reedy, National Center on Systemic Improvement
(NCSI)
3
Goals for the Discussion
Expose participants to a range of approaches for measuring fidelity of TA.
Pique interest among participants in creating fidelity frameworks for their TA projects.
Provide opportunities for participants to ask experts for guidance and advice.
4
Definition of Terms: Fidelity of Implementation vs. Fidelity of Intervention
Fidelity of implementation answers:– Are key components of a program being implemented as
intended?
Fidelity of intervention answers:– Did the degree of adoption of key components lead to
desired outcomes?
5
Sample Research Questions
Fidelity of Implementation– Were all elements of the technical assistance delivered by
program staff?– Were all of the mentoring services provided to related
services professionals according to the specified TA plan?
Fidelity of Intervention– To what extent did related services professionals engage
in the mentoring services with the TA team?– To what extent did the mentoring meetings cover the
specified TA topics? Were professionals satisfied with the quantity and quality of the mentoring services provided?
6
Measuring Fidelity in the Context of Evaluating TA
Measuring Fidelity
Recipients’ Use of New Knowledge
& Skills Recipients’
LearningRecipients’ Reactions
Organizational Support & Change
Ultimate Outcomes
7
Framework for Creating a Fidelity Measurement System
1. Specify the program logic model
2. Operationally define key components
3. Select data sources and measures
4. Establish fidelity thresholds
5. Calculate fidelity ratings
6. Determine how ratings will be used
8
Sample TA Logic Model
Monthly mentor/protégé meetings
Content of mentoring meetings
5 coaches, 1 for each related service profession
Length, number & frequency of coaching sessions
Content of coaching sessions
Inputs/ Strategies
Outputs/Activities/ Participation
Coaching Students’/ children’s
skills/ knowledge
increase, as measured by appropriate
assessments
Mentoring
Peer-to-Peer TA
Biweekly 1-on-1 peer meetings
Content of peer meetings
Online community of practice (CoP)
Outcomes Intermediate
Outcomes Long-term
Professionals participate in
TA in the required amounts
TA covers the specified
content
Outcomes Short-term
Professionals are satisfied with quantity & quality of
TA
Professionals apply skills & knowledge in
work with students/ children
Professionals increase
knowledge & practice skills
Consultation Length, number &
frequency of consultation meetings
Content of consultation meetings
9
Questions for Panelists
In what ways does your expert judgment guide how you think about measuring fidelity of TA? – What to measure? – What data to use? – How to approach calculating ratings? – How to decide which ratings to consider as evidence of
faithful implementation?
How do you approach measuring fidelity of TA components/activities that are intended to vary or be adapted (e.g., coaching at a range of intensity levels)?
10
Questions for Audience
Based on what you’ve heard today, what do you think will be the most challenging for your project in devising a fidelity framework?
What issues raised by the panelists seem particularly relevant for your project?
Based on what you heard today, how would you approach creating a logic model and identifying the key components for your TA project?