Post on 11-Aug-2020
Coaching with FidelityKaci Fleetwood, M.Ed, BCBA, LBAJodie Soracco, M.Ed, BCBA, LBA
APBS 2020 J 11
Lens of Implementation
State Level
District Level
Doesn’t fit in a box
Building Level
Identify a Buddy
• Make eye contact with someone close to you
• Tap elbows or give each other a thumbs up to confirm you’ve noticed each other
• Plan to talk to your buddy during think, pair, share prompts
5
“There was a moment in sports when employing a coach was unimaginable – and then came a time when not doing so was unimaginable. We care about results in sports, and if we care half as much about results in schools and in hospitals, we may reach the same conclusion.”
Atul Gawande
Why is Coaching Important?
• More accurate implementation fidelity scores (McIntosh et al., 2017)
• Greater improvements in outcomes (Artman-Meeker, Hermeter, & Snyder, 2014; Capella et al., 2012; Flynn, Lissy, Alicea, Tarzates, & Mckay, 2017)
• Sustainability and scale-up (Fixen & Blasé, 1993; Horner et al., 2014)
6
“You would never have a team without a
coach”-Shauna Bake (Pershing County MTSS Coordinator), 2019
7
Research to Practice Gap
• Research* has demonstrated that coaching is essential for translating training into effective implementation.
• However, little explicit guidance exists on how to structure the roles and activities of external coaches who support multiple implementation sites.
*(Joyce & Showers, 2002; LaPolla & Scott, 2019; McDaniel & Bloomfield, 2019; Sugai et al., 2012)
Default Approach
• Those who have been responsible for recruiting, training, and supervising external coaches at the district, regional, or state level often resort to simply hiring candidates with the most robust coach-related experience and permit a type of “free-range coaching”.
• This results in a great deal of variability in what coaching activities are performed, how school sites are supported.
Diverse Backgrounds
10
In Nevada Coaching Positions are filled by individuals with diverse professional histories• Licensed Teachers• School Psychologists• Board Certified Behavior
Analysts• Social Workers• RN/Psychiatric Nurses• School Administrators• Others
What are the Roles of a Coach?
• “External Coaches”– Coach teams– Train content– Evaluate outcomes– Coordinate implementation– Expand LEA capacity– Other duties as assigned
SOLUTION: OUTLINE PERFORMANCE EXPECTATIONS FOR DIVERSE COACHING FUNCTIONS
Leveraging a School-wide Approach
• Expectations = professional values• Locations/Contexts= job functions• Coaching Competencies= observable behaviors
Valu
e/Ex
pect
atio
nVa
lue/
Ex
pect
atio
nVa
lue/
Expe
ctat
ion
Safe Respectful
Responsible
What expectations/
values work for your coach(es)?
What professional values apply to your context?
Insert Nevada Coach Matrix
MTSSExternalCoachExpectationMatrix
Overall(O)
CoachingTeams(C)
Training(T)
Responsible
1. Attendmeetingsandcoachtrainingswithcoordinator______
2. Self-monitortasksassignedandcompletion_____
3. Engageinformaldissemination______
1. Attendsite-teammeetings(data-driven)______
2. Collectmonthlyartifactsfromteams______3. Facilitateestablishmentofdatacollectionsystems
______4. Consultationnotescompleted______
1. Askforclarificationorassistancefor
concepts/topicswithwhichyouarenotyetfluent______
2. Schedule&presenttargetedtrainings_____
Data -
Driven
4. Reportsanddataaredeliveredtothe
stateonorbeforeduedate______
5. Usedatatoallocatetimeandresources______
5. UseTFItocollectdataanddriveactionplanning
______
6. Usevariousdatasourcesforassessment,planning,andreporting______
3. Collectevaluationsonallformal
presentations______
4. Review&reflectonfeedbackfrom:participants,districtadministrators,andcoordinators______
Professional
6. Assistdistricttobuildcapacityfor
sustainedimplementation(redefineyourroleovertime)______
7. Readprofessionalliteratureata
minimumof30min/week______8. Engageinprofessionalandethical
standards(seeCoachesCode)______
7. Advocateforsmallestchangethatwillhavethe
largestimpact______8. Buildcredibility&rapport______
5. EngageinEffectiveSpokesperson
techniqueswhilepresenting______
Bonus:PresentonMTSS/PBISatdistrict,state,andnationalforums______
Professional Values/ Expectations
Insert Nevada Coach Matrix
MTSSExternalCoachExpectationMatrix
Overall(O)
CoachingTeams(C)
Training(T)
Responsible
1. Attendmeetingsandcoachtrainingswithcoordinator______
2. Self-monitortasksassignedandcompletion_____
3. Engageinformaldissemination______
1. Attendsite-teammeetings(data-driven)______
2. Collectmonthlyartifactsfromteams______3. Facilitateestablishmentofdatacollectionsystems
______4. Consultationnotescompleted______
1. Askforclarificationorassistancefor
concepts/topicswithwhichyouarenotyetfluent______
2. Schedule&presenttargetedtrainings_____
Data -
Driven
4. Reportsanddataaredeliveredtothe
stateonorbeforeduedate______
5. Usedatatoallocatetimeandresources______
5. UseTFItocollectdataanddriveactionplanning
______
6. Usevariousdatasourcesforassessment,planning,andreporting______
3. Collectevaluationsonallformal
presentations______
4. Review&reflectonfeedbackfrom:participants,districtadministrators,andcoordinators______
Professional
6. Assistdistricttobuildcapacityfor
sustainedimplementation(redefineyourroleovertime)______
7. Readprofessionalliteratureata
minimumof30min/week______8. Engageinprofessionalandethical
standards(seeCoachesCode)______
7. Advocateforsmallestchangethatwillhavethe
largestimpact______8. Buildcredibility&rapport______
5. EngageinEffectiveSpokesperson
techniqueswhilepresenting______
Bonus:PresentonMTSS/PBISatdistrict,state,andnationalforums______
Locations/Contexts
delineate job functions
Insert Nevada Coach Matrix
MTSSExternalCoachExpectationMatrix
Overall(O)
CoachingTeams(C)
Training(T)
Responsible
1. Attendmeetingsandcoachtrainingswithcoordinator______
2. Self-monitortasksassignedandcompletion_____
3. Engageinformaldissemination______
1. Attendsite-teammeetings(data-driven)______
2. Collectmonthlyartifactsfromteams______3. Facilitateestablishmentofdatacollectionsystems
______4. Consultationnotescompleted______
1. Askforclarificationorassistancefor
concepts/topicswithwhichyouarenotyetfluent______
2. Schedule&presenttargetedtrainings_____
Data-
Driven
4. Reportsanddataaredeliveredtothe
stateonorbeforeduedate______
5. Usedatatoallocatetimeandresources______
5. UseTFItocollectdataanddriveactionplanning
______
6. Usevariousdatasourcesforassessment,planning,andreporting______
3. Collectevaluationsonallformal
presentations______
4. Review&reflectonfeedbackfrom:participants,districtadministrators,andcoordinators______
Professional
6. Assistdistricttobuildcapacityfor
sustainedimplementation(redefineyourroleovertime)______
7. Readprofessionalliteratureata
minimumof30min/week______8. Engageinprofessionalandethical
standards(seeCoachesCode)______
7. Advocateforsmallestchangethatwillhavethe
largestimpact______8. Buildcredibility&rapport______
5. EngageinEffectiveSpokesperson
techniqueswhilepresenting______
Bonus:PresentonMTSS/PBISatdistrict,state,andnationalforums______
Continuous Improvement
• Contexts updated annually as needed• Competencies revised annually based on feedback/field test • V1 (2015); v2 (2016); v3 (2017); v4 (2018); v5 (2019)
2015 2016 2017 2018 2019
State/OverallDistrictTeamsContent/Presentations
OverallDistrictTeamsContent/Presentations
OverallDistrictTeamsContent/Presentations
OverallDistrictTeamsIndividual
OverallCoaching TeamsTraining
TEACHING COACHING COMPETENCIES
Leveraging a School-Wide Approach
• Annual Coaches “Kick-Off” event at the beginning of each school year
• Reveal the current matrix• Use a Cool Tool format to teach each competency:
– Include rationale for competency– Include examples of that competency in observable behaviors– Include non-example of that competency
23
T1: Attend site-team meetings (data-driven)
Rationale:Training alone is a death sentence for initiatives. On-site technical assistance is key to
translating what teams have learned in training into practice.
Examples: Non-Examples:
• Attending meetings to give guidance and feedback
• Allocating your time to support teams based on their need
• Proactively scheduling time to attend their meeting (pre-arranged)
• Being the leader and running meetings for teams
• Avoiding teams who are difficult to work with
• Attending meeting(s) and not providing feedback
24
O6: Assist district to build capacity for sustained implementation (redefine your role over time)
Rationale:For an initiative to be durable and sustain, it will require multiple champions/ local experts
Examples: Non-Examples:
• Coordinating trainings for PBISMTSS across departments (not a one person training)
• Continuously working on sustainability (not as a goal, but as a value)
• Relooking at organizational structures as needed
• Advocating for schools & district needs
• Leaving training to a lone soldier• Relying on others to make the change• Operating in a silo because it’s easier• Being the superhero who does everything
for everyone always
25
T5: Use TFI to collect data and drive action planning
Rationale:Using the TFI is required of all sites who are formally identified as implementing
PBIS in Nevada. Site based activities should be in pursuit of critical implementation components of the tier they are on to move forward.
Examples: Non-Examples:
• Following up with teams after TFI administrations to help support them in action planning
• Requesting copies of team action plans
• Letting teams drift and tend to topics/activities that are far removed from components of implementation
• Not prompting sites to action plan
Notes about teaching the Competencies
• Articulating the rationale is extremely important• Allow for veteran coaches to provide supplemental
examples/non-examples and reinforce or provide corrective feedback based on what they shared to enhance discrimination of appropriate ways to engage in the competencies
• A through discussion of each competency is required. The cool tools alone are insufficient
HOW TO MEASURE COMPETENCIES AND SET CRITERIA FOR HIGH-FIDELITY PERFORMANCE
How might you measure these behaviors?
Next Challenge
Needed a way to:• Quantifiably measure the activities and job duties executed by
external coaches• Capture variance within each competency
– More than yes/no
Training(T)CoachExpectations Red Yellow Green1 Askfor
clarificationorassistanceforconcepts/topicswithwhichyouarenotyetfluent
Assistancewasneededbutnotsoughtafter
Assistancewasneededbutinconsistentlysoughtafter
Assistancewasneededand
soughtafter/notapplicable
2 Schedule&presenttargeted
trainings
Trainingwasnotscheduled/deliveredordidnotaddressthedatadriven
target
Trainingwasdeliveredbutdidnotmeettheneedsofthetopic/group
Trainingwashighquality,
mettheneedsofthegroup;topicwasrelevantandbasedon
data
3 Collectevaluationsonall
formalpresentations
Evaluationswerenotcollected
Evaluationswerepartiallycollectedand/ornotturnedin
Allevaluationswerecollectedandturnedinon
time4 Review&reflect
onfeedbackfrom:participants,district
administrators,andcoordinators
Couldnotprovideaconcreteexample
Feedbackwasreceived,butdidnotdriveaction/changeinpractices
Provided1(+)concreteexampleoffeedbackthatpromptedadaptation
5 EngageinEffective
Spokespersontechniqueswhile
presenting
Couldnotprovideanexample
Canprovideoneexamplefroma
training,meeting,orother
presentationopportunity
Canprovidetwoexamplesfroma
training,meeting,orother
presentationopportunity
Bonus PresentonMTSS/PBISat
district,state,andnationalforums
Opportunitywasavailablebutdidnot
seek
Soughtopportunitybutwasdenied
PresentedonMTSS/PBIS
CoachingTeams(C)CoachExpectations Red Yellow Green1 Attendsite-team
meetings(data-driven)
Attends1meeting
Attends2
meetingsbasedondata
Attends3meetings,atleast2foremergingor
developingsites2 Collectmonthly
artifactsfromteams
0artifactscollectedper
team
1artifactcollectedperteamforlessthan80%ofschools
2relevantartifacts
collectedperteamfor80%of
schools
3 Facilitateestablishmentofdatacollection
systems
0–4contacts
5–7contacts
8–10contacts
4 Consultationnotescompleted
Documentedlessthan50%of
TA
Documented50-79%ofTA
Documented80%ormoreof
TA
5 UseTFItocollectdataanddriveactionplanning
Couldnotprovideaconcreteexample
Provided1concreteexample
Provided2concreteexamples
6 Usevariousdatasourcesforassessment,planning,andreporting
Couldnotprovideaconcreteexample
Provided1concreteexample
Provided2concreteexamples
7 Advocateforsmallestchangethatwillhavethelargestimpact
Couldnotprovideaconcreteexample
Provided1concreteexample
Provided2concreteexamples
8 Buildcredibility&rapport
Couldnotprovideaconcreteexample
Provided1concreteexample
Provided2concreteexamples
Overall(O)CoachExpectations Red Yellow Green
1 Attendmeetingsandcoachtrainingswithcoordinator
Attendlessthan2
Attend2-3
Attendedatleast4meetings/trainings
2 Self-monitortasksassignedandcompletion
Lessthan50%ofactivitiesacrossweeklytargetsheetsarecompleted
50-79%ofactivitiesacrossweeklytargetsheetsarecompleted
80%(+)ofactivitiesacrossweeklytargetsheetsarecompleted
3 Engageinformaldissemination
Noformaldisseminationofinformation/presentations
Newsletterpublishedlate/orlackingrelevantcontent,data,orrecognition
High-qualitynewsletterwithdata,content,andrecognition
4 Reportsanddataaredeliveredtothestateonorbeforeduedate
Dataforlessthan80%ofteams
submittedontime
Dataforallteamssubmittedlate(within1week)
Dataforallteamssubmittedontimeinrelevantmonths
5 Usedatatoallocatetimeandresources
Evidencedlessthan2weeks(50%)oftimeacrosstargetsheetexpectations
Evidenced2/4weeks(50%)oftimeacrosstargetsheetexpectations
Evidencedatleast3/4weeks(75%orgreater)andactivitiesare
distributedacrossthematrix
6 Assistdistricttobuildcapacityforsustainedimplementation(redefineyourroleovertime)
Cannotprovideaconcreteexampleforthismonth
Canprovideatleast1concreteexampleforthismonth
Canprovideatleast2concreteexamplesforthis
month
7 Readprofessionalliteratureataminimumof30min/week
Participated/readlessthan1hourofprofessional
literature/webinarsthismonth
Participated/readunder2hoursofprofessional
literature/webinarsthismonth
Readatleast2hoursof
professionalliteraturethismonthor/andattendedawebinar
8 Engageinprofessionalandethicalstandards(seeCoachesCode)
Cangivenoexamplesof
adherencetotheethicalstandardsorcoachescodeorhadaviolationofethicalstandards
Cangive1exampleofadherencetotheethicalstandardsor
coachescode
Cangive2examplesof
instanceswherethecoachescodewasusedin
coaching,training,orpresentations
PROGRESS MONITORING
Feedback Rubrics
• Monthly Feedback Session with External Coach
• Provides feedback on performance with respect to:– Expectations– Roles/Functions– Overall competency distribution
How might you use these data?
Same data, different purposes
• Individual progress monitoring tool• Provides a view of districts coaching activities in action• Measurement of adherence to coaching activities by “coaches”
– Data leveraged for advocacy when “other duties as assigned” chronically interrupts coaching activities
• At times, the data reveal district barriers to effective coaching behaviors
PAYING HOMAGE: BEHAVIOR SCIENCE, OBM
Total Performance System
• “Management Without Supervision”• Management with positive reinforcement• Reduces the need for direct supervision• Managers may spend more time:
– Planning– Coordinating– Optimizing performance
39Abernathy (2000, 2011, 2014)
Alignment with Abernathy’s TPS
40
Observable, measureable
strategic objectives
Expectation Matrix
Self-Management
Tools
Feedback Rubrics
Personal scorecards with specific
measures and goals
Feedback Scorecard
Managers assist in pinpointing ongoing opportunities and
designing improvement plans
Performance Charts
Tactical Improvement
Plan
Employees share in the organization’s
success
Performance-Based
Incentives
TACTICAL IMPROVEMENT PLAN Flowchart Pinpointing Obstacles to Maximum Performance
41
Abernathy Theory
• How we typically address poor performance– Look for problems inside the person– Managers point to characteristics like cooperativeness, dedication,
commitment, work ethic
• It is easier for managers to blame their employees than themselves– “If I only had good students, I could teach!”
42
Pinpointing Obstacles to Maximum Performance
43
Target High Opportunity
Measure
Measure Validity Issue?
Performance Improvement
Issue?
Behavior Issue?
Selection or Training Behavior Prompts Behavior
FeedbackBehavior
Consequences
Process Issue
Coach Utilization Work Methods
Adapted from Abernathy, 2000
Pinpointing Obstacles to Maximum Performance
44
Target High Opportunity
Measure
Measure Validity Issue?
Performance Improvement
Issue?
Behavior Issue?
Selection or Training Behavior Prompts Behavior
FeedbackBehavior
Consequences
Process Issue
Coach Utilization Work Methods
Adapted from Abernathy, 2000
Relevant and attainable
measures and goals
Skills and/or training
Cues, signals,
instructions, job aids, or
other antecedents
to work actions
Scorecards, coaching sessions,
etc.
Praise, recognition, feedback, criticism,
etc.
Pulled away for other
projects/activities
Work process-
es, quality
of input, budget,
etc.
LESSONS LEARNED & FUTURE DIRECTIONS
Lessons Learned
• Be open to annual revisions – Keep notes from monthly feedback sessions to drive future changes
to contexts, competencies, and especially rubric criteria
• Need to depersonalize individual performance– Many activities are confounded by system barriers outside of
coaches control
• Additional lessons
Future Directions
• Desire to investigate empirical questions– How/does coaching performance on the TPS impact:
• School outcomes (fidelity, implementation, sustainability)• District outcomes (capacity, sustainability, etc)
– Is there a threshold for maximum impact?– Are coaches utilizing this system more efficacious than those who do
not have this system? (explore beyond anecdotal reports)• Analysis beyond individual progress• Additional things we want to research
What future directions would you recommend?
Questions
We appreciate you being here!
Kaci Fleetwood; Kacif@unr.edu
Jodie Soracco; Jodie@unr.edu