Training is Not Enough: Intervention and Evaluation Supplements
description
Transcript of Training is Not Enough: Intervention and Evaluation Supplements
![Page 1: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/1.jpg)
Training is Not Enough:Intervention and Evaluation
Supplements
Christina J. Groark and Robert B.
McCallUniversity of Pittsburgh
Office of Child Development
![Page 2: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/2.jpg)
Overview—Intervention
• Training alone is a minimally effective intervention
• Need:−Hands-on technical assistance in
situ−Supportive work environment−Continuous monitoring, positive
supervision
![Page 3: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/3.jpg)
• Assessing trainee’s satisfaction and
learning is a minimum evaluation
• Need:−Changes in trainee’s on-the-job behavior−Changes in the behavior/outcomes of
clients of the trainees−Analyses to show trainees behavior
change mediated the intervention's effect on clients’ behavior/outcomes
Overview—Evaluation
![Page 4: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/4.jpg)
Intervention
Training alone is minimally
effectiveWhy? Trainees often do not change
their behavior
Need:• On-the-ground technical assistance• Supportive work environment• Monitoring and positive supervision
![Page 5: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/5.jpg)
Intervention
On-the-ground technical
assistance
•Observe (videotape) client on job•Constructive, positive feedback•Modeling with client on job•Daily review
![Page 6: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/6.jpg)
Intervention
![Page 7: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/7.jpg)
Intervention
Work environment
•Supports behaviors taught in training•Hand washing—need a sink, waterless soap handy•Early care and education—need small group size, low children:caregiver ratio, a few consistent caregivers
![Page 8: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/8.jpg)
Chaotic Child Environment
![Page 9: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/9.jpg)
Orderly Child Environment
9
![Page 10: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/10.jpg)
Monitoring and positive
supervision•Sets expectations, standards, reminds, rewards•Continuing system, supervisor, staff meetings
•Hand washing•Early care and education—teachable moments, responsiveness to child-directed initiatives
![Page 11: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/11.jpg)
Illustrative Example ILatin American Orphanage Intervention (birth-6 years)•Training—sensitive, responsive interactions
◦ 6 one-day training sessions
•On-the-ground technical
assistance◦ 12 days more than planned
because caregivers were not implementing
![Page 12: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/12.jpg)
BUT –
• Limited environmental change
◦ 4 primary caregivers per ward
◦ 8-10 children per caregiver
• No continuing supervision
![Page 13: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/13.jpg)
![Page 14: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/14.jpg)
1
5
7
Intervention
Pre- Post-
Results—Caregiving behavior/environment (ITERS/ECERS)
2
3
4
Younge
r
Older
6
Typical USA Early Care
![Page 15: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/15.jpg)
60
70
80
Intervention
Pre- Post-
Results—Children’s development (Battelle)
Young
er
Older
90
100Typical Parent-Reared
![Page 16: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/16.jpg)
Conclusion
• Needed hands-on technical assistance to promote implementation
• Limited supportive work environment
• No supervisory system• Some improvement in caregivers,
children, but NOT MUCH
![Page 17: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/17.jpg)
Illustrative Example IISt. Petersburg (Russia) Orphanage Intervention
•More intensive, comprehensive
•All major components
![Page 18: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/18.jpg)
• Minimum warm, sensitive, contingently-responsive caregiver-child interactions
• Perfunctory, business-like; little talking
• Caregiver directed
• Conformity
04/21/23 18
St. Petersburg-USA Orphanage Research Team, 2005, 2008
![Page 19: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/19.jpg)
![Page 20: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/20.jpg)
• 12-14 children/ward
• 9-12 caregivers/week
• Homogeneous age, disability groups
• Periodic graduations
• 60-100 caregivers before age 19
months
04/21/23 20
St. Petersburg-USA Orphanage Research Team, 2005, 2008
![Page 21: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/21.jpg)
Intervention
•Training◦ Train Trainers◦ 12 sessions, 2-3 hrs.◦ Warm, sensitive, responsive◦ “Love these kids…”
•Hands-on technical assistance
![Page 22: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/22.jpg)
![Page 23: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/23.jpg)
Intervention—Supportive Work Environment
• Reduced group size to 6-7• Assigned primary, secondary
caregivers—6 vs 9 cgrs.• Changed work schedules• Integrated groups by age,
disabilities• Eliminated periodic graduations to
new groups• Family hour
![Page 24: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/24.jpg)
Intervention—Monitoringand Positive Supervision
• Training for supervisors• Trainer observed, coached both
caregivers and supervisors• Supervisors met periodically to
problem solve, case reviews, mutual support
![Page 25: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/25.jpg)
Three orphanages (Baby Homes), birth – 4 years
• T+SC/M&S—Training plus Structural Changes, Monitoring, Supervision
• TO—Training Only
• NoI—No Intervention
Evaluation
![Page 26: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/26.jpg)
Evaluation—Trainee Learning
![Page 27: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/27.jpg)
Longitudinal Sample
Evaluation—Improved Caregiving
T+SC
![Page 28: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/28.jpg)
Evaluation – Children’s Development
40
50
60
70
80
90
First 4-9 mos 9+ mos
Time in Intervention
Dev
elo
pm
enta
l Qu
oti
ent T+SC
TO
NoI
100
Typical Parent Reared
![Page 29: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/29.jpg)
Evaluation--Mediation Analysis
• Intervention improved caregiving• Intervention improved children’s
development
• But was children’s
developmental improvement
associated with improved
caregiving?
![Page 30: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/30.jpg)
Evaluation—Mediation Analysis
![Page 31: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/31.jpg)
Evaluation--Mediation Analysis
Independent Variable T+SC vs
NoIIntervention
Outcome Battelle
Total EffectĈ =
14.14(2.07)***
MediatorHOME
Independent Variable T+SC vs
NoIIntervention
OutcomeBattelle
Effect of Mediator on Outcome Controlling for Indep. Variable
Direct effect Controlling for Mediator
Ĉ = 8.00(2.87)***
Mediator accounts for 43% of Total Effect
Effect of Indep. Var. on Mediator= 3.41(.22)*** = 1.80
(.59)**
![Page 32: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/32.jpg)
• Intervention◦Training alone is often minimally
effective◦Has greater effect if accompanied by
On-the-ground technical assistance Supportive work environment Monitoring and supervision
Conclusion
![Page 33: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/33.jpg)
• Evaluation is improved if◦Some training vs. No Training comparison
◦Measures of–Learning–Work-place behavior change–Outcome behavior of clients
◦Mediation analysis
Conclusion
![Page 34: Training is Not Enough: Intervention and Evaluation Supplements](https://reader036.fdocuments.in/reader036/viewer/2022062806/56814f56550346895dbd0271/html5/thumbnails/34.jpg)
Evaluation Mediation Analysis:MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating mediated effects in prevention studies. Evaluation Review, 17(2), 144-158.