Training is Not Enough:Intervention and Evaluation
Supplements
Christina J. Groark and Robert B.
McCallUniversity of Pittsburgh
Office of Child Development
Overview—Intervention
• Training alone is a minimally effective intervention
• Need:−Hands-on technical assistance in
situ−Supportive work environment−Continuous monitoring, positive
supervision
• Assessing trainee’s satisfaction and
learning is a minimum evaluation
• Need:−Changes in trainee’s on-the-job behavior−Changes in the behavior/outcomes of
clients of the trainees−Analyses to show trainees behavior
change mediated the intervention's effect on clients’ behavior/outcomes
Overview—Evaluation
Intervention
Training alone is minimally
effectiveWhy? Trainees often do not change
their behavior
Need:• On-the-ground technical assistance• Supportive work environment• Monitoring and positive supervision
Intervention
On-the-ground technical
assistance
•Observe (videotape) client on job•Constructive, positive feedback•Modeling with client on job•Daily review
Intervention
Intervention
Work environment
•Supports behaviors taught in training•Hand washing—need a sink, waterless soap handy•Early care and education—need small group size, low children:caregiver ratio, a few consistent caregivers
Chaotic Child Environment
Orderly Child Environment
9
Monitoring and positive
supervision•Sets expectations, standards, reminds, rewards•Continuing system, supervisor, staff meetings
•Hand washing•Early care and education—teachable moments, responsiveness to child-directed initiatives
Illustrative Example ILatin American Orphanage Intervention (birth-6 years)•Training—sensitive, responsive interactions
◦ 6 one-day training sessions
•On-the-ground technical
assistance◦ 12 days more than planned
because caregivers were not implementing
BUT –
• Limited environmental change
◦ 4 primary caregivers per ward
◦ 8-10 children per caregiver
• No continuing supervision
1
5
7
Intervention
Pre- Post-
Results—Caregiving behavior/environment (ITERS/ECERS)
2
3
4
Younge
r
Older
6
Typical USA Early Care
60
70
80
Intervention
Pre- Post-
Results—Children’s development (Battelle)
Young
er
Older
90
100Typical Parent-Reared
Conclusion
• Needed hands-on technical assistance to promote implementation
• Limited supportive work environment
• No supervisory system• Some improvement in caregivers,
children, but NOT MUCH
Illustrative Example IISt. Petersburg (Russia) Orphanage Intervention
•More intensive, comprehensive
•All major components
• Minimum warm, sensitive, contingently-responsive caregiver-child interactions
• Perfunctory, business-like; little talking
• Caregiver directed
• Conformity
04/21/23 18
St. Petersburg-USA Orphanage Research Team, 2005, 2008
• 12-14 children/ward
• 9-12 caregivers/week
• Homogeneous age, disability groups
• Periodic graduations
• 60-100 caregivers before age 19
months
04/21/23 20
St. Petersburg-USA Orphanage Research Team, 2005, 2008
Intervention
•Training◦ Train Trainers◦ 12 sessions, 2-3 hrs.◦ Warm, sensitive, responsive◦ “Love these kids…”
•Hands-on technical assistance
Intervention—Supportive Work Environment
• Reduced group size to 6-7• Assigned primary, secondary
caregivers—6 vs 9 cgrs.• Changed work schedules• Integrated groups by age,
disabilities• Eliminated periodic graduations to
new groups• Family hour
Intervention—Monitoringand Positive Supervision
• Training for supervisors• Trainer observed, coached both
caregivers and supervisors• Supervisors met periodically to
problem solve, case reviews, mutual support
Three orphanages (Baby Homes), birth – 4 years
• T+SC/M&S—Training plus Structural Changes, Monitoring, Supervision
• TO—Training Only
• NoI—No Intervention
Evaluation
Evaluation—Trainee Learning
Longitudinal Sample
Evaluation—Improved Caregiving
T+SC
Evaluation – Children’s Development
40
50
60
70
80
90
First 4-9 mos 9+ mos
Time in Intervention
Dev
elo
pm
enta
l Qu
oti
ent T+SC
TO
NoI
100
Typical Parent Reared
Evaluation--Mediation Analysis
• Intervention improved caregiving• Intervention improved children’s
development
• But was children’s
developmental improvement
associated with improved
caregiving?
Evaluation—Mediation Analysis
Evaluation--Mediation Analysis
Independent Variable T+SC vs
NoIIntervention
Outcome Battelle
Total EffectĈ =
14.14(2.07)***
MediatorHOME
Independent Variable T+SC vs
NoIIntervention
OutcomeBattelle
Effect of Mediator on Outcome Controlling for Indep. Variable
Direct effect Controlling for Mediator
Ĉ = 8.00(2.87)***
Mediator accounts for 43% of Total Effect
Effect of Indep. Var. on Mediator= 3.41(.22)*** = 1.80
(.59)**
• Intervention◦Training alone is often minimally
effective◦Has greater effect if accompanied by
On-the-ground technical assistance Supportive work environment Monitoring and supervision
Conclusion
• Evaluation is improved if◦Some training vs. No Training comparison
◦Measures of–Learning–Work-place behavior change–Outcome behavior of clients
◦Mediation analysis
Conclusion
Evaluation Mediation Analysis:MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating mediated effects in prevention studies. Evaluation Review, 17(2), 144-158.
Top Related