Thinking Strategically About and Practically About ... · 6 Things Any Program Need to Know… 1....
Transcript of Thinking Strategically About and Practically About ... · 6 Things Any Program Need to Know… 1....
Thinking Strategically About and Practically About BioSafety Training
2
Today
1. Think of evaluation and planning together2. Gathering “process use” insights3. Understanding how to set an evaluation
focus
3
Why We Care! The Continuous Quality Improvement Cycle
Planning—What actions will best reach our goals and objectives.
Performance measurement— How are we doing?
Evaluation—Why are we doing well or poorly?
What do we do?
Why are we
doing well or poorly?
How are we
doing?
How do we do it?
4
6 Things Any Program Need to Know…1. Big “need” to which it is contributing2. “Accountable” outcome3. ST outcomes that accountable
outcome4. Outputs: “Strong” activities that ST
outcomes, AND what “strong” means5. Moderators: Contextual factors that
help/hobble
55
A (Simple) Roadmap/Logic Model Helps!
Activities Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will changebecause of the program…
6
Payoff for Evaluation
1. Accountable outcome is key one to measure
2. What ST/MT drive it, IF we don’t think we’ll get there?
3. Outputs=Process measures for our activities
4. Moderators=Ability to explain (lack of) progress
7
BUT, Immediate Payoff in Process Use…What do we need to do NOW to:1. Improve ability of activities to drive
accountable outcomes (Outputs)2. ‘Own” factors in the larger environment
and address them so they don’t get in our way (Moderators)
3. OR, ACCOMMODATE to factors in the environment that are “laws of gravity” (Moderators)
88
(Simple) Logic Models are Roadmaps for ANY Effort
Activities Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will changebecause of the program…
Big “need” to which program is contributing
99
(Simple) Logic Models are Roadmaps for ANY Effort
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
Big “need” to which Lab Safety Training effort is contributing
1010
(Simple) Logic Models are Roadmaps for ANY Effort
Activities Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will changebecause of the program…
“Accountable” outcome
1111
(Simple) Logic Models are Roadmaps for ANY Effort
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
“Accountable” outcome for Lab Safety Training
effort
12
(Selected) Behaviors Sustained on Job Proper use of PPE Proper use of BSC Conduct formal risk assessments Ability to recognize hazards in any
situation Complete appropriate documentation Proper waste management
1313
Simple Logic Models as Roadmaps
Activities Inputs Outputs Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will changebecause of the program…
What the program needs…
Outputs: What “strong” activities means
1414
(Simple) Logic Models are Roadmaps for ANY Effort
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
Outputs: What “strong” activities means for Lab Safety Training efforts
15
(Selected) Key Attributes of Activities to Ensure Outcomes are Reached Content of training should explain why it
should be done Focus on actual work practices/evidence-
based work practices Real-time experience during the training Include reinforcing materials Personalizing content to the situation of
the person as opposed to general regs
1616
Simple Logic Models as Roadmaps
Activities Inputs Outputs Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will changebecause of the program…
What the program needs…
Context and Assumptions
External factors that influence getting to outcomes
5. What “strong” activities means
Killer assumptions: Outside factors that can
help or hobble
17
(Selected) Moderating Factors, Even if Our Training is Great! Supervisors’ attitudes and expectations Lack of resources for quality and safety Lack of reinforcement (decay of KAB) Time/safety trade-off, particularly during
public health emergency (time constraint) Competing priorities
18
(Selected) Actions We Can Take to Mitigate Moderating Factors Supervisors’ attitudes and expectationsAdd consistent messages in supervisor
trainingPush others to create SOPs or policies to
induce supervisor cooperation. Lack of reinforcement (decay of KAB)Refresher training; written materials
19
(Selected) Actions We Can Take to Mitigate Moderating Factors (cont’d) Time/safety trade-off, particularly during
public health emergency (time constraint) Competing prioritiesHighlight consequences of not doing things
right Lack of resources for quality and safetyArray solutions in class from
cheapexpensivePush others for more budget for these efforts
20
Payoff for Evaluation1. Accountable outcome is key one to
measure1. % of trainees maintaining (selected)
behaviors on the job 3/6/12 months post-training?
2. What ST/MT drive it, IF we don’t think we’ll get there?
1. % trainees exhibiting appropriate knowledge and skills on key competencies post-training
3. Outputs=Process measures for our activities
21
Payoff for EvaluationOutputs=Process measures for our activities1. Degree to which the trainings:Focus content on why it should be doneFocus on actual work practices/evidence-
based work practicesProvide real-time experience during the
training Include reinforcing materialsAre personalized to the situation of the person
as opposed to general regs
22
Payoff for EvaluationModerators=Ability to explain (lack of) progress Did supervisors’ exhibit consistent
attitudes and expectations? Were there sufficient resources for quality
and safety improvements? Did KAB decay over time? Did trainees see safety as important even
in the face of time constraints and competing priorities?
23
Questions?