Program Evaluation Webinar Series Part 1:
description
Transcript of Program Evaluation Webinar Series Part 1:
![Page 1: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/1.jpg)
Program Evaluation Webinar Series Part 1:“Top Roadblocks on the Path to Good Evaluation– And How to Avoid Them”
Presented by: Tom Chapel
![Page 2: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/2.jpg)
Top Roadblocks on the Path to Good Evaluation– And How to Avoid Them
Thomas J. Chapel, MA, MBAChief Performance Officer (Acting)CDC/Office of the Director/OCOO
Presented November 20, 2008
![Page 3: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/3.jpg)
Objectives
Program evaluation and typical “roadblocks” in doing good evaluation.
CDC’s Evaluation Framework as way to surmount roadblocks.
![Page 4: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/4.jpg)
Key Points
In today’s session we will discuss:
What is important about CDC’s framework?
Why does it lead to better use of findings?
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEPS
StandardsUtility
FeasibilityProprietyAccuracy
![Page 5: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/5.jpg)
Why We Evaluate…
“... The gods condemned Sisyphus to endlessly roll a rock up a hill, whence it would return each time to its starting place.
They thought, with some reason…
![Page 6: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/6.jpg)
Why We Evaluate…
…there was no punishment more severe than eternally futile labor....”
The Myth of Sisyphus
![Page 7: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/7.jpg)
The Problem
The stuff I do doesn't make a difference!
Why don't things get better?!
![Page 8: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/8.jpg)
Implementing Program Evaluation
How do I
motivate?
What gets in the way?
Not this… This…
![Page 9: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/9.jpg)
Today’s Focus
Top Roadblocks on the Road to
Good Evaluation
![Page 10: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/10.jpg)
Defining Evaluation
Evaluation is the systematic investigation of the merit, worth, or significance of any “object”.
Michael Scriven
![Page 11: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/11.jpg)
Use the Findings!
If the findings don’t get used…
the program will not improve.
![Page 12: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/12.jpg)
What is “Evaluation?”
Evaluation is not…
Evaluation is…
A specific set of tools or
techniques.
An orientation to your program.
The idea of continuous reflection.
![Page 13: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/13.jpg)
Defining Evaluation
Evaluation is the systematic investigation of the merit, worth, or significance of any “object”.
Michael Scriven
![Page 14: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/14.jpg)
What is a “Program”?
Not only:Big training programsCommunity interventions
But also:Recommendations and guidelinesSurveillance systems
In other words, a program is anything with an intended outcome.
![Page 15: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/15.jpg)
Roadblock #6
Not understanding where evaluation “fits in” …
![Page 16: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/16.jpg)
The “Silo” Model
![Page 17: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/17.jpg)
The Integrated or “CQI” Model
To achieve “continuous quality improvement” planners, performance measurers, and evaluators must communicate with each other.
![Page 18: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/18.jpg)
The Customer is the KeyProgram evaluation must:
See planning, performance measurement, and evaluation as being integrated.
Start with the idea of having a customer or an intended user of findings.
Direct the evaluation with the customer in mind.
![Page 19: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/19.jpg)
Roadblock #5
Making the “perfect” the enemy of the “good”.
![Page 20: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/20.jpg)
Roadblock #5
What if you said, “To be cardiovascularly fit, you must run a marathon.”?
![Page 21: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/21.jpg)
Thanks, but…
That's not me. I don't have
that expertise.
I don't have those skills.
I don't have the money to
do that.
![Page 22: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/22.jpg)
Do What You Can!
There’s always an evaluation worth doing.
The biggest mistake is doing nothing because you can only do a little.
Even a little bit is going to yield some benefit.
![Page 23: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/23.jpg)
Roadblock #4
Evaluating only what you can “measure”…
… because those are the things we can measure with validity, reliability and accuracy.
![Page 24: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/24.jpg)
Upstream Questions
How many brochures?How many trainees? How many people showed up?
Did we get a lot of product out there?
![Page 25: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/25.jpg)
Downstream Questions
What have you done for me lately?
How has it mattered?
What have you done for public health?
![Page 26: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/26.jpg)
Measuring the Right Thing…
“…Sometimes, what counts can’t be counted. And what can be counted doesn’t count….”
Albert Einstein
![Page 27: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/27.jpg)
Evaluation Starts By Saying…
What are the important things that need to be measured? Can I measure them with enough rigor to meet the needs of this situation this time?
Sometimes the answer is “NO!”
![Page 28: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/28.jpg)
You Get What You Measure…
“…In Poland in the 1970s, furniture factories were rewarded based on pounds of product shipped. As a result, today Poles have the world’s heaviest furniture…”
(New York Times, 3/4/99)
![Page 29: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/29.jpg)
Roadblock #3Neglecting Intermediate Outcomes….
Nothing has advanced the evaluation cause in public health more than preaching
this idea of intermediate outcomes.
![Page 30: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/30.jpg)
Intermediate OutcomesContribute to Downstream Success
How is it that my program will make a contribution to that downstream outcome?
We call these “intermediate outcomes”.
![Page 31: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/31.jpg)
What is the Program Logic?
What needs to happen to achieve the desired outcome?
What is the “program logic”?
My action
Desired outcome
![Page 32: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/32.jpg)
What are the markers that tell me I’m on the right road?
Don’t just ask: Did it work?
How many tomatoes did I get?
![Page 33: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/33.jpg)
What are the markers that tell me I’m on the right road?
Ask: Is it working?
Are planting, watering, and weeding taking place?
Have the blossoms “set”?
Are there nematodes on the plants?
![Page 34: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/34.jpg)
Research Model
Develop Theory
Measure Outcome
Program Activities
![Page 35: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/35.jpg)
Research Model
Develop Theory
Measure Outcome
Program Activities
If I achieved the outcome– great!If I didn’t achieve the outcome– why?
![Page 36: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/36.jpg)
Evaluation Unpacks the “Black Box”
My action
Desired outcome
![Page 37: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/37.jpg)
The World’s BestChildren’s Soccer Program
![Page 38: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/38.jpg)
But We Never Won a Game
![Page 39: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/39.jpg)
Focus on Intermediate Outcomes
Can we:• pass the ball?• spread out?• spend more time
on the opponent’s side of the field?
![Page 40: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/40.jpg)
Forgetting Intermediate Outcomes
Ó ScienceCartoonsPlus.com
![Page 41: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/41.jpg)
What’s In the Box?
My program:• training• technical
assistance• funding• partnerships
Desired outcome:• less morbidity• fewer mortalities
![Page 42: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/42.jpg)
What’s In the Box?
My program:• training• technical
assistance• funding• partnerships
Desired outcome:• less morbidity• fewer mortalities
Intermediate outcomes
![Page 43: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/43.jpg)
The Power of Evaluation
Establishing intermediate outcomes allows you to determine if you are making progress in the right direction.
![Page 44: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/44.jpg)
Diabetes Intermediate Outcomes
![Page 45: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/45.jpg)
Why Intermediate Outcomes?
I’m making progress in the right direction.I am contributing to the downstream outcome.
![Page 46: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/46.jpg)
Identifying Intermediate Outcomes
What is the ultimate outcome I’m seeking?Who (besides me) needs to take action to achieve it?What action do they need to take?
These are the intermediate outcomes that populate the “black box” or the “program logic”.
![Page 47: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/47.jpg)
Roadblock #2
Confusing attribution and contribution…
“I can’t make the case that my program was responsible for that change.”
![Page 48: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/48.jpg)
The Role of Public Health
a mobilizer and convener
Public health is not … Public health is…
a direct deliverer of services
Based on: The Future of Public Health, Institute of Medicine, 1988.
![Page 49: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/49.jpg)
“Networked” Interventions
Agency AProgram A-n
Program A-1
Agency B Program B-1
Agency C
Program C-n
Program C-1
Agency DProgram D-n
Program D-1
OUTPUTS
SHORT-TERMOUTCOMES
LONG-TERMOUTCOMES
SYSTEMOUTCOME
![Page 50: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/50.jpg)
Attribution
Program A-n
Program A-1
Agency B Program B-1
Agency C
Program C-n
Program C-1
Agency DProgram D-n
Program D-1
OUTPUTS
SHORT-TERMOUTCOMES
LONG-TERMOUTCOMES
SYSTEMOUTCOME
Agency A
![Page 51: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/51.jpg)
Contribution
Agency AProgram A-n
Program A-1
Agency B Program B-1
Agency C
Program C-n
Program C-1
Agency DProgram D-n
Program D-1
OUTPUTS
SHORT-TERMOUTCOMES
LONG-TERMOUTCOMES
SYSTEMOUTCOME
![Page 52: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/52.jpg)
Identify Your Contributionsby Asking “Why?”
Why fewer diabetes amputations ?
![Page 53: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/53.jpg)
Identify Your Contributionsby Asking “Why?”
Because physicians are doing more timely foot exams.
Why?
Why fewer diabetes amputations ?
![Page 54: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/54.jpg)
Identify Your Contributionsby Asking “Why?”
Because physicians doing more timely foot exams.
Because the insurance reimbursement climate has changed.
Why?
Why fewer diabetes amputations ?”
![Page 55: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/55.jpg)
Identify Your Contributionsby Asking “Why?”
Because physicians doing more timely foot exams.
Because insurance reimbursement climate has changed.
Because the standards of practice have changed.
Why?
Why fewer diabetes amputations ?”
![Page 56: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/56.jpg)
A “Chain of Causation”
Because:We formed a coalition.We helped incent those standards. We helped change the reimbursement climate.
![Page 57: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/57.jpg)
Contributions Count!
Attribution
We formed a coalition.
Contribution
We helped change the standards.
We helped incent reimbursement.
![Page 58: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/58.jpg)
Establish the “Chain of Causation”
Ask providers. Ask insurance reimbursers. Ask physicians.
Was the coalition influential in making these changes?
![Page 59: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/59.jpg)
Roadblock #1Not asking:
“Who (else) cares…..”
![Page 60: Program Evaluation Webinar Series Part 1:](https://reader035.fdocuments.in/reader035/viewer/2022081604/56816842550346895dde14c2/html5/thumbnails/60.jpg)
Ask the Right Questions
Who matters for this intervention besides me?
Who else needs the information from this evaluation?
The stakeholders!
Engage stakeholders
Step 1