Post on 07-Jul-2018
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
1/26
Why Evaluate? Gemma Stevenson
Associate Director, CERP
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
2/26
Quiz: Why Evaluate?a) To work out the best design of a programme
b) To identify which programme delivers a desired outcome
c) To determine how to reach a desired outcome in the most cost-effective way
d) To be accountable to those paying for the programme (donors / tax
payers / etc.)
e) To settle an argument; show the impact of a program to skeptics or showthe inefficacy of a program to believers
f) All of the above
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
3/26
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
4/26
Aid optimists“I have identified the specificinvestments that are needed [toend poverty]; found ways to planand implement them; [and] shownthat they can be affordable.”
Jeffrey Sachs
End of Poverty
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
5/26
Aid pessimists
5
“After $2.3 trillion over 5 decades, why are the desperate needs of the world's poor still so tragically unmet?
Isn't it finally time for an end to theimpunity of foreign aid?”
Bill Easterly The White Man’s Burden
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
6/26
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
7/26
Debates surrounding development projects! Impact:
• Do we agree on what the impact should be?• Can this impact be measured?• How do we know if a programme is effective?
! Prioritize:• How should we allocate/prioritise a limited budget?
! Benefit-Cost Calculations:• What is the cost-effectiveness of a program relative to other programmes?
! What role can evidence play in policymaking?• What is the most effective way to design a programme?• How to generate short and long run success?
Can these questions be resolved?
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
8/26
Needsassessment
Programdesign - withM&Esystem
embedded
Pre-programlogistics/Program
roll-out
Processevaluation/
Monitorprogresstowardstarget
Learning&
ongoingimprove-
ments
Reportingfindings -impact,progress
Usingfindings
toimproveprogram
design &delivery
Yes they can!By embedding M&E in policy-making
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
9/26
Needsassessment
Programdesign - withM&Esystem
embedded
Pre-programlogistics/Programroll-out
Processevaluation/
Monitorprogresstowardstarget
Learning&
ongoingimprove-
ments
Reportingfindings -impact,progress
Usingfindings
toimproveprogramdesign &
delivery
Highlight no.1 Do we agree on what the impactshould be?
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
10/26
Needsassessment
Programdesign forimplemen
tation
Pre-programlogistics/Programroll-out
Processevaluation/
Monitorprogress
towardstarget
Learning&
ongoingimprove-
ments
Reportingfindings -impact,
progress
Usingfindings
toimproveprogramdesign &delivery
Highlight no.2
Use Monitoring and evaluation to continually improve program design
What is the most effective way todesign a program?
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
11/26
Example: Property Tax Project! Tax systems throughout the developing world have poor performance
! Problems with tax inspectors (assess and collect taxes)• Corruption: tax inspector and taxpayer collude to reduce taxpayer liability• Effort: weak incentives for tax collectors lead to low effort and tax evasion
! Historically, problem solved by high-powered incentives (tax farming) for theseofficials
! But concerns about over-taxation and extortion led to a move away from high-powered incentives and towards balanced compensation for tax staff
! Project: Design optimal incentive schemes to improve tax collection and customerservice by CERP researchers engaging in a rigorous collaboration with the Punjab,Pakistan property tax department
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
12/26
Process Evaluation! Is the program implemented as planned?
• E.g.; tax collector incentive schemes in place & understood
! Are basic tasks being completed?• E.g.; tax collection data being regularly reported
! Is the intervention reaching the target population?• E.g.; tax inspectors, constables, and clerks informed and engaged
! Are the services being delivered?• E.g.; Challans issued, payments collected etc.
! Is the intervention being completed well or efficiently and to the beneficiaries’satisfaction?
• E.g.; Payments made on time and on the agreed on criteria
12
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
13/26
Evaluation design! The program happened, how did it change lives?
o Tax staff rewarded for efforto Taxpayer paid due and fairly treated
! Did we achieve our goals?o Tax collections improved
o Taxpayer satisfaction/service quality raised
! Also distributional questions such as, did the program benefit somemore than others?
o Some tax staff did bettero Some taxpayers benefitted moreo Collections and service improvements varied across tax circles/regions
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
14/26
Policy design! Through (real-time) testing and evaluating different
iterations of a program, you can compare theimpact levels to find the optimal design of the
program
! The evaluation design should be developed inconjunction with the design of the intervention
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
15/26
Needsassessment
Programdesign - withM&Esystem
embedded
Pre-programlogistics/Programroll-out
Processevaluation/
Monitorprogresstowardstarget
Learning&
ongoingimprove-
ments
Reportingfindings -impact,progress
Usingfindings
toimproveprogramdesign &
delivery
Pre-emptive M&E
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
16/26
What is Impact?! IMPACT
• What happened because of the program?
• What would have happened in the absence of theprogram? ( counterfactual )
! Take the difference between what happened (with the program) …and
- what would have happened (without the program)= IMPACT of the program
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
17/26
Intervention
Time
P r i m a r y o u t c o m e
Counterfactual
Impact
What is Impact?
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
18/26
Intervention
Time
P r i m a r y
O u t c o m e
Counterfactual
Impact
Beware!
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
19/26
Intervention
Time
P r i m a r y
O u t c o m e
Without the counterfactual
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
20/26
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
21/26
Intervention
Time
P r i m a r y o u t c o m e
Without the counterfactual
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
22/26
How to evaluate?! For any policy question of interest, need to be able
to construct a plausible counterfactual
! Different ways to construct the counterfactual:• Pre-post• Simple Difference• Difference-in-Difference• Randomization
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
23/26
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
24/26
Evaluate for cost-effectiveness
! Only once you accurately measure the impact of aprogram can you accurately calculate cost-
effectiveness
! Though a comparative cost-effectiveness analysis,
you can find the least expensive way of achieving adesired outcome
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
25/26
Comparative cost effectiveness
8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final
26/26
So why evaluate?! For more focused and effective programme design! For understanding what works and what doesn’t
work in a programme! For determining the impact of a programme! For understanding how far a programme is cost-
effective