Making Impact Evaluations Happen World Bank Operational Experience 6 th European Conference on...
-
date post
21-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of Making Impact Evaluations Happen World Bank Operational Experience 6 th European Conference on...
Making Impact Evaluations Happen World Bank
Operational Experience
6th European Conference onEvaluation of Cohesion Policy
30 November 2009Warsaw
Joost de Laat and Kaspar Richter
World Bank’s Result Agenda
• Results-based approach to ensure that WB contributes to improved country outcomes• Demand for evidence of the results of development
assistance is increasing – limited resources, donor fatigue, client expectations
• Multiple policy options to address needs• Rigorous evidence often lacking to prioritize policy options
• Among monitoring and evaluation techniques, impact evaluation provides an important tool to show the effect of interventions
• Given the power of this tool, World Bank increasing number of impact evaluations
World Bank’s Development Impact Evaluation Initiative
• Objectives:• To increase number of Bank projects with impact
evaluation components• To increase staff capacity to design and carry out such
evaluations• To build a process of systematic learning based on
effective development interventions with lessons learned from completed evaluations
• 12 Clusters:• Conditional Cash Transfers, Early Childhood
Development, Education Service Delivery, HIV/AIDS Treatment and Prevention, Local Development, Malaria Control, Pay-for-Performance in Health, Rural Roads, Rural Electrification, Urban Upgrading, ALMP and Youth Employment
Figure 1: World Bank Impact Evalautions, by year and status
2837
80
13
28
41
0
20
40
60
80
100
120
140
Before 2004 After 2004 Current (Ongoing*)
Nu
mb
er o
f Im
pac
t E
valu
atio
ns
Non-Bank Projects
Bank Projects
Impact evaluation differs from M&E
INPUTS OUTCOMESOUTPUTS
MONITOR EFFICIENCY
EVALUATE EFFECTIVENESS
$$$, activities
BEHAVIORBEHAVIOR
Impact Evaluation Informs
Strategy
Whether we are doing the right thingsRationale/justificationClear theory of change
Operatio
n
Whether we are doing things right Effectiveness in achieving expected outcomes Efficiency in optimizing resources Client satisfaction
Learning
Whether there are better ways of doing itAlternativesBest practicesLessons learned
Impact Evaluation Methods
• Experimental methods/Randomization• Quasi-experimental methods
• Propensity score matching (PSM)• Regression discontinuity design (RDD)
• Other Econometric methods • Before and After (Reflexive comparisons)• Difference in Difference (Dif in Dif)• Instrumental variables • Encouragement design
Global Growth Industry - Ongoing Randomized Impact Evaluations
From MIT Poverty Action Lab Website (2009)
IE are easier in some sectors than in others
World Bank Lending (Annual average FY05-FY07) by sector
Health and Other Social Services
14%
Finance14%
Agriculture, Fishing, and Forestry
15%
Industry and Trade; Infromation and
Communication; Law and Justice and Public
Administration18%
Energy and Mining, Transport, Water, Urban
Upgrading18%
Education21%
Impact Evaluations by Bank Sector
Education29%
Health and Other Social Services
39%
Agriculture, Fishing, and Forestry
5%
Energy and Mining, Transport, Water, Urban
Upgrading17%
Finance8%
Industry and Communication; Law and Justice and Public
Administration2%
Randomization in Infrastructure?
• Very hard to do this mainly due to engineering constrains• Unit of observation are often communities rather than HHs• Self selection: local communities have to be eligible,
prepare a project, apply for funds, and commit some project value (in kind and cash);
• Some projects already ongoing and the government has no capacity to start everywhere at the same time
10 Steps to Making Impact Evaluations Relevant for Practitioners
Make policy question the starting point Take seriously ethical objections and political
sensitivities Take comprehensive approach to sources of
bias Look for spillover effects Take a sectoral approach Look for impact heterogeneity Take scaling up seriously Understand what determines the impact Don’t reject theory Develop within-country capacity
Active Labor Market Programs – DIME Examples
Financial Support: WB Spanish Impact Evaluation Fund Funded by Spain (€10.4 mn) and UK (€1 mn)
Active Labor Markets and Youth Employment Country Budget
National Rural Employment Guarantee Evaluation (NREG)
India € 261,745
Steps to Work Jamaica € 268,456
Youth Development Project (YDP) DR € 167,534
“First Employment” (Mi Primer Empleo) Program
Honduras € 200,430
Northern Uganda Social Action Fund (NUSAF) Uganda € 134,134
Steps for ALMP Impact Evaluation• Undertake prior quantitative analysis to identify priority areas
• Skills are particularly low among which (age, ethnic, etc.) groups?• Unemployment is particularly high among which groups?
• Undertake qualitative analysis that may answer the why questions (why is unemployment high among certain groups?). Inquire with Government, Employers, Employees, Unemployed
• Design a pilot and evaluate impact before scale-up• Select one group to receive treatment (job training, counseling etc.)• Find a comparison group to obtain counterfactual
• Treatment & comparison groups with identical initial characteristics so that only difference is the ALMP
• Hence, differences in unemployment rate arise only due to ALMP• Collect baseline data
• can ensure proper targeting of ALMP• allows verification that treatment and comparison groups are
statistically identical prior to intervention• enables ex-post evaluation of heterogeneous program effects (i.e.
was the job training program more effective among certain types of subgroups?)
• Implement and monitor outcomes in treatment & comparison groups
Evidence from Randomized IE of ALMPsIbarrarán and Shady (2008)
• Considerable heterogeneity with none to modest employment impacts overall.
• Considerable heterogeneity with substantial employment impacts effects on some subgroups (e.g. women, adults) but not others.
• European evidence is far more uncertain in part because of the lack of experimental studies and the wide variation in evaluation methods.
• Given the considerable heterogeneity, it is important to pilot and evaluate.
Some Resources on Impact Evaluations
• www.worldbank.org/sief
• www.worldbank.org/dime
• www.worldbank.org/impactevaluation
• http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evaluation_en.htm
• www.povertyactionlab.org
• http://evidencebasedprograms.org/
• “Using Randomization in Development Economics Research: A Toolkit” (2006). By: E. Duflo, M. Kremer and R. Glennerster
(At: www.povertyactionlab.org/research/rand.php )• “Institutionalizing Impact Evaluation Within the Framework of a
Monitoring and Evaluation System” (2009): By: World Bank(At: www.worldbank.org/ieg/ecd/docs/inst_ie_framework_me.pdf )
What we need for an IE
1. The difference in outcomes with the program versus without the program – for the same unit of analysis (e.g. individual, community etc.)
2. Problem: individuals only have one existence
3. Hence, we have a problem of a missing counter-factual, a problem of missing data