Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez...
-
Upload
lionel-ray -
Category
Documents
-
view
214 -
download
0
Transcript of Setting the framework: Designing interventions to achieve impact Esteban Diez Roux Patricia Yanez...
Setting the framework: Designing interventions to achieve impact
Esteban Diez RouxPatricia Yanez PagansInter-American Development Bank
June 22nd 2015
Setting the framework: Designing
interventions to achieve impact
• Impact Evaluations and the Transport Sector
• Main Issues in Designing IEs for Transport
Effectiveness: What works and what doesn’t work?
Why do we evaluate?
Efficiency: Most efficient way to achieve an outcome?
Evidence-based policy making
What do we want to achieve with our projects?Is it more than reducing transportation costs?Traditional outcomes in transport:– Travel times– Vehicle operating costs
Non-traditional outcomes? – New development agenda– EI’s could be key to understand our impact on
development – But…how far can we get with our investments?– Attribution critical
How can we design projects that are more effective in achieving these non-traditional outcomes and impacts?
First step…
Understand what works and what doesn’t work
How?Impact evaluation!
Attribution: Let’s be careful in getting the right answers
• Traditional approaches in transport:
– Before-and-after methods– Ex-post cost-benefit analysis
• Limitations of these methods?
Impact evaluations complement other approaches
They complement process evaluations and ex-post cost benefit analyses
However, different evaluation instruments answer slightly different questions
8
Process evaluation
Did we achieve all products at the planned costs ?Did outcome indicators move in the right direction?
Ex-post cost-benefit analysis
Did project economic benefits exceed project costs?
Impact evaluation
What would be the situation of beneficiaries without the project?
Learning, accountability, and complementarities
9
Process evaluation
Generates information about costs and trends in indicators. Helps for project accountability.
Ex–post cost-benefit analysisExtracts information of costs and benefits from monitoring. Ideally extracts benefits information from IE. Justifies investments based on their economic return.
Impact evaluationMainly for learning, helps to understand what strategies are more effective. Provides information on actual benefits for economic analysis.
Using instruments in isolation might leave important open questions
Process evaluation
Are there any other external factors that influenced results?
Ex-post cost-benefit analysis
What are the real costs and benefits of the project?
Impact evaluation
Is the program cost-effective?
Setting the framework: Designing
interventions to achieve impact
• Impact Evaluations and the Transport Sector
• Main Issues in Designing IEs for Transport
Designing impact evaluations (IEs) for the transport sector
• Causality or attribution• Mechanisms• Spillovers
=> Need a clear understanding of the theory of change
The theory of change is key for the design of projects and for IEs
Resources at the disposal of the project
Actions taken to convert inputs into outputs
Tangible goods and services project activities produce
Results likely to be achieved once beneficiaries use outputs
Final project goals
StaffBudget
ConstructionRehabilitation
Km of roads paved
Reduction in travel timesReduction in operation costs
Poverty reduction
Inputs Activities Outputs Outcomes Impacts
=> IE questions?
Impact evaluation needed where program attribution is unclear
Resources at the disposal of the project
Actions taken to convert inputs into outputs
Tangible goods and services project activities produce
Results likely to be achieved once beneficiaries use outputs
Final project goals
StaffBudget
ConstructionRehabilitation
Km of roads pavedKm of new roads
Reduction in travel timesOperation costsIncreased mobility?
Poverty reductionIncreased productivity
Under project control
Inputs Activities Outputs Outcomes Impacts
Household level outcomes
Firm level outcomes
Community/Regional level outcomes
What have we learned about non-traditional outcomes in transport?
Picture: Horia Varlan
Some aspects we have learned about household-level outcomes
Road improvement increases…
Household investments in durable assets (Mexico - Gonzales-Navarro and Quintana-Domeneque, 2012)
Changes household production decisions(Peru - Valdivia, 2009)
Use of health and education services (Bangladesh - Khandker et al., 2009)
Some aspects we have learned about firm-level outcomes
Expansion rail system
Changes hiring decisions of firms located close to stations (US - Holzer et al. 2003)
Improved highways
Helps firms reduce stock of input inventories and re-optimize suppliers choice (India - Datta, 2012)
Some aspects we have learned about community & regional level outcomes
Rehabilitation of rural roads
Reduces local prices (Sierra Leone - Casaburi et al., 2013)
Railway construction
Increases wage gaps through increased specialization (Central Europe - Walker, 2015)
Electronic toll collection
Reduces vehicle emissions improving overall child health (US -Currie and Walker, 2011)
Attribution: When compared to other instruments only impact evaluation seeks for causality
Process evaluationNO attribution
Ex-post cost-benefit analysisNO attribution
Impact evaluationYES attribution
How do we seek for causality with an impact evaluation?
What would have happened to our beneficiaries in the absence of the
program?
=> Find the right counterfactual (comparison group)
• Roads and public transportation systems are not randomly placed
• Benefits are difussed, implications on how we define the zone of intervention
• Impacts may take a long time to emerge
IEs in transport can be challenging…
Mechanisms: We usually assume that roads reduce poverty, but how?
Povertyreduction
New employment opportunities?
Market prices?
Increased production?
Access to public services?
Picture by: Klaus Leidorf
Others?
Mechanisms: Distribution of impacts and complementarities
• Are benefits different for men and women?
• Are benefits different for poor and less poor populations?
• How do transport investments interact with other interventions?
Picture: Bob Miller
Picture: Bob Miller
Spillovers: Are we having positive/negative impacts on non-beneficiaries?
Picture: Rakesh Rocky
Spillovers 1: Project externalities
Benefits or costs that are transferred directly from beneficiaries to non-beneficiaries
• Positive? Reducing travel times for users of BRT systems may reduce travel time of non-users in less congested roads
• Negative? Relocation of economic activity closer to highways may have negative impacts on areas further away
Spillovers 2: General equilibrium effects
Program effects are transmitted to the local economy
• Rehabilitation of rural roads reduces local prices (Sierra Leone - Casaburi et al., 2013)
• Railway construction reduces price gaps between regions, but increased specialization increases wage gaps (Central Europe - Walker, 2015)
Spillovers 3: Interactions
Non-beneficiaries may be affected by the program through social and economic
interactions with beneficiaries
• Road safety education programs might have impacts beyond those that receive education
• ICT messages that promote savings might affect the savings decisions of others (friends, family, neighbors)
Spillovers 4: Behavioral effects
Program affects behavioral or social norms within the area of the intervention affecting
both beneficiaries and non-beneficiaries.
• Awareness campaigns to promote safe driving in some neighborhoods might change social norms in a city
• ICTs to promote transparency in some governments offices might promote a general culture of transparency
How to successfully implement impact evaluations?
• Lots of discussion and consensus between key stakeholders
• Continued financial support • Strong local capacity for monitoring and
implementation• Local ownership to answer relevant questions and
guarantee success
Picture: Steve Davis
Key messages
1) Find the right ways to identify causality
2) Design IEs that allow understanding mechanisms and spillovers
3) Work together to guarantee success
Let’s start!
THANK YOU