Www.nsp-ie.org. Evaluation Team Andrew Beath (Harvard University) Fotini Christia (M.I.T.) Ruben...
-
Upload
anthony-wilson -
Category
Documents
-
view
219 -
download
1
Transcript of Www.nsp-ie.org. Evaluation Team Andrew Beath (Harvard University) Fotini Christia (M.I.T.) Ruben...
National Solidarity Programme
Phase-II
Impact Evaluation
w w w. n s p - i e . o r g
Interim Estimates from 1st Follow-Up Survey
Evaluation TeamAndrew Beath (Harvard University)Fotini Christia (M.I.T.)Ruben Enikolopov (New Economic School, Moscow)Shahim Kabuli (World Bank)Sakhi Frozish (World Bank)
PartnersVulnerability Analysis Unit (MRRD)AfghanAid, CHA, IC, IRC, NPO/RRAA, Oxfam, PiNNational Solidarity Program (NSP)
FundingNational Solidarity ProgramFood and Agriculture Organisation(FAO)World Bank TFs
National Solidarity Programme
Executed by Ministry of Rural Rehabilitation and Development (MRRD)
Implemented by 28 Facilitating Partners (local and int’l NGOs)
Funded by Multilateral, Bilateral Donors
2003 – 2010: Covered 22,500 communities at a cost of $929 million
Phase-I: 2003-7; Phase-II: 2007-11; Phase-III: 2010+
Create Gender-Balanced Community
Development Councils (CDCs)
through Secret Ballot, Universal Suffrage
Election
Fund Projects Selected by CDCs and Villagers and Managed by CDCs
(Average Grant: $33,000; Max.:
$60,000)
Community Development Council (CDC)
Projects
National Solidarity Programme
Two Principal Village-Level Interventions:
6 months 18-24 months
Water Supply 24%
Roads & Bridges
25%
Irrigation 18%
Electricity 13%
What areas does NSP potentially impact?
Community Development Council (CDC)
Projects
Local Governance
Access to Services
Economic Activity
NSP
Social Cohesion & Conflict
Political Attitudes
Gender
Local Governance
Access to Services
Economic Activity
The evaluation measures impact of NSP-II on . . .
The evaluation estimates these impacts by . . .Social Cohesion &
ConflictGender
Political Attitudes
Similarity of Treatment and Control Villages
Create
CDCs
Treatment
Villages (NSP)
SelectSub-
Projects
Implement Sub-
Projects
Sub-Project
s Finish
ed Control Villages (Non-NSP)
May – Oct. 2009 Spring 2011
Interim Estimates
(18% of Projects Complete at
Survey)
Final Estimates
Baseline
Survey
Aug. – Sep. 2007
1st Follow-Up Survey
2nd Follow-Up Survey
The evaluation estimates these impacts by . . .collecting data over 3½ years in 500
villages: 250 NSP (treatment) & 250 non-NSP
(control)Structure of Evaluation and Data Collection
The evaluation uses this data to . . .
Baseline
Survey
1st Follow-
Up Survey
1st Follow-
Up Survey
Change in Treatment
Villages
Change in Control Villages
Baseline
Survey
Treatment
Villages (NSP)
Control Villages (Non-NSP)
1st Follow-Up Survey
The evaluation uses this data to . . .compare changes in treatment villages (NSP) with changes in control villages
(non-NSP)
-- =
=
- = Impact of NSP=-
Estimation of Impacts of NSP
Baseline
Survey
Differences estimation is accurate because . . .
Difference-in-Difference EstimatesDifference EstimatesIf treatment and control villages are
identical at baseline . . .
Control Group (Non-NSP)
Treatment Group (NSP)
Differences estimation is accurate because . . .250 treatment villages selected
randomly from 500 surveyed villages - other villages to control group
study is a randomized control trialVillage AVillage BPartitioned Randomization: FPs
denoted 15 villages to be excluded from randomization and evaluation
Partitioned Randomization & Contractual Embedding Improved Chance of
Successful Randomization
Due to funding constraints and lack of village-level data, randomization was
fairest way to decide which villages received NSP
Randomization embedded in FP contracts
• Local Governance• Access to Services• Economic Activity
• Gender
Interim Estimates ofProgram Impact
• Social Cohesion & Conflict
• Political Attitudes
Local GovernanceCreates Functional Councils which
Assume Authority over Different Aspects of Local Governance
Creates Avenues for Women’s Participation;
Increases Responsiveness to Women’s Needs;
Increases Satisfaction of Women with Village Leaders
Increases Participation by Men in Local Governance;
No Evidence of Impact on Men’s Satisfaction with Village Leadership
Access to ServicesIncreases Access to Safe Drinking Water
Increases Access to Electricity
Mixed Evidence of Impact on Medical Care
No Evidence of Impact on Boys’ Schooling (but
Possible Increase in Girls’ Enrolment Rates)
Weak Impact on Transportation (at this stage)
Economic Activity
Improves Perceptions of Economic Situation and Optimism
No Evidence (at this stage) of Impact on Household Income, Consumption, Assets or Borrowing
Limited Impacts on Production
Social Cohesion & Conflict
No Evidence of Impact on Disputes & Feuds
No Evidence of Impact on Attacks, but Improves Perceptions of Security
Decreases Out-Migration of Households
No Evidence of Impact on Trust among Villagers
GenderIncreases Involvement in Economic
Activities, but Doesn’t Affect Involvement in Decision-Making
Increases Inter-Village Connections and Availability of Support Groups
Increases Respect for Women and Acceptance of Female Participation in Local Governance
Decreases Incidence of Extreme Unhappiness among Women
Political Attitudes
Connects Villages with Central Government
Improves Perceptions of Government & NGOs
No Evidence of Impact on Acceptance of Government Jurisdiction, but Increases Support for Electing Local Leaders
Presiden
t
Provin
cial G
overn
or
Distric
t Administ
rator
Mem
bers of P
arliam
ent
Centra
l Gov't
Officia
ls
Governmen
t Judge
sPolic
e
NGO Worke
rs
ISAF /
US A
rmy S
oldiers
76%65%
59%50%
64%
46%
71%64%
26%
Presiden
t
Provin
cial G
overn
or
Distric
t Administ
rator
Mem
bers of P
arliam
ent
Centra
l Gov't
Officia
ls
Governmen
t Judge
sPolic
e
NGO Worke
rs
ISAF /
US A
rmy S
oldiers
80%71%
65%56%
69%
51%
73% 68%
29%
+ 4%
Perceptions of Gov’t, Civil Society, and Military
+ 6%+ 7%
+ 5%
+ 5%
+ 5%
+ 4%
+ 3%
Male Villager Believes Official Works for the Benefit of All Villagers
PoliticalAttitudes
GenderSocial
Cohesion Economic
ActivityAccess
to ServicesLocal
Governance
Community Development Council (CDC) Election Type
Method of Sub-ProjectSelection
Sub-Treatment Interventions (STIs)
STIs test different implementation strategies or changes in program design
STIs provide real-time evidence-based feedback on how to
improve program effectiveness
NSP-II impact evaluation incorporates two STIs which test changes in two program
components:
Test of Two Different Types of CDC Elections
Neighborhood
Election
Village Election
Treatment Village ATreatment Village B125
Villages
125 Villag
es
250 Treatme
nt Villages
Neighborhood Election: One male & one female from each neighborhood elected to CDC
Village Election: Highest male & female vote-getters in entire village elected to CDC
→ Guarantees representation
→ Preferred candidates are elected
Test of Two Different Types of CDC Elections
250 treatment villages randomly assigned to elect CDC either by neighborhood or village election
Effect of Different Types of CDC Elections
82%
48%
67%72% 75%
97%
81%
64%
51%
95%
60%
85%Neighborhood Election
Village Election
Village Elections (compared to Neighborhood Elections) increase
electoral competitiveness
Effect of Different Types of CDC Elections
Test of Two Different Types of Project Selection
. . . on Electoral Competitiveness
Consultation Meeting
Secret Ballot
Referendum
125 Villag
es
125 Villag
es
250 Treatme
nt Villages
Consultation Meeting: Villagers hold meeting and decide collectively which project is best
Secret Ballot Referendum: Villagers select projects through selecting preferred project
→ consensus-based
→ directly democratic
Test of Two Different Types of Project Selection
250 treatment villages randomly assigned to select project either
by consultation meeting or secret-ballot referendum
Impact of Different Types of Project Selection
Demographic Sub-Project Selection Method
Stage of Sub-Project Selection
Proposal Selection Prioritization
Male VillagersMeeting ~ + +
Referendum ~ + +Village Leaders
Meeting + + +Referendum ~ ~ ~
Female Villages
Meeting ~ ~ ~Referendum ~ ~ ~
Preferences of male villagers influence selection and prioritization in both
meetings and referendaPreferences of village leaders influence
proposals, selection, and prioritization in meetings, but not referenda
Preferences of female villages do not influence proposals, selection, or
prioritization
Preferences of village leaders more able to influence selection in consultation meetings (but influence of villagers in
affecting selection is not affected)
Baseline Survey asked male villagers, female villagers, and village leaders which village projects they preferred Obtained information from FPs on which projects were proposed, selected, and prioritized for implementationCompared projects preferred by different groups with projects proposed, selected, and prioritized to find out who is influential in selection processCompare influence of different groups under two selection processes to find out how selection type affects elite capture
Impact of Different Types of Project Selection
Test of Election / Project Selection Interactions
. . . on Elite Capture of Project Selection
250 Treatme
nt Villages Consultatio
n Meeting
Secret Ballot
Referendum
125 Villag
es125
Villages
250 Treatme
nt Villages
Neighborhood
ElectionVillage
Election
125 Villag
es125
Villages
250 Treatme
nt Villages
62 Villag
es
63 Villag
es
63 Villag
es
62 Villag
es
Neighborhood
Election & Referendu
mVillage Election & Consultation Meeting
Village Election & Referendu
m
Randomization of Election
Type
Randomization of Project
Selection Type
Randomization of Election Type and Project Selection Type Done Separately
Four Randomly-Assigned Combinations of Election and Selection Type
Test of Election / Project Selection Interactions
Use to Test Interactions between Election and Selection Type
Neighborhood
Election & Consultation Meeting
Instrument Combination Proposal Selection Prioritization
Male Villager
Neighborhood Election / Meeting ~ + +Neighborhood Election / Referendum ~ + ~
Village Election / Meeting ~ + ~Village Election / Referendum ~ ~ +
Difference between Types ~ ~ ~
Village Leaders
Neighborhood Election / Meeting ~ ~ ~Neighborhood Election / Referendum ~ ~ ~
Village Election / Meeting + + +Village Election / Referendum ~ ~ ~
Difference between Types + + +
Male villagers influence selection and prioritization
Combinations of election and referendum type do not affect influence of male villagers
Impact of Combinations on Selection Outcomes
Villager leaders influence proposals, selection, and prioritization when village elections are
combined with consultation meetings
Probability of Elite Capture Maximized by Combination of Village Elections with
Consultation Meetings
. . . on Elite Capture of Project Selection
Problems with SolutionsCDD is interesting b/c it can change behaviors, attitudes, institutions, social cohesion etc.
Very difficult to measure well
‘Parrot Bias’ in Surveys: Does program really change attitudes and behaviors or does it just responses to survey questions?→ measure actual behaviors rather than simply asking questions
Randomization: theoretically easy, practically difficult→ tweak procedure to be resistant to pressures & communicate→ successful quasi-experimental eval. better than failed RCT
Prospective Evaluations take ages . . . → STIs can provide (quicker) real-time feedback to program
→ invest in extensive pilot-testing and consult widely
→ Manage expectations and communicate . . .
Issues That RemainIndependence of Researchers vs. Accountability of Evaluations to Programs, Implementers, and Donors
Importance of Non-Results vs. Lack of Interest and Lack of Incentives for Researchers or Program to DisseminateSmall proportion of programs are subjected to prospective IEs. How do we ensure the evaluated ones aren’t punished for non- or bad results?
Who will synthesize IE results for policy-makers and make sure the results are used in policy decisions and program design?Who will coordinate researchers to make sure evaluations address questions useful for policy decisions and program design?