National Solidarity Programme

Phase-IIImpact Evaluation

w w w. n s p - i e . o r gInterim Estimates from 1st Follow-Up

Survey

Evaluation TeamAndrew Beath (Harvard University)Fotini Christia (M.I.T.)Ruben Enikolopov (New Economic School, Moscow)Shahim Kabuli (World Bank)Sakhi Frozish (World Bank)

PartnersVulnerability Analysis Unit (MRRD)AfghanAid, CHA, IC, IRC, NPO/RRAA, Oxfam, PiNNational Solidarity Program (NSP)

FundingNational Solidarity ProgramFood and Agriculture Organisation(FAO)World Bank TFs

Background & Methodology

National Solidarity ProgrammeExecuted by Ministry of Rural Rehabilitation and Development (MRRD)

Implemented by 28 Facilitating Partners (local and int’l NGOs)

Funded by Multilateral, Bilateral Donors

2003 – 2010: Covered 22,500 communities at a cost of $929 million

Phase-I: 2003-7; Phase-II: 2007-11; Phase-III: 2010+

Create Gender-Balanced Community

Development Councils (CDCs)

through Secret Ballot, Universal Suffrage

Election

Fund Projects Selected by CDCs and Villagers and Managed by CDCs

(Average Grant: $33,000; Max.:

$60,000)

Community Development Council (CDC)

Projects

National Solidarity Programme

Two Principal Village-Level Interventions:

6 months 18-24 months

Water Supply 24%

Roads & Bridges

25%

Irrigation 18%

Electricity 13%

What areas does NSP potentially impact?

Community Development Council (CDC)

Projects

Local Governance

Access to Services

Economic Activity

NSP

Social Cohesion & Conflict

Political Attitudes

Gender

Local Governance

Access to Services

Economic Activity

The evaluation measures impact of NSP-II on . . .

The evaluation estimates these impacts by . . .Social Cohesion &

ConflictGender

Political Attitudes

Similarity of Treatment and Control Villages

Create

CDCs

Treatment

Villages (NSP)

SelectSub-

Projects

Implement Sub-

Projects

Sub-Project

s Finish

ed Control Villages (Non-NSP)

May – Oct. 2009 Spring 2011

Interim Estimates

(18% of Projects Complete at

Survey)

Final Estimates

Baseline

Survey

Aug. – Sep. 2007

1st Follow-Up Survey

2nd Follow-Up Survey

The evaluation estimates these impacts by . . .collecting data over 3½ years in 500

villages: 250 NSP (treatment) & 250 non-NSP

(control)Structure of Evaluation and Data Collection

The evaluation uses this data to . . .

Baseline

Survey

1st Follow-

Up Survey

1st Follow-

Up Survey

Change in Treatment

Villages

Change in Control Villages

Baseline

Survey

Treatment

Villages (NSP)

Control Villages (Non-NSP)

1st Follow-Up Survey

The evaluation uses this data to . . .compare changes in treatment villages (NSP) with changes in control villages

(non-NSP)

-- =

= - = Impact of NSP=-

Estimation of Impacts of NSP

Baseline

Survey

Differences estimation is accurate because . . .

Difference-in-Difference EstimatesDifference EstimatesIf treatment and control villages are identical at baseline . . .

Control Group (Non-NSP)

Treatment Group (NSP)

Differences estimation is accurate because . . .250 treatment villages selected

randomly from 500 surveyed villages - other villages to control group

study is a randomized control trialVillage AVillage BPartitioned Randomization: FPs

denoted 15 villages to be excluded from randomization and evaluation

Partitioned Randomization & Contractual Embedding Improved Chance of

Successful Randomization

Due to funding constraints and lack of village-level data, randomization was

fairest way to decide which villages received NSP

Randomization embedded in FP contracts

• Local Governance• Access to Services• Economic Activity

• Gender

Interim Estimates ofProgram Impact

• Social Cohesion & Conflict

• Political Attitudes

Local GovernanceCreates Functional Councils which

Assume Authority over Different Aspects of Local Governance

Creates Avenues for Women’s Participation;

Increases Responsiveness to Women’s Needs;

Increases Satisfaction of Women with Village Leaders

Increases Participation by Men in Local Governance;

No Evidence of Impact on Men’s Satisfaction with Village Leadership

Access to ServicesIncreases Access to Safe Drinking Water

Increases Access to Electricity

Mixed Evidence of Impact on Medical Care

No Evidence of Impact on Boys’ Schooling (but

Possible Increase in Girls’ Enrolment Rates)

Weak Impact on Transportation (at this stage)

Economic ActivityImproves Perceptions of Economic

Situation and Optimism

No Evidence (at this stage) of Impact on Household Income, Consumption, Assets or Borrowing

Limited Impacts on Production

Social Cohesion & Conflict

No Evidence of Impact on Disputes & Feuds

No Evidence of Impact on Attacks, but Improves Perceptions of Security

Decreases Out-Migration of Households

No Evidence of Impact on Trust among Villagers

GenderIncreases Involvement in Economic

Activities, but Doesn’t Affect Involvement in Decision-Making

Increases Inter-Village Connections and Availability of Support Groups

Increases Respect for Women and Acceptance of Female Participation in Local Governance

Decreases Incidence of Extreme Unhappiness among Women

Political Attitudes

Connects Villages with Central Government

Improves Perceptions of Government & NGOs

No Evidence of Impact on Acceptance of Government Jurisdiction, but Increases Support for Electing Local Leaders

Presiden

t

Provincia

l Gove

rnor

Distric

t Administ

rator

Members

of Parl

iamen

t

Centra

l Gov't

Officials

Governmen

t Judge

sPolice

NGO Worke

rs

ISAF /

US Arm

y Soldier

s

76%65%

59%50%

64%

46%

71%64%

26%

Presiden

t

Provincia

l Gove

rnor

Distric

t Administ

rator

Members

of Parl

iamen

t

Centra

l Gov't

Officials

Governmen

t Judge

sPolice

NGO Worke

rs

ISAF /

US Arm

y Soldier

s

80%71%

65%56%

69%

51%

73% 68%

29%

+ 4%

Perceptions of Gov’t, Civil Society, and Military

+ 6%+ 7%

+ 5%

+ 5%

+ 5%

+ 4%

+ 3%

Male Villager Believes Official Works for the Benefit of All Villagers

PoliticalAttitudesGenderSocial

Cohesion Economic

ActivityAccess

to ServicesLocal

Governance

Sub-Treatment Interventions

Community Development Council (CDC) Election Type

Method of Sub-ProjectSelection

Sub-Treatment Interventions (STIs)

STIs test different implementation strategies or changes in program design

STIs provide real-time evidence-based feedback on how to

improve program effectiveness

NSP-II impact evaluation incorporates two STIs which test changes in two program

components:

Test of Two Different Types of CDC Elections

Neighborhood

Election

Village Election

Treatment Village ATreatment Village B125

Villages

125 Villag

es

250 Treatme

nt Villages

Neighborhood Election: One male & one female from each neighborhood elected to CDC

Village Election: Highest male & female vote-getters in entire village elected to CDC

→ Guarantees representation

→ Preferred candidates are elected

Test of Two Different Types of CDC Elections

250 treatment villages randomly assigned to elect CDC either by neighborhood or village election

Effect of Different Types of CDC Elections

82%

48%

67% 72% 75%

97%

81%

64%51%

95%

60%

85%Neighborhood Election

Village Election

Village Elections (compared to Neighborhood Elections) increase

electoral competitiveness

Effect of Different Types of CDC Elections

Test of Two Different Types of Project Selection

. . . on Electoral Competitiveness

Consultation Meeting

Secret Ballot

Referendum

125 Villag

es

125 Villag

es

250 Treatme

nt Villages

Consultation Meeting: Villagers hold meeting and decide collectively which project is best

Secret Ballot Referendum: Villagers select projects through selecting preferred project

→ consensus-based

→ directly democratic

Test of Two Different Types of Project Selection

250 treatment villages randomly assigned to select project either

by consultation meeting or secret-ballot referendum

Impact of Different Types of Project Selection

Demographic Sub-Project Selection Method

Stage of Sub-Project Selection

Proposal Selection Prioritization

Male VillagersMeeting ~ + +

Referendum ~ + +Village Leaders

Meeting + + +Referendum ~ ~ ~

Female Villages

Meeting ~ ~ ~Referendum ~ ~ ~

Preferences of male villagers influence selection and prioritization in both

meetings and referendaPreferences of village leaders influence

proposals, selection, and prioritization in meetings, but not referenda

Preferences of female villages do not influence proposals, selection, or

prioritization

Preferences of village leaders more able to influence selection in consultation meetings (but influence of villagers in

affecting selection is not affected)

Baseline Survey asked male villagers, female villagers, and village leaders which village projects they preferred Obtained information from FPs on which projects were proposed, selected, and prioritized for implementationCompared projects preferred by different groups with projects proposed, selected, and prioritized to find out who is influential in selection processCompare influence of different groups under two selection processes to find out how selection type affects elite capture

Impact of Different Types of Project Selection

Test of Election / Project Selection Interactions

. . . on Elite Capture of Project Selection

250 Treatme

nt Villages Consultatio

n Meeting

Secret Ballot

Referendum

125 Villag

es125

Villages

250 Treatme

nt Villages

Neighborhood

ElectionVillage

Election

125 Villag

es125

Villages

250 Treatme

nt Villages

62 Villag

es

63 Villag

es

63 Villag

es

62 Villag

es

Neighborhood

Election & Referendu

mVillage Election & Consultation Meeting

Village Election & Referendu

m

Randomization of Election

Type

Randomization of Project

Selection Type

Randomization of Election Type and Project Selection Type Done Separately

Four Randomly-Assigned Combinations of Election and Selection Type

Test of Election / Project Selection Interactions

Use to Test Interactions between Election and Selection Type

Neighborhood

Election & Consultation Meeting

Instrument Combination Proposal Selection Prioritization

Male Villager

Neighborhood Election / Meeting ~ + +Neighborhood Election / Referendum ~ + ~

Village Election / Meeting ~ + ~Village Election / Referendum ~ ~ +

Difference between Types ~ ~ ~

Village Leaders

Neighborhood Election / Meeting ~ ~ ~Neighborhood Election / Referendum ~ ~ ~

Village Election / Meeting + + +Village Election / Referendum ~ ~ ~

Difference between Types + + +

Male villagers influence selection and prioritization

Combinations of election and referendum type do not affect influence of male villagers

Impact of Combinations on Selection Outcomes

Villager leaders influence proposals, selection, and prioritization when village elections are

combined with consultation meetings

Probability of Elite Capture Maximized by Combination of Village Elections with

Consultation Meetings

. . . on Elite Capture of Project Selection

Conclusion / Issues for Discussion

Problems with SolutionsCDD is interesting b/c it can change behaviors, attitudes, institutions, social cohesion etc.

Very difficult to measure well

‘Parrot Bias’ in Surveys: Does program really change attitudes and behaviors or does it just responses to survey questions?→ measure actual behaviors rather than simply asking questions

Randomization: theoretically easy, practically difficult→ tweak procedure to be resistant to pressures & communicate→ successful quasi-experimental eval. better than failed RCT

Prospective Evaluations take ages . . . → STIs can provide (quicker) real-time feedback to program

→ invest in extensive pilot-testing and consult widely

→ Manage expectations and communicate . . .

Issues That RemainIndependence of Researchers vs. Accountability of Evaluations to Programs, Implementers, and Donors

Importance of Non-Results vs. Lack of Interest and Lack of Incentives for Researchers or Program to DisseminateSmall proportion of programs are subjected to prospective IEs. How do we ensure the evaluated ones aren’t punished for non- or bad results?

Who will synthesize IE results for policy-makers and make sure the results are used in policy decisions and program design?Who will coordinate researchers to make sure evaluations address questions useful for policy decisions and program design?

www.nsp - i e . org