Impact Evaluation for Evaluating Social Safety Nets at the World Bank

Javier E. BaezEconomist

IEG World Bank

AEA ConferenceSan Antonio, TX

November 12, 2010

Overview

1. Impact Evaluations – What are they?

2. Using Impact Evaluations to Inform Large Thematic Evaluations

I. A Comprehensive Review of Existing Impact Evaluation Evidence in SSNs

II. Conducting new IE in relevant areas with limited evidence

3. Conclusions

Impact Evaluations – What are they?

What is Impact Evaluation?

► Attribution: • A method that seeks to identify the impact

attributable to an intervention in contrast to just measuring changes in the variables of interest

► Counterfactual: • Outcomes are compared with a counterfactual

situation – what would have happened without the program

► Methods• Techniques vary according to the program setting

and available data, but can be broadly classified into two categories: (1) experimental, and (2) quasi-experimental

Using Impact Evaluations to Inform Large Thematic

Evaluations

First exercise: a meta-review of IEs in Social Safety Nets …

► How? • An exhaustive search, filtering and

systematic organization of evidence from 137 completed IEs

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

0

5

10

15

20

25

Nu

mb

er

of

IEs

to draw lessons about: (1) the short-term impacts of the programs, …

► Example: effects on school participation • SSNs consistently increase the use of

educational services and progression

Scho

ol E

nrol

lmen

t

Scho

ol A

tten

danc

e

Drop

outs

Grade

Rep

ititio

n

Grade

Pro

gres

sion

0102030

# Positive Total Evaluations

88%

90%

72%

73%

88%

(2) whether the impacts remain over time, …

► Example: effects on future incomes • Some evidence suggests that SSNs increase

incomes over time through various channelsO

pport

un

idades

Fam

ilia

s en

Accio

n

PA

CE

S

Food f

or

Edu

cati

on

Ch

ild S

upport

Gra

nt

Fam

ilia

s en

Accio

n

Through School Attainment Through Nutrition & Health

0%

10%

20%

30%

(3) if the transfers of the programs change other behaviors, …

► Example: sexual, marriage and fertility decisions• Some programs encouraged adolescent

girls to: (1) adopt safer sexual behaviors and (2) delay early marriage and childbearing

and, (4) cases in which impacts may hold in different contexts

► Example for comparable conditional cash transfers programs across different settings:• Impacts (direction and size) on some outcomes

very stable across various contexts

Turk

ey (C

CT)

Mex

ico (O

ppor

tuni

dade

s)

Colo

mbi

a (F

A)

Hond

uras

(PRA

F II)

El S

alva

dor (

RS)

Nica

ragu

a (R

PS)

Colo

mbi

a (F

A)

Mex

ico (O

ppor

tuni

dade

s)

Turk

ey (C

CT)

Primary Level Secondary Level

05

1015202530

School Enrollment

Second exercise: Primary IEs of SSN programs

► Motivation: large knowledge gaps • Even though SSNs are justified on the

basis of their long-term objectives to reduce poverty, they are often evaluated only against short-term outcomes

► Goal: help fill in this gap and inform the broader evaluation• IEG carried out ex-post IEs of two SSN

programs in Pakistan and Colombia

The Familias en Acción CCT program in Colombia

► Main aspects of the program

• Cash transfers to 20 percent poorest households conditional upon certain behaviors (use of education and health services)

• Started in 2002 to help mitigate the effects of the economic crisis and protect investments in human capital

• Supported by the World Bank and the Inter-American Development Bank

• Expanded between 2002-2007; covers approximately 2 million households

Does the program increase human capital?

► What did we know? • Program increased school attendance and

enrollment

► What did not we know?• If increased school inputs actually translate into

more human capital

► How did we get the answer? • Assess the effects on high school completion by:

– Using various sources of existing data– Exploiting the rules of eligibility to the program– Tracking registration in a national test given at the

end of high school

Various sources of data were carefully merged to …

► 2002

► 03

► 04

► 05

► 06

► 08

► 07

► 2009

► Test scores of all students reaching 11th grade (T and C)

► Administrative systems of the program (census of participants, only T)

► Poverty Census I (T and C)

track comparable participant and non-participant children over time ...

and find that participants are more likely to finish high school► Evidence is consistent and robust • No jumps in other variables that should not

be affected by the program (placebo test)

Conclusions

Do impact evaluation complement other evaluation approaches?► Yes – Provides causal evidence for evaluation

questions regarding “effectiveness”• Can assess what works, why, for whom, and at

what cost

► But be aware of its limitations• Results may be program- and context-specific –

especially in topics with little evidence accumulated• Requires high quality data, could be expensive and

take time to deliver findings to inform program and policy decisions

► Large thematic evaluations should be conceived to balance the strengths and limitations of each evaluation approach

Impact Evaluation for Evaluating Social Safety Nets at the World Bank

Javier E. BaezEconomist

IEG World Bank

AEA ConferenceSan Antonio, TX

November 12, 2010