povertyactionlab.org

Managing Threatsto Evaluations and Data Analysis

Michael KremerHarvard University

mkremer@fas.harvard.edu

2

Outline

• Background: Worms and the Primary School Deworming Project

• Threats to Estimating the Impact of Deworming– Externalities– Compliance

• Deworming Treatment Effects on Health, Nutrition, and School Participation

• Cost Effectiveness• Conclusion

3

Background on Worms

• Worms infect ¼ of people worldwide:• Hookworm 1.3 billion• Roundworm 1.3 billion• Whipworm 900 million• Schistosomiasis 200 million

• Spread through poor hygiene and sanitation

• Prevalent among school-age children in developing countries

4

Impacts of Worms

• Health– listlessness, diarrhea, abdominal pain, anemia,

malnutrition– Light infections often asymptomatic

• Education: – Seen as a key issue in whether deworming should be

a priority in poor countries– Hypothesized to negatively affect cognitive ability and

academic performance– Little previous evidence on impacts of deworming on

school attendance

5

Prevention and Treatment

• Treatment:– Low-cost single-dose oral therapies (50 cents)– Every 6 months– Virtually no side effects.– Girls over 12 typically excluded

• Although there is effective treatment…– Worms often seen as ubiquitous, non-life threatening– Rapid re-infection rates implies treatment must be

regular

• Likely substantial externalities to treatment

6

Research Question

What are the impacts of deworming on health and education?

• A few randomized studies, but overall evidence on impact of deworming on cognition is inconclusive.

• Dickson et al., 2000 conclude that they cannot recommend massive school-based programs.

7

Previous Studies

Previous studies randomize deworming treatment within schools

– Suppose that deworming prevents the transmission of disease, what problems does this create?

– If external benefit of 10% less infection after 6 months, how much underestimate benefit?

– Suppose externalities are local? How can we measure total impact?

8

Externalities: Other contexts

• Programs may create spillover effects on untreated individuals– Physical spillovers

• De-worming program: disease reduction externalities

– Prices• Vermeersch and Kremer (2004): school meals

provision led other schools to reduce school fees

– Learning and imitation effects• Duflo and Saez (2004)

9

Meauring Treatment Effect with Local Externalities

• Form two villages

• Randomly assign $1 in economic development assistance within village

• 25 cent sharing

• What is measured treatment effect?

10

Measuring Externalities

• If we expect externalities that are all within school:– Randomization at the level of the school

allows for estimation of the overall effect of deworming on a school

11

Measuring Externalities

• If we expect externalities across schools or outside of schools– Need to use random variation in density of

treatment

12

Kenyan Deworming Program

• Primary School Deworming Project (PSDP)– 75 primary schools and 30,000 children– Single region of Kenya (Busia)

• Randomize deworming across schools:– Schools stratified by geographic zone and other assistance – Listed alphabetically– Every 3rd school assigned to groups 1,2 & 3

• Group 1: treatment starting in 1998 and 1999• Group 2: treatment starting in 1999• Group 3: treatment starting in 2001

– Balanced Control and Treatment schools

Treatment Group 1998 1999 2000 2001 Group 1 (25 schools) T T T T Group 2 (25 schools) C T T T Group 3 (25 schools) C C C T

13

Treatment Program• Education on worm prevention, hygiene and sanitation.• Treatment delivered by Kenyan MoH officials and ICS officers:

– Deworming drugs at 6-month intervals – All schools with geohelminth prevalence over 50% mass treated with

albendazole.– All schools with schistosomiasis prevalence over 30% mass treated with

praziquantel.– Treatment schools which did not meet these prevalence requirements,

infected pupils were treated individually.

• Not everyone treated in treatment schools– Girls 13 years and older not treated – Consent procedures in treatment schools

• 1998: ICS obtained community consent• 1999: Kenyan MoH required signed parental consent to receive treatment

14

Estimating Impacts of the Program

• Randomization implies program participation not correlated in expectation with individual characteristics or infection status.

• Controls include school average 1996 exam results, average prevalence in zone, indicators for school involvement in other NGO assistance, and grade cohort dummies.

• School-level random effect

15

Table 6: The direct health impact of deworming, January to March 1999, Group 1 schools (1998 Treatment) versus Group 2 schools (1998 Comparison)

Group 1 (Treatme

nt)

Group 2 (Comparis

on)

Group 1 – Group

2 Any moderate-heavy infection, 1998

0.38 - -

Any moderate-heavy infection, 1999 0.27 0.52 -0.25*** (0.06)

Hookworm moderate-heavy infection, 1999

0.06 0.22 -0.16*** (0.03)

Roundworm moderate-heavy infection, 1999

0.09 0.24 -0.15*** (0.04)

Schistosomiasis moderate-heavy infection, 1999

0.08 0.18 -0.10*

(0.06) Whipworm moderate-heavy infection, 1999

0.13 0.17 -0.04 (0.05)

Other Nutritional and Health Outcomes Sick in past week (self-reported), 1999

0.41

0.45

-0.04** (0.02)

Sick often (self-reported), 1999 0.12 0.15 -0.03** (0.01)

Height-for-age Z-score, 1999 (low scores denote undernutrition)

-1.13 -1.22 0.09* (0.05)

Weight-for-age Z-score, 1999 (low scores denote undernutrition)

-1.25 -1.25 -0.00 (0.04)

Hemoglobin concentration (g/L), 1999 124.8 123.2 1.6 (1.4)

Proportion anemic (Hb < 100g/L), 1999 0.02 0.04 -0.02**

(0.01) Worm Prevention Behaviors Clean (observed by field worker), 1999

0.59

0.60

-0.01 (0.02)

Wears shoes (observed by field worker), 1999

0.24 0.26 -0.02 (0.03)

Days contact with fresh water in past week (self-reported), 1999

2.4 2.2 0.2 (0.3)

Health impact after 1st round of treatment:

•Prevalence of worms significantly lower in Group 1

•27% versus 52% moderate-to-heavy infection

•Fewer Group 1 report having been sick

•Group 1 students have better height-for-age

•Worm prevention techniques education failed

16

Health Externalities(Two types: Within-school and between school)

• Localized, within school externalities– Rates of moderate-to-heavy infections substantially

lower in Group 1 pupils who did not receive medical treatment than among comparison students in Group 2 schools

– Incoming pre-school students had only half infection rate

– Moderate-to-heavy infection rates 10% lower for girls>12

17

Deworming treatment effects on health:

0

10

20

30

40

50

60

TreatedStudents

Comparablestudents incomparison

schools

Untreatedstudents intreatmentschools

Comparablestudents incomparison

schools

Moderate-to-Heavy Helminth Infection Rates

18

Cross-school externalities

– Can you look at number of treatment schools nearby?

– What is potential problem?– What need to control for?

19

Yij is health outcome of individual i in village j,

Ti is the indicator variable for individual assignment to the program,

Xij is a vector of village and individual characteristics,

Nit is the total number of individuals at distance d from i

is the number of individuals at distance d from i randomly assigned to the program TitN

ijtiitTitijtititijt euNNXTTY 2211

Empirical Specification

20

Empirical Specification

Treatment externality:

In words:

Change in outcome with each additional treated person living d distance away

Average effect of de-worming on treatment group in year 1:

Average effect of de-worming on treatment group in year 2:

ijtiitTitijtititijt euNNXTTY 2211

d

TitN1

TitN2

21

Results, controlling for density

– Infection rates 26 percentage points lower per 1000 pupils in Group 1 schools within 3 km

– Infection rates 14 percentage points lower per 1000 pupils between 3-6 km away

22

Compliance: Treatment

Some students in treatment schools not treated• 1998: 78% of pupils assigned to receive

treatment received at least some treatment• 1999: around 72%• Absence from school the major cause of non-

compliance• 19% of girls over 13 received treatment in 1998

– Confusion over age– Nurses

Some students in comparison schools treated• 5% received treatment outside of program

23

Compliance: Transfers

24

Potential Comparisons

• What if drop non-compliers from analysis?

• What if analyze based on actual treatment?

• What do you measure if compare based on assigned treatment?

25

Intention to Treat (ITT)

• Y(T) = The average number of days children in a deworming school came to school

• Y(C) = The average number of days children in comparison schools came

• Y(T)-Y(C) = the difference between the two is the “intention to treat” effect of the program.

• This is the difference between the people you intended to treat and those who you did not intend to treat.

26

• Intention to treat: “What happened to the average child who is in a treated school in this population?”

• What does “intention to treat” measure?• Is this the right number to look for?• Remember: In the deworming case many

children in treatment schools were not treated and many children in the control treatment were.

Intention to treat (ITT)

27

When ITT is useful?

May relate more to actual programs For example, we may not be interested in

the medical effect of deworming treatment, but what would happen under an actual deworming program.

If students often miss school and therefore don't get the deworming medicine, the intention to treat estimate may actually be most relevant.

28

What ITT does not measure

• Why might ITT not give good sense of what would happen with real program?

Why might take up be lower?

Higher?

Transfers?

29

What if you wanted to know effect of the medicine on the children who took the medicine?”

30

From ITT to Effect of Treatment of Treated (ITT)

• Initial assignment an instrumental variable for treatment– Correlated with actual treatment, uncorrelated

with other factors affecting outcomes

31

“Treatment on the Treated”

• The effect of the treatment on those who got the treatment:

• Suppose children who got the treatment attended school A days, irrespective of whether they were in a treatment or a control school

• Suppose children who got no treatment attended school B days, again in both kinds of schools

32

“Treatment on the Treated”

Then…• Y(T)=A*Prob[treated|T]+B(1-Prob[treated|T])• Y(C)=A*Prob[treated|C]+B(1-Prob[treated|C])• A-B= (Y(T)-Y(C))

(Prob[treated|T] – Prob[treated|C])• =The “treatment on the treated” effect.

33

How do we calculate this?

• We need to observe the number of people getting treated

Prob[treated|T] – Prob[treated|C]. • This may not always be easy. Why?• We don’t always need the entire sample:

– It may be enough to get it from some representative sub-sample.

• General technique for when compliance is not perfect, for example, in an encouragement design.

34

Impacts on Education:

Treatment associated with dramatic gains in school participation, large externalities

• Baseline absence rate (30%) reduced by ¼ • Total effect: Increase in school participation of

~0.14 years per child• Cost-effectiveness: $3.50/additional year of

schooling

35

Figure 2: School participation rate May 1998 to November 1999 for girls under 13 and all boys, difference between Group 1 and Group 3 (diamonds), and

difference between Group 2 and Group 3 (squares)

36

Estimating the Overall Effect

• Comparison schools: 1.5 percentage point increase in school participation, three Group 2 & 3 pupils for every treated child in Group 1.

• Treatment schools: 7 percentage point increase in school participation for all children, one untreated child for every 2 treated children.

• (.015 * 3) + (.07 * .5) + (.07 * 1) = .15 years• For each child treated, school participation

increased by .15 years.

37

Cost Effectiveness and Welfare Analysis

• Health cost effectiveness

• Educational cost effectiveness

• Human capital investment

• Externalities

38

Health Cost Effectiveness

• One of many worthy health interventions competing for scarce resources—annual government per capita expenditure on health is $5.

• DALY’s

• PCD cost per pupil treated is $0.49

39

Health Cost Effectiveness

• Treatment of schistosomiasis is very cost effective, but only meets the strictest standard when take into account the externalities present.

• $12 per DALY saved• The externality benefits account for 61.1% of the • DALY reduction• Treatment of geohelminths would not meet

standard cost– Effectiveness criteria in the poorest countries based

on health impact alone: $155 per DALY saved.

40

Educational Cost Effectiveness

What is the cost of an additional year of school participation?

• School participation increased by .15 years per treated child

• Cost is $.49 per child in a large-scale program.• Costs only $3.27 per additional year of school

participation.• Much cheaper than other ways of increasing

schooling.

41

Human Capital Investment What is the rate of return to deworming in future earnings?• Labor market benefits alone may far outweigh cost.• Excludes health and nutritional gains, thus understating benefits.• Knight and Sabot (1987) estimate the avg. labor market return to an

additional year of primary schooling in Kenya is 17%.• Years of schooling alone account for approximately 40% of the total

return to education controlling for scores.• If deworming increases participation but has minimal cognitive

gains, the direct return to an additional year of primary education would be 7%.

• Output per worker is $570• Discount rate of 10%• Assume opportunity cost is half adult wage• Deworming increases net present value of wages by more than $20

per treated child at a cost of only $0.49.

42

Externalities

• What subsidy would lead people to fully internalize the externalities present?

• Total net externality gain per child treated is at least $11.70, when cost of treating that child was $0.49.

• Government should heavily subsidize.

• Hard to know the optimal level of a subsidy without an epidemiological model.

43

Deworming program: follow-up

• “Sustainable” financing of deworming?– Health education, commitment ineffective– cost recovery leads to much less take up– Suggests a need for subsidies for de-worming

• Networks and social learning– Children with (randomly) more social links to

early treatment schools are themselves significantly less likely to take deworming drugs

44

References

• Edward Miguel and Michael Kremer (2004) “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities,” Econometrica, 72(1): 159-217.– Kremer and Miguel (2004) “The Illusion of

Sustainability,” mimeo.– Miguel and Kremer (2004) “Networks and

Social Learning about Health,” mimeo.