1

AADAPT Workshop South AsiaGoa, December 17-21, 2009

Using Randomized Evaluations to Improve Policy

Nandini Krishnan

2

Objective

To Identify the causal effect of an intervention separate the impact of the program from

other factors Need to find out what would have

happened without the program Cannot observe the same person with and

without the program at the same point of time>> Create a valid counterfactual

Correlation is not causation

High Yield

Fertilizer Use

OR ?

2) ?

1) ?

High Yield

Knowledge of

Technologies Fertilizer

Use

Before After02468

101214

Treatment GroupTreatment Group

4

(+6) BIASED measure of the program impact

Illustration: Fertilizer Voucher Program (Before-After)

5

Hard to distinguish causation from correlation from statistical analysis of existing data However complex the statistics can only see that X

moves with Y Hard to correct for unobserved characteristics, like

motivation/ability May be very important- also affect outcomes of interest

Selection bias a major issue for impact evaluation Projects started at specific times and places for particular

reasons Participants may be selected or self-select into programs First farmers to adopt a new technology are likely to be

very different from the average farmer, looking at their yields will give you a misleading impression of the benefits of a new technology

Motivation

Before After0

2

4

6

8

10

12

14Control GroupTreatment Group

6

(+4) Impact of the program

(+2) Impact of other (external) factors

Illustration: Fertilizer Voucher Program (Valid Counterfactual)

7

All those in the study have the same chance of being in the treatment or comparison group

By design, treatment and comparison have the same characteristics (observed and unobserved), on average Only difference is treatment

With large sample, all characteristics average out

Unbiased impact estimates

Experimental Design

8

Lottery (0nly some receive) Lottery to receive information about a new

agricultural technology Random phase-in (everyone gets it eventually)

Train some farmers groups each year Variation in treatment

Some get information on new seed, others get credit, others get demonstration plot on their land etc

Encouragement design One farmers support center per district Some farmers get travel voucher to attend the center

Options for Randomization

Lottery among the qualified

Must receive the program

Not suitable for the program

Randomize who gets the program

10

Budget constraint prevents full coverage Random assignment (lottery) is fair and

transparent Limited implementation capacity

Phase-in gives all the same chance to go first

No evidence on which alternative is best Random assignment to alternatives

with equal ex ante chance of success

Opportunities

11

Opportunities for Randomization Take up of existing program is not

complete Provide information or incentive for some to

sign up- Randomize encouragement Pilot a new program

Good opportunity to test design before scaling up

Operational changes to ongoing programs Good opportunity to test changes before

scaling them up

12

Individual Farm Farmers’ Association Irrigation block

Village level Women’s

association Youth groups School level

Different levels you can randomize at

Group or individual randomization? If a program impacts a whole group--

usually randomize whole community to treatment or comparison

Easier to get big enough sample if randomize individuals

Individual randomization Group randomization

14

Randomizing at higher level sometimes necessary: Political constraints on differential treatment

within community Practical constraints—confusing to implement

different versions Spillover effects may require higher level

randomization

Randomizing at group level requires many groups because of within community correlation

Unit of Randomization

15

Elements of an experimental design

16

External validity The evaluation sample is representative of the total population The results in the sample represent the results in the population We can apply the lessons to the whole population

Internal validity The intervention and comparison groups are truly comparable estimated effect of the intervention/program on the evaluated population reflects the real impact on that population

External and Internal Validity (1)

17

External and Internal Validity (2) An evaluation can have internal validity without external validity

Example: A randomized evaluation of encouraging women to stand for elections in urban areas may not tell us much about impact of a similar program in rural areas

An evaluation without internal validity, can’t have external validity If you don’t know whether a program works in

one place, then you have learnt nothing about whether it works elsewhere.

18

Internal & external validity

Random Sample- Randomization

Randomization

National Population

Samples National Population

19

Internal validity

Stratification

Randomization

Population

Population stratumSamples of Population

Stratum

20

Representative but biased: useless

National Population

Biased assignmentUSELESS!

Randomization

21

Efficacy & Effectiveness Efficacy

Proof of concept Smaller scale Pilot in ideal conditions

Effectiveness At scale Prevailing implementation arrangements --

“real life”

Higher or lower impact? Higher or lower costs?

22

Advantages of “experiments” Clear and precise causal impact Relative to other methods

Much easier to analyze- Difference in averages

Cheaper (smaller sample sizes) Easier to explain More convincing to policymakers Methodologically uncontroversial

Randomly assigning wheat plots to receive regular irrigation water

Wheat plants do NOT Raise ethical or practical concerns about

randomization Fail to comply with Treatment Find a better Treatment Move away—so lost to measurement Refuse to answer questionnaires

Human beings can be a little more challenging!

What if there are constraints on randomization?

Some interventions can’t be assigned randomly

Partial take up or demand-driven interventions: Randomly promote the program to some Participants make their own choices about

adoption Perhaps there is contamination- for

instance, if some in the control group take-up treatment

24

Random Promotion(Encouragement Design) Those who get promotion are more likely to

enroll But who got promotion was determined

randomly, so not correlated with other observables/non-observables Compare average outcomes of two groups:

promoted/not promoted Effect of offering the encouragement (Intent-To-

Treat) Effect of the intervention on the complier population

(Local Average Treatment Effect)▪ LATE= ITT/proportion of those who took it up

RandomizationAssigned to treatment

Assigned to control

Difference Impact: Average treatment effect on the treated

Non-treated

Treated

Proportion treated

100% 0% 100%Impact of assignment

100%

Mean outcome

103 80 23Intent-to-treat estimate

23/100%=23Average treatment on the treated

Random encouragement

RandomlyEncouraged

Not encouraged

Difference Impact: Average treatment effect on compliers

Non-treated(did not take up program)Treated(did take up program)

Proportion treated

70% 30% 40%Impact of encouragement

100%

Outcome 100 92 8Intent-to-treat estimate

8/40%=20Local average treatment effect

Common pitfalls to avoid Calculating sample size incorrectly

Randomizing one district to treatment and one district to control and calculating sample size on number of people you interview

Collecting data in treatment and control differently

Counting those assigned to treatment who do not take up program as control—don’t undo your randomization!!

28

29

When is it really not possible? The treatment already assigned and

announcedand no possibility for expansion of treatment

The program is over (retrospective) Universal take up already Program is national and non

excludable Freedom of the press, exchange rate

policy(sometimes some components can be

randomized) Sample size is too small to make it

worth it

30

Thank You