Povertyactionlab.org How to Randomize? Abhijit Banerjee Massachusetts Institute of Technology.

povertyactionlab.org

How to Randomize?

Abhijit BanerjeeMassachusetts Institute of Technology


Different types of randomization

Randomization Designs

• Lottery design

• Phase-in design

• Encouragement design


What should we worry about when we randomize?

Key constraints

• Cannot interfere significantly with program operations

• Cannot be perceived as unfair

• Must be politically feasible

• Must be ethical

One Trick: Limited Resources

• Many programs only have so much resources

• Many more people are eligible than they could serve

• Examples:– Training program– Limited textbooks to distribute

• How to allocate resources?

Lottery design

• Randomly choose from applicant pool• In this design participants are explicitly told

that “winners” and “losers” will end up with different outcomes

• Use when there is no good reason to discriminate among a subset of applicants

• Lottery is perceived as fair• Transparent• Politically more feasible often


What are other types of lotteries?

Lottery Design

• Lottery need not be over individuals

• Can be over groups

• Which micro-credit groups?

• Can be over communities

• Which villages should we enter?

• Can be over schools

• Balsakhi

A lottery over program sites

• Works if there are multiple sites

• How are the program sites chosen?

• Some choices arbitrary?

• What is the affected population?

Ethicality and Political Feasibility

• Transparency often helps

• Can we give something to the control sites?

• What happened in the balsakhi intervention?


What to do if program has 500 applicants for roughly 500 slots?

Or less?

A less obvious lottery design

• One trick: Advertise program and greatly increase applicants– Now lottery is feasible

• Is this ethical? • Would it be ethical we added slots to the

program and only gave these to the new applicants?

• Or desirable from the evaluation point of view?


When is a lottery impossible?

Where a lottery fails

• Suppose there are 2000 applicants.

• Screening of applications/need produces 500 “worthy” candidates

• Lottery infeasible


What can be done in this situation?

What can be done?

• Potential Solution: – What are they screening for? – What elements are essential?– What elements are arbitrary?

• Example: Training program– 2000 candidates– 1000 fit criteria such as poor enough to be worthy, qualified

enough to take advantage of training– 250 chosen of out of the 1000 based on particular attributes– NGO willing to allocate remaining 250 by lottery

• Lesson: Many parts of a selection rule are designed simply to weed out candidates


Can we think of any other trick that will work here?

Phase-in Design

• In this design everyone is told that they will end up with the same outcome but some later than others.

• Use the fact that the program going to expand• Example:

– In 5 years, 500 schools will be covered.• What determines which schools will be covered?

– Some choices may be based on need, potential impact, etc.– Some choices largely arbitrary

• We can therefore choose who gets program early at random– Do this on population about which choices are arbitrary

• What is the affected population?• How are the comparisons made?


Yet another trick?

Encouragement design

• In this design everyone is immediately eligible to receive the program—there is enough funding to cover all of them.

• However not all of them will necessarily take it up

• Pick some people at random and encourage them to use program

• Use non-encouraged as control• Ethical?• What population is this a treatment effect on?


Level of Randomization

The unit of randomization

• Can be:– Individual: A person/A household/A selected

group of people– Collective: A community/An institution/ A

village


What are the key concerns in this decision?

Key concerns

• Political feasibility

• Practicality of implementation

• Scope of the impact

• Data collection costs

Individual versus collective randomization

• Pros of individual level randomization: – Data collection is easier: counterexample?– We can measure the impact of individual

characteristics on what the program does

• Cons: – The program may naturally operate at another level

• In the balsakhi case how would we do an individual level randomization?

– May create conflict within a group/community– Even if the treatment is on the individual the effect

may go beyond him.

Randomize evenly?

• Suppose 100 slots are available in a training program.

• Two hundred men and women apply• You are able to randomize these 100 slots• One strategy:

– Give each person a lottery number 1 to 200. – Randomly pick 100 numbers. – Those people are chosen

• Anything wrong with this strategy?

A Simplified India

• For simplification, we reference only the sixteen largest Indian states

• These 16 states are divided into four geographic regions

• We also assume there are exactly 20 districts in each Indian state, implying 320 districts in all of India.

Regions of India

Northern StatesHimachal Pradesh Haryana Punjab Uttar Pradesh

Western StatesGujaratMadhya PradeshMaharashtraRajasthan

Southern StatesAndhra PradeshKarnatakaKeralaTamil Nadu

Eastern StatesAssamBiharOrissaWest Bengal

Experiment Design

• For Program X, we plan to run an intervention with 40 treatment districts and 40 control districts. These districts are randomly selected from the entire pool of 320 Indian districts.

• The composition of the treatment group, broken down by geographic region, varies considerably each time we randomize the sample.

Randomized Placement of Treatment Groups

Randomize


Why is this a problem?

Stratification

• Divide India into four regions of 80 districts each.

• Randomly choose 10 control and 10 treatment in each.

• Why not 70 in control?

Problems with unbalanced treatment and control groups

• Shocks hitting some regions but not others.

• Suppose you wanted to know the treatment effect in each region?

Advantages of Stratification

• Good control groups. Ensure that each area is represented in treatment and control– Important if some shocks are hitting men or

women specifically

• Allows estimation of treatment effects for each group– Important if you care about differential

treatment for men and women


Would estimates be biased if there were no stratification?

No. Stratification increases power

Variables to stratify on

• Context specific

• Groups affected by different shocks– Different regions– Different income groups– Different occupations

• Groups with different treatment effects

• What happened in the balsakhi case?


Poverty Action Lab: Translating Research into Action

MIT

Povertyactionlab.org How to Randomize? Abhijit Banerjee Massachusetts Institute of Technology.

Documents

Transcript of Povertyactionlab.org How to Randomize? Abhijit Banerjee Massachusetts Institute of Technology.