Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient...
Transcript of Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient...
![Page 1: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/1.jpg)
Lecture 3
RANDOMISED EVALUATIONS
Agriculture Sector Dialogue Phase II
![Page 2: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/2.jpg)
Overview
• What is randomization and how does it solve the causality problem?
• Choosing the level of randomization
• Ways to randomize
• When to do a randomized evaluation
• Case Study: Nudging farmers to use fertilizer
Africa Infrastructure Projects 2
![Page 3: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/3.jpg)
What is randomization and how does it solve the causality problem?
![Page 4: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/4.jpg)
Randomization and causation
• The challenge for impact evaluation is to “construct” a comparison group that mimics the group receiving the intervention
• In randomized evaluations: Participants and nonparticipants are chosen at random
• There is no reason, other than chance, that they are selected into the program
• On average, participants and nonparticipants have the same characteristics– they would, on average, have the same outcomes
• Any difference at the end is due to the program
![Page 5: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/5.jpg)
Randomization creates groups with similar characteristics
Study sample
Treatment group 1
Comparison group
Treatment group 2
![Page 6: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/6.jpg)
Random assignment vs. random sampling
• Random assignment– Units (people, schools etc.) are randomly assigned to different groups
(e.g. treatment and comparison)
– Creates two or more comparable groups
– Basis of randomized evaluation
• Random sampling– Want to measure the characteristics of a group (e.g. average height)
– Measure a random sample of the group
– Often used during randomized evaluations, especially group level randomization
![Page 7: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/7.jpg)
Randomly samplefrom area to get study sample
Random sampling
![Page 8: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/8.jpg)
Randomly samplefrom area to get study area
Randomly assignCommunities to treatmentand comparison
Random sampling and Random Assignment
Randomly sampleIndividuals to survey from both treatment and comparison
![Page 9: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/9.jpg)
Steps in random assignment
• Define those eligible for a program
• Randomly assign which units are in the treatment and the comparison group
• Implement the policy or program for the treatment group
• Compare outcomes for those in treatment and comparison groups
![Page 10: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/10.jpg)
Steps to randomization: 1
![Page 11: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/11.jpg)
Steps to randomization: 2
![Page 12: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/12.jpg)
Steps to randomization: 3
![Page 13: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/13.jpg)
Steps to randomization: 4
![Page 14: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/14.jpg)
Non-random assignment
HQMonthly income, per capita
1000
500
0Treatment Control
1457
947
![Page 15: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/15.jpg)
Random assignment
Monthly income, per capita
1000
500
0Treatment Control
1257 1242
HQ
![Page 16: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/16.jpg)
Key advantage of random assignment
Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,
any difference that subsequently arises between them can be attributed to the program rather than to other factors.
16
![Page 17: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/17.jpg)
Choosing the level of randomization
![Page 18: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/18.jpg)
Unit of Randomization: Individual?
![Page 19: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/19.jpg)
Unit of Randomization: Individual?
![Page 20: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/20.jpg)
Unit of Randomization: Clusters?
![Page 21: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/21.jpg)
Unit of Randomization: Class?
![Page 22: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/22.jpg)
Unit of Randomization: Class?
![Page 23: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/23.jpg)
Unit of Randomization: School?
![Page 24: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/24.jpg)
Unit of Randomization: School?
![Page 25: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/25.jpg)
Considerations for level of randomization
• Unit of measurement– Generally, best to randomize at the level at which the treatment is
administered
• Feasibility and fairness
• Spillovers
• Attrition
• Compliance
• Statistical power
• Clustering on the ground
![Page 26: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/26.jpg)
Ways to randomise
![Page 27: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/27.jpg)
What can be randomized?
• Different ways to randomize come from three basic elements of a program which can be randomized
• Access: we can choose which people are offered access to a program
• Timing: we can choose when people are offered access
• Encouragement: we can choose which people are given encouragement to participate
![Page 28: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/28.jpg)
Simple lottery
• Most simple design
• Existing pool of potential participants
• Given number of slots
• Randomly assign potential participant to a treatment group or a control group
• Very useful and often the fairest way to allocate a program when resources are limited
![Page 29: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/29.jpg)
Monthly income, per capita
1000
500
0Treatment Control
1257 1242
HQ
Simple Lottery
![Page 30: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/30.jpg)
Randomization around a cut off (partial lottery)
• Sometimes a partner may not be willing to randomize among eligible people.
• Partner might be willing to randomize around a cut off or “at the margin”
• People “around the cut off” are people who are borderline in terms of eligibility– Just above the threshold not eligible, but almost
• What treatment effect do we measure? • The effect of the program for those around the cut off
![Page 31: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/31.jpg)
Partial Lottery
Around the
cut off,
compare
treatment to
control
ParticipantsNon-participants
Treatment
Control
![Page 32: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/32.jpg)
Phase-in design
• Everyone gets program eventually
• Natural approach when expanding program faces resource constraints
• What usually determines which schools, branches, etc. will be covered in which year?
![Page 33: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/33.jpg)
Phase-in design
Round 3Treatment: 3/3
Control: 0 1
1
11
1
1
1
1
1
11
1
1
1
2
2
22
2
2
22
2
2
2
22
2
2
2
3
333
3
3
3
33
3
33
3
3
3 3
3
Round 1Treatment: 1/3
Control: 2/3
Round 2Treatment: 2/3
Control: 1/3
Randomized
evaluation ends
![Page 34: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/34.jpg)
Encouragement design
• Sometimes a program is open to all and it’s impossible to randomize program access
• If there is low take up, there is still an opportunity to evaluate
• A treatment group can be created through randomly assigning individuals or groups to receive encouragement to take up the program
• Encouragement – e.g. phone call, text message reminder etc
• The idea is to increase the probability of take-up by the encouraged
• Outcomes you compare: encouraged group to the not encouraged group
![Page 35: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/35.jpg)
When to do a randomized evaluation?
• When there is an important question you want/need to know the answer to that will affect many:– To establish impact and cost effectiveness (value for money)
• Timing – not too early and not too late
• Program is representative – not gold plated– Or tests a basic concept you need tested
• Time, expertise, and money to do it right
![Page 36: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/36.jpg)
When NOT to do an RE
• When evaluating macro policies
• When it is unethical or politically unfeasible to deny a program to a control group.
• If the program is changing during the course of the experiment
• If the program under experimental conditions differs significantly from how it will be under normal conditions.
• If a RCT is too time-consuming or costly and therefore not cost-effective.
• If threats such as attrition and spill-over are too difficult to control for and hurt the integrity of the experiment
• If sample size is too small
![Page 37: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/37.jpg)
Case Study: A Well-Timed Nudge
37
Duflo, Esther, Michael Kremer and Jonathan Robinson, 2011. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya" American Economic Review
![Page 38: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/38.jpg)
Needs Assessment: Low Fertilizer Adoption
38
• Studies in Western Kenya showed that limited fertilizer use increased farmers’ annualized returns by 52-85 percent.
• But less than a third farmers reported using fertilizer in the previous two growing seasons.
• Question: If using fertilizer is profitable to farmers, why aren’t they investing in it?
![Page 39: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/39.jpg)
Reasons for low fertilizer adoption?
39
• Limited access to cash or credit – Response: fertilizer subsidies
• Issues of timing and impatience– Farmers may have the money immediately after harvest
– But they do not need the fertilizer then so may not be motivated to buy it or pre-purchasing may be inconvenient
– Response: What happens if we make it easier for farmers to buy fertilizer after harvest when they have cash?
![Page 40: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/40.jpg)
Program Evaluated
Savings and Fertilizer Initiative (SAFI) in Western Kenya
• In season 1 - Basic SAFI:
• Farmers visited immediately after harvest, and offered fertilizer voucher at market price, but with free delivery at a date of their choice.
• Farmers had to make a decision and purchase immediately
• In season 2, different variations of the program were tested including how a subsidy influences fertilizer adoption
40
![Page 41: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/41.jpg)
Treatments in Season 2
41
![Page 42: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/42.jpg)
Randomisation in season 2
• Sample Frame: built from parents of 5th and 6th grade children at 16 schools in Kenya’s Busia district
• Stratified sample (by school, class, participation in prior agricultural programmes/ treatments)
• Random assignment: Individual level - farmers
42
![Page 43: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/43.jpg)
Balance
43
SAFI Season 2: Baseline Characteristics: Means
Comparison Basic SAFISAFI with
timing choice
Full price & free delivery, late
season50% subsidy late season
(0) (1) (2) (3) (4)
Income (1,000 Kenyan Shillings) 2.19 2.82 2.76 2.94 2.29
Years of Education of household head 7.47 6.96 6.84 7.09 7.12
Household had used fertilizer prior to season 1 0.53 0.41 * 0.40 * 0.45 0.37 *
Home has mud walls 0.88 0.88 0.88 0.90 0.86
Home has mud floor 0.87 0.82 0.87 0.89 0.85
Home has thatch roof 0.50 0.54 0.53 0.54 0.51
Observations 121 213 213 145 147
* Differences in means from comparison group significant at the 5% level
Source: Adapted from Table 3, Duflo, Esther, Michael Kremer and Jonathan Robinson, 2011.
"Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya" American Economic Review 101:2350-2390
![Page 44: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/44.jpg)
Results
44
![Page 45: Lecture 3 RANDOMISED EVALUATIONS - cabri-sbo.org · buy it or pre-purchasing may be inconvenient ... What happens if we make it easier for farmers to buy fertilizer after harvest](https://reader031.fdocuments.in/reader031/viewer/2022030913/5b5d3d927f8b9ac6028ddeef/html5/thumbnails/45.jpg)
Results
45
• SAFI increased fertilizer adoption, but only for the duration of the program
• Free delivery during the growing season has, at most, a much smaller effect on fertilizer adoption
• The impact of SAFI was comparable to that of a 50% discount at fertilizer application time