How to Run an Impact Evaluation: An Example from Malawi€¦ · How to Run an Impact Evaluation: An...
Transcript of How to Run an Impact Evaluation: An Example from Malawi€¦ · How to Run an Impact Evaluation: An...
1
Dean Yang
University of Michigan
How to Run an Impact Evaluation: An Example from Malawi
Fertilizer use, cash crop farmers in central Malawi
2
39%
8%
22%
9%
2%
17%
1% 1% 0% 0%
2%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 to 9 10 to 19 20 to 29 30 to 39 40 to 49 50 to 59 60 to 69 70 to 79 80 to 89 90 to 99 100 +
KG per acre categories
Urea per Acre of Maize
Note: Per-acre fertilizer recommendation for central Malawi is ~60kg urea, ~40 kg 23:21 (per-hectare recommendation is 150 kg urea and 100 kg 23:21).
Official government recommended level
Raising farm output with rural finance
• Insure farmers against adverse events
– Provide insurance against poor rainfall
• Facilitate credit for agricultural inputs
– Improve repayment via biometric identification
• Encourage farmers to save for their own input purchases
– Facilitate access to ordinary and “commitment” savings accounts
3
Raising farm output with rural finance
• Insure farmers against adverse events
– Provide insurance against poor rainfall
• Facilitate credit for agricultural inputs
– Improve repayment via biometric identification
• Encourage farmers to save for their own input purchases
– Facilitate access to ordinary and “commitment” savings accounts
4
Agenda
• Motivation – MRFC’s perspective
• Intervention: fingerprinting
• Sample
• Randomized assignment to treatment
– Choice of club-level randomization
• Data for analysis
– Baseline survey
– Follow-up survey
– MRFC administrative data on loan repayment
5
Malawi Rural Finance Company
• Malawian microfinance institution
• One of most important providers of credit in rural areas
• Group-liability loans to rural entrepreneurs as well as farmers
6
MRFC loan officer
• Club visits for loan orientation and repayment collection carried out by MRFC loan officer on motorcycle
7
Repayment problems
• Serious repayment problems among farmers
– Default rate often >10%
• Among other factors, perception that many defaults are due to imperfect personal identification
– MRFC can’t tell if a person applying for a new loan has defaulted in the past
– A borrower can take out a loan, default, and then apply for a new loan in future by using a different name
– If MRFC could tell who had defaulted in past, would not lend to them in future
• Problem: no formal system to identify people
– Loan officers can use their personal knowledge, but imperfect
8
Intervention
• To identify past defaulters, lenders in other countries use a variety of methods
– E.g., national ID cards
• Approach used in our study with MRFC:
– Collect fingerprints of farmers applying for a loan
– Tell farmers that prior to all future loans, they will be fingerprinted again
• Fingerprint will be used to look up individual’s past repayment history
• If they repay this loan on time, MRFC will be able to see that, and will approve next loan
• If they default on this loan, MRFC can see that as well, and they will be denied future loans
9
Fingerprinting
• Aug-Sep 2007
10
Demonstrating fingerprint identification
11
• What is the impact of fingerprinting on loan repayment?
Our evaluation question
12
intervention outcome
The loan program
• For paprika farmers in Central Malawi
• MRFC in-kind loans of ~MK17,000 (US$120) for paprika farming inputs
• Cheetah Paprika provides extension services and purchases paprika output
– Loan repayment forwarded to MRFC by Cheetah
– Farmers paid remainder
• How can farmers “cheat” MRFC?
– Take inputs, but don’t grow paprika
– Sell inputs for cash value
– Or sell paprika to other buyer, not to Cheetah
13
Malawi Study Areas
N
Young paprika in the field (Jan 2008)
15
A healthy paprika crop
16
Packing paprika for sale (May-Jun 2008)
17
Where did the sample come from?
• Cheetah Paprika provided us with list of farmers (and clubs) who had sold paprika to them in past in central Malawi
• MRFC agreed to consider all clubs for loans
• Study population:
– 3,206 MRFC borrowers in central Malawi
– In 214 farmer “clubs”
18
Paprika extension officer (plus our research manager)
19
A paprika club
20
Treatment and control groups
• Control farmers:
– Educational module on credit history
• Treatment farmers:
– Educational module on credit history (identical to module given to control group)
– plus:
• Biometric fingerprint collected from all farmers as part of loan application
• Use of fingerprints for unique identification explained
• Fingerprint identification demonstrated within group
• Question: Why give educational module to both groups? 21
Randomization of treatment
• Key question in study design:
• At what level do we randomize treatment?
• First idea: randomize at individual level
– What are the pros and cons of randomizing at individual level?
22
Group-level randomization
• We randomized at the level of the farmer club
• 214 clubs randomly assigned to treatment (fingerprinting) or control
• Rationale for doing this:
– Farmers may be upset by knowing they were treated differently from others
• And this may affect repayment
• (Although not clear whether farmers would be more upset if they were fingerprinted or not…)
• Clubs are geographically spread out, so clubs were generally not aware that other clubs might have had different treatment
23
Study schedule
July 2007
August 2007
Sep. 30, 2008
Clubs organized
Baseline survey and fingerprinting begin
November 2007
Loans disbursed
Loans due
September 2007
Baseline survey and fingerprinting end
Follow-up survey
August 2008
Baseline surveys (Aug–Sep 2007)
25
Study design incorporated from start
• Question: Why was it important that randomized evaluation was part of plan from the beginning?
July 2007
August 2007
Sep. 30, 2008
Clubs organized
Baseline survey and fingerprinting begin
November 2007
Loans disbursed
Loans due
September 2007
Baseline survey and fingerprinting end
Follow-up survey
August 2008
What were the impacts?
• Data for analysis:
– Administrative data from MRFC on loan repayment
– Follow-up survey
• In treatment and control groups, same percentage of farmers took out loans
• Other outcomes to compare between treatment and control:
– Total borrowed
– Repayment performance
27
Total borrowed
28
16 590 17 279
0
2 000
4 000
6 000
8 000
10 000
12 000
14 000
16 000
18 000
20 000
Treatment Control
Total borrowed (MK)
Repayment rate (on-time)
29
88%
80%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Treatment Control
% of loan repaid
Repayment rate (eventual)
30
90% 88%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Treatment Control
% of loan repaid
Are effects different for different types of borrowers?
• Overall differences between treatment and control groups are relatively modest
• So next step in analysis: are effects different for different types of individuals?
• Question: What different types of individuals would you look at?
• What we did: look at different types of borrowers by risk of default
– Calculate a “credit score” for each borrower
– Examine treatment vs. control for borrowers in 5 different categories
31
1%
3%
6%
8%
3%
6%
9%
7%
18%
39%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
less than
10%
10% to
20%
20% to
30%
30% to
40%
40% to
50%
50% to
60%
60% to
70%
70% to
80%
80% to
90%
more than
90%
Predicted Repayment for Loan Recipients
Predicted percentage repaid (“credit score”)
Repayment: % of balance paid on-time
33
88%
79%
91% 93%
89%
26%
74%
92%
96% 98%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Worst 2nd quintile 3rd quintile 4th quintile Best
Fingerprinted
Control
Repayment: % of balance paid (eventual)
34
92%
83%
93% 94% 92%
67%
77%
93%
96% 99%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Worst 2nd quintile 3rd quintile 4th quintile Best
Fingerprinted
Control
Fraction of land allocated to paprika
35
19%
15%
21% 22%
23%
11%
16%
19%
21%
23%
0%
5%
10%
15%
20%
25%
Worst 2nd quintile 3rd quintile 4th quintile Best
Fingerprinted
Control
Market inputs used on paprika (MK)
36
9 600
8 381
9 858
8 088
8 874
2 503
4 911
11 803
11 262
12 378
0
2 000
4 000
6 000
8 000
10 000
12 000
14 000
Worst 2nd quintile 3rd quintile 4th quintile Best
Fingerprinted
Control
In sum
• A simple RCT in microfinance found a cheap and easy way to raise loan repayment performance
• Lessons for RCT design:
– Randomization at group is feasible and avoids possible problems of individual-level randomization
– Randomized study design needs to be built into program from the beginning
37