Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public...
Transcript of Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public...
![Page 1: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/1.jpg)
1
Module 12
Advanced Evaluation Designs (IEs)
*
Certain slides used with permission from the Abdul Latif Jameel Poverty Action Lab (JPAL) and Center of Evaluation for Global Action (CEGA)
![Page 2: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/2.jpg)
By the end of this module participants will be able to:
1• Understand experimental and quasi-experimental IE designs
2• Recognize the shortcomings of common ‘counterfeit’ counterfactuals
3• Identify the advantages and challenges of advanced IE designs
4• Plan mitigation strategies for common threats in IEs
5• Propose multiple approaches to contracting IEs
2
Module Objectives
![Page 3: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/3.jpg)
Today’s Roadmap
3
Defining Impact Evaluation and
the Counterfactual
Experimental & Quasi-Experimental
MethodsRandomized Controlled Trials (RCTs)Regression DiscontinuityDifferences-in-DifferencesMatching
Planning and Managing
Threats to IEs
![Page 4: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/4.jpg)
Why do we do Impact Evaluations? Review
• Accountability to stakeholders– Measure effectiveness, relevance and efficiency– Disclose findings to stakeholders
• Learning to improve effectiveness– Select evaluation questions to test assumptions underlying project
designs– Utilize appropriate methods to generate valid findings– Share findings – Integrate conclusions and recommendations into decision-making
4
![Page 5: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/5.jpg)
How to Answer Impact Questions
1. Convert your question into a testable hypothesis– IE module
2. Develop and implement an evaluation design – IE module
3. Collect Data – Quantitative and Qualitative Methods Module
4. Test your hypothesis– Quantitative and Qualitative Methods Module
5
![Page 6: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/6.jpg)
Impact Evaluation Questions – Review
Which of the following are typical IE questions?• Was it the intervention that caused observed results?• What proportion of women participated in the program?• Were women affected differently than men?• Was the program implemented on time and budget?• Which approach is most effective in achieving the desired
result?• Under what conditions would the intervention produce similar
results if it were replicated?
6
Are these other questions still important for IEs?
![Page 7: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/7.jpg)
Converting Questions to Hypotheses
• How do we answer the evaluation questions?– Turn them into hypotheses that can be
tested empirically (i.e. through data and evidence)
• Often written as if…then statements– If the intervention, then the outcome
• In research terminology:– If the independent variable (IV), then the
dependent variable (DV)The DV is caused by the IV…it is
dependent on the IV7
![Page 8: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/8.jpg)
Hypotheses to Test with IEs
• Examples:
• Do microfinance schemes impact suicide rates?– If a small business owner cannot repay a loan, then risk of committing
suicide rises
• Does increased elementary school attendance impact family income?– If a child goes to school, then the family loses labor opportunity
• Do increased literacy rates impact local government participation?– If a citizen learns to read, then he/she will vote in the next election
8
![Page 9: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/9.jpg)
USAID Definition of Impact Evaluation -Review
• Impact Evaluations are based on models of cause and effect …
– Requires a causal theory of change• E.g. results framework, logical framework, development
hypothesis
• … and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change.
– The counterfactual… identifies what would have happened to the beneficiaries absent the program.
• The ‘impact’ of a program is defined as changes relative to this counterfactual
9
![Page 10: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/10.jpg)
Counterfactual…in Pictures
10Time
Prim
ary
Out
com
e
Impact
Intervention
![Page 11: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/11.jpg)
11
The Counterfactual
• Can we directly measure the counterfactual?– Unless you have a time machine, you can’t be in two places at one
time– You can’t both participate and not-participate in the program at the
same time
• Since we cannot, how can we estimate it in an unbiased way?
Let’s start by looking at a couple ‘counterfeit’ counterfactuals…
![Page 12: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/12.jpg)
‘Counterfeit’ Counterfactual 1: Before-After
12
Time
Out
com
e of
Inte
rest
Before – After Evaluation
Intervention
• What is the ‘impact’?
• What is the counterfactual?
- Assumes no change
Consider another scenario…
‘Impact’
Before After
Legend
Participants
Counterfactual
Impact
‘Counterfactual’
![Page 13: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/13.jpg)
‘Counterfeit’ Counterfactual 1: Before-After
13
Time
Out
com
e of
Inte
rest
True CounterfactualIntervention
• What if participants would have been worse off without the program?
• Before – After gave a biased impact estimate True
Impact
Estimated Impact
Legend
Participants
Counterfactual
Impact Before After
‘Counterfeit Counterfactual’
True (unknown) Counterfactual
![Page 14: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/14.jpg)
Before and After Example: Food Aid
• We observe that income after food aid is lower than income before food aid
• Did the program fail?
• Suppose that the road got washed away in the floods that triggered the food aid?
The household’s income before the floods is a poor estimate of the counterfactual (i.e. the household’s income without the food aid)
14
![Page 15: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/15.jpg)
‘Counterfeit’ Counterfactual 1: Example
15
Time
Inco
me
Income effects of food aid Food Aid
• With food aid income decreased…
• But what would it have been without the food aid?
• Before-after would have given a biased estimate
• What are the DV and IV?
True Impact
LegendWith Food-Aid
Without Food AidImpact
Before After
True (unknown) Counterfactual
(IV)
(DV)
![Page 16: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/16.jpg)
‘Counterfeit’ Counterfactual 2
• Compare participants to non-participants:
16
Those who are NOT selected or choose NOT to participate
Comparison
Those who are selected or choose to participate
Treatment
![Page 17: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/17.jpg)
Why might non-participants be a biased counterfactual?
• Potential Problem:– Why do our participants participate?
They (or their communities) are often selected because they are poorer or better connected, for example
Or may choose to participate because they are highly motivated
• Many times reasons are directly related to the outcome of interest– E.g. more motivated=more likely to increase income even without the
program
• Selection bias: – Differences between treatment and comparison group which
influence the outcome of interest– Is the comparison group really comparable?
17
![Page 18: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/18.jpg)
Counterfeit Counterfactual 2: An Example• Policy Question: Does VCT reduce risky behavior?
• What if the treatment group went for VCT because they know they engage in high-risk behaviors?- In the absence of VCT, they still have a higher chance of HIV
• Comparison group not a good counterfactual- They have different expected outcomes even without the program
High risk type Low risk type
Treatment Group: VCT
Comparison Group: No VCT
18
![Page 19: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/19.jpg)
Exercise I – Impact Evaluation Critique (Exercise 12-1)
• Read through Exercise 12-1 and answer the following questions:
– How credible are the results of this impact evaluation?
– What is the counterfactual assumed in this impact evaluation?
– Bonus Question: Assuming that we could start over, how would you improve the design to generate more credible results?
19
![Page 20: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/20.jpg)
Good Counterfactuals Exist
• …but they require careful planning
• Important that our ‘stand-ins’ are comparable to beneficiaries– Both initially– And during the project
• You need to be convinced that the remaining difference is ONLY caused by the intervention
20
![Page 21: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/21.jpg)
Main Types of Impact Evaluation Designs
1. Experimental designs:• Units are randomly assigned to the project and control groups• Provides the strongest counterfactual
2. Quasi-experimental designs:• Units are either self-selected or selected by the project• A comparison group is selected to match as closely as possible the
project group• Design quality varies in terms of how closely the two groups are matched
(selection bias)
3. Non-experimental designs: • Used when it is not possible to select a comparison group• The analysis is statistically much weaker but the findings may be credible
and useful to managers
21
![Page 22: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/22.jpg)
22
IE Designs: Experimental Designs
Experimental Designs(Randomized Evaluations, RCTs)
![Page 23: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/23.jpg)
• Eligible individuals, communities, schools etc are randomly assigned to:
- The project group (that receives the services) or- The control group (that does not have access to
the project services)
23
IE Designs: Experimental Designs
![Page 24: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/24.jpg)
• Helps to ensure no systematic differences between the project beneficiaries and the control group at the start of the project.– None of the initial characteristics which might influence the
outcomes of interest are correlated with participation.
• Removes the following sources of bias that are difficult to control for in non-randomized designs:– Self-selection bias: individuals decide to join the project (or not)– Agency-selection bias: the implementing agency selects units to
participate (or not)
24
Why Randomize? Review
![Page 25: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/25.jpg)
HQIncome per person
5000
0Treat Compare
8,990
12,470
Blue villages don’t make a good counterfactual. These villages are different. They are better off.
Targeted Assignment
25
![Page 26: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/26.jpg)
Randomly samplefrom area of interest (select some eligible communities)
Remove ineligible communities from sample
Randomly assignto treatment and control out of sample
Randomly samplefor surveys (from both treatment and control)
Income per person before the program
5000
0
10,05710,024
Randomly samplesamplefrom area of interest from area of interest (select some eligible communities)
Remove ineligible Remove ineligible communities from communities from sample
Randomly assignassignto treatment and and control out of sampleout of sample
Randomly samplesamplefor surveys (from both for surveys (from both treatment and control)treatment and control)
from area of interest from area of interest (select some eligible (select some eligible
assignassignassignassign
Income per person Income per person communities from Income per person communities from communities from Income per person communities from before the programbefore the programsamplebefore the programsample
5000Randomly 5000Randomly to 5000to treatment5000treatment
0for surveys (from both 0for surveys (from both
from area of interest (select some eligible communities)
Randomly Randomly treatment
control
Randomly
from area of interest (select some eligible
assignassignand
out of sample
sample
Randomly assignassignRandomly assign
from area of interest from area of interest 10,057sample10,057sample
10,02410,024Randomly
10,024Randomly
Income per person after the program
5000
0
Randomly samplefrom area of interest
10,02410,024Randomly
10,024Randomly sample
10,024sample
from area of interest from area of interest (select some eligible
assign
from area of interest (select some eligible communities)
Randomly assign
from area of interest
assignassignand
out of sample
sample
Randomly assignRandomly assigntreatment
out of sample
Randomly sample
assignRandomly assign
from area of interest from area of interest from area of interest (select some eligible
assign
from area of interest
out of sample
assignassignand
out of sample
sample
assignRandomly Randomly assignassignassignassignRandomly assignassign
from area of interest from area of interest from area of interest from area of interest 10,057
12,486
Random Sampling and Assignment
HQ
26
![Page 27: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/27.jpg)
How do we Implement Random Assignment?
• What does the term “random” mean here?– Equal chance of participation for every unitUnits with ‘favorable’ characteristics not more (or
less) likely to receive services
• How do we do this?– Typically by computer (random number generator)
Permits stratification and balance checking
– Draw numbers out of a hat or public lottery More transparent – and credible to evaluation sample
– Exploit naturally occurring randomnessVeteran status (draft) depended on date of birth Last digit of vehicle number plate odd/even
27
![Page 28: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/28.jpg)
Multi-media
Esther Duflo TED talk
28
![Page 29: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/29.jpg)
• Random assignment can occur at different levels with different types of ‘units’:
1.Randomizing at the individual level2.Randomizing at the group level
– E.g. class, school, clinic, community, district– “Cluster Randomized Trial”
29
Unit of Randomization: Options
![Page 30: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/30.jpg)
Unit of Randomization: Individual?
30
![Page 31: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/31.jpg)
Unit of Randomization: Individual?
31
![Page 32: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/32.jpg)
Unit of Randomization: Class?
32
![Page 33: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/33.jpg)
Unit of Randomization: Class?
33
![Page 34: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/34.jpg)
Unit of Randomization: School?
34
![Page 35: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/35.jpg)
Unit of Randomization: School?
35
![Page 36: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/36.jpg)
Unit of Randomization: Considerations
• Which level to randomize?– What unit does the program target for treatment?
If the program must be implemented at the school level, we can’t assign at the individual level
Cannot assign at a unit lower than the unit of treatment– What is the unit of analysis?
If we are looking at school level outcomes, we can’t assign at the individual level
Cannot assign at a unit lower than the unit of analysis
• Other considerations– Spillovers– Power– Ethical/Political
36
![Page 37: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/37.jpg)
• Suppose there are 2000 applicants• Screening of applications produces 500 “worthy”
candidates• There are 500 slots• A simple lottery will not work
• What are our options?
Sometimes screening matters
![Page 38: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/38.jpg)
• What are they screening for?• Which elements are essential?• Selection procedures may exist only to reduce eligible
candidates in order to meet a capacity constraint• If certain filtering mechanisms appear “arbitrary”
(although not random), randomization can serve the purpose of filtering and help us evaluate
Consider the screening rules
![Page 39: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/39.jpg)
• Sometimes a partner may not be willing to randomize among eligible people.
• Partner might be willing to randomize at the margins• People “at the margins” are people who are borderline in
terms of eligibility– Just above the threshold not eligible, but almost
• What treatment effect do we measure? What does it mean for external validity?
Randomization at the margins
![Page 40: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/40.jpg)
Randomization at the margins
At the margins, compare treatment to control
participantsNon-participants
Treatment
Control
![Page 41: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/41.jpg)
Quasi-Experimental Designs
41
![Page 42: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/42.jpg)
Quasi-Experimental Designs
• Quasi-experimental methods offer a wider set of evaluation strategies
• 3 Examples:
1. Regression Discontinuity Design
2. Difference-in-differences
3. Matching42
![Page 43: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/43.jpg)
Quasi-Experimental Design 1:Regression Discontinuity
43
![Page 44: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/44.jpg)
I. Regression Discontinuity – Review
• What is it?
• Treatment and comparison group assigned based on cut-off score on a quantitative variable that is believed to be correlated with intended outcome
- Examples of cutoffs? Students must score above 80 on entrance exam Household income below $1,000/yr Farms below 1.5 hectares
44
![Page 45: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/45.jpg)
I. Regression Discontinuity: An Example
• Agricultural Loan Program
• Loans provided to farmers to increase farm productivity and income
• To be eligible, farmers must have a credit rating of at least 100
• What are the IV, DV, and development hypothesis in this example?
45
![Page 46: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/46.jpg)
Example of RD: Impact of Agricultural Loans
Assignment variable (Credit Score)
Bas
elin
e Fa
rm In
com
e
10 20 30 40 50 60 70 80 90 100 110 120 130 140
Prior to Agricultural Loan Program
46
![Page 47: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/47.jpg)
Comparison group Treatment group
Assignment variable (Credit Score)
Year
3 F
arm
Inco
me
Cut-off score =
100
10 20 30 40 50 60 70 80 90 100 110 120 130 140
Example of RD: Impact of Agricultural Loans
48
![Page 48: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/48.jpg)
Defining the Selection Cut-off Point
• The cut-off point is determined by an analysis of the distribution of scores on the scale.– Ideally there will be approximately equal numbers of subjects
above and below the cut-off point
• However, this if often strongly influenced by resources or implementation concerns
50
![Page 49: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/49.jpg)
Challenges to RD
• It will sometimes be difficult to construct an ordinal or interval scale.– The information may not be available on all subjects.– The selection criteria used by management or experts
may not be easy to convert into an ordinal scale.
• Generalizability– Only measure impact at the cutoff point
51
![Page 50: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/50.jpg)
Quasi-Experimental Design 2:Difference in Differences
(Double Difference)
52
![Page 51: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/51.jpg)
2. Difference in Differences – Review
• What is it? • Compare the change in treatment group with change in a
comparison group• 2 level comparison:
1. observed changes in outcomes (Before-After) 2. for a sample of participants and non-participants (With-Without)
• How are the treatment and comparison groups selected?• Treatment group is usually purposefully selected• Comparison Group either:
• Matched• Judgmental or Statistical
• Unmatched• Can be used with Randomization (method of analysis for RCT) 53
![Page 52: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/52.jpg)
First Differences
54
Time
Out
com
e of
Inte
rest
Intervention
Participants (T)
Non-Participants (C)
2. Difference in Differences
∆T
∆C
First DifferencesDifferences in Differences
mpact
![Page 53: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/53.jpg)
• Hopetown HIV/AIDS Program (2008-2012)
• Objectives
- Reduce HIV transmission
• Intervention: Peer education
- Target group: Youth 15-24
• Indicator: Teen pregnancy rate (proxy for unprotected sex)
55
2. Difference in Differences - Example
![Page 54: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/54.jpg)
?62.9 66.4
46.37
57.5
40
50
60
70
80
2008 2012
Teen
Pre
gnan
cy(p
er 1
000)
ParticipantsComparison
56
2. Difference in Differences
Before After
![Page 55: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/55.jpg)
?62.9 66.4
46.37
57.5
40
50
60
70
80
2008 2012
Teen
Pre
gnan
cy(p
er 1
000)
ParticipantsComparison
57
2. Difference in Differences
Before After
∆ ‘Before’ 62.9 – 46.37 = 16.53
∆ ‘After’ 66.4 – 57.5 = 8.87
Impact = Difference in Differences8.87 – 16.53 = -7.66
![Page 56: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/56.jpg)
58
Impact
2. Difference in Differences
Counterfactual (Parallel Trends)
Teen
Pre
gnan
cy
![Page 57: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/57.jpg)
59
2. Difference in Differences
![Page 58: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/58.jpg)
• Advantages
– QEDs allow targeted selection of participantsAvoids some of the ethical or political objections to random
assignment
– In special circumstances, can be done retrospectivelyBaseline and outcome data must be available for potential
comparison units• E.g. when secondary data sources are used
60
2. Difference in Differences
![Page 59: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/59.jpg)
• Challenges– Reliance on assumption of parallel trends
Generally, how comfortable are we with this assumption? Improved when groups are similar at baseline
– Counterfactual less reliable than other IE designs (RCT, RD, PSM)
61
2. Difference in Differences
![Page 60: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/60.jpg)
Quasi-Experimental Design 3:Matching Designs
62
![Page 61: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/61.jpg)
3. Matching – Review
• What is it? • Construct a comparison group that looks as similar as
possible to the treatment group• Pair each program participant with one or more non-
participants, based on observable characteristics
• How do we construct the comparison group?• Selection
• Judgmental• Match on one variable• Match on multiple variables (e.g. propensity score matching)
• Individual vs group level matching
63
![Page 62: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/62.jpg)
HQIncome per person
5000
0Treat Compare
12,520 12,470
Comparison village are matched on observed characteristics but may differ on unobserved.
Income per personIncome per personSelect TreatmentCommunities
12,520 12,470Identify ComparisonCommunities Similar to Treatments
3. Matching
64
![Page 63: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/63.jpg)
65
3. Matching
Before After
![Page 64: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/64.jpg)
• What is it?– Extension of matching technique to efficiently match on multiple
variables
• How is it done?– After treatment group is selected, uses regression analysis to
identify the baseline characteristics correlated with selection – Each treatment unit and potential match is given a propensity
score based on the degree to which it shares these baseline characteristicsThe propensity or likelihood of being in the treatment group Intuitively: How similar is this participant to the ‘typical’
treatment unit? – Select “nearest neighbors” (usually around 3-5) from
comparison group who most closely match a participant.Note: Other matching methods exist
66
3. Matching: Propensity Score Matching (PSM)
![Page 65: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/65.jpg)
67
3. Matching: Propensity Score Matching (PSM)
Project farm 1 Project farm 2Farm 1
Dollars/month
Average for 5 nearest neighbors (nearest matches)
Dollars/month
Farm 2
Dollars/month
Average for 5 nearest neighbors (nearest matches)
Dollars/month
Before project 350 360 410 405After project 500 430 520 530Change +150 +70 +110 +125Impact estimate: 150 – 70 = + 80 110 – 125 = - 15
Estimated total project impact = Sum of impacts for each farm/ number of project farms
Estimating project impact on farm income using propensity score matching and DD
![Page 66: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/66.jpg)
• Assumption• Unobserved characteristics do not affect outcomes of interest
Things that we do not or cannot measure Examples?
• Can we test this assumption? By definition…no But we can try to measure proxies
68
3. Matching
![Page 67: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/67.jpg)
• Advantages…similar to DD– Similar to DD but stronger matching techniques are able to control
for observed characteristics – QEDs allow targeted selection of participants– In special circumstances, can be done retrospectively
Baseline and outcome data must be available for potential comparison units
If done retrospectively without baseline date, rely on ‘covariates of convenience’
• Variables that are not affected by treatment and can be observed at outcome measurement
» (age, sex, marital status, race, etc)• Leaves lots of unobservables
» Bias?
69
3. Matching
Prospective whenever possible
![Page 68: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/68.jpg)
• Challenges– Cannot control for unobservables
Generally, how comfortable are we with this assumption?– Often need large data set to identify appropriate comparisons– Relies on finding similar comparison units
Can we find comparison units that are similar to all treatment units?
• Region of common support
70
3. Matching
![Page 69: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/69.jpg)
Note:
• In order to do PSM well, you will need to find manycomparisons, which will require a very large sample size.
– This is not easy!
71
3. Matching: Propensity Score Matching (PSM)
![Page 70: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/70.jpg)
Source: CEGA
3. Matching: Propensity Score Matching (PSM)
What can we say about those outside the region of common support?
![Page 71: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/71.jpg)
• Gold standard is randomization – minimal assumptions needed, intuitive estimates
• Quasi-experimental approaches require assumptions– Hard to defend them sometimes– But can be very useful when randomization is not possible
Summary of IE Designs - Review
73
Complete Exercise 12-2 IE Design Options – Scenarios 1-4 and Bonus Questions in groups based on the IE designs
you have reviewed this module.
![Page 72: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/72.jpg)
Summary – Which Design Do I Choose?
74
Can a comparison group be identified?
Yes
Can assignment be randomized?
Non-experimental methods (e.g. before after)
Is a cutoff used to determine participation?
No
Yes
Regression Discontinuity (Quasi- Experimental)
Experimental Evaluation Yes No Combine DD with:
Propensity Score MatchingOther Matching
Unmatched(Quasi-Experimental)
No
![Page 73: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/73.jpg)
Exercise III – Which Design is it? (Exercise 12-3)
• Read each of the executive summaries and write down
– The key evaluation design being used
– What concerns you have about the design being used
– Note: ex ante: planned beforeex post: planned after
75
![Page 74: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/74.jpg)
76
Threats in Designing and Managing Impact Evaluations
![Page 75: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/75.jpg)
Types of Threats
• Design Threats– Ethical and Political Constraints on Comparison Group– Limitations to External Validity– Sufficient Sample Size to Yield Significant Results
• Implementation Threats– Spillovers– Non-Compliance– Attrition– Other Threats to Internal Validity
• Have you actually analyzed those threats?
77
![Page 76: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/76.jpg)
Ethical and Political Constraints
78
Target Population
Not in evaluation
Evaluation Sample Assignment
Treatment group
Comparison group
What happens when we can’t create a comparison group?
![Page 77: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/77.jpg)
Ethical and Political Constraints
• Is participation offered to all?– Usually not…resource constraints
If not, we next ask: Is random assignment the ‘best’ way to determine participation?
• When offered to all, we still might have design options:– Phased– Multiple Treatment
• Some projects by their nature affect whole populations– E.g. passing laws at the national level– In these cases, we often cannot identify a comparison group and
must use other evaluation designs
79
![Page 78: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/78.jpg)
Randomized-phase in example: Double Shifting in Secondary Schools in Uganda
Timing of implementation
2011/12
Group A:100
schoolsProgram
Group B:100
schools
•Assignment to groups A and B random
•Group B schools serve as controls for Group A in 2011/12…
80
![Page 79: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/79.jpg)
Randomized-phase in example: Double Shifting in Secondary Schools in Uganda
Timing of implementation
2011/12 2012/13
Group A:100
schoolsProgram Program
Group B:100
schoolsProgram
…and Group B schools also participate in 2012/3
• Is there a comparison group for 2012/3?
81
![Page 80: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/80.jpg)
Limitations on External Validity
82
Target Population
Not in evaluation
Evaluation Sample Assignment
Treatment group
Comparison group
Are the Target Population and Context similar to ‘new project’?
Is the evaluation sample representative of the target
population?
If conditions are truly unique, learning from IE will be limited
![Page 81: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/81.jpg)
Small Sample Size
83
Target Population
Not in evaluation
Evaluation Sample Assignment
Treatment group
Comparison group
What happens when we are constrained by a ‘small’ sample? • Might not be able to identify an
impact even if there really was one
![Page 82: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/82.jpg)
Spillover
84
Target Population
Not in evaluation
Evaluation Sample Assignment
Treatment group
Comparison group
Treatment
• Program impacts ‘spillover’ to comparison group
• Comparison group benefits indirectly from program
• Special case: donor gap filling
![Page 83: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/83.jpg)
• What are the implications of positive spillovers for impact estimates?
• What do we do about spillovers?– Measure
Construct or use differences in treatment intensity
– LimitDoesn’t mean we have to limit all spillovers…just to comparison
unitsSelect unit of assignment to encompass spillovers
– Example of Deworming project
Spillover
85
![Page 84: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/84.jpg)
Non-Compliance
86
Target Population
Not in evaluation
Evaluation Sample Assignment
Treatment group
Participants
No-ShowsControl group
Non-ParticipantspCross-overs
• Some Treatment units don’t participate…and some comparison units do
• How do we analyze results?– Based on actual participation, or– Original assignment
86
![Page 85: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/85.jpg)
Attrition
87
Target Population
Not in evaluation
Evaluation Sample Assignment
Treatment group
Comparison group
• Some people can’t be found for follow-ups
• Why not?– Move and can’t be found– Refuse to respond– Fatalities
Drop-outs
group
Drop-outs
![Page 86: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/86.jpg)
• What are the implications of attrition for impact estimates?– Depends on whether attrition was:
Random or Differential
• Random Attrition reduces sample size but internal validity is maintained
• Differential Attrition: attrition is correlated with:– Treatment (e.g. more attrition in comparison group)– Other Characteristics (e.g. the poorest drop out)
• What are the implications for differential attrition?
Attrition
88
![Page 87: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/87.jpg)
• How to handle attrition
– Limiting AttritionCollect good location informationConsider (carefully) incentives for responding
– Dealing with AttritionAssess whether attrition was differential
• Allocate resources for identifying drop-outs• Use baseline data
Attrition
89
![Page 88: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/88.jpg)
Exercise IV–Kinerja Case Study (Exercise 12-4)
Read and answer the study questions in your groups.
90
![Page 89: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/89.jpg)
Managing Impact Evaluations
![Page 90: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/90.jpg)
Evaluation Timing
• Contracting• Lead time before implementation
Team formation Background research Design Baseline planning
• Collaboration between IE and project partner
Impact evaluations require sufficient timing to ensure:
IE design should occur during the program design stage
92
![Page 91: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/91.jpg)
93
InterventionStarts
InterventionCeases
Start Up
Illustrative Prospective Impact Evaluation Timeline
Monitor Delivery, Attrition, Spillovers/Crossovers; CollectAdditional Data From Groups
Project and Impact Evaluation Timelines Must be Coordinated
PR
OJE
CT
EVA
LUAT
ION
![Page 92: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/92.jpg)
LOE and Budgeting Implications
• Evaluation design• Sample size and geographic
dispersion• Number of survey rounds• Local costs (enumerators,
materials, transport, security)
• Total cost is highly variable– Anywhere from <$100k to more
than $2M
94
![Page 93: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/93.jpg)
Triangulation
• Qualitative studies– Contracted in tandem with
Impact Evaluation to interpret and enrich findings
• Prior studies– USAID or third-party
evaluations– Qualitative or quantitative
• Extant data
95
High-Quality Evaluation
Extant Data
Prior Studies
Qualitative Studies
Impact Evaluation to interpret
Extant Data
Studies
![Page 94: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/94.jpg)
Contracting Impact Evaluations
• Hierarchical – IE component is subcontracted to the IP– Evaluation Policy now says this mechanism is NOT to be used
• Why?
• Parallel – IE contractor operates independently of the IP– Timing/Planning implications
• Mission-level M&E Partner– Approach used by a number of missions– Partner contracted for M&E work, including IEs, across multiple
projects96
![Page 95: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/95.jpg)
Baselines
• Baseline data is critical to IEs
• Baselines allow you to measure the progress of your intervention against a starting point
• Especially useful for midterm measurement– Allows you to modify intervention mid-way if necessary
97
![Page 96: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/96.jpg)
Review Questions
• What is a counterfactual? Why is it important?
• What are the common IE designs? Name one strength and weakness of each.
• Name two common threats to IEs and how to mitigate those threats.
• Based on what you now know, how does the design of your project change? How will you phase in IE?
98
![Page 97: Advanced Evaluation Designs (IEs) - USAID Learning Lab...–Draw numbers out of ahat or public lottery More transparent –and credible to evaluation sample –Exploit naturally occurring](https://reader033.fdocuments.in/reader033/viewer/2022042217/5ec039e33b809807835ea24d/html5/thumbnails/97.jpg)
Exercise: PROGRESA (Exercise 12-5)
“Economic Scene: A model for evaluating the use of development dollars, south of the border”
Alan B. Krueger
The Power of Credible Evaluation
99