Making the Most out of Discontinuities
description
Transcript of Making the Most out of Discontinuities
![Page 1: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/1.jpg)
AADAPT Workshop South AsiaGoa, December 17-21, 2009
Making the Most out of Discontinuities
Bilal Zia World Bank
![Page 2: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/2.jpg)
Introduction• Many times random assignment is not possible
– Universal take-up – Non-excludable intervention– Treatment already assigned
• When randomization is not feasible, how can we exploit implementation features of the program to “measure” its impact?
• Answer: Quasi-experiments– Example: Regression Discontinuity Design.
![Page 3: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/3.jpg)
Regression Discontinuity Design
• RDD closer to randomized experiments than other quasi-experimental methods
• Relies on knowledge of the selection process– Need to know quantifiable selection criteria – a “score”
• Assignment to “treatment” depends discontinuously on this “score”– Example: A matching grants policy that applies to firms with annual
sales less than or equal to $5,000. – A firm with sales = $5,001 would not be treated but would be very
similar to a firm with sales = $5,000. – RDD would compare firms just above and just below the $5k
threshold.
![Page 4: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/4.jpg)
Another RDD Example
• Policy: US minimum legal drinking age is 21 alcohol consumption illegal for people younger than 21
• Observation: –People aged 20 years, 11 months and 29 days–21 year olds
–Treated differently under policy because of arbitrary age cut off–But not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc)
![Page 5: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/5.jpg)
Effect of Alcohol on Mortality In effect: This policy rule assigns people to “treatment” and “comparison” groups– Treatment group: People between ages 20 years and 11 months and 20 years 11 months and 29 days– Comparison group: individuals who just turned 21 and can now legally drink alcohol.–They should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates).
This way, it is possible to isolate the causal impact of alcohol consumption on mortality rates among young adults.
![Page 6: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/6.jpg)
Graphical Depiction
![Page 7: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/7.jpg)
Increased alcohol consumption causes higher mortality rates around the age of 21All deaths
All deaths associated with injuries, alcohol or drug use
All other deaths
![Page 8: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/8.jpg)
Sharp and Fuzzy Discontinuities
• Sharp discontinuity – Discontinuity precisely determines treatment status
• All people 21 and older drink alcohol and no one else does• All firms with less than $5,000 in sales receive vouchers and larger
firms do not.
• Fuzzy discontinuity– Percentage of participants changes discontinuously at cut-off, but not
from 0% to 100% (or from 100% to 0%)• Some people younger than 21 end up consuming alcohol and/or
some older than 21 don’t consume at all• Policy rule determines eligibility but among firms with less than
$5,000 in sales, there is only partial compliance / take-up
![Page 9: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/9.jpg)
0.2
5.5
.75
1
tre
atm
en
t pro
bab
ility
assignment variable
Sharp Design for Voucher receipt
0.2
5.5
.75
1
assignment variable
Fuzzy Design for Voucher receipt
Probability of Participation: Sharp ruleand “fuzzy” rule
100%
0%
75%
0%
![Page 10: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/10.jpg)
Internal Validity• General idea
– If cut-off is arbitrary, individuals to the immediate left and right of the cut-off should be very similar.
– Differences in outcomes can be attributed to the policy.
• Major assumption
– Nothing else is happening: in absence of policy, we would not observe a discontinuity in the outcomes around this particular cut off.
– Might not be the case if • Bike-helmet policy stops also stops applying at age 21.• Another policy gives equipment to firms with sales less than
$5,000
![Page 11: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/11.jpg)
Outcome Profile Before and After the Intervention
outc
om
e
assignment variable
Baseline
assignment variable
Follow-up
![Page 12: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/12.jpg)
External Validity• Would the results generalize past the two groups you are
comparing?
• Counterfactual group in RDD– Individuals marginally excluded from benefits– Examples: people less than 21 but older than 20 years and 10
months, firms with sales less than $5,000 but more than $4,500.
• Causal conclusions are limited to individuals, households, villages, near the cut-off– The effect estimated is for individuals marginally or just eligible
for benefits– Extrapolation beyond this point needs additional, often
unwarranted, assumptions (or multiple cut-offs)
• Fuzzy designs exacerbate the problem
![Page 13: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/13.jpg)
RDD Implementation: Nuts and Bolts
• Major advantage of the RDD – Transparency– Illustrated possible using graphical methods
• Major disadvantage– Requires many observations around cut-off – All observations away from the cut-off should have less
weight– Why?
• Only near the cut-off can we assume that people find themselves by chance to the left and to the right of the cut-off.
• Think about firm with annual sales = $5,000 vs firm with sales = $500.
• Or a 16 year old vs a 25 year old.
![Page 14: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/14.jpg)
Graphical Analysiso
utc
om
e
assignment variable
![Page 15: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/15.jpg)
Wrap Up
• RDD as a powerful tool to identify causal effects – Pros
• RDDs share the same properties as an experiment locally at the cut-off
• Can be used to evaluate interventions ex post by treating cut-offs as “natural experiments” (approximate properties of an experiment but not designed as experiments)
– Cons• Estimated program effects are representative only of
individuals/firms near the cut off• RDDs require a lot of data• Individuals/firms may adjust their behavior in subsequent years in
response to the cut-off– Example: Stop reporting sales above $5,000.
![Page 16: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/16.jpg)
Wrap Up
• Can be used to design a prospective evaluation when randomization is not feasible– The design applies to all means tested programs– Multiple cut-offs to enhance external validity
• Can be used to evaluate ex-post interventions using discontinuities as “natural experiments”.
![Page 17: Making the Most out of Discontinuities](https://reader035.fdocuments.in/reader035/viewer/2022062518/56814499550346895db141c9/html5/thumbnails/17.jpg)
Thank You
17