Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial...
-
Upload
alessandro-chown -
Category
Documents
-
view
220 -
download
2
Transcript of Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial...
![Page 1: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/1.jpg)
Evaluation Designs in the IMATCHINE project:
Regression DiscontinuityCluster Randomized Trial
Presentation by Manoj Mohanan, Duke University
1
CO
HE
SIV
E-I
ND
IA
![Page 2: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/2.jpg)
Commercial Break! A word about COHESIVE-India• COHESIVE-India: Collaboration for Health Systems Impact
Evaluation in India.• Jerry La Forgia (World Bank)• Grant Miller (Stanford U. & NBER)• Manoj Mohanan (Duke U.)• Marcos Vera-Hernandez (U. College London & IFS)
• Focus on evaluation in health sector using a combination of quasi-experimental and experimental methods
• Provide critical input into the design of policies and interventions, to provide rigorous evidence on how to improve performance as part of evaluation.
• Collaborating with SAMBODHI and DFID-India on IMATCHINE project
2
CO
HE
SIV
E-I
ND
IA
![Page 3: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/3.jpg)
Key Question in Impact Evaluation• What is the counterfactual? What would have happened if
this program did not exist?• Identifying CAUSAL effects is the central to impact
evaluation• Two common fallacies in making causal inference:
• Cum hoc ergo propter hoc• Post hoc ergo propter hoc
![Page 4: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/4.jpg)
Cum hoc ergo propter hoc
Source: http://ssgreenberg.name/PoliticsBlog/2009/04/03/diversion-highway-fatalities-and-lemons/
![Page 5: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/5.jpg)
The post hoc fallacy• Observe people on the street @ 9AM to predict weather?• In medicine, there is a disease progression, hence you can
use temporal changes to make causal claims. In social sciences, less so.
![Page 6: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/6.jpg)
Visualizing problems in evaluation
Participants
Counter factual
This is why the Before-After method is sometimes called The “Counterfeit Counterfactual” method!!
TIME -------------------------------------
OU
TC
OM
ES
---
----
----
----
----
----
----
![Page 7: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/7.jpg)
Outline of Today’s Presentation
• Projects being evaluated in the IMATCHINE project• Gujarat: (Regression Discontinuity)
Chiranjeevi Yojana (CY)
• Karnataka: (Experimental Evaluation)Thayi Bhagya Yojana (TBY)
7
CO
HE
SIV
E-I
ND
IA
![Page 8: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/8.jpg)
Gujarat: Chiranjeevi Yojana (CY)
• Introduced in 2005• Response to acute shortage of OBGYNS in
public sector• Leveraging presence of private providers in
rural areas• Pays approx Rs. 1700 to accredited provider
per delivery• Eligibility: BPL card holder or BPL eligible
(~23% of population; total pop 55 million)8
CO
HE
SIV
E-I
ND
IA
![Page 9: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/9.jpg)
More on CY
• First introduced in 5 backward districts 2005-2007 and then rolled out across the state Jan 2007-08 onwards
• B/w 2005 – Feb 2008, CY had covered over 165,000 deliveries provided by 852 providers
• Claims: (in 2009)Has increased institutional births from a national
average of 57% to over 80%Has reduced MMR & IMR Won WSJ Innovations Award & is now widely
looked upon as the “model” 9
CO
HE
SIV
E-I
ND
IA
![Page 10: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/10.jpg)
Previous ‘Evaluations’ of CY
Source: Mavalankar, D. et al. 2009. Saving mothers and newborns through an innovative partnership with private sector obstetricians: Chiranjeevi scheme of Gujarat, India. International Journal of Gynecology and Obstetrics 107: 271–276.
• Have typically used data from CY facilities to extrapolate estimated utilization and health benefits in the population
10
CO
HE
SIV
E-I
ND
IA
![Page 11: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/11.jpg)
Gujarat: Regression Discontinuity Evaluation Design
• The program uses the BPL line as the eligibility criteria, which allows a quasi-experimental RD design
Since the BPL score is continuous, households immediately next to each other across the BPL line are comparably similar to each other
Discontinuity in program eligibility across the BPL line allow us to test for differences in outcomes that can be attributed to the program
11
CO
HE
SIV
E-I
ND
IA
![Page 12: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/12.jpg)
Gujarat: RD Design …contd…Two main challenges with BPL criteria:
Lots of other state run programs use BPL eligibility (such as JSY, food subsidies),:
− SOLUTION: We use a “Difference-in-Difference” framework by relying on timing of introduction and expansion of the program to 5 districts in 2005 and all over the state in 2007 to try to identify program effects of CY.
•
Manipulation of BPL criteria, resulting in misclassification− SOLUTION: We rely on a “Fuzzy” regression discontinuity
strategy, where we calculate the “true” eligibility and then instrument for CY participation using simulated eligibility
12
CO
HE
SIV
E-I
ND
IA
Year Pilot Districts Expanded Districts
2005 BPL + CY BPL
2008 BPL + CY BPL + CY
![Page 13: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/13.jpg)
Gujarat: RD Analysis• First Stage: (instrumenting for CY Participation)
• Second Stage: (Change in avg. outcomes)
• Key outcomes:Rates of institutional delivery Study is not powered to detect effects on IMR or MMR,
but we will collect data on these measures anyway in addition to measures of morbidity that are more common. Also will collect data on HH characteristics
13
CO
HE
SIV
E-I
ND
IA
ihk khhih CONTROLSBPLBELOWeparticipat
ikikk khihih CONTROLSBPLeparticipatoutcome
![Page 14: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/14.jpg)
Gujarat: RD Analysis using the Geographic Spread
• Accounting for CY roll out in time and space (2nd difference estimate)
• Second Stage: (change on avg. outcomes : same equation accounting for roll out)
14
CO
HE
SIV
E-I
ND
IA
ihdyydk ikkh
dyhdyhihdy
CONTROLSBPL
xCYBELOWCYBELOWeparticipat
)(
ihdyydk ikk
hihdydyhihdy
CONTROLS
BPLeparticipatCYBELOWoutcome
![Page 15: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/15.jpg)
BREAK B/W RD & CRT
15
CO
HE
SIV
E-I
ND
IA
![Page 16: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/16.jpg)
Experimental Methods• The Randomized Controlled Trial
• Clearly, the gold standard• An obvious solution to the Prog Eval problem, although not
always practical:• Assigns treatment in a manner that is unrelated to outcomes
• Two important steps in randomized evaluation1. Randomly selecting potential participants from population
2. Randomly assign treatment to the group
• The RCT in IMATCHINE Project…
![Page 17: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/17.jpg)
Karnataka: Thayi Bhagya Yojana (TBY)
• New Conditional Cash Transfer program of Rs. 1000 for women who prefer to give birth in the private sector
• Two components • (1) Prospective evaluation• Since the program uses a BPL eligibility like the
CY program, we will use a RD based method, combined with a difference-in-difference
We are working with the Govt. of KN to implement a baseline survey 17
CO
HE
SIV
E-I
ND
IA
![Page 18: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/18.jpg)
KN : Experimental Evaluation of Provider Incentives (2nd
component)• Cluster randomized trial of incentives for
providers to estimate effect of incentives for improvements in process measures v/s outcome measuresArm 1: Provider incentives evaluated based on
performance on quality of care indicators
Arm 2: Provider incentives evaluated based on improvement of MCH outcomes in catchment area population
Arm 3: Control group, with no incentives
18
CO
HE
SIV
E-I
ND
IA
![Page 19: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/19.jpg)
KN Incentives Experiment: Design Issues
• 180 rural clusters, in 3 groups of 60 each
• Each cluster is approximately at the level of the HOBLI (called kasba in N. India)
• On average: 3 OBGYN providers in each cluster
• Sample for the study: approx 550 doctors and 18,000 households (100 women who have a baby in catchment area of each cluster)
19
CO
HE
SIV
E-I
ND
IA
![Page 20: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/20.jpg)
Key challenge from the RCT perspective• Recall the two important steps in randomized evaluation
1. Randomly selecting potential participants from population
2. Randomly assign treatment to the group
• The second one is relatively easy – just write a STATA code• The first one is the big challenge
• Need to identify providers whom we can include in the study• Both a conceptual and logistic challenge• Need to define eligibility based on objective of the
experiment
20
CO
HE
SIV
E-I
ND
IA
![Page 21: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/21.jpg)
KN Incentives Experiment: Design Issues …contd…
• There are three key issues related to measurement in this study:
Definition of clusters and catchment area
Measurement of process measures of quality and health outcomes in a population
Identification of women who have had a baby and interviewing them in time 21
CO
HE
SIV
E-I
ND
IA
![Page 22: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/22.jpg)
Measurement Issues – contd..• Some things to keep in mind:
• Sample Sizes in Cluster Randomized Trials … the devil called Intra-Cluster Correlation
• Careful attention to defining eligible population – mapping is VERY effort and resource intensive, but has HUGE pay offs
• How practical is it to do RCTs? • “Politically Robust Randomization” – Gary King et al.
• Policy Relevance….• Very important topic, but we don’t have time for this today.• Working in conjunction w state governments to ensure buy-
in and policy impact.• Findings from our research in 2011.
22
CO
HE
SIV
E-I
ND
IA
![Page 23: Evaluation Designs in the IMATCHINE project: Regression Discontinuity Cluster Randomized Trial Presentation by Manoj Mohanan, Duke University 1 COHESIVE-INDIA.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649ca35503460f94963dcd/html5/thumbnails/23.jpg)
Thanks
• 3ie• Government of Gujarat• Government of Karnataka• DFID• World Bank • My colleagues at Sambodhi• For further details on project contact:
Ms. Manveen Kohli, Project Manager, IMATCHINE, [email protected] 23
CO
HE
SIV
E-I
ND
IA