New research methods for the evaluation of policy changes

Sanjay Basu, MD, PhD

basus@stanford.edu

O L D P R O B L E M S , N E W S O L U T I O N S

Three new methods to discuss

• If you have individual-level data, but an imperfect control group:• Near-far matching

• If you have population-level data, but an imperfect control group:• Synthetic control analysis

• If you have either type of data, and want to estimate disparities:• Distributional decomposition

Example: Is the school meal program “worsening child health”?

In a recent analysis: “Rural children in the meals program had a significantly higher probability of

being stunted than those not in the program, even after controlling for income differences.” (IIPS, 2014)

Our typical solutions

Propensity score matching • Problem: unobserved confounders

Our typical solutions

An instrumental variable Problem of “weak” instruments

Method: near-far matching

Baiocchi, et al., Health Serv Outcomes Res Method 2012

Re-analysis of food programs

Re-analysis of India school mean program and stunting Without matching: OR = 1.28 (1.12,1.44) With propensity score matching: OR= 0.96 (0.80, 1.12) With near-far matching: OR = 0.84 (0.70,0.98)

For a worked example, see: Lorch et al, Pediatrics, 2012

The ‘individualistic fallacy’

Some of our policies are designed to focus on a population-level outcome, not just an individual-level one

And many of our most interesting policies are ‘case studies’ of one group performing a policy, with no natural ‘control group’

Typical solution: difference-in-differences analysis

Example of synthetic control

For worked example, see Abadie et al., Am J Pol Sci, 2014 In Stata: ssc install synthIn R: install.packages("Synth")

Decomposition

As compared to standard regression

Distributional decomposition

For proof and worked example, see Basu et al., Am J Epi, 2015 In Stata: download distdecomp package from sdr.stanford.edu

References

Near-far matching Mike Baiocchi, http://web.stanford.edu/~baiocchi/ See: Baiocchi, et al., Health Serv Outcomes Res Method 2012

Synthetic control Jens Hainmueller, http://web.stanford.edu/~jhain/ See: Abadie et al., Am J Pol Sci, 2014

Distributional decomposition Sanjay Basu, http://web.stanford.edu/~basus/ See: Basu et al., AJE (in press), 2015

Additional slides

Regression discontinuity

Advantages:

only post-policy data

needed

Disadvantages:

people can ‘cheat’

only informs the margins