Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment

Amanda Kowalski

Associate Professor, Department of Economics, YaleFaculty Research Fellow, NBERVisiting Associate Professor, SIEPR

October 2015

Experiments Address Internal Validity,But What About External Validity?

• Explosion in use of experiments• Are they externally valid?• Quandary: what to do if the results from

randomized experiments do not conform to results of seemingly convincing natural experiments?

• Policy-relevant example: Oregon Health Insurance Experiment vs. Massachusetts Health Reform

• My results show that there may not be a conflict after all…

Oregon Health Insurance Experiment (OHIE) Shows ER Visits Increased

• Do these results generalize to other settings?• Massachusetts Health Reform showed – ER visits decreased • Miller (2012), Smulowitz et al. (2011)

– ER visits stayed the same• Chen, Scheffler, Chandra (2011)

– Admissions from ER (proxy for ER visits) decreased• Kolstad & Kowalski (2012)

I Apply MTE Methods to Examine External Validity of the OHIE

• Prior literature– Bjorklund and Moffitt (1987)– Heckman Vytlacil (1999, 2005, 2007)– Brinch, Mogstad, Wiswall (2015)

• Extended Heckman Vytlacil to allow discrete instrument

• MTEs in experiments (binary instruments) are now possible• To my knowledge, these methods have not been applied to experiments

(Brinch et al. application is to quantity-quality of children in IV framework)

• Key insight: unless everyone assigned to lottery gets intervention (and everyone else does not), then external validity will be affected by selection

I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon

• Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization

• Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate

• MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

Selection into treatment given lottery is key model element

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

MTE is treatment effect for individuals indifferent to treatment

• Can recover LATE from MTE:

Can identify linear MTE from standard LATE assumptions, nonlinear MTE requires additional separability

• Assumption 1 sufficient for linear MTE– Standard LATE conditional independence and monotonicity– Key is additional moment: separation of compliers from always takers

and never takers• Assumption 2 yields nonlinear MTE

– Additive separability between observed and unobserved heterogeneity often imposed in applied work

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

All

Always Takers and

Treated Compliers

Always Takers

Never Takers

Never Takers and

Control Compliers

Treated Compliers

Control Compliers

All Compliers

D = 1 D = 0Z = 1 Z = 0 Z = 1 Z = 0 Z = 1 Z = 0 Z = 1 Z = 0

Female 0.56 0.55 0.56 0.58 0.66 0.50 0.52 0.51 0.55 0.53Age in 2009 40.7 40.7 40.7 41.4 40.2 39.6 40.9 42.5 42.6 42.6English 0.91 0.91 0.91 0.92 0.91 0.90 0.91 0.92 0.92 0.92Y (Any ER visit) 0.37 0.38 0.37 0.43 0.44 0.31 0.34 0.41 0.37 0.38Y (Number of ER visits) 1.12 1.16 1.09 1.36 1.38 0.85 0.97 1.35 1.13 1.20Number of Observations 19,643 6,755 12,888 4,096 3,747 2,659 9,141 2,132 4,068 6,200D = 1 if any insuranceZ = 1 if randomized into lottery

D = 1 D = 0

Source: Oregon Administrative Data, 1 Lottery Entrant in Household.

Compliers, Always Takers, Never Takers

• Standard formulas for average characteristics

Always takers: D=1 regardless of Z=1 or Z=0Compliers: D=1 if Z=1 but D=0 if Z=0Never takers: D=0 regardless of Z=1 or Z=0Defiers: D=1 if Z=0 but D=0 if Z=1 (not present by assumption)

New Test of Internal Validity

Test of External Validity(1) (2) (3) (4) (5) (6) (7) [(4) - (5)] - [(6) - (7)]

All

Always Takers and

Treated Compliers

Always Takers

Never Takers

Never Takers and

Control Compliers

Difference-in-difference Test of External Validity

Z = 1 Z = 0 Z = 1 Z = 0 Z = 1 Z = 0Female 0.56 0.55 0.56 0.58 0.66 0.50 0.52 -0.058

(0.016)†††(0.014)***

Age in 2009 40.7 40.7 40.7 41.4 40.2 39.6 40.9 2.586(0.388)†††(0.383)***

English 0.91 0.91 0.91 0.92 0.91 0.90 0.91 0.013(0.009)(0.009)

Y (Any ER visit) 0.37 0.38 0.37 0.43 0.44 0.31 0.34 0.007(0.015)(0.017)

Y (Number of ER visits) 1.12 1.16 1.09 1.36 1.38 0.85 0.97 0.103(0.081)(0.087)

Number of Observations 19,643 6,755 12,888 4,096 3,747 2,659 9,141††† p<0.01, †† p<0.05, † p<0.1; asymptotic standard errors*** p<0.01, ** p<0.05, *p<0.1; bootstrapped standard errorsD = 1 if any insuranceZ = 1 if randomized into lottery

D = 1 D = 0

Source: Oregon Administrative Data, 1 Lottery Entrant in Household.

The Geometry of the LATE and MTE

Always takers Always takers and

treated compliers

Never takers

Never takers and

control compliers

The Geometry of the LATE and MTE

Always takers Always takers and

treated compliers

Never takers

Never takers and

control compliers

The Geometry of the LATE and MTE

The Geometry of the LATE and MTE

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

Any Emergency Room Visits(1) (2) (3) (4) (5) (6) (7)

Medicaid 0.0697(0.024)†††

(0.024)***

Any insurance 0.0610 0.0665 -0.0124 0.0633 0.207 0.0453(0.021)††† (0.026)†† (0.022) (0.022)††† (0.087)†† (0.023)††

(0.021)*** (0.023)*** (0.022) (0.022)*** (0.094)** (0.022)**

Control variablesPre-visits,

Lottery Entrants

Pre-visits, Lottery

Entrants

Lottery Entrants

No Controls

Lottery Entrants, Common Controls

Common Controls

Common Controls

Regression sampleFull sample Full sample Full sample Full sample Full sample

2+ Lottery Entrants

1 Lottery Entrant

Observations 24,646 24,646 24,646 24,646 24,646 5,003 19,643R-squared 0.151 0.146 0.023 0.036 0.050 0.033 0.033Mean of Y in control group 0.34 0.34 0.34 0.34 0.34 0.21 0.37

Bringing MTE to OHIE

• Control for lottery entrants matters• Otherwise, replicate well with covariates• Results for number of visits similar

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

MTE Results

MTE Weights

Weighted MTE

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

Propensity Scores in MA

Propensity Scores in MACompliers Only, Rescaled

Massachusetts Weights

Oregon MTE/Massachusetts WeightsNegative ER Impacts Plausible in MA

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

Treatment Effect HeterogeneityBinary Subgroup Analysis

• Point estimates all positive except non-English speakers

Treatment Effect HeterogeneityMultivariate Subgroup Analysis

• Many negative point estimates

38

Who Are the People with Reductions in ER Utilization?

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

Experimental Design for External Validity

• Collect data on treatment (endogenous variable)• Collect a wide array of covariates– Use standard definitions

• Only stratify on covariates that can be obtained elsewhere

• Collect data on experimental subjects randomized out of treatment who get treatment via other means

• Design experiment so that always takers and never takers are possible

Lessons for Experimentalists

• Going to great lengths to encourage “attendance” could decrease external validity– In the limiting case, if full attendance, cannot

estimate MTE• Going to great lengths to encourage

“attendance” could decrease estimated effect size

Outline

• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions

I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon

• Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization

• Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate

• MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments

Appendix Slides

Appendix: Bringing OHIE to MTENumber of ER visits

• Return to similar results for any visits

Appendix:Formulas for Complier Characteristics

• Treated Compliers

• Control Compliers

• Return to results with complier characteristics