Econometrics with Observational Data: Research Design Todd Wagner.

Econometrics with Econometrics with Observational Data: Observational Data:

Research DesignResearch Design

Todd WagnerTodd Wagner

Research DesignResearch Design

Goal: evaluate behaviors and identify Goal: evaluate behaviors and identify causationcausation– Policy X caused effect YPolicy X caused effect Y

– Medication A resulted in B hospitalizationsMedication A resulted in B hospitalizations

Unit of analysis can be individual or Unit of analysis can be individual or organizationalorganizational

Research MethodsResearch Methods

Random assignment?

Intent to Treat?

Yes


Random assignment?

Intent to Treat?

Yes

Basic RCT Analysis

Yes No

On Treatment

On TreatmentOn Treatment

RCT comparing drug A to drug BRCT comparing drug A to drug B Adherence for drugs Adherence for drugs

– A is 70%A is 70%

– B is 40%B is 40% What does a comparison of A versus B What does a comparison of A versus B

tell us?tell us?


Random assignment?

Intent to Treat? Is there a control group?

Yes No


Is thererandom assignment

RandomizedTrial

Is there a control group

Quasi-experimentalDesign

DescriptiveStudy

Quasi-Experimental DesignsQuasi-Experimental Designs

Difference-in-differencesDifference-in-differences Regression discontinuityRegression discontinuity Switching replicationsSwitching replications Non-equivalent dependent variablesNon-equivalent dependent variables

Most commonIn health

Difference-in-DifferencesDifference-in-Differences

AKA: DD, D in D, or Diff in DiffAKA: DD, D in D, or Diff in Diff

Differences across time and armsDifferences across time and arms– Usually two arms: treatments, Usually two arms: treatments,

controlscontrols

– In theory can be used with 3+ armsIn theory can be used with 3+ arms

Methods for Identifying ControlsMethods for Identifying Controls

Inherent matchingInherent matching: Find similar : Find similar individuals not getting treatment to serve individuals not getting treatment to serve as controls (as controls (e.g.e.g., twins), twins)

StatisticalStatistical: use statistical techniques to : use statistical techniques to identify best comparison groupsidentify best comparison groups

LocationLocation: use other geographic sites, : use other geographic sites, states or regions as controlsstates or regions as controls

Unit of AnalysisUnit of Analysis

D in D works for different units of D in D works for different units of analysisanalysis– Person–people followed over timePerson–people followed over time– Site– sites followed over timeSite– sites followed over time– State– states followed over timeState– states followed over time

May need to make some analytical May need to make some analytical changes depending on unit of analysischanges depending on unit of analysis

Diff in Diff exampleDiff in Diff example

Gruber, Adams and Newhouse (1997)Gruber, Adams and Newhouse (1997) Tennessee increased Medicaid fees for Tennessee increased Medicaid fees for

primary care services (goal encourage primary care services (goal encourage office care; decrease hospital-based office care; decrease hospital-based ambulatory care)ambulatory care)

What is the effect of this policy change?What is the effect of this policy change?

Research DesignsResearch Designs

Difference-in-differencesDifference-in-differences Regression discontinuityRegression discontinuity Switching replicationsSwitching replications Nonequivalent dependent variablesNonequivalent dependent variables

Regression DiscontinuityRegression Discontinuity Participants are assigned to program or Participants are assigned to program or

comparison groups solely on the basis of comparison groups solely on the basis of an observed measure (education test or an observed measure (education test or means test)means test)

Appropriate when we wish to target a Appropriate when we wish to target a program or treatment to those who most program or treatment to those who most need or deserve itneed or deserve it

Regression DiscontinuityRegression Discontinuity Partial coverage (not everyone gets the treatment)Partial coverage (not everyone gets the treatment)

Requires the selection mechanism to be fully knownRequires the selection mechanism to be fully known

Selection mechanism must be consistently applied to Selection mechanism must be consistently applied to all personsall persons

RD Design GraphicallyRD Design Graphically

Source: Urban Institute

Threshold MUST be known and consistently applied

Test forsignificance

Switching ReplicationsSwitching Replications Has two groups and three waves of Has two groups and three waves of

measurementmeasurement

AKA waitlist control groupAKA waitlist control group

This design is sometimes used in This design is sometimes used in randomized trialsrandomized trials

Example from Pap Smear StudyExample from Pap Smear Study

treat

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 > 12

Months since Initial Pap

Cu

mu

lati

ve %

Fo

llo

wed

Up

Intervention Control

Immediate treatment

delayed treatment

Non-Equivalent DVsNon-Equivalent DVs

Analyze dependent variable that should Analyze dependent variable that should not be affected by the interventionnot be affected by the intervention

Example: Intervention is designed to Example: Intervention is designed to affect quality of diabetes care, but could affect quality of diabetes care, but could also see if intervention affected quality of also see if intervention affected quality of asthma careasthma care

Notes on the AnalysisNotes on the Analysisof DD dataof DD data

Analytical MethodsAnalytical Methods

Plot or graph unadjusted dataPlot or graph unadjusted data

Graduate to more complex modelsGraduate to more complex models

Address, if possible, model Address, if possible, model

limitationslimitations

DD Raw DataDD Raw Data

Standard deviations in parentheses+DD = (Expfollowup- Expbaseline)-(Controlfollowup- Controlbaseline)† unadjusted estimates

Baseline 1-Year Follow-Up Exp. Control Exp Control DD+ -----------------------------------------------------------------------------------------Utilization Entry (% yes) 84.5% 86.1% 88.9% 86.8% 3.7 (36.2) (34.6) (31.4) (33.9)

No. of visits (0-16) 3.69 3.84 3.73 3.67 0.21 (4.28) (4.36) (4.00) (4.07) ------------------------------------------------------------------------------------------

Diff n Diff ModelDiff n Diff Model

Y = Y = + + G + G + T + T + GT+ GT+ XX + +

Y=outcomeY=outcomeG = group (0=control, 1=treatment) G = group (0=control, 1=treatment)

T= time (0=baseline, 1=follow-up)T= time (0=baseline, 1=follow-up)

X = characteristics of person, place, etc.X = characteristics of person, place, etc.

= error term = error term

Program EffectProgram Effect

ff33 = 0 then the program has no effect = 0 then the program has no effect

Limited statistical power. Testing interactions increases risk of type 2 Limited statistical power. Testing interactions increases risk of type 2 error.error.

Outcome = + 1G + 2T + 3GT + X +

+------------------------------+avgcost sta3n exp yr_d year--------------------------------. 358 0 0 93. 358 0 1 94318.2305 402 1 0 93323.2815 402 1 1 94472.0291 405 1 0 93480.1368 405 1 1 94364.0456 436 0 0 93398.9824 436 0 1 94369.9669 437 0 0 93346.4565 437 0 1 94270.0007 438 0 0 93322.2588 438 0 1 94292.7632 442 1 0 93. 442 1 1 94475.6746 452 1 0 93494.9601 452 1 1 94

Note: Data Listed in Stata

Organizing the Dataset

IdentificationIdentification

How do you obtain an unbiased estimate of How do you obtain an unbiased estimate of 33??

For an unbiased estimate of GT, G must not For an unbiased estimate of GT, G must not be correlated with be correlated with that is, G must be that is, G must be exogenousexogenous

Outcome = + 1G + 2T + 3GT + X +

IdentificationIdentification

G may be endogenousG may be endogenous Selection biasSelection bias

– Selection bias is type of endogeneitySelection bias is type of endogeneity

– Caused by non-random assignmentCaused by non-random assignment

– Outcome and G (group) affect each other -- Outcome and G (group) affect each other -- causality runs both wayscausality runs both ways

– Impact: Impact: 33 is biased is biased

Outcome = + 1G + 2T + 3GT + X +

Wagner, T. H., & Chen, S. (2005). An economic evaluation of inpatient residential treatment programs in the department of veterans affairs. Med Care Res Rev, 62(2), 187-204.

Example: VA Residential Example: VA Residential TreatmentTreatment

Residential Treatment ProgramsResidential Treatment Programs

RTPs provide mental health and RTPs provide mental health and substance use treatmentsubstance use treatment

RTPs were designed toRTPs were designed to– treat eligible veterans in a less-treat eligible veterans in a less-

intensive and more self-reliant setting.intensive and more self-reliant setting.– to provide cost-effective care that to provide cost-effective care that

“promotes independence and fosters “promotes independence and fosters responsibility.”responsibility.”

ObjectivesObjectives

1.1. Did the RTPs save money?Did the RTPs save money?

2.2. Were savings a “one-time” event or do Were savings a “one-time” event or do they continue to accrue?they continue to accrue?

Design ChoiceDesign Choice

Selection mechanism is not observed– Selection mechanism is not observed– can’t use regression discontinuitycan’t use regression discontinuity

We know who adopted RTP and when– We know who adopted RTP and when– DD is feasibleDD is feasible

MethodsMethods Built a longitudinal dataset for 1993-1999 for Built a longitudinal dataset for 1993-1999 for

all VA medical centersall VA medical centers

Tracked approved RTP programs (N=43)Tracked approved RTP programs (N=43)

We merged data from the PTF and CDR to We merged data from the PTF and CDR to tracktrack– Total MH inpatient days (PTF) and dollars (CDR)Total MH inpatient days (PTF) and dollars (CDR)– Total SA inpatient days (PTF) and dollars (CDR)Total SA inpatient days (PTF) and dollars (CDR)

Outcomes Outcomes

Department-level costsDepartment-level costs– Average cost per MH dayAverage cost per MH day

– Average cost per SA dayAverage cost per SA day

– Total MH/SA department costsTotal MH/SA department costs Sensitivity analysis Sensitivity analysis

– Outpatient MH/SA costsOutpatient MH/SA costs

– FTEFTE

Multivariate modelsMultivariate models Fixed-effects modelsFixed-effects models11

– DV: Department-level costsDV: Department-level costs

– Controlled for medical center sizeControlled for medical center size

– Inflation adjusted to 1999 using CPIInflation adjusted to 1999 using CPI

– Year dummiesYear dummies

– Wage indexWage index

1 Random effects were similar; Hausman tests were not significant. Fixed effects were more conservative.

Results: Mental HealthResults: Mental Health

Average cost savings of $81 per day Average cost savings of $81 per day (p<0.01).(p<0.01).

Savings do not appear to be increasing Savings do not appear to be increasing over time.over time.

Mental Health CostsMental Health Costs

0

200

400

600

800

1993

1994

1995

1996

1997

1998

1999

Fiscal year

RTP

No RTP

Results: Substance AbuseResults: Substance Abuse

Average cost savings of $112 per day Average cost savings of $112 per day (p<0.01).(p<0.01).

Savings do not appear to be increasing Savings do not appear to be increasing over time.over time.

Mental Health CostsMental Health Costs

0

100

200

300

400

500

1993 1994 1995 1996 1997 1998 1999

Fiscal year

RTP

No RTP

Sensitivity AnalysisSensitivity Analysis

RTPs were associated with a slight RTPs were associated with a slight decrease in the costs of outpatient decrease in the costs of outpatient psychiatry.psychiatry.

RTPs were associated with a decrease in RTPs were associated with a decrease in FTEFTE

LimitationsLimitations

Not clear if RTPs could be better– are Not clear if RTPs could be better– are they treating the right patient?they treating the right patient?

Endogeneity of RTPsEndogeneity of RTPs– 1 and 2 year lags (medical centers with 1 and 2 year lags (medical centers with

RTPs in 1994 and 1995) are not associated RTPs in 1994 and 1995) are not associated with costswith costs

– There does not appear to be self-selection in There does not appear to be self-selection in RTPs.RTPs.

Any Questions?Any Questions?

Design ReferencesDesign References

Trochim, W. Research Methods Knowledge Database http://www.socialresearchmethods.net/kb/

Rossi, PH, and HE Freeman. Evaluation: A systematic approach. 5th ed. New York: Sage, 1993.

Regression ReferencesRegression References

Wm. Greene. Wm. Greene. Econometric AnalysisEconometric Analysis. .

J Wooldridge. J Wooldridge. Econometric Analysis Econometric Analysis of Cross Section and Panel Dataof Cross Section and Panel Data..

You’ve Almost Made ItYou’ve Almost Made It

June11June11thth Mark Smith, Endogeneity Mark Smith, Endogeneity TBA: Todd Wagner: Using StataTBA: Todd Wagner: Using Stata

Econometrics with Observational Data: Research Design Todd Wagner.

Documents

Transcript of Econometrics with Observational Data: Research Design Todd Wagner.