Simplifying Causal Inference

Gary King

Institute for Quantitative Social ScienceHarvard University

(Talk at the Center for Population and Development Studies,Harvard University, 11/8/2012)

Gary King (Harvard, IQSS) Matching Methods 1 / 58

Overview

Problem: Model dependence (review)

Solution: Matching to preprocess data (review)

Problem: Many matching methods & specifications

Solution: The Space Graph helps us choose

Problem: The most commonly used method can increase imbalance!

Solution: Other methods do not share this problem

(Coarsened Exact Matching is simple, easy, and powerful)

Lots of insights revealed in the process

Overview

Model Dependence Example

Data: 124 Post-World War II civil wars

Dependent variable: peacebuilding success

Treatment variable: multilateral UN peacekeeping intervention (0/1)

Control vars: war type, severity, duration; development status; etc.

Counterfactual question: UN intervention switched for each war

Data analysis: Logit model

The question: How model dependent are the results?

Model Dependence ExampleReplication: Doyle and Sambanis, APSR 2000

Two Logit Models, Apparently Similar Results

Original “Interactive” Model Modified ModelVariables Coeff SE P-val Coeff SE P-valWartype −1.742 .609 .004 −1.666 .606 .006Logdead −.445 .126 .000 −.437 .125 .000Wardur .006 .006 .258 .006 .006 .342Factnum −1.259 .703 .073 −1.045 .899 .245Factnum2 .062 .065 .346 .032 .104 .756Trnsfcap .004 .002 .010 .004 .002 .017Develop .001 .000 .065 .001 .000 .068Exp −6.016 3.071 .050 −6.215 3.065 .043Decade −.299 .169 .077 −0.284 .169 .093Treaty 2.124 .821 .010 2.126 .802 .008UNOP4 3.135 1.091 .004 .262 1.392 .851Wardur*UNOP4 — — — .037 .011 .001Constant 8.609 2.157 0.000 7.978 2.350 .000N 122 122Log-likelihood -45.649 -44.902Pseudo R2 .423 .433

Doyle and Sambanis: Model Dependence

Model Dependence: A Simpler Example

What to do?

Preprocess I: Eliminate extrapolation region

Preprocess II: Match (prune bad matches) within interpolation region

Model remaining imbalance

Model Dependence: A Simpler Example(King and Zeng, 2006: fig.4 Political Analysis)

What to do?

Matching within the Interpolation Region(Ho, Imai, King, Stuart, 2007: fig.1, Political Analysis)

Education (years)

12 14 16 18 20 22 24 26 28

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

Matching reduces model dependence, bias, and variance

How Matching Works

Notation:

Yi Dependent variableTi Treatment variable (0/1, or more general)Xi Pre-treatment covariates

Treatment Effect for treated (Ti = 1) observation i :

TEi = Yi (Ti = 1)−Yi (Ti = 0)

= observed −unobserved

Estimate Yi (Ti = 0) with Yj from matched (Xi ≈ Xj) controls

Yi (Ti = 0) = Yj(Ti = 0) or a model Yi (Ti = 0) = g0(Xj)

Prune unmatched units to improve balance (so X is unimportant)

QoI: Sample Average Treatment effect on the Treated:

SATT =1

∑i∈{Ti=1}

or Feasible Average Treatment effect on the Treated: FSATT

How Matching Works

Notation:

Yi Dependent variableTi Treatment variable (0/1, or more general)Xi Pre-treatment covariates

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

Notation:Yi Dependent variable

Ti Treatment variable (0/1, or more general)Xi Pre-treatment covariates

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

Notation:Yi Dependent variableTi Treatment variable (0/1, or more general)

Xi Pre-treatment covariates

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

Notation:Yi Dependent variableTi Treatment variable (0/1, or more general)Xi Pre-treatment covariates

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

How Matching Works

TEi = Yi (Ti = 1)−Yi (Ti = 0)

SATT =1

∑i∈{Ti=1}

Method 1: Mahalanobis Distance Matching

1 Preprocess (Matching)

Distance(Xi ,Xj) =√

(Xi − Xj)′S−1(Xi − Xj)Match each treated unit to the nearest control unitControl units: not reused; pruned if unusedPrune matches if Distance>caliper

2 Estimation Difference in means or a model

(Xi − Xj)′S−1(Xi − Xj)

Match each treated unit to the nearest control unitControl units: not reused; pruned if unusedPrune matches if Distance>caliper

(Xi − Xj)′S−1(Xi − Xj)Match each treated unit to the nearest control unit

Control units: not reused; pruned if unusedPrune matches if Distance>caliper

(Xi − Xj)′S−1(Xi − Xj)Match each treated unit to the nearest control unitControl units: not reused; pruned if unused

Prune matches if Distance>caliper

Mahalanobis Distance Matching

Education (years)

12 14 16 18 20 22 24 26 28

Education (years)

12 14 16 18 20 22 24 26 28

Education (years)

12 14 16 18 20 22 24 26 28

Education (years)

12 14 16 18 20 22 24 26 28

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

TT TT T TTTTT TT

CC C CC

CCC CCCCC

C CCCC

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

TT TT T TTTTT TT

CC C CC

Education (years)

12 14 16 18 20 22 24 26 28

T TT T

TT TT T TTTTT TT

CC C CC

Method 2: Propensity Score Matching

Reduce k elements of X to scalar πi ≡ Pr(Ti = 1|X ) = 11+e−Xi β

Distance(Xi ,Xj) = |πi − πj |Match each treated unit to the nearest control unitControl units: not reused; pruned if unusedPrune matches if Distance>caliper

Distance(Xi ,Xj) = |πi − πj |

Match each treated unit to the nearest control unitControl units: not reused; pruned if unusedPrune matches if Distance>caliper

Distance(Xi ,Xj) = |πi − πj |Match each treated unit to the nearest control unit

Control units: not reused; pruned if unusedPrune matches if Distance>caliper

Distance(Xi ,Xj) = |πi − πj |Match each treated unit to the nearest control unitControl units: not reused; pruned if unused

Prune matches if Distance>caliper

Propensity Score Matching

Education (years)

12 16 20 24 28

Education (years)

12 16 20 24 28

PropensityScore

Education (years)

12 16 20 24 28

PropensityScore

CCCCCCCCC

Education (years)

12 16 20 24 28

PropensityScore

CCCCCCCCC

Education (years)

12 16 20 24 28

PropensityScore

CCCCCCCCC

Education (years)

12 16 20 24 28

PropensityScore

Education (years)

12 16 20 24 28

C TTTT

PropensityScore

Education (years)

12 16 20 24 28

C TTTT

Method 3: Coarsened Exact Matching

Temporarily coarsen X as much as you’re willing

e.g., Education (grade school, high school, college, graduate)Easy to understand, or can be automated as for a histogram

Apply exact matching to the coarsened X , C (X )

Sort observations into strata, each with unique values of C(X )Prune any stratum with 0 treated or 0 control units

Pass on original (uncoarsened) units except those pruned

Need to weight controls in each stratum to equal treatedsCan apply other matching methods within CEM strata (inherit CEM’sproperties)

Temporarily coarsen X as much as you’re willing

1 Preprocess (Matching)Temporarily coarsen X as much as you’re willing

e.g., Education (grade school, high school, college, graduate)

Easy to understand, or can be automated as for a histogram

Sort observations into strata, each with unique values of C(X )

Prune any stratum with 0 treated or 0 control units

Need to weight controls in each stratum to equal treateds

Can apply other matching methods within CEM strata (inherit CEM’sproperties)

Coarsened Exact Matching

Education

12 14 16 18 20 22 24 26 28

CCC CCC CC

C CC C CCC CCCCCCC CC CCC CCCCCC

C CCC CC C

T TT T

TT TT T TTTTT TT

Education

HS BA MA PhD 2nd PhD

Drinking age

Don't trust anyoneover 30

The Big 40

Senior Discounts

Retirement

CCC CCC CC

C CCC CC C

T TT T

TT TT T TTTTT TT

Education

Drinking age

The Big 40

Senior Discounts

Retirement

CCC CCC CC

C CCC CC C

T TT T

TT TT T TTTTT TT

Education

Drinking age

The Big 40

Senior Discounts

Retirement

CC C CCC CC CCCC

TTT T TT

TTT TT

Education

Drinking age

The Big 40

Senior Discounts

Retirement

CCCC C CC

C CC CCCC

TTT T TT

TTT TT

Education

12 14 16 18 20 22 24 26 28

CCCC C CC

C CC CCCC

TTT T TT

TTT TT

The Bias-Variance Trade Off in Matching

Bias (& model dependence) = f (imbalance, importance, estimator) we measure imbalance instead

Variance = f (matched sample size, estimator) we measure matched sample size instead

Bias-Variance trade off Imbalance-n Trade Off

Measuring Imbalance

Classic measure: Difference of means (for each variable)Better measure (difference of multivariate histograms):

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Measuring Imbalance

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Measuring Imbalance

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Measuring Imbalance

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Measuring Imbalance

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Measuring Imbalance

Classic measure: Difference of means (for each variable)

Better measure (difference of multivariate histograms):

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Measuring Imbalance

L1(f , g ;H) =1

∑`1···`k∈H(X)

|f`1···`k− g`1···`k

Comparing Matching Methods

MDM & PSM: Choose matched n, match, check imbalance

CEM: Choose imbalance, match, check matched n

Best practice: iterate

But given the matched solution matching method is irrelevant

Our idea: Identify the frontier of lowest imbalance for each given n,and choose a matching solution

A Space Graph: Foreign Aid Shocks & ConflictKing, Nielsen, Coberley, Pope, and Wells (2012)

Mahalanobis Discrepancy L1

Imbalance Metric

Difference in Means

N of Matched Sample

2500 1500 500 0

published PSM

published PSM with 1/4 sd caliper

N of Matched Sample

2500 1500 500 0

●●

published PSM

published PSM with1/4 sd caliper

N of Matched Sample

2500 1500 500 0

published PSM

published PSM with 1/4 sd caliper

● Raw DataRandom Pruning

"Best Practices" PSMPSM

MDMCEM

A Space Graph: Healthways DataKing, Nielsen, Coberley, Pope, and Wells (2012)

A Space Graph: Called/Not Called DataKing, Nielsen, Coberley, Pope, and Wells (2012)

A Space Graph: FDA Drug Approval TimesKing, Nielsen, Coberley, Pope, and Wells (2012)

A Space Graph: Job Training (Lelonde Data)King, Nielsen, Coberley, Pope, and Wells (2012)

A Space Graph: Simulated Data — Mahalanobis

MDM: 1 Covariate

N of matched sample

500 250 0

HighMedLow

Imbalance:

MDM: 2 Covariates

N of matched sample

500 250 0

HighMedLow

Imbalance:

MDM: 3 Covariates

N of matched sample

500 250 0

HighMedLow

Imbalance:

A Space Graph: Simulated Data — CEM

CEM: 1 Covariate

N of matched sample

500 250 0

HighMedLow

Imbalance:

CEM: 2 Covariates

N of matched sample

500 250 0

HighMedLow

Imbalance:

CEM: 3 Covariates

N of matched sample

500 250 0

HighMedLow

Imbalance:

A Space Graph: Simulated Data — Propensity Score

PSM: 1 Covariate

N of matched sample

500 250 0

HighMedLow

Imbalance:

PSM: 2 Covariates

N of matched sample

500 250 0

HighMedLow

Imbalance:

PSM: 3 Covariates

N of matched sample

500 250 0

HighMedLow

Imbalance:

PSM Approximates Random Matching in Balanced Data

Covariate 1

−2 −1 0 1 2

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

PSM MatchesCEM and MDM Matches

CEM Weights and Nonparametric Propensity Score

CEM Weight: wi =mT

(+ normalization)

CEM Pscore: Pr(Ti = 1|Xi ) =mT

mTi + mC

Gives a better pscore than PSM

Doesn’t match based on crippled information

CEM Weight: wi =mT

(+ normalization)

mTi + mC

CEM Weight: wi =mT

(+ normalization)

mTi + mC

CEM Weight: wi =mT

(+ normalization)

mTi + mC

CEM Weight: wi =mT

(+ normalization)

mTi + mC

Destroying CEM with PSM’s Two Step Approach

Covariate 1

−2 −1 0 1 2

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

● ●●

●●

● ●

CEM MatchesCEM−generated PSM Matches

Conclusions

Propensity score matching:

The problem:

Imbalance can be worse than original dataCan increase imbalance when removing the worst matchesApproximates random matching in well-balanced data(Random matching increases imbalance)

The Cause: unnecessary 1st stage dimension reductionImplications:

Balance checking requiredAdjusting for potentially irrelevant covariates with PSM: mistakeAdjusting experimental data with PSM: mistakeReestimating the propensity score after eliminating noncommonsupport: mistake1/4 caliper on propensity score: mistake

In four data sets and many simulations:CEM > Mahalanobis > Propensity Score(Your performance may vary)CEM and Mahalanobis do not have PSM’s problemsYou can easily check with the Space Graph

Conclusions

Propensity score matching:

The problem:

Conclusions

Propensity score matching:The problem:

Conclusions

Imbalance can be worse than original data

Can increase imbalance when removing the worst matchesApproximates random matching in well-balanced data(Random matching increases imbalance)

Conclusions

Imbalance can be worse than original dataCan increase imbalance when removing the worst matches

Approximates random matching in well-balanced data(Random matching increases imbalance)

Conclusions

The Cause: unnecessary 1st stage dimension reduction

Implications:

Conclusions

Balance checking required

Adjusting for potentially irrelevant covariates with PSM: mistakeAdjusting experimental data with PSM: mistakeReestimating the propensity score after eliminating noncommonsupport: mistake1/4 caliper on propensity score: mistake

Conclusions

Balance checking requiredAdjusting for potentially irrelevant covariates with PSM: mistake

Adjusting experimental data with PSM: mistakeReestimating the propensity score after eliminating noncommonsupport: mistake1/4 caliper on propensity score: mistake

Conclusions

Balance checking requiredAdjusting for potentially irrelevant covariates with PSM: mistakeAdjusting experimental data with PSM: mistake

Reestimating the propensity score after eliminating noncommonsupport: mistake1/4 caliper on propensity score: mistake

Conclusions

Balance checking requiredAdjusting for potentially irrelevant covariates with PSM: mistakeAdjusting experimental data with PSM: mistakeReestimating the propensity score after eliminating noncommonsupport: mistake

1/4 caliper on propensity score: mistake

Conclusions

In four data sets and many simulations:CEM > Mahalanobis > Propensity Score

(Your performance may vary)CEM and Mahalanobis do not have PSM’s problemsYou can easily check with the Space Graph

Conclusions

In four data sets and many simulations:CEM > Mahalanobis > Propensity Score(Your performance may vary)

CEM and Mahalanobis do not have PSM’s problemsYou can easily check with the Space Graph

Conclusions

In four data sets and many simulations:CEM > Mahalanobis > Propensity Score(Your performance may vary)CEM and Mahalanobis do not have PSM’s problems

You can easily check with the Space Graph

Conclusions

For papers, software (for R, Stata, & SPSS), tutorials, etc.

http://GKing.Harvard.edu/cem

Data where PSM Works Reasonably Well — PSM & MDM

−4 −2 0 2 4

Unmatched Data: L1 = 0.685

Covariate 1

−4 −2 0 2 4

PSM: L1 = 0.452

Covariate 1

−4 −2 0 2 4

MDM: L1 = 0.448

Covariate 1

Data where PSM Works Reasonably Well — CEM

−4 −2 0 2 4

Bad CEM: L1 = 0.661

Covariate 1

100% of the treated units

−4 −2 0 2 4

Better CEM: L1 = 0.188

Covariate 1

−4 −2 0 2 4

Even Better CEM: L1 = 0.095

Covariate 1

Simplifying Causal Inference - Harvard University

Transcript of Simplifying Causal Inference - Harvard University

Simplifying Causal Inference - Harvard University

Documents

Transcript of Simplifying Causal Inference - Harvard University

Mediation: The Causal Inference Approach

Causal Inference and Data-Fusion in Econometrics · 2019-03-29 · Causal Inference in Econometrics I Despite a strong interest in causal inference in general, graphical models of

Lecture 1: Introduction, Regressions and Causal Inference 1.pdf · Lecture 1: Introduction, Regressions and Causal Inference January 10, 2016. Introduction Regressions Causal Inference

Causal inference in practice

Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference MSM

Applying Causal Inference Techniques to Strengthen Dual ... · Modern Causal Inference Techniques Modern causal inference techniques can be used to account for the absence of random

Experimental Causal Inference

Causal Inference with Noisy and Missing Covariates via ...papers.nips.cc/paper/7924-causal-inference-with-noisy...Causal Inference with Noisy and Missing Covariates via Matrix Factorization

How Big Data and Causal Inference Work Together in Health ... · –Three Ideas for Causal Inference –Ideas that aim at solving the problem of causal inference in observational

Introduction to Causal Bayesian Inference Chris Holmes ... · Causal Bayesian Inference 3 Overview of Course – Introduction to causal inference and statistical association – The

Causal Inference and Ambiguous Manipulations

Causal Inference Overview - Harvard University · Fundamental Problem of Causal Inference (ctd). I The fundamental problem of causal inference is that at most only one of the two

Causal Inference v16

Causal Inference and Machine Learning - uni-potsdam.de · 2019-09-09 · Causal Inference Course 1 September 2019 Potsdam Causal Inference and Machine Learning Guido Imbens, imbens@stanford.edu

Generalizability in Causal Inference - Carlos Cinelli and Bareinboim...Cinelli, Bareinboim, SoCal 2019 - Generalizability in Causal Inference Causal inference: causal models Inputs:

Lecture 1: Causal Inference in Social Science€¦ · Outlines 1 Course Overview 2 Causal Inference in Social Science Causal Inference: The Core of Empirical Studies in Economics

Machine Learning and Causal Inference

THE FOUNDATIONS OF CAUSAL INFERENCE

Diff: simplifying the causal inference analysis with difference … · Diff: simplifying the causal inference analysis with difference-in-differences Juan M. Villa PhD Student –