Propensity Scores Friday, June 1 st , 10:15am-12:00pm
description
Transcript of Propensity Scores Friday, June 1 st , 10:15am-12:00pm
Propensity ScoresFriday, June 1st, 10:15am-12:00pm
Deborah Rosenberg, PhD Kristin Rankin, PhDResearch Associate Professor Research Assistant Professor
Division of Epidemiology and BiostatisticsUniversity of IL School of Public Health
Training Course in MCH Epidemiology
22
Propensity Scores
The goal of using propensity scores is to more completely and efficiently address observed confounding of an exposure-outcome relationship.
Program evaluation – Addresses selection bias
Epidemiology – Addresses non-randomization of exposure
Propensity scores are the predicted probabilities from a regression model of this form:
Exposure = pool of observed confounders
“Conditional probability of being exposed or treated (or both)”
33
Propensity Scores
When exposed and unexposed groups are not equivalent such that the distribution on covariates is not only different, but includes non-overlapping sets of values, then the usual methods for controlling for confounding may be inadequate.
Non-overlapping distributions (lack of common support) means that individuals in one group have values on some of the covariates that don’t exist in the other group and vice versa.
4Sturmer, et al 2006, J Clin Epidemiol
Area of “Common Support”
5
Benefits of Propensity Score Methods
The accessibility of multivariable regression methods means they are often misused, with reporting of estimates that are extrapolations beyond available data.
The process of generating propensity scores:– focuses attention on model specification to account for
covariate imbalance across exposure groups, and support of data with regard to “exchangeability” of exposed and unexposed
– Allows for trying to mimic randomization by simultaneously matching people on large sets of known covariates
– Forces researcher to design study/check covariate balance before looking at outcomes
Oakes and Johnson, Methods in Social Epidemiology
6
Propensity Scores
Propensity scores might be used in three ways:
1. as a covariate in a model along with exposure, or as weights for the observations in a crude model (not recommended due to possible off-support inference)
2. as values on which to stratify/subclassify data to form more comparable groups
3. as values on which to match an exposed to an unexposed observation, then using the matched pair in an analysis that accounts for the matching
77
Propensity Scores
Propensity scores are the predicted probabilities from a regression model of this form:
Exposure = pool of observed confounders
proc logistic data=analysis desc;
class &propenvars / param=ref ref=first;
model adeq=&propenvars;
output out=predvalues p=propscore; run;
Once the propensity scores are generated, they are used to run the real model of interest:
outcome = exposure*Note: Make sure you start with a dataset with no missing values on outcome, or you will end up with unmatched pairs
8
Generating Propensity Scores
• Consider only covariates that are measured pre-program/intervention/exposure or do not change over time; value shouldn’t be affected by exposure or in causal pathway between exposure and outcome
• Covariates should be based on theory or prior empirical findings; never use model selection procedures such as stepwise selection for these covariates – if conceptually based, they should stay in the model regardless of statistical significance
• Include higher order terms and interactions to get best estimated probability of exposure and balance across covariates; trade-off between fully accounting for confounding and including so many unnecessary variables/terms that common support becomes an issue and PS distributions are more likely to be non-overlapping
8Oakes and Johnson, Methods in Social Epidemiology
9
Propensity Score Distributions
Examine the distribution of propensity scores in exposed and unexposed
• If there is not enough overlap (not enough “common support”), then these data cannot be used to answer the research question
• Observations with no overlap cannot be used in matched analysis
• If there are areas that don’t overlap, the matched sample may not be representative (examine characteristics of excluded individuals to assess this)
9
10
Propensity Scores
• Sometimes propensity scores are used to verify that pre-defined comparison groups are actually equivalent;
• If they are, then the propensity scores may not have to be used in analysis
11
Propensity ScoresFlorida Healthy Start Evaluation: from Bill Sappenfield
.5 .6 .7 .8 .9 1Propensity Score
Reference 1 Care Coordination
12
.2 .3 .4 .5 .6 .7Propensity Score
Reference 2 Care Coordination
Propensity ScoresFlorida Healthy Start Evaluation: from Bill Sappenfield
13
Analysis Approach 1: Propensity Score as a Covariate or Weight in Model
• Use the propensity score as a covariate in model–1 degree of freedom as opposed to 1 or more for each
original covariate; particularly useful when the prevalence of outcome is small relative to the number of covariates that must be controlled, leading to small cell sizes
• Weight data using the propensity scores–the weight for an “exposed” subject is the inverse of the
propensity score–the weight for an “unexposed” subject is the inverse of 1
minus propensity score; weights must be normalizedThese approaches do not handle the issue of off-support data unless
data are restricted to the range of propensity scores common to both the exposed and unexposed
13
14
Analysis Approach 2: Subclassification by Categories of the Propensity Scores
Stratifying by quintiles of the overall distribution of propensity scores can remove approx 90% of the bias caused by the propensity score
The measure of effect is then computed in each stratum and a weighted average is estimated based on the number of observations in each stratum
14
15
Analysis Approach 3: Propensity Score Matching
Several matching techniques are available:• Nearest Neighbor (with or without replacement)• Caliper and Radius• Kernal and Local Linear
Several software solutions available to perform matching. Two examples include:• PSMATCH2 in STATA • GREEDY macro in SAS
15
16
Analysis Approach 3: Propensity Score Matching
PSMATCH2 (STATA):•PSMATCH2 is flexible and user-controlled with regard to matching techniques
GREEDY (51 digit) macro in SAS:• The GREEDY (51 digit) Macro in SAS performs one to one
nearest neighbor within-caliper matching:• First, matches are made within a caliper width of 0.00001
(“best matches”), then caliper width decreases incrementally for unmatched cases to 0.1
• At each stage, “unexposed” subject with “closest” ; propensity score is selected as the match to the exposed; in the case of ties, the unexposed is randomly selected
• Sampling is without replacement
16
17
After Matching…
1. Check for balance in the covariates between the exposed and unexposed groups
2. If not balanced, re-specify the model and re- generate propensity scores; consider adding interactions or higher order terms for variables that were not balanced
3. If balanced, calculate a measure of association from an analysis that accounts for matched nature of data
• Relative Risk / Odds Ratio / Hazard Ratio/ Rate Ratio and 95% CI
• Risk Difference (Attributable Risk) and 95% CI
17
Matched Analysis
Analysis to estimate effect of exposure on outcome should account for matched design in estimation of standard errors, since matched pairs are no longer statistically independent
Estimates of effect need not be adjusted for matching because exposed are matched to unexposed; therefore a selection bias is not imposed on the data as it is in a matched case- control study where conditional logistic regression is needed
18
Matched Analysis
Multivariable regression not necessary (but GEE can be used) since matching addresses confounding, so a simple 2x2 table can be used, but this 2x2 table must reflect the matched nature of the data
19
Unexposed Develops Oucome? Yes No
Yes
a
b
a + b
Exposed Develops
Outcome? No c d
c + d
a + c b + d a + b + c + d (n pairs)
Exposed Experiences Outcome
Unexposed Experiences Outcome
Matched Analysis: Measures of Effect (95% CI)
Relative Risk (RR) = (a+c)/(a+b)
SE (lnRR) = sqrt [(b+c) / {(a+b)(a+c)}]
95% CI = exp[lnRR ± (1.96*SE)]
Risk Difference (RD) / Attributable Risk (AR) = (b-c)/n
SE (RD) = ((c + b)−(b−c)2/n)/n2
95% CI = RD ± 1.96(SE)
Note: Measures of effect from propensity score-matched analyses are often called “Average Treatment Effect in the Treated (ATT)” in the propensity score literature. This usually refers to RD, but sometimes ATTratio is reported
20
21
Propensity Scores Using the 2007 National Survey of Children’s Health (NSCH) for Illinois
2222
Example: Association between receiving care in a medical home and reported overall health
Exposure
Outcome
Output from
SAS proc surveryfreq
Children (age 0-17) Receiving Care that Meets the Medical Home Criteria
Medical Home Freq
WeightedFreq
Weighted Percent
Yes 1059 1730663 55.9095
No 801 1364811 44.0905
Total 1860 3095474 100.000
Frequency Missing = 72
Description of Child’s General Health (Recode of k2q01)
general health FreqWeighted
FreqWeighted
Percent
Excellent,Very good 1650 2715176 84.9019
Good, Fair, Poor 282 482840 15.0981
Total 1932 3198016 100.000
2323
Example: Association between medical home (Y/N) and reported overall health
% of children whose
overall health was
reported as excellent or
very good, according
to whether the care they
received met the
medical home criteria.
Medical Home by General Health
Medical Home
General Health Freq
WeightedFreq
WeightedRow
Percent
Yes EVG 981 1594691 92.1434
GFP 78 135972 7.8566
Total 1059 1730663 100.000
No EVG 616 1039346 76.1531
GFP 185 325465 23.8469
Total 801 1364811 100.000
Total EVG 1597 2634037
GFP 263 461437
Total 1860 3095474
Frequency Missing = 72
2424
Crude Logistic Regression ModelOutput from SAS proc surveylogistic
The odds of a child’s overall health being described as at least very good are 3.7 times greater for those who receive care that met the medical home criteria compared to those whose care did not.
Odds Ratio Estimates
EffectPoint
Estimate95% Wald
Confidence Limits
Medical Home 3.67 2.51 5.37
2525
Creating Propensity Scores for the Medical Home
Many factors—sociodemographic as well as medical—are likely to confound the association between medical home and reported overall health.
It may not be feasible to adjust for all of these factors in a conventional regression model.
Instead, propensity scores will be generated to simultaneously account for many factors.
26
Creating Propensity Scores for the Medical Home: 3 Versions
1. 12 variables—demographic variables only
2. 14 variables—12 demographic variables plus a composite variable used to identify children with special health care needs (CSHCN) and a composite variable indicating severity of any health conditions
3. 38 variables—12 demographic variables plus 5 individual CSHCN screener variables and 21 indicators of condition severity
27
Distribution of Propensity Scores Before Matching
Version 3 – 38 Variables
Before Matching (n=1428)Propensity score distributions (PSCORE 1, 2, 3) by Medical Home Status - before matching
0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Per
cent
Car
e D
OE
S N
OT
mee
t med
ical
hom
e cr
iteria
0 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00
0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Per
cent
Car
e M
EE
TS
med
ical
hom
e cr
iteria
Estimated Probability
Chi
ldre
n (a
ge 0
-17)
rec
eivi
ng c
oord
inat
ed, o
ngoi
ng, c
ompr
ehen
sive
car
e w
ithin
a m
edic
al h
ome
Medical Home = NO
Medical Home = YES
2828
Creating Propensity Scores for the Medical Home: 3 Versions
Pool of Variables Used to Create Propensity scores—Predicted Probabilities from Modeling: medical home (Y/N) = pool of variables
# obs. used
12 variablesageyr_child racernew msa_stat totkids4 sex planguage coveragetotadult3 famstruct k9q16r marstat_par neighbsupport
1629
14 variablesageyr_child racernew msa_stat totkids4 sex planguage coveragetotadult3 famstruct k9q16r marstat_par neighbsupport screenscale severityscale
1629
38 variablesageyr_child racernew msa_stat totkids4 sex planguage coveragetotadult3 famstruct k9q16r marstat_par neighbsupport k2q12_s k2q15_s k2q18_s k2q21_s k2q23_sK2Q30_s K2Q31_s K2Q32_s K2Q33_s K2Q34_s K2Q35_s K2Q36_s K2Q37_s K2Q38_s K2Q40_s K2Q41_s K2Q42_s K2Q43_s K2Q44_s K2Q45_s K2Q46_s K2Q47_s K2Q48_s K2Q49_s K2Q50_s K2Q51_s
1578
2929
Creating Propensity Scores for the Medical Home
Sample SAS code for outputting the predicted values that are the propensity scores:
proc surveylogistic data=datasetname;
title1 “text”;
strata state;
cluster idnumr;
weight nschwt;
class classvars (ref=“ “)/ param=ref;
model medical_home (descending) = confounder pool;
output out=outputdataset p=name for pred. value;
run;
3030
Creating Propensity Scores for the Medical Home: Excerpt from SAS proc print
Obs. pscore1 pscore2 pscore3
811 Medical Home Yes 0.82314 0.82344 0.77917
812 Medical Home Yes 0.79093 0.80706 0.79674
813 Medical Home No 0.57322 0.45131 .
814 Medical Home No . . .
815 Medical Home Yes 0.82352 0.82899 0.83309
816 Medical Home No 0.31732 0.37460 0.36290
817 Medical Home Yes 0.81300 0.82409 0.82015
818 Medical Home No 0.72170 0.76384 0.78867
819 Medical Home No . . .
820 Medical Home No 0.09905 0.11217 0.11435
821 Medical Home Yes 0.44107 0.50713 0.47309
822 Medical Home Yes 0.75459 0.76151 0.77425
823 Medical Home Yes 0.87060 0.89112 0.88204
3131
Modeling General Health: 3 approaches for each of 3 pools of Variables Modeling the Impact of Having a Medical Home on the
Respondent’s Rating of Child’s General Health# obs. used OR 95% CI
Crude Model: genhealth = medical home(Y/N)genhealth = medical home (Y/N) – for non-miss covariates
18601629
3.67 (2.51, 5.37)3.72 (2.44, 5.66)
Using 12 variable version of the propensity scores:genhealth = medical home(Y/N) + 12 orig. varsgenhealth = medical home(Y/N) + prop score (12)genhealth = medical home(Y/N) (matched on prop score)*
16291629
509 pairs
1.99 (1.22,3.24)1.89 (1.16,3.08)2.52 (1.72,3.70)
Using 14 variable version of the propensity scores:genhealth = medical home(Y/N) + 14 orig. varsgenhealth = medical home(Y/N) + prop score (14)genhealth = medical home(Y/N) (matched on prop score)*
16291629
503 pairs
1.49 (0.90,2.47)1.44 (0.89,2.34)1.55 (1.09,2.22)
Using 38 variable version of the propensity scores:genhealth = medical home(Y/N) + 38 orig. varsgenhealth = medical home(Y/N) + prop score (38)genhealth = medical home(Y/N) (matched on prop score)*
15781578
482 pairs
1.75 (0.99,3.08)1.57 (0.93,2.65)1.93 (1.30,2.86)
*SAS Greedy Macro used for matches;PROC GENMOD used for GEE logistic regression with no weights or survey design variables.
3232
Modeling General Health: 3 approaches for each of 3 pools of Variables
Example of
statistical results
when including
the medical home
plus 12 covariates:
3333
Modeling General Health: 3 approaches for each of 3 pools of Variables
As the number of variables increases, it becomes more difficult to implement a conventional model.
With the medical home plus 38 variables, there were convergence problems:
Warning: Ridging has failed to improve the loglikelihood. You may want to increase the initial ridge value (RIDGEINIT= option), or use a different ridging technique (RIDGING= option), or switch to using linesearch to reduce the step size (RIDGING=NONE), or specify a new set of initial estimates (INEST= option).
Warning: The SURVEYLOGISTIC procedure continues in spite of the above warning. Results shown are based on the last maximum likelihood iteration. Validity of the model fit is questionable.
Fortunately, convergence was not a problem when using the 38 variables to create the propensity scores.
3434
Modeling General Health: 3 approaches for each of 3 pools of Variables
Using the propensity scores
as a covariate in the model
only requires 1 df making it
feasible to account for many
variables simultaneously
Odds Ratio EstimatesMedical Home + Propensity Scores (12 Vars)
Predicting General Health (EVG V. GFP)
EffectPoint
Estimate95% Wald
Confidence Limits
ind4_8_07 1.886 1.156 3.075
pscore1 24.222 8.481 69.182
Odds Ratio EstimatesMedical Home + Propensity Scores (14 Vars)
Predicting General Health (EVG V. GFP)
EffectPoint
Estimate95% Wald
Confidence Limits
ind4_8_07 1.44 0.89 2.337
pscore2 65.614 23.088 186.470
Odds Ratio EstimatesMedical Home + Propensity Scores (38 Vars)
Predicting General Health (EVG V. GFP)
EffectPoint
Estimate95% Wald
Confidence Limits
ind4_8_07 1.567 0.928 2.647
pscore3 38.073 13.230 109.565
Distribution of Propensity Scores Before and After Matching
Version 3 – 38 Variables
Before AfterPropensity score distributions (PSCORE 1, 2, 3) by Medical Home Status - before matching
0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Per
cent
Car
e D
OE
S N
OT
mee
t med
ical
hom
e cr
iteria
0 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00
0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Per
cent
Car
e M
EE
TS
med
ical
hom
e cr
iteria
Estimated Probability
Chi
ldre
n (a
ge 0
-17)
rec
eivi
ng c
oord
inat
ed, o
ngoi
ng, c
ompr
ehen
sive
car
e w
ithin
a m
edic
al h
ome
Propensity score distributions (PSCORE 1, 2, 3) by Medical Home Status - after matching
0
2
4
6
8
10
12
14
Per
cent
Car
e D
OE
S N
OT
mee
t med
ical
hom
e cr
iteria
0 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00
0
2
4
6
8
10
12
14
Per
cent
Car
e M
EE
TS
med
ical
hom
e cr
iteria
Estimated Probability
Chi
ldre
n (a
ge 0
-17)
rec
eivi
ng c
oord
inat
ed, o
ngoi
ng, c
ompr
ehen
sive
car
e w
ithin
a m
edic
al h
ome
Medical Home = NO
Medical Home = YES
Medical Home = NO
Medical Home = YES
3636
Modeling General Health: Stratified by Whether the Child is Screened as CSHCN
12 Variable VersionModeling the Impact of Having a Medical Home on the
Respondent’s Rating of Child’s General Health # obs. used
OR 95% CI
Among Children WITHOUT Special Health Care NeedsUsing 12 variable version of the propensity scores^:genhealth = medical home(Y/N) + 12 orig. varsgenhealth = medical home(Y/N) + prop score (12)genhealth = medical home(Y/N) (matched on prop score)*
13091309
389 pairs
1.28 (0.69,2.34)1.31 (0.76,2.26)2.12 (1.26,3.56)
Among Children WITH Special Health Care NeedsUsing 12 variable version of the propensity scores^:genhealth = medical home(Y/N) + 12 orig. varsgenhealth = medical home(Y/N) + prop score (12)genhealth = medical home(Y/N) (matched on prop score)*
320320
114 pairs
2.76 (1.21,6.29)2.26 (1.05,4.88)2.49 (1.40,4.41)
*PROC GENMOD was used for GEE logistic regression with no weights or survey design variables; Matching was performed separately within CSHCN and non-CSHCN
^Stratum-specific estimates for the unmatched analyses were obtained using a DOMAIN statement in PROC SURVEYLOGISTIC in SAS 9.2
3737
Modeling General Health: Stratified by Whether the Child is Screened as CSHCN
Rather than stratified analysis, obtain stratified results by including a product term in the model:
genhealth = medical home(Y/N) + prop score (12) + medical home*cshcn
Use contrast statements in SAS to generate the stratum-specific results:contrast 'odds ratio among cshcn y' medicalhome 1 medicalhome*cshcn 1 / estimate=exp;contrast 'odds ratio among cshcn n' medicalhome 1 / estimate=exp;
These results attenuated compared to the matched, stratified results.
Contrast Estimate Confidence Limits
odds ratio among cshcn n 1.55 0.89 2.70
odds ratio among cshcn y 1.96 0.93 4.14
Propensity Score Example:Using 2003 Natality Data for Illinois
38
3939
Example: Association between receiving adequate prenatal care and Preterm Birth
Exposure
Outcome
Output from
SAS PROC FREQ
Prenatal Care Adequacy (Kotelchuck) for Mothers of Singleton Infants (PNC)
PNC Freq Percent
Intermediate/Adequate/Adeq Plus 147,416 90.5
Inadequate/No PNC 15,503 9.5
Total 162,919 100.0
Frequency Missing =9,439
Preterm Birth (PTB)
Freq Percent
Preterm Birth (<37 wks) 16,923 10.4
Term Birth 145,996 89.6
Total 162,919 100.0
Frequency Missing =9,439
4040
Crude Measures of Effect
proc freq data=analysis order=formatted;
tables adeq*ptb/relrisk riskdiff;
format adeq ptb yn.; run;
Measures of Effect and 95% Cis
Type of Study Value 95% Confidence Limits
Case-Control (Odds Ratio)Cohort (Col 1 Risk)
Risk Difference
0.76 0.78-0.03
0.72 0.75-0.03
0.80 0.82-0.02
PTB
PNC Preterm Birth Term Birth Total
Adequate 14,919 (10.1) 132,497 (89.9) 147,416
Not Adequate 2,004 (12.9) 13,499 (87.1) 15,503
Total 17,454 (10.5) 148,423 (89.5) 162,919
4141
Creating Propensity Scores for PNC Adequacy
Variable Name Description ValuesAGECAT Maternal age at delivery 1=<20, 2=20-34, 3=35+RACEETH Race/Ethnicity 1=White, 2=Af-Am, 3=Hisp, 4=OtherEDUCAT Education 1=<HS, 2=HS, 3=>HSPARITY2 Parity 0=Primp, 1=1-2 previous LB, 3=3+MARRIED Marital Status 1=Married, 0=Not MarriedSMOKE Smoking Status 1=Smoker, 0=Non-smokerRISKFAN Anemia (HCT.<30/HGB.<10) 1=Yes, 0=NoRISKFCAR Cardiac Disease 1=Yes, 0=NoRISKFLUN Acute or Chronic Lung Disease 1=Yes, 0=NoRISKFDIA Diabetes 1=Yes, 0=NoRISKFHER Genital Herpes 1=Yes, 0=NoRISKFHEM Hemoglobinopathy 1=Yes, 0=NoRISKFCHY Hypertension, Chronic 1=Yes, 0=NoRISKFPHY Hypertension, Pregnancy-Associated 1=Yes, 0=NoRISKFINC Incompetent Cervix 1=Yes, 0=NoRISKFPRE Previous Infant 4000+ Grams 1=Yes, 0=NoRISKFPRT Prev Preterm or SGA 1=Yes, 0=NoRISKFREN Renal Disease 1=Yes, 0=NoRISKFRH RH Sensitization 1=Yes, 0=NoRISKFUTE Uterine bleeding 1=Yes, 0=NoRISKFOTH Other Medical Risk Factors 1=Yes, 0=No
How might variables be different if exposure was entry into PNC?
4242
Creating Propensity Scores for PNC Adequacy
Sample SAS code for outputting the predicted values that are the propensity scores:
proc logistic data=datasetname desc;
title1 “text”;
class classvars / param=ref ref=first;
model adeq = confounder pool;
output out=outputdataset p=name for pred. value;
run;
4343
Creating Propensity Scores for PNC Adequacy: Excerpts from SAS proc print
n=160,642
ID Adeq propscore
1 0 0.79507
2 1 0.87975
3 1 0.88361
4 1 0.96668
5 0 0.94172
6 0 0.77970
7 1 0.95197
8 0 0.87975
9 1 0.85336
10 1 0.95197
11 1 0.97350
12 1 0.95197
44
Distribution of Propensity Score by PNC Adequacy, before Matching
44On Support = 0.386-0.988
Adequate (range): 0.366-0.995
Inadequate (range): 0.386-0.988
38 observations at top and 2 at bottom of distribution in Adequate group
45
Analyzing Data: Four Approaches
45
Approach SAS Code
1. Model adequacy of PNC plus all 28 covariates
Proc genmod data=OUTPUTDATASET desc; class CLASSVARS / param=ref ref=first; model PTB = ADEQ AGECAT…RISKFOTH/link=log dist=bin; run;
2. Model adequacy of PNC plus the propensity score
proc genmod data=OUTPUTDATASET desc; model PTB = ADEQ PROPSCORE/link=log dist=bin; run;
3. Weight analysis on propensity score
proc genmod data=OUTPUTDATASET desc; model PTB = ADEQ/link=log dist=bin; weight pweight; run;
4. Match women with adequate PNC to those without by propensity score and conduct matched analysis
Call GREEDY macro: %GREEDMTCH(work,outputdataset,adeq,matched,propscore,idnumr); proc genmod data=matched desc; class matchto; model ptb = adeq/dist=bin link=log; repeated subject=matchto/type=IND corrw covb; estimate 'adeq' adeq 1/exp;run;
46
Checking Covariate Balance Before Propensity Score Matching (GREEDY 1:1 Match)
Selected Variables
Before PS Match Standardized Difference*
Adequate(n=147,416)
Inadequate(n=15,503)
Age Mean (SD) Mean (SD)
<20 0.09 (0.21) 0.21 (0.41) -34.61
20-34 0.76 (0.43) 0.70 (0.46) 14.72
35+ 0.15 (0.36) 0.10 (0.30) 16.96
Race/Ethnicity
NH White 0.57 (0.50) 0.32 (0.47) 53.04
NH African American
0.15 (0.36) 0.347 (0.48) -46.37
Hispanic 0.23 (0.42) 0.30 (0.46) -16.73
Other 0.05 (0.22) 0.04 (0.19) 6.94
Preg-Induced Hypertension
0.03 (0.18) 0.02 (0.15) 7.06
46
*Calculated as:
100*(meanexp - meanunexp)
SQRT((s2exp + s2
unexp) / 2 )
where s=std dev of mean
Commonly, a Standardized Difference of >=10% or indicates imbalance
Note: All factors are significantly associated with adequate PNC at p<0.0001
47
Checking Covariate Balance Before and After Propensity Score Matching (GREEDY 1:1 Match)
Selected Variables
After PS Match (GREEDY in SAS)
Standardized Difference
% Bias Reduction^
Adequate(n=15,002)
Inadequate(n=15,002)
Age Mean (SD) Mean (SD)
<20 0.21 (0.41) 0.21 (0.41) 0.03 99.9%
20-34 0.70 (0.46) 0.70 (0.46) 0.48 96.7%
35+ 0.09 (0.29) 0.09 (0.29) -0.80 95.3%
Race/Ethnicity
NH White
NH African American
0.35 (0.48) 0.35 (0.48) 0.0 100%
Hispanic 0.30 (0.46) 0.30 (0.46) 0.04 99.8%
Other 0.04 (0.19) 0.04 (0.18) 0.44 93.7%
Preg-Induced Hypertension
0.02 (0.14) 0.02 (0.15) -1.61 77.2%
47
^Calculated as:
unmatched
matched
StdDif
StdDif1
48
Distribution of Propensity Score by PNC Adequacy, after Matching (GREEDY)
48
4949
Modeling the Impact of Having Adequate PNC on Preterm Birth
# obs. used
RR (95% CI) RD (95% CI)
Crude Model: PTB = Adequate PNC (Y/N) 162,919 0.78 (0.75, 0.82) -0.03 (-0.03, -0.02)
Using 26 variable version of the propensity scores:PTB = Adeq PNC (Y/N)+ 26 orig. vars
PTB = Adeq PNC (Y/N) + prop score
PTB = Adeq PNC (Y/N) (weighted to inverse of propensity score)
PTB = Adeq PNC (Y/N) (matched on prop score using GREEDY macro (1:1 match)
160,642
160,642
160,642
15,010 pairs
0.94 (0.90, 0.99)
0.99 (0.95, 1.04)
1.04 (1.01, 1.07)
0.98 (0.93, 1.04)
-0.007 (-0.01, -0.002)
0.0003 (-0.005, 0.006)
0.004 (0.001, 0.006)
-0.00247(-0.0249, 0.00244)
Results: Four Approaches Using SASIs PNC Associated with Reduced Risk of Preterm Birth?
Results: Restructuring data for matched 2x2 table
/*Restructuring data from one observation per infant to one observation per matched pair (n obs from 30020 15010)*/
data adeq (rename=(ptb=InAdeqPTB));
set matched; where adeq=0; run;
proc sort data=adeq; by matchto; run;
data inadeq (rename=(ptb=AdeqPTB));
set matched; where adeq=1; run;
proc sort data=inadeq; by matchto; run;
data matchedpair;
merge adeq inadeq;
by matchto;
run;50
Results: Matched Analysis from 2x2 Table
/*Producing 2x2 table for matched pairs, with McNemar test*/
proc freq data=matchedpair order=formatted;
table InadeqPTB*AdeqPTB/norow nocol;
exact mcnem; format AdeqPTB InadeqPTB yn.;
run;
RR = (a+c) / (a+b)SE (lnRR) = sqrt [(b+c) / {(a+b)(a+c)}] 95% CI = exp[lnRR ± (1.96*SE)]
RR = (288+1623) / (288+1660) = 0.981SE = sqrt [(1660+1623) / {(288+1660)(288+1623)}] = 0.029795% CI = 0.926, 1.040
52
Some Limitations of Propensity Score Methods
Like multivariable regression:• Cannot account for unobserved characteristics
(unmeasured confounders)• Must consider how to approach the issue of missing data
on covariates of interest (complete-case analysis, separate dummy variable for missing, imputation)
Unlike multivariable regression:• In most accessible form, methods are limited to binary
exposures (though work is being done in this area)• Mis-specification of model to generate propensity score
can have a large impact on resulting estimates52
53
Some Limitations of Propensity Score Methods
Propensity score techniques may not result in different findings than multivariable regression; it’s not always clear that there is a benefit to performing the analysis in this way
Some exceptions include:• Datasets in which sample size is limited or the outcome
is rare, and multiple covariates need to be controlled; propensity scores provide a way to adjust for all covariates with fewer degrees of freedom
• Datasets in which some of the data is off-support; though care must be taken in interpretation as generalizability is affected and, in some cases, bias can be introduced when sample is restricted
53Sturmer, et al 2006, J Clin Epidemiol.
5454
Questions and Challenges
1. What if there is interest in the independent effects of a few other variables besides the 'exposure' – as in any matched design, should these variables not be included in the pool used to create the propensity scores so that they can then be included as covariates in a final model?
5555
Questions and Challenges
2. While the model to create the propensity scores can include many variables regardless of their statistical significance, the number of observations lost due to missing values likely increases as the number of variables used increases. What is the balance here? Does this call for imputation?
5656
Questions and Challenges
3. For a given sample size, at some point the model to produce the propensity scores will get too big, so although theoretically many variables can be included, mechanically there may be convergence problems. With very small samples, this may mean that fully controlling for observed confounding may not be possible even with propensity scores. With a small number of variables, is it still worth it to gain the efficiency of matching—creating comparable groups.
5757
Questions and Challenges
4. One approach to using propensity scores is to weight the observations. Is this possible with a complex sampling design in which the observations are already weighted?
5858
Questions and Challenges
5. Choices about level of measurement might be made differently when modeling to generate propensity scores. For example, variables might be left in continuous form even though they might be categorized when assessing their independent effect on outcome (e.g. child's age).
Similarly, for categorical variables, there is no need to collapse categories even when modeling results indicate it would be appropriate since parsimony is not critical (e.g. not combining "multiracial" with "other").
5959
Questions and Challenges
6. For stratified analysis, should propensity scores be created first for all observations in a single model (of course not including the stratification variable), or should stratum-specific models be run to create the propensity scores?
And, if the scores are generated within strata, should identical pools of variables be used, or might those pools also be stratum-specific ?
60
Resources
Software
SAS GREEDY MACRO – code and documentation: http://www2.sas.com/proceedings/sugi26/p214-26.pdf
STATA PSMATCH2: http://ideas.repec.org/c/boc/bocode/s432001.html
Other Matching Programs: http://www.biostat.jhsph.edu/~estuart/propensityscoresoftware.html
Select Methods Articles
Austin, Peter. Comparing paired vs non-paired statistical methods of analyses when making inferences about absolute risk reductions in propensity-score matched Samples Statist. Med. 2011, 30 1292—1301. (Plus any other recent Austin papers).
Caliendo and Kopeinig , 2005 “Some Practical Guidance for the Implementation of Propensity Score Matching” Available at: http://repec.iza.org/dp1588.pdf
Oakes JM and Johnson P. Propensity Score Matching for Social Epidemiology. Oakes JM, Kaufman JS (Eds.), Methods in Social Epidemiology. San Francisco, CA: Jossey-Bass.
Stürmer T, Joshi M, Glynn RJ, Avorn J, Rothman KJ, Schneeweiss S. A Review of Propensity Score Methods Yielded Increasing Use, Advantages in Specific Settings, but not Substantially Different Estimates Compared with Conventional Multivariable Methods. J Clin Epidemiol. 2006 May; 59(5): 437-447. 60
61
Resources
Some MCH Applications
Bird TM, Bronstein JM, Hall RW, Lowery CL, Nugent R, Mays GP. Late preterm infants: birth outcomes and health care utilization in the first year. Pediatrics (2):e311-9. Epub 2010 Jul 5.
Brandt S, Gale S, Tager IB. Estimation of treatment effect of asthma case management using propsensity score methods. Am J Mang Care, 16(4): 257-64, 2010.
Cheng YW, Hubbard A, Caughey AB, Tager IB. The association between persistent fetal occiput posterior position and perinatal outcomes: An example of proensity score and covariate distance matching. AJE, 171(6): 656-663, 2010.
Johnson P, Oakes JM, Anderton DL. Neighborhood Poverty and American Indian Infant Death: Are the Effects Identifiable? Annals of Epidemiology 18(7), 2008: 552-559.
61