Bayesian Propensity Score Analysis: Simulation … Propensity Score Analysis: Simulation and Case...

Bayesian Propensity Score Analysis: Simulation and Case Study

David Kaplan

Cassie J. S. Chen

Department of Educational Psychology

❖ Introduction❖ Bayesian PropensityScore Approaches

❖ StrataSub-classification one(z)

❖ Propensity ScoreWeighting

❖ Optimal Matching onthe Propensity Score

❖ Design and Results ofSimulation Study I

❖ Steps in Study I

❖ Design and Results ofSimulation Study II

❖ Steps in Study II

❖ Conclusion

SREE 2011 2 / 28

The research reported here was supported by the Institute of EducationSciences, U.S. Department of Education, through Grant R305D110001 toThe University of Wisconsin - Madison. The opinions expressed are thoseof the authors and do not represent views of the Institute or the U.S.Department of Education.

Introduction









❖ Conclusion

SREE 2011 3 / 28

● This talk considers the use of propensity scores for equating groupson the basis of pre-treatment variables with the goal of strengtheningcausal inferences in observational studies.

● Propensity score analysis has been used extensively in fields suchepidemiology, education, and sociology.

● Typically, however, propensity score analysis has been implementedwithin the conventional frequentist perspective of statistics.

● This perspective does not allow for encoded prior informationregarding either the parameters of the model that generates thepropensity scores or the model that provides the causal estimand(the outcomes model).

● A Bayesian approach to propensity score analysis provides asolution to the problem.

Bayesian Propensity Score Approaches









❖ Conclusion

SREE 2011 4 / 28

● A review of the extant literature reveals very few studies examiningBayesian approaches to propensity score analysis.

● An earlier paper by Rubin (1985) argued that because propensityscores are, in fact, randomization probabilities, these should be ofgreat interest to the applied Bayesian analyst.

● In the context of propensity score analysis, Rubin (1985) argued thatunder the condition of strong ignorability and assuming that theestimated propensity score e(z) is an adequate summary of theobserved covariates z, then the applied Bayesian will bewell-calibrated (Dawid, 1982), in the sense that posterior predictionsshould match up with what happens in reality.

● Although Rubin (1985) provides a justification for why an appliedBayesian should be interested in propensity scores, his analysisdoes not address the actual estimation of the propensity scoreequation or the subsequent outcomes equation from a Bayesianperspective.










❖ Conclusion

SREE 2011 5 / 28

● A paper by McCandless et al. (2009) provides an approach toBayesian propensity score analysis for observational data.

● Their approach involves treating the propensity score as a latentvariable and modeling the joint distribution of the data and theparameters for the propensity score and outcomes equationssimultaneously via an MCMC algorithm.

● From there, the marginal posterior probability of the treatment effectthat directly incorporates uncertainty in the propensity score can beobtained.

● A recent paper by An (2010) also puts forth a joint modelingapproach to Bayesian propensity score analysis.

● Gelman, et al. (2003) have argued that the propensity score shouldprovide information only regarding study design and not regardingthe treatment effect, as is the case with the Bayesian procedureadvocated by McCandless et al. (2009) and An (2010).










❖ Conclusion

SREE 2011 6 / 28

● Recent work by Kaplan and Chen (2010) examined conventional andBayesian propensity score approaches in a real data setting.

● The Bayesian propensity score was implemented via stratasubclassification, weighting, and optimal matching and compared tothe conventional propensity score.

● Results indicated that estimates of causal effects were similar acrossmethods, but posterior probability intervals were wider, as expected.

● Kaplan and Chen (2010) did not examine these approaches in acontrolled simulation setting, nor did they examine the combinationof a Bayesian propensity score equation with a Bayesian outcomesequation.

● We extend Kaplan and Chen (2010) by examining a Bayesianapproach to propensity score analysis in a comprehensivesimulation study. We also examine the implications of specifying aBayesian model for the treatment effect.

Strata Sub-classification one(z)









❖ Conclusion

SREE 2011 7 / 28

1. Create strata on the estimated propensity score.

2. Estimate treatment effect within each strata.

3. Average the treatment effect and standard errors across strata usingthe “Rubin” approach.

● Cochran (1968) and also Rosenbaum and Rubin (1983) found thatsubclassification into five strata on continuous distributions such asthe propensity score has been observed to remove approximately90% of the bias due to non-random selection effects (Cochran,1968).

● Rosenbaum and Rubin (1983) proved that when subclass units arehomogeneous with respect to e(z) then the treatment and controlunits in the same subclass will have the same distribution on z.

Propensity Score Weighting









❖ Conclusion

SREE 2011 8 / 28

● Propensity score weighting is based on the idea ofHorvitz-Thompson sampling weights and is designed to reweight thetreatment and control group participants in terms of their propensityscores.

● Let e(z) be the estimated propensity score, and let T indicatewhether an individual is treated (T = 1) or not (T = 0).

● The weight used to estimate the ATE can be defined as

ω1 =T

e(z)+

1− T

1− e(z). (1)

● We see that when T = 1, ω1 = 1/e(z) and when T = 0,ω1 = 1/[1− e(z)].

● This approach weights the treatment and control group up to the fullsample.

Optimal Matching on the Propensity Score









❖ Conclusion

SREE 2011 9 / 28

● Optimal matching is an improvement on the so-called greedymatching algorithm.

● Greedy matching essentially does not revisit the match. It does notattempt to provide the lowest overall “cost” for the match.

● Optimal matching might reconsider a match if the total distanceacross all matches is less than if the algorithm proceeded.

● Optimal matching is as good and often better than greedy matching.

● Greedy matching can provide a good answer, but there is noguarantee that the answer will be tolerable - and it can be quite bad.

Design and Results of Simulation Study I









❖ Conclusion

SREE 2011 10 / 28

● We propose a two-step Bayesian propensity score analysisapproach, with a Bayesian propensity score model in the first stepand Bayesian outcomes model in the second step, and compare itwith the conventional propensity score analysis (PSA).

● Also, we fit the simple linear regression and Bayesian simpleregression without any propensity score adjustment for comparativepurposes.

● The Bayesian simple regression utilizes the Gibbs sampler within theMCMCregress package in R to simulate the posterior distribution ofthe outcomes model.









❖ Conclusion

SREE 2011 11 / 28

● The simple outcomes model is written as

Y = µ+ γT + ǫ1, (2)

where µ is the intercept, Y is the outcome, γ is the causal effect, andT is the treatment indicator.

● We assume ǫ1 ∼ N(0, σ2

1In) where In is the n dimensional identitymatrix.

● Noninformative uniform priors are used for Bayesian simpleregression and an inverse gamma prior is used for σ2

1 , with shapeparameter and scale parameter both 0.001.









❖ Conclusion

SREE 2011 12 / 28

● For both PSA and BPSA, two models are specified. The first is thepropensity score model, specified as a logit model.

Log

(

e(z)

1− e(z)

)

= α+ β′Z, (3)

where α and β are unknown parameters, Z represents a designmatrix of chosen covariates.

● For BPSA, we utilized the R function MCMClogit to simulate from theposterior distribution of a logistic regression using a random walkMetropolis algorithm.

● After estimating the conventional or Bayesian propensity scores, weuse the outcomes model in the second step to estimate the causaleffect via the three approaches: stratification, weighting, andmatching.









❖ Conclusion

SREE 2011 13 / 28

● In the two simulation studies and the case study, the frequentistbased average treatment effect γate and standard error σ areestimated via ordinary least squares regression (OLS).

● For the conventional PSA, propensity score stratification isconducted by forming strata on the propensity score, calculating theOLS treatment effect within stratum, and averaging over the stratausing “Rubin’s” rules.

● Propensity score weighting is performed by fitting a weightedregression with 1/e(z) and 1/[1− e(z)] as the weights for the treatedand control group, respectively. These are the ATE weights.

● Propensity score matching utilizes the full optimal matching methodproposed by Rosenbaum (1989).

Steps in Study I









❖ Conclusion

SREE 2011 14 / 28

● Study I examines the effects of the Bayesian propensity score modeland OLS outcomes model via different sample sizes, true treatmenteffects and priors. Data are generated according to the followingsteps:

✦ 1. Randomly generate three covariates z1, z2 and z3 withsample sizes n = 100 and n = 250, respectively as

z1 ∼ Normal(1, 1)

z2 ∼ Poisson(2)

z3 ∼ Binomial(n, 0.5).

✦ 2. Obtain propensity score “true model” by

e(z) =exp(0.2z1 + 0.3z2 − 0.2z3)

1− exp(0.2z1 + 0.3z2 − 0.2z3), (4)

that is, the propensity score generating model has true α = 0and true β = (0.2, 0.3,−0.2).

Steps in Study I









❖ Conclusion

SREE 2011 15 / 28

● 3. Calculate a treatment assignment vector T by comparing theestimated propensity score ei(z) to a random variable Ui generatedfrom the Uniform(0, 1) distribution, where i = 1, . . . , n. AssignTi = 1 if Ui <= ei(z), Ti = 0 otherwise.

● 4. Generate the outcome Y1, . . . , Yn using the causal model:

Y = 0.4z1 + 0.3z2 + 0.2z3 + γT + ǫ3, (5)

where ǫ3 ∼ N(0, 0.1) and γ is the true treatment effect taking twodifferent values 0.25 and 1.25.

● 5. Data = {(Yi, Zi, Ti), i = 1, . . . , n, n = 100 or 250}.

● 6. Perform 100 replications for the PSA model.

Steps in Study I









❖ Conclusion

SREE 2011 16 / 28

● We specify a uniform prior on the intercept α and a multivariatenormal prior on β:

β ∼ Normal(bβ , B−1

β ),

with bβ as the prior mean vector and Bβ as the prior precisionmatrix.

● For Study 1, bβ is set to 0 to imitate the case of having littleinformation on the mean, the same as what McCandless et al.(2009) chose in their study.

● Furthermore, we examine different prior precisions at Bβ = 1, 10,and 100 to explore the relation between the choice of prior precisionsand the causal effects.

Steps in Study I









❖ Conclusion

SREE 2011 17 / 28

● The MCMC sampling of the Bayesian propensity score model has104 iterations with 1000 burnin and thinning interval of 10.

● There are m = 1000 sets of propensity scores e1i(z), . . . , emi(z)generated from the predictive distribution of responses.

● For j = 1, . . . ,m, a treatment effect estimate γj is obtained using thejth generated Bayesian propensity score ej(z) by the conventionalstratification, weighting and optimal matching method as in thetraditional PSA.

Steps in Study I









❖ Conclusion

SREE 2011 18 / 28

● The final estimate of the treatment effect is

γ =

∑m

j=1γj

m. (6)

● We show in our paper a new variance estimation formula of theestimated treatment effects for the two-stage BPSA approach,

V ar(γ) =m−1

∑m

j=1σ2

j + (m− 1)−1∑m

j=1

(

γj −m−1∑m

j=1γj)

2

m.

(7)

which accounts for variation in the treatment effect and variation inthe propensity scores.









❖ Conclusion

SREE 2011 19 / 28

Table 1:The γATE s & S.E.s for Conventional PSA and BPSA

with Different True γATE s, Sample Sizes and Prior Precisions in Simulation Study 1

N = 100 N = 250

First Step Second Step γATE =.25 γATE=1.25 γATE =.25 γATE =1.25

PSA-1repStratification .30 (.13) 1.30 (.13) .29 (.07) 1.29 (.07)Weighting .31 (.12) 1.31 (.12) .28 (.08) 1.28 (.08)Matching .23 (.11) 1.23 (.11) .29 (.07) 1.29 (.07)

PSA-100repStratification .28 (.10) 1.28 (.10) .28 (.06) 1.28 (.06)Weighting .25 (.12) 1.25 (.12) .25 (.08) 1.25 (.08)Matching .27 (.09) 1.27 (.09) .25 (.05) 1.25 (.05)

BPSA Bβ = 0Stratification .32 (.13) 1.32 (.13) .32 (.09) 1.32 (.09)Weighting .31 (.18) 1.31 (.18) .27 (.13) 1.27 (.13)Matching .27 (.13) 1.27 (.13) .31 (.09) 1.31 (.09)




No AdjustmentSLR-1rep .35 (.12) 1.35 (.12) .54 (.09) 1.54 (.09)

SLR-100rep .48 (.13) 1.48 (.13) .47 (.08) 1.47 (.08)Bayes SLR .35 (.12) 1.35 (.12) .54 (.09) 1.54 (.09)

Design and Results of Simulation Study II









❖ Conclusion

SREE 2011 20 / 28

● We conduct the second simulation study for the BPSA with bothBayesian propensity score model and Bayesian outcomes model, inwhich uniform priors were compared to normal priors with varyingprecision.

● Also, the effects of different sample sizes and true γate on the causalinference are studied.

● The generated data for this study is the same as the one inSimulation Study 1.

● In addition, a Bayesian outcomes model equation is developedaccording to equation 2 using MCMCregress function in R, whichreplaces the regular OLS outcomes model for stratification andoptimal matching in simulation study 1.

● We are not aware of a program to conduct Bayesian weightedregression. Possible to program it in R.

Steps in Study II









❖ Conclusion

SREE 2011 21 / 28

● For the Bayesian propensity score model, multivariate normal priorsare chosen for both α and β:

α ∼ Normal(bα, B−1

α )

β ∼ Normal(bβ , B−1

β ),

where bα and bβ are prior means, and Bα and Bβ are priorprecisions.

● In the Bayesian outcomes model, we also assume multivariatenormal priors on the intercept µ and the treatment effect γ:

µ ∼ Normal(bµ, B−1

µ )

γ ∼ Normal(bγ , B−1

γ ),

with bµ and bγ as the prior mean, and Bµ and Bγ as the priorprecisions.

● Note that when a prior precision takes value 0, MCMCregress usesa uniform prior.

Steps in Study II









❖ Conclusion

SREE 2011 22 / 28

● Study II contains two conditions to examine the performance of BPSA whenthere is little prior information or abundant information, respectively.

● Condition A: Little prior information

✦ bα,bβ ,bµ and bγ set to 0.

✦ Bα = Bβ = 0, 1, 10, 100 for PSA model.

✦ Bµ = Bγ = 0, 1, 10, 100 for outcomes model.

● Condition B: Abundant prior information. Hyperparameter values based ondata generating parameter values

✦ bα = 0 and bβ = (0.2, 0.3,−0.2) for PSA model.

✦ bµ = Bµ = 0 for outcomes model to indicate no prior information on µ.

✦ bγ = 0.25 or 1.25 for the treatment effect.

✦ Bα = Bβ = 0, 1, 10, 100 in the PSA model.

✦ Bγ = 0, 1, 10, 100 for outcomes model.









❖ Conclusion

SREE 2011 23 / 28

Table 2:The γATE s & S.E.s for Conventional and Bayesian Stratification and Matching of Design I

with True γATE=0.25 in Simulation Study 2

N = 100 N = 250

First Step Second Step Stratification Matching Stratification Matching

Bβ=0

Bγ = 0 .32 (.14) .27 (.13) .32(.09) .31 (.09)Bγ = 1 .34 (.13) .28 (.13) .33 (.09) .32 (.08)Bγ = 10 .39 (.09) .31 (.11) .40 (.08) .39 (.08)Bγ = 100 .11 (.05) .38 (.08) .29 (.04) .52 (.06)

Bβ=1

Bγ = 0 .31 (.14) .26 (.13) .31 (.09) .30 (.08)Bγ = 1 .33 (.13) .26 (.13) .32 (.09) .31 (.08)Bγ = 10 .38 (.09) .30 (.11) .39 (.08) .38 (.08)Bγ = 100 .11 (.05) .38 (.08) .29 (.04) .52 (.06)

Bβ=10

Bγ = 0 .29 (.13) .24 (.12) .29 (.07) .27 (.07)Bγ = 1 .31 (.12) .24 (.12) .30 (.07) .28 (.07)Bγ = 10 .37 (.09) .29 (.10) .36 (.07) .34 (.07)Bγ = 100 .11 (.05) .38 (.07) .29 (.04) .51 (.06)

Bβ=100

Bγ = 0 .28 (.12) .25 (.11) .29 (.07) .27 (.06)Bγ = 1 .30 (.11) .25 (.11) .30 (.06) .28 (.06)Bγ = 10 .37 (.08) .29 (.10) .35 (.06) .32 (.06)Bγ = 100 .12 (.05) .38 (.07) .31 (.04) .51 (.06)

PSA-1rep .30 (.13) .23 (.11) .29 (.07) .29 (.07)PSA-100rep .28 (.10) .27 (.09) .28 (.06) .25 (.05)









❖ Conclusion

SREE 2011 24 / 28

Table 3:The γATE s & S.E.s for Conventional and Bayesian Stratification and Matching of Design I


N = 100 N = 250


Bβ=0

Bγ = 0 1.32 (.14) 1.27 (.13) 1.32 (.09) 1.31 (.09)Bγ = 1 1.27 (.13) 1.27 (.13) 1.31 (.09) 1.32 (.08)Bγ = 10 1.00 (.12) 1.22 (.11) 1.22 (.06) 1.35 (.07)Bγ = 100 .08 (.05) .62 (.11) .27 (.06) 1.15 (.06)

Bβ=1

Bγ = 0 1.31 (.14) 1.26 (.13) 1.31 (.09) 1.30 (.08)Bγ = 1 1.26 (.13) 1.25 (.13) 1.30 (.08) 1.31 (.08)Bγ = 10 1.00 (.12) 1.21 (.11) 1.22 (.06) 1.34 (.07)Bγ = 100 .08 (.05) .62 (.11) .27 (.06) 1.15 (.06)

Bβ=10

Bγ = 0 1.29 (.13) 1.24 (.12) 1.29 (.07) 1.27 (.07)Bγ = 1 1.25 (.12) 1.23 (.12) 1.28 (.07) 1.28 (.07)Bγ = 10 1.00 (.12) 1.20 (.10) 1.22 (.06) 1.31 (.06)Bγ = 100 .08 (.05) .62 (.11) .30 (.07) 1.15 (.06)

Bβ=100

Bγ = 0 1.28 (.12) 1.25 (.11) 1.29 (.07) 1.27 (.06)Bγ = 1 1.25 (.11) 1.25 (.11) 1.29 (.06) 1.27 (.05)Bγ = 10 1.06 (.13) 1.22 (.10) 1.25 (.05) 1.30 (.05)Bγ = 100 .08 (.05) .62 (.11) .34 (.09) 1.16 (.06)

PSA-1rep 1.30 (.13) 1.23 (.11) 1.29 (.07) 1.29 (.07)PSA-100rep 1.28 (.10) 1.27 (.09) 1.28 (.06) 1.25 (.05)









❖ Conclusion

SREE 2011 25 / 28

Table 4:The γATE s & S.E.s for Conventional and Bayesian Stratification and Matching of Design II


N = 100 N = 250


Bβ=0

Bγ = 0 .32 (.14) .27 (.13) .32(.09) .31 (.09)Bγ = 1 .31 (.13) .27 (.13) .32 (.09) .31 (.09)Bγ = 10 .29 (.10) .27 (.12) .30 (.08) .31 (.08)Bγ = 100 .26 (.04) .26 (.08) .26 (.04) .29 (.07)

Bβ=1

Bγ = 0 .30 (.14) .25 (.13) .31 (.09) .30 (.08)Bγ = 1 .30 (.13) .25 (.13) .31 (.09) .30 (.08)Bγ = 10 .28 (.10) .25 (.12) .30 (.07) .30 (.08)Bγ = 100 .26 (.04) .25 (.08) .26 (.04) .28 (.06)

Bβ=10

Bγ = 0 .28 (.13) .23 (.11) .28 (.07) .27 (.07)Bγ = 1 .27 (.12) .23 (.11) .28 (.07) .27 (.07)Bγ = 10 .26 (.09) .23 (.11) .27 (.06) .27 (.07)Bγ = 100 .25 (.04) .24 (.07) .25 (.04) .26 (.06)

Bβ=100

Bγ = 0 .24 (.09) .23 (.09) .26 (.06) .24 (.05)Bγ = 1 .24 (.09) .23 (.09) .26 (.06) .24 (.05)Bγ = 10 .25 (.08) .23 (.08) .25 (.06) .24 (.05)Bγ = 100 .25 (.04) .24 (.06) .24 (.04) .24 (.05)

PSA-1rep .30 (.13) .23 (.11) .29 (.07) .29 (.07)PSA-100rep .28 (.10) .27 (.09) .28 (.06) .25 (.05)









❖ Conclusion

SREE 2011 26 / 28

Table 5:The γATE s & S.E.s for Conventional and Bayesian Stratification and Matching of Design II


N = 100 N = 250


Bβ=0

Bγ = 0 1.32 (.14) 1.27 (.13) 1.32(.09) 1.31 (.09)Bγ = 1 1.31 (.13) 1.27 (.13) 1.32 (.09) 1.31 (.09)Bγ = 10 1.29 (.10) 1.27 (.12) 1.30 (.08) 1.31 (.08)Bγ = 100 1.26 (.04) 1.26 (.08) 1.26 (.04) 1.29 (.07)

Bβ=1

Bγ = 0 1.30 (.14) 1.25 (.13) 1.31 (.09) 1.30 (.08)Bγ = 1 1.30 (.13) 1.25 (.13) 1.31 (.09) 1.30 (.08)Bγ = 10 1.28 (.10) 1.25 (.12) 1.30 (.07) 1.30 (.08)Bγ = 100 1.26 (.04) 1.25 (.08) 1.26 (.04) 1.28 (.06)

Bβ=10

Bγ = 0 1.28 (.13) 1.23 (.11) 1.28 (.07) 1.27 (.07)Bγ = 1 1.27 (.12) 1.23 (.11) 1.28 (.07) 1.27 (.07)Bγ = 10 1.26 (.09) 1.23 (.11) 1.27 (.06) 1.27 (.07)Bγ = 100 1.25 (.04) 1.24 (.07) 1.25 (.04) 1.26 (.06)

Bβ=100

Bγ = 0 1.24 (.09) 1.23 (.09) 1.26 (.06) 1.24 (.05)Bγ = 1 1.24 (.09) 1.23 (.09) 1.26 (.06) 1.24 (.05)Bγ = 10 1.25 (.08) 1.23 (.08) 1.25 (.06) 1.24 (.05)Bγ = 100 1.25 (.04) 1.24 (.06) 1.24 (.04) 1.24 (.05)

PSA-1rep 1.30 (.13) 1.23 (.11) 1.29 (.07) 1.29 (.07)PSA-100rep 1.28 (.10) 1.27 (.09) 1.28 (.06) 1.25 (.05)

Conclusion









❖ Conclusion

SREE 2011 27 / 28

● To summarize, Study I reveals that greater precision in thepropensity score equation yields better recovery of thefrequentist-based causal effect compared to traditional PSA andcompared to no adjustment.

● Study I also reveals a very small advantage to the Bayesianapproach for N = 100 versus N = 250.

● Study II-A reveals that greater precision around the wrong causaleffect can lead to seriously distorted results.

● Study II-B reveals that greater precision around the correct causalparameter yeilds quite good results, with slight improvement seenwith greater precision in the propensity score equation.

● This study also suggests that optimal matching is preferred in termsof bias and precision.









❖ Conclusion

SREE 2011 28 / 28

● To conclude, we propose a simple and reasonable strategy forBayesian propensity score analysis based on a two step approach.

● This approach preserves the basic idea that the propensity scoreshould provide information only regarding study design and notregarding the treatment effect.

● Bayesian PSA is easy to implement and addresses the encoding ofprior information in both the propensity score equation andoutcomes model equation.

● Research on the elicitation of priors will be essential to furtherdemonstrate the value of the Bayesian approach (O’Hagan et al.,2006).

Bayesian Propensity Score Analysis: Simulation … Propensity Score Analysis: Simulation and Case...

Documents

Transcript of Bayesian Propensity Score Analysis: Simulation … Propensity Score Analysis: Simulation and Case...