Bayes Factors - Mathematics & Statisticsmath.bu.edu/people/sray/talk/bic.pdf · 2006. 8. 17. ·...

Bollen, Ray, Zavisca ASA, April 22 , 2005 - slide #1

Bayes Factorsin

Structural Equation Models (SEMs):Schwarz BIC and Other Approximations

.

Kenneth A. BollenUniversity of North Carolina, Chapel Hill

Surajit RaySAMSI and University of North Carolina, Chapel Hill

Jane ZaviscaSAMSI and University of Arizona

Bayes Factor in SEM

Bayes factor in SEM

Model Fit in SEM’s

Approximating Bayes Factor

Simulation

Results

Conclusion


Bayes factor in SEM

■ Model Fit in SEMs

■ Bayes Factor

■ Approximating Bayes Factor

■ Simulation

■ Results

■ Conclusions

Bayes Factor in SEM

Bayes factor in SEM


SEM

Hypothesis testing

Fit Indices

Model Comparisons


Simulation

Results

Conclusion


SEM

Latent Variable Model

η = Bη + Γξ + ζ

Measurement Model

y = Λyη + �

x = Λxξ + δ

Bayes Factor in SEM

Bayes factor in SEM


SEM

Hypothesis testing

Fit Indices

Model Comparisons


Simulation

Results

Conclusion


Hypothesis testing

H0 : Σ = Σ(θ)

Chi square Test Statistic

T = (N − 1)FML ∼ χ2 in large samples

■ excess power when big N

■ excess kurtosis influence on T

■ exact H0, approximate model

Bayes Factor in SEM

Bayes factor in SEM


SEM

Hypothesis testing

Fit Indices

Model Comparisons


Simulation

Results

Conclusion


Fit Indices

RMSEA, CFI, TLI, IFI, Etc.

■ Cutoff values?

■ Nonnested models?

■ Small N issues?

■ Behavior across estimators?

Bayes Factor in SEM

Bayes factor in SEM


SEM

Hypothesis testing

Fit Indices

Model Comparisons


Simulation

Results

Conclusion


Model Comparisons

■ Chi square difference (LR) tests◆ Power, excess kurtosis, N issues◆ Nested Models Only.

■ Fit indices differences◆ Cutoff values for differences◆ Behavior across estimators◆ Properties across N

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)

Posterior Odds and BF

Marginal Likelihood

Laplace Approximation

Laplace Approximation - MLE

substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


Previous Work

■ Bayesian work small part of SEMs

■ Bayes factor, largely discussed via BIC in SEM literature◆ Cudeck and Browne (1983)◆ Bollen (1989)◆ Raftery (1993, 1995)◆ Haughton, Oud, and Jansen (1997)

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


Bayes Factor (BF)

■ Y : Data

■ Mk : Model

■ Bayes Theorem

P (M1|Y ) =P (Y |M1)P (M1)

P (Y |M1)P (M1) + P (Y |M2)P (M2)

■ Comparing Model M1 and M2Choose the model with higher posterior probability.

P (M1|Y )

P (M2|Y )=

P (Y |M1)

P (Y |M2)

P (M1)

P (M2)

= Bayes Factor × prior odds

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion



P (M1|Y )

P (M2|Y )=

P (Y |M1)

P (Y |M2)

P (M1)

P (M2)

If P (M1) = P (M2) then prior odds=1

=⇒ Posterior Odds = BF

Define Bayes Factor as

BF12 =P (Y |M1)

P (Y |M2)

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


Marginal Likelihood

■ P (θ|Mk): Prior Distribution of θ given the models Mk.

■ dk dimension of θk■ In general the Marginal Likelihood is

P (Y |Mk) =

∫

θk

P (Y |Mk, θk)P (θk|Mk)dθk

■ If the pdf P (Y |Mk)’s are completely specified ( no free parameters).—– Bayes Factor= Likelihood ratio.

■ Sensitivity to prior is more critical in calculation in BF than in otherBayesian Analysis. ( Raftery 1993)

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion



P (Y |Mk) =

∫

θk

P (Y |Mk, θk)P (θk|Mk)︸︷︷︸

dθk

■ Laplace Approximation on Likelihood × Prior

P (Y |Mk) ≈ (2π)dk/2|Ĩ(θ̃k)|

− 12 P (y|θ̃k, Mk)P (θ̃k)

■ Error of approximation : O( 1n )

Ĩ(θ̃k) =d2

dθdθ′log(P (y|θ, Mk)P (θ))

∣∣∣θ=θ̃k

=d2

dθdθ′l̃(θ)

∣∣∣θ=θ̃k

■ θ̃ not readily available from software outputs

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


Laplace Approximation - MLE substitution

■ Replacing θ̃ by θ̂ (MLE)

P (Y |Mk) ≈ (2π)dk/2|I(θ̂k)|

− 12 P (y|θ̂k, Mk)P (θ̂k)

I(θ̂k) =d2

dθdθ′log(P (y|θ, Mk))

∣∣∣θk=θ̂k

=d2

dθdθ′l(θ)

∣∣∣θk=θ̂k

◆ Error: O( 1n )

◆ Less accurate than using θ̃

◆ Using E(I(θ̂k)) in place of I(θ̂k) has error: O(1

n12)

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


BIC

P (Y |Mk) ≈ (2π)dk/2|I(θ̂k)|

− 12 P (y|θ̂k, Mk)P (θ̂k)

■ Choosing Unit Information prior i.e.

P (θk) ∼ N

θok ,

[

I(θ̂k)

n

]−1

=⇒ P (Y |Mk) ≈ el(θ̂k|y)(n)−dk/2

=⇒ 2 log P (Y |Mk) ≈ 2l(θ̂k|y) −dk

log(n)

2 log BF = BIC

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


ABF 1

P (Y |Mk) =

∫

θk

P (Y |Mk, θk)︸︷︷︸

P (θk|Mk)︸︷︷︸

dθk

Likelihood Prior

■ Laplace Approximation only on P (y|θ̂k, Mk)

■ Prior

P (θk|Mk) ∼ N

θ0k,

[

c

I(θ̂k)

n

]−1

Using the c to maximize P (Y |Mk)

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


ABF 1 and BIC

Using the c

2 log P (Y |Mk) ≈ 2l(θk) − dk

(

1 + log

[

dk

θ̂kTI(θ̂k)θ̂k

])

=⇒ ABF 1 = BIC − d1 log

[

d1

θ̂1T I(θ̂1)

nθ̂1

]

+ d2 log

[

d2

θ̂2T I(θ̂2)

nθ̂2

]

Bayes Factor in SEM

Bayes factor in SEM



Previous Work

Bayes Factor (BF)


Marginal Likelihood



substitution

BIC

ABF 1

ABF 1 and BIC

ABF 1 and BIC

Simulation

Results

Conclusion


ABF 1 and BIC

BIC ABF 1

Prior implicit Yes Yes: has more flexibility thanthe unit information prior

Uses standard software output Yes Yes

Need for defining n. a Yes No: Enters through

θ̂kTI(θ̂k)θ̂k

aSee Raftery 1993 and 1995 on uncertainty about n in SEM’s

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)

Missing Crossloading (M1)



Correlated errors Replace

Crossloadings (M4)

Over-specified Model (M5)



Extra Latent Variable (M8)

Missing Latent Variable

Wrong Structural Model



Missing Indicator

Switched Loadings

Results

Conclusion


Simulation Study

■ We tested model selection properties on a simulated data set.

■ 20 replicate data sets were created at each of 4 sample sizes (N=100,N=250, N=500, N=1000). Full scale Monte Carlo simulation study stillneeds to be done.

■ We fit a variety of under- and over-specified models.

For further details on the simulated data, see Paxton et “Monte CarloExperiments” Structural Equation Modeling, 2001.

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)





Crossloadings (M4)









Missing Indicator

Switched Loadings

Results

Conclusion


True Model (MT)

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)





Crossloadings (M4)









Missing Indicator

Switched Loadings

Results

Conclusion



Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)





Crossloadings (M4)









Missing Indicator

Switched Loadings

Results

Conclusion


Correlated errors Replace Crossloadings (M4)

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)





Crossloadings (M4)









Missing Indicator

Switched Loadings

Results

Conclusion



Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)





Crossloadings (M4)









Missing Indicator

Switched Loadings

Results

Conclusion


Missing Indicator

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Simulation Study

True Model (MT)





Crossloadings (M4)









Missing Indicator

Switched Loadings

Results

Conclusion


Switched Loadings

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results

Distn. (%) of Selected Models


% of Samples Selecting Over vs Under

Specified Models

Ranking of Models 1-3

Conclusion



Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results




Specified Models


Conclusion



Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results




Specified Models


Conclusion


% of Samples Selecting Over vs Under Specified Models

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results




Specified Models


Conclusion



Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results

Conclusion

Concluding Remarks

Future Direction


Concluding Remarks

■ BIC 6= ABF 6= Bayes Factor.

■ Other approximations are possible◆ ABF2, GBIC, Houghton’s BICR

■ PERFORMANCE: Small-scale simulation suggests◆ ABF performs better than BIC in small samples◆ ABF performs better than PVAL based conclusion in large samples◆ ABF performs as good or better than the best performance of BIC or

PVAL for all sample size.

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results

Conclusion

Concluding Remarks

Future Direction


Future Direction

■ Study the sensitivity of Bayes Factors to priors.

■ Study the consistency property of ABF.

■ Choosing more flexible priors. (ABF 2, GBIC)

■ Do a large simulation study

Bayes Factor in SEM

Bayes factor in SEM



Simulation

Results

Conclusion

Acknowledgement

Acknowledgement


Acknowledgement

We thank theSAMSI Model Averaging Group

. Slides Prepared by LATEX HA-prosper Package.

A Statisticians OverviewBayes factor in SEM MFBSEM Hypothesis testing Fit Indices Model Comparisons

ABFPrevious Work Bayes Factor (BF) Posterior Odds and BF Marginal Likelihood Laplace Approximation Laplace Approximation - MLE substitution BIC ABF 1 ABF 1 and BIC ABF 1 and BIC

SIMSimulation Study True Model (MT) Missing Crossloading (M1) Missing Crossloading (M2) Missing Crossloading (M3) Correlated errors Replace Crossloadings (M4) Over-specified Model (M5) Over-specified Model (M6) Over-specified Model (M7) Extra Latent Variable (M8) Missing Latent Variable Wrong Structural Model Wrong Structural Model Wrong Structural Model Missing Indicator Switched Loadings

RESDistn. (%) of Selected Models Distn. (%) of Selected Models % of Samples Selecting Over vs Under Specified Models Ranking of Models 1-3

CONCConcluding Remarks Future Direction

ACKAcknowledgement

Bayes Factors - Mathematics & Statisticsmath.bu.edu/people/sray/talk/bic.pdf · 2006. 8. 17. ·...

Documents

Transcript of Bayes Factors - Mathematics & Statisticsmath.bu.edu/people/sray/talk/bic.pdf · 2006. 8. 17. ·...