Issues of Generalized Causal Inference and Methods for Single Studies
description
Transcript of Issues of Generalized Causal Inference and Methods for Single Studies
![Page 1: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/1.jpg)
Issues of Generalized Causal Inference and Methods for Single
Studies
Shadish, Cook and CampbellChapters 10 and 11
Joy and Olivia
![Page 2: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/2.jpg)
9-9:30: Principals for Generalized Causal Inference and Typical/Heterogeneous Sampling (Joy)
9:30-55: Activity 1 (You all)
9:55-10:15: Methods for Generalizing from Purposive Samples and Studying Causal Explanation in Single Studies (Olivia)
10:15-10:25: Break
10:25-10:50: Activity 2 (You all)
10:50-11:00: Wrap-up on single study methods (Olivia)
11:00-11:50: Standards of Evidence (Brian)
Agenda (approximation)
![Page 3: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/3.jpg)
Involves generalizing about a study’s UTOS: Units, Treatments, Outcomes, Settings
Researchers make these generalizations intuitively The authors’ purpose is to describe and promote an
explicit, standardized process for generalizing This improves:
◦ Improves dissemination of results◦ Construct and external validity
Generalized Causal Inference
![Page 4: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/4.jpg)
Formal (random) sampling is ideal But, real-world constraints can make random sampling
difficult◦ Random sampling of units difficult when:
Attrition or non-compliance are high Exact parameters of the population are not known
◦ Samples of settings limited by cost and convenience◦ Researchers generally prefer that treatments and outcome
measures be based in theory, and limited availability may make randomizing difficult
The alternative: Purposive sampling
Why not Formal Sampling?
![Page 5: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/5.jpg)
Derived from authors’ observations of how generalizations are made (chart pg. 357)◦ Surface Validity◦ Ruling Out Irrelevancies◦ Making Discriminations◦ Interpolation and extrapolation◦ Causal Explanation
Purposive Sampling helps to locate heterogeneous or typical cases to study interactions
Principles of Generalized Causal Inferencefor Purposive Sampling
![Page 6: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/6.jpg)
Judging apparent similarities between what was studied and the targets of generalization
Constructs: ◦ ensuring face validity of constructs ◦ matching surface characteristics to prototypical
characteristics (ex. Unemployment) External validity: easiest where proximal
similarity is most evident (ex. Cancer treatment)
Principle 1: Surface Similarity
![Page 7: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/7.jpg)
Identify attributes that don’t impact generalizability
Constructs:◦ Ruling out construct irrelevancies related to study design◦ Convergent validity – outcomes on different measures
for similar constructs should correlate◦ Multiple operationalism differentiates between relevant
and irrelevant features of measures◦ Heterogeneity of irrelevancies provides insight to which
aspects of design are truly irrelevant External validity:
◦ Including heterogeneity of presumed irrelevancies strengthens design (Fisher, 1935)
May result in modifiers to the causal relationship
Principle 2: Ruling Out Irrelevancies
![Page 8: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/8.jpg)
Discriminating between those UTOS
that can and cannot be generalized to Constructs:
◦ Discriminating between constructs of interest and alternatives (discriminant validity, ex. Schizophrenia vs. Alzheimer’s)
External Validity: ◦ Determining which UTOS might
change causal relationships’ direction or strength
Principle 3: Making Discriminations
![Page 9: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/9.jpg)
Generalizing to unsampled values of the UTOS both within and outside the range of the sampled data
Constructs: ◦ Interpolation requires accurate representation of the population◦ Confounding constructs with levels◦ Floor and ceiling effects
External validity:◦ Interpolation (ex. BMI)◦ Extrapolation
Confidence is highest closest to the range of the data
Can you think of an example of when it is acceptable to extrapolate?
Principle 4:Interpolation and Extrapolation
![Page 10: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/10.jpg)
Developing and testing explanatory theories
◦ Esp. for selecting treatments, measures
Constructs:◦ Deep or Structural Similarities may not always be
apparent at the surface level◦ Construct Network describes relationships between
constructs External validity: in order to generalize, need to
specify (1) which parts of treatment (2) affect which outcomes (3) through which causal mechanisms◦ Complete causal knowledge is a rarity
Principle 5: Causal Explanation
![Page 11: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/11.jpg)
Typical instances are those that represent the mean, median, or mode of the UTOS of interest◦ Requires clear definitions of the “type” of interest
Heterogeneous instances are diversified on important characteristics◦ Lowers power, but can demonstrate a stronger relationship◦ Increases ability to identify irrelevancies, make discriminations,
determine mediators, and inter/extrapolate Surface similarities, ruling out irrelevancies, making
discriminations, and inter/extrapolation can help determine what characteristics should be typical or heterogeneous
Purposive Sampling of UTOS
![Page 12: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/12.jpg)
Form groups of 3-4 and select a program to evaluate from the list Discuss the relevant UTOS (Units, Treatments, Outcomes, Settings). How do you
know which ones to choose? How do you use the Principles of Causal Generalization (Surface Similarity, Ruling out
Irrelevancies, Making Discriminations, Inter/Extrapolation, Causal Explanation) and PSI-Typ/PSI-Het to design the study? (you don’t have to use every principle, just touch on the most useful ones for your case)
Be ready to report back!
Programs:
Time for an activity!
1. School-based program for STI prevention in teens
6. Evaluate the implementation of the Coordinated Care Organization legislation in Oregon
2. A hospital-based parenting program for new parents
7. A program to reduce hypertension in middle-aged women
3. A nutrition-based obesity prevention program for children
8. A work-place wellness program
4. A program to increase social supports for older adults
9. A program to promote healthy relationships in LGBT couples
5. A program to reduce suicide rates in middle-aged Japanese males
10. A childhood vaccination awareness program for parents
![Page 13: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/13.jpg)
![Page 14: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/14.jpg)
The classic method in stats for generalizability is formal probability sampling, but…
◦ This is rarely feasible in experimental and quasi-experimental research.
Because of this we can strengthen the generalizability of causal inference by applying SCC’s five principles using
◦ Statistical methods for generalizing from purposive samples, and/or
◦ Methods for studying causal explanation.
Caveat: Described with respect to single studies.
Let’s begin with the end
![Page 15: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/15.jpg)
The main idea = by applying weights to our sample data (ideally on a purposive sample), we can generate population-ish estimates.
How close we come to this approximation depends on our knowledge of the pertinent population characteristics (i.e. if the info exists).
We also usually think of this only in terms of our units/cases/obs, namely people, vs. the other TOS.
Generalizing from Purposive Samples: Sample Reweighting
![Page 16: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/16.jpg)
RSM logic: generalized causal inference is a prediction about the likely effect that would be obtained with the UTOS specified by the analyst as the target of generalization.
For example, we may want to understand what level(s) of x predict an optimal y.
Think “forecasting” or if you’ve had exposure to engineering “optimization.”
This is essentially regression modeling, but you need to have collected the right data to do it ◦ (e.g. different levels of your main independent variables of interest).
Generalizing from Purposive Samples: Response Surface Modeling (RSM)
![Page 17: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/17.jpg)
RSM Example
Source: Arch Intern Med. 2004;164(10):1121-1127. doi:10.1001/archinte.164.10.1121
![Page 18: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/18.jpg)
Qualitative methods—discovery focus
Statistical methods—measuring, not directly manipulating the UTOS during the study period
Manipulation of explanatory variables in experiments—strengthening the basis for asserting the causal roles of mediating variables
Causal Explanation Methods
![Page 19: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/19.jpg)
Help us understand how and/or why interventions/programs/treatments/etc. work (or don’t).
May include participant observation, interviews, focus groups, etc.
Discrepancies in findings may arise when using mixed-methods (quantitative and qualitative).
How have you used qualitative methods in your work or for someone else’s to study causal explanation?
Qualitative Methods
![Page 20: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/20.jpg)
Main idea = We have a set of independent, dependent, mediating and moderating variables that we can use to model causal relationships with.
aka: Causal modeling, structural equation modeling, or covariance structure analysis.
◦ Specific applications include path analysis, cross-lagged panel designs and latent variable structural models.
Statistical Methods (Yay!)
![Page 21: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/21.jpg)
Two examples
What is the difference between mediation and moderation/interaction?
Diagrams adopted from: http://academic.csuohio.edu/kneuendorf/c63111/hand22.pdf
![Page 22: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/22.jpg)
Main idea = “[A] graphical model that [structures] hypothesized causal and correlational relationships among all measures” (p. 393).
Force us to thoughtfully specify models and test the goodness/lack of fit of what we hypothesized based on the results.
Paths may be direct or indirect.
Path Diagrams
![Page 23: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/23.jpg)
Two Examples
Exercise/weight loss figures adopted from: http://crab.rutgers.edu/~goertzel/pathanal.htmRejection/identification model from article by Branscombe et al. 1999.
![Page 24: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/24.jpg)
Path coefficient = the parameter being estimated
Endogenous variable = variables that receive causal inputs from other variables in the model (they have an arrow going into them).
Exogenous variable = no causal inputs
Error terms = additional variance influencing endogenous variables (e.g. measurement error)
Non-recursive causation = when two variables influence one another (i.e. the relationship is bi-directional).
Semantics
![Page 25: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/25.jpg)
Get back into your group from earlier and create a simple path diagram for the intervention/program you picked.
You don’t need to include all of the potential variables, just select an outcome, predictor, mediator and/or modifier.
Extra credit for writing your path equations (p. 397).
Be ready to draw it and explain briefly.
You have 15 minutes to work on this.
Now you give it a whirl
![Page 26: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/26.jpg)
Measurement error may bias path coefficients.
Omitted variables may bias models.
Misspecification of relationships and/or their functional form may bias estimates.
Make sure the assumptions of tests you use are met.
Tests of individual model parts may lower power, so sample size is important.
Things to keep in mind
![Page 27: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/27.jpg)
Cross-lagged panel correlational design = measuring cause and effect together at two times to see if cross-lagged correlations differ. ◦ If a stronger association exists between Time 1 Cause and Time 2 Effect than
Time 1 Effect and Time 2 Cause, then we can better infer a causal relationship. (image p. 412)
Instrumental variable approaches = Including factors in analyses hypothesized to indirectly influence outcome(s). See Urban Institute Tool Kit for more info: http://www.urban.org/toolkit/data-methods/instrumental.cfm
Latent variable modeling = separating variance due to the latent construct from variance due to unique and random error components (e.g. factor analysis).◦ Can help us to suss out if our measures are indicative of what we think they are,
or show multiple latent variables in a single measure◦ Examples?
More applications
![Page 28: Issues of Generalized Causal Inference and Methods for Single Studies](https://reader035.fdocuments.in/reader035/viewer/2022062323/568166ee550346895ddb433a/html5/thumbnails/28.jpg)
Why? Helps to determine the effects of potential mediators within our causal explanation framework.
Blockage models = block the mediator to see if the relationship between cause and effect changes as hypothesized.
Enhancement models = test of cause and effect followed by another test in which the mediating factor is added to see if enhancement of effect occurs as hypothesized.
Pattern matching = testing meditational paths hypothesized to result in different outcomes and seeing if the observed pattern matches any hypothesized pattern(s).
Manipulation of explanatory variables in experiments