PSY 1950 Meta-analysis December 3, 2008
-
Upload
cain-mcconnell -
Category
Documents
-
view
18 -
download
1
description
Transcript of PSY 1950 Meta-analysis December 3, 2008
PSY 1950Meta-analysis
December 3, 2008
Definition“the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature.”– Glass (1976)
“Mega-silliness”– Eysenck (1977)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
History• Pre-history
– Pearson (1904), Fisher (1948), Cochran’s (1955)
• The Great Debate– 1952: Eysenck concluded that psychotherapy was bunk
– 20 years of research did not settle debate
– 1978: Glass & Smith statistically aggregated findings from 375 studies, concluding that psychotherapy works
• Necessity is the mother of invention– Psychology abounds!
Rationale• Meta-analyses avoid the limitations of qualitative/narrative/traditional reviews:– Weak effects overloooked
• Meta-analyses are more powerful
– Differences between studies over-interpreted• In meta-analysis, heterogeneity assessed statistically
– Moderating variables overestimated or overlooked• In meta-analysis moderators assessed statistically
– Limited, subjective sampling of studies• In meta-analysis, exhaustive search and defined inclusion/exclusion criteria
– Overwhelmed by large database• Meta-analyses can summarize hundreds of effects
– Subjective assessment• In meta-analysis, any subjectivity is discernible
Example: Finding Weak Effects
• Cooper, H. M., & Rosenthal, R. (1980). Statistical versus traditional procedures for summarizing research findings. Psychological Bulletin, 87, 442-449.– 32 grad students, 9 faculty members– Randomly assigned to statistical and traditional review technique
– Given 7 studies that examined sex differences in persistence•For 2 studies, females more persistent than males (ps = .005, .001)
•For other studies, no significant difference
– Does evidence presented support the conclusions that females are more persistent?
actual p = .016
Example: Assessing Moderators Statistically
Criticisms• Weak
– Apples and oranges– Flat Earth society– Garbage in, garbage out– File-drawer problem
• Strong– Post-hoc
Apples and Oranges• Critique
– Meta-analyses add together apples and oranges
• Response– Glass: “in the study of fruit, nothing else is sensible”
– Analogy with single experiments– Empirical question resolved through examination of moderating variables
Flat Earth Society• Critique
– Cronbach: "...some of our colleagues are beginning to sound like a kind of Flat Earth Society. They tell us that the world is essentially simple: most social phenomena are adequately described by linear relations; one-parameter scaling can discover coherent variables independent of culture and population; and inconsistences among studies of the same kind will vanish if we but amalgamate a sufficient number of studies.... The Flat Earth folk seek to bury any complex hypothesis with an empirical bulldozer.”
• Response– Code and analyze moderating variables
QuickTime™ and a decompressor
are needed to see this picture.
Garbage In, Garbage Out• Critique
– The inclusion of flawed studies “dirties” the database, obscures the truth, and invalidates meta-analytic conclusions
• Response– Glass: “I remain staunchly committed to the idea that meta-analyses must deal with all studies, good bad and indifferent, and that their results are only properly understood in the context of each other, not after having been censored by some a priori set of prejudices.”
– Empirical question: Study quality (or better yet, related variables) can be coded and analyzed as moderators
File-drawer Problem• Critique
– Meta-analytic database is biased sampling of studies
– Significant findings are more likely to be published than nonsignificant findings
• Response– Less bias than narrative reviews– File-drawer analyses (e.g., funnel plots) can empirically address the presence and influence of missing studies
Post-hoc• Criticism
– By definition, meta-analysis is a post-hoc endeavor, i.e., an observational study•Moderating variables may be confounded, sometimes extremely so
•Effects may be correlational
• Response– Confounding may be interested in its own right
– Statistical control– Hypothesis generation versus hypothesis testing
Type of publication
Type of meditation
JournalDissertationOtherType of meditation
Type of meditation
TM-0.15<. 010.19Yoga0.10-0.140.03Other0.090.12-0.25
Author affiliation-0.020.03<. 010.160.15-0.31Control
group/ cond-0.040.07-0.03-0.09-0.080.17-0.20State0.06-0.190.14-0.050.010.05-0.100.22Trait-0.080.20-0.110.03-0.01-0.030.15-0.22Both0.07-0.05-0.030.030.001-0.04-0.120.05 p < .001
General intelligence0.06-0.090.030.09-0.09-0.030.11-0.04-0.050.07-0.05 p < .0001
Fluid intelligence-0.070.020.060.08-0.07-0.04-0.120.02-0.130.100.03Crystallized intelligence-0.060.10-0.040.10-0.08-0.050.030.03-0.080.040.08
Memory and learning-0.100.060.07-0.060.030.05-0.10<. 010.01-0.010.001Visual
perception0.110.02-0.15-0.060.050.030.11-0.06-0.040.09-0.10Auditory
perception-0.08-0.040.140.17-0.09-0.12-0.150.120.25-0.20-0.05Retrieval/ creativity0.08-0.04-0.050.02-0.110.06-0.020.11-0.020.05-0.06Cognitive speediness-0.020.11-0.08-0.080.17-0.050.00-0.020.04-0.01-0.05Processing
speed0.04-0.080.030.09-0.06-0.050.15-0.070.01-0.040.07Psychomotor-0.05-0.100.17-0.050.17-0.08-0.01-0.060.10-0.180.19
Attention0.050.06-0.13-0.230.010.26-0.040.04-0.060.08-0.05Between
participants-0.080.12-0.030.13-0.220.02-0.10-0.27-0.140.110.04-0.210.01-0.070.030.030.050.01-0.070.030.070.04Within
participants-0.06-0.110.200.000.13-0.110.080.110.31-0.26-0.040.05-0.06-0.04-0.08-0.050.25-0.060.03-0.010.09-0.06Mixed0.10-0.08-0.05-0.130.170.020.060.220.02<. 01-0.020.180.010.09<. 01-0.01-0.150.020.06-0.03-0.10-0.01
Random assignment0.030.10-0.15-0.16-0.050.220.170.25-0.030.04-0.030.150.050.100.12-0.09-0.140.050.07-0.13-0.130.01-0.38n/a0.38
Age-0.240.41-0.14-0.02-0.080.090.14-0.04-0.070.040.04-0.11-0.09-0.02-0.03-0.010.02-0.040.040.030.080.130.230.01-0.23-0.08Meditative experience-0.060.08-0.010.50-0.27-0.360.18-0.29-0.020.05-0.08-0.050.03-0.02-0.07-0.020.14-0.08-0.010.090.01-0.010.52-0.02-0.51-0.360.28
Impact factorn/an/an/a-0.09-0.210.290.140.24-0.140.120.020.040.020.09-0.01-0.11-0.070.04-0.08-0.04-0.170.36-0.27-0.110.30-0.360.11-0.21
Broad cognitive ability
Type of meditation
Type of effect
AgeRandom assignment
StateTraitBoth
Type of design
Between participantsWithin participantsMixed
Meditative experience
Cognitive speedinessProcessing speedPsychomotorAttentionMemory and learningVisual perceptionAuditory perceptionRetrieval/ creativityGeneral intelligenceFluid intelligenceCrystallized intelligence
Control group/ condAuthor affiliation
OtherYogaTMType of meditation
Type of effect
Broad cognitive ability
Type of design
Steps of a Meta-analysis• Define question• Search literature• Determine inclusion/exclusion criteria
• Code moderating variables• Analyze data
• This is an iterative process!
Defining Meta-analytic Question
• Interestingness– Establish presence of effect– Determine magnitude of effect– Resolve differences in literature– Test competing theories
• e.g., psychotherapy, imagery v1
Inclusion/Exclusion Criteria
• Theoretical considerations– Scope/generalizability– Quality
• Practical considerations– Power– Missing data– Time
• Studies were included if they– had written published or unpublished reports in English available by March 1, 2008
– presented original data from between-participants, within-participants (i.e., single-group pretest-posttest, or PP), or mixed design (i.e., independent-groups pretest-posttest, or IG-PP) experiments or quasi-experiments
– objectively, quantitatively evaluated performance on at least one cognitive task as a function of meditative experience or state
• Studies were excluded if they– used a psychopathologically or neurologically disordered population
– confounded meditation with other mental training (e.g., education), maturation, or practice and used measures susceptible to such confounding (e.g., academic achievement test)
– did not report data on or contained data that allow estimation of participants’ age or meditative experience
– did not contain basic methodological information (e.g. type of task administered)
Literature Search• Types of searches
– Keyword– Ancestor– Descendent
• Available Resources– Electronic
• e.g., PsychInfo, SCI, Google scholar
– Physical• Conference proceedings• Bibliographies• Key journals
– Mental• Experts in the field
Harvard’s Electronic Resources
• SSCI/SCI (Social/Science Citations Index)– http://nrs.harvard.edu/urn-3:hul.eresource:scicitin
• PsychInfo– http://nrs.harvard.edu/urn-3:hul.eresource:psycinfo
• Google Scholar– http://scholar.google.com/
• E-Journals @ Harvard– http://sfx.hul.harvard.edu/sfx_local/az/
• HOLLIS– http://holliscatalog.harvard.edu/
• Interlibrary Loan– https://illiad.hcl.harvard.edu/
• Dissertations (Proquest)– http://nrs.harvard.edu/urn-3:hul.eresource:psycinfo
0
0.0005
0.001
0.0015
0.002
0.0025
0.003
1968 1973 1978 1983 1988 1993 1998 2003
Year
Meditation publications (%)
TM Buddhist Yoga
Coding• What to code
– Anything possibly interesting•e.g., control group/condition, participant variables
– Anything possibly confounding•e.g., publication year, journal impact factor
– How you coded effect sizes
• How to code– Using explicit coding scheme– Set measurement scale– Multiple coders– Calculate reliability
Coded VariableType of data Categories (if applicable)
Potential Moderator?
First author Nominal no
Publication year Interval no
Type of publication Nominal peer-reviewed journal, dissertation/thesis, other yes
Journal impact factor Ratio yes
Type of meditation Nominal TM, yoga, other yes
Author affiliation Nominal yes, no yes
Control group/conditionOrdinal 1, 2, 3 yes
Type of effect Nominal state, trait, both yes
Task Nominal no
Narrow cognitive abilityNominal see Carroll (1993) no
Broad cognitive abilityNominal
general intelligence (g), fluid intelligence (2F), crystallized
intelligence (2C), memory and learning (2Y), visual perception (2V),
auditory perception (2U), retrieval/creativity (2R), cognitive speediness (2S), processing speed (2T), psychomotor ability (2P),
attention
yes
Experimental design Nominal between-participants, within-participants, mixed yes
Random assignment Nominal yes, no yes
# participants (meditation)Ratio no
# participants (control)Ratio no
Mean age (meditators)Interval 0-18, 18-25, 25-40, 40+ (years old)yes
Meditative experienceInterval 0-3 months, 3-12 months, 1-3 years, 3+ years yes
Analysis• Calculate effect size• Weight effect size• Describe• Infer
– Univariate analyses– Multivariate analyses
Calculating Effect Size• Only one ES per construct per study– Balance between dependency and thoroughness
• Typically d or r
• Can be calculated in lots of way (from raw data to graphs)
• Effect size calculator
Weighting Effect Size• Why weight?
– Studies vary significantly in size– Studies with large n have more reliable effect sizes than studies with small n
• How weight?– Simple approach: weight by sample size– Better approach: weight by precision
• What is precision weighting?– Each effect size has associated SE– Hedges showed that best meta-analytic estimate of precision is weight by inverse sampling variance
Describing Distribution• Central tendency• Spread• Shape
0
5
10
15
20
25
30
-1.6-1.4-1.2-1
-0.8-0.6-0.4-0.20
0.2 0.4 0.6 0.81
1.2 1.4 1.6 1.82
2.2 2.4
Effect size
Proportion total (%)
NonpartisanPartisan
Inferencial Statistics• Select a model
– Fixed effects– Random effects
• Univariate analyses– Analogous to one-way ANOVA– Examine how much variation in effect sizes is explained by one (categorical) variable
• Multivariate analyses– Analogous to multivariate regression– Examine how much variation in effect sizes is explained by set of (categorical or continuous) variables
– Examine how much unique variation in effect sizes is explained by one (categorical or continuous) variable
k d lowerupperQb Qw p k d lowerupperQbQw p k d lowerupperQb Qw pOverall2180.390.330.44 <.00011290.200.140.26 <.0001940.630.560.71 <.0001
Type of publication 44.91 <.0001 1.63 0.20 12.78 0.002Journal1260.340.270.42 8.78<.0001930.230.150.30 5.78<.0001330.610.480.74 8.96<.0001
Dissertation/Thesis510.150.030.28 2.410.02360.130.020.25 2.250.02150.23-0.040.49 1.690.09Other410.770.630.90 11.12<.0001 380.760.620.90 10.93<.0001
Type of meditation 9.26 0.01 1.18 0.55 5.94 0.01TM1460.390.310.46 9.69<.0001690.170.080.25 3.690.0002750.590.490.68 11.81<.0001
Yoga270.580.410.75 6.52<.0001150.250.080.41 2.910.004110.930.671.19 7.02<.0001Other450.230.080.37 3.080.002450.230.120.34 3.990.0001
Author affiliation 20.34 <.0001No1590.290.220.37 7.93<.0001Yes590.610.490.72 10.26<.0001
Control group/cond 6.02 0.05 3.38 0.18 5.64 0.0611260.400.320.49 9.28<.0001650.210.110.30 4.42<.0001580.600.490.71 10.46<.00012 380.510.350.67 6.15<.0001260.310.160.47 3.950.0001120.920.661.17 7.00<.00013 540.270.140.39 4.26<.0001380.140.030.24 2.450.01160.550.350.75 5.43<.0001
Type of effect 4.10 0.13 2.66 0.27 5.00 0.03State310.440.270.62 5.07<.0001190.16-0.010.33 1.800.07110.900.641.16 6.86<.0001Trait1790.380.310.46 10.60<.00011030.220.150.29 6.05<.0001740.590.490.69 11.48<.0001Both8 0.06-0.270.39 0.350.72 7 0.01-0.240.27 0.110.911 0.89
Broad cognitive ability 12.48 0.25 11.46 0.32 12.49 0.25General intelligence130.450.210.70 3.630.00035 0.12-0.180.43 0.780.448 0.620.360.88 4.63<.0001
Fluid intelligence190.350.140.57 3.210.001120.240.040.44 2.380.027 0.520.200.84 3.210.001Crystallized intelligence9 0.13-0.170.43 0.860.39 5 0.05-0.230.33 0.370.714 0.27-0.160.69 1.230.22
Memory and learning270.550.360.73 5.84<.0001170.320.150.49 3.650.00039 0.760.471.06 5.12<.0001Visual perception440.410.270.55 5.67<.0001260.180.040.32 2.450.01160.770.580.97 7.69<.0001
Auditory perception120.630.330.93 4.14<.00017 0.480.140.83 2.730.015 0.740.361.13 3.840.0001Retrieval/creativity170.320.100.54 2.830.005110.11-0.110.32 0.970.336 0.650.350.95 4.26<.0001
Cognitive speediness150.20-0.040.43 1.650.10100.14-0.060.35 1.360.175 0.34-0.030.71 1.800.07Processing speed270.400.220.58 4.36<.0001120.330.130.54 3.220.001150.480.270.70 4.41<.0001
Psychomotor200.310.090.53 2.780.01120.02-0.190.23 0.190.858 0.850.531.17 5.14<.0001Attention150.300.050.55 2.380.02120.210.0030.42 1.990.053 0.720.191.26 2.670.01
Full sample Nonpartisan sample Partisan sample95% CI 95% CI 95% CI
B SEB Q p B SEB Q p B SEB Q pType of publicationa 58.15<.0001 4.44 0.04 13.200.001
Dissertation/Thesis-0.240.0520.56<.0001-0.180.084.44 0.04-0.270.115.71 0.02
Other0.320.0537.59<.0001 0.230.087.49 0.01
Type of meditationb 2.78 0.25 0.04 0.98 3.83 0.05
Yoga0.100.062.39 0.12 0.010.070.02 0.89 0.360.193.83 0.05
Other-0.040.060.39 0.53-0.010.060.02 0.89
Author affiliation0.340.0724.35<.0001
Control Group/Cond-0.020.040.20 0.65 0.010.040.05 0.82-0.150.074.55 0.03
Type of effectc 1.40 0.50 3.66 0.16 0.33 0.85
State0.040.070.29 0.59 0.040.080.25 0.62 0.090.220.18 0.67
Both-0.110.101.11 0.29-0.180.103.41 0.06 0.160.400.16 0.69
Broad cognitive abilityd 14.690.14 14.000.17 23.830.008
General intelligence-0.010.100.01 0.92-0.120.140.71 0.40 0.120.130.94 0.33
Crystallized intelligence-0.130.121.19 0.28-0.070.130.28 0.60-0.240.191.58 0.21
Memory and learning0.190.085.79 0.02 0.140.092.51 0.11 0.170.131.65 0.20
Visual perception0.020.060.16 0.69-0.050.070.39 0.53 0.170.112.59 0.11
Auditory perception0.220.142.60 0.11 0.300.163.45 0.06-0.140.230.36 0.55
Retrieval/creativity-0.020.090.04 0.84-0.140.101.88 0.17 0.210.132.36 0.12
Cognitive speediness-0.130.101.79 0.18-0.030.100.06 0.80-0.420.175.98 0.01
Processing speed-0.030.080.20 0.65 0.140.101.84 0.17-0.040.100.17 0.68
Psychomotor-0.160.102.84 0.09-0.160.102.59 0.11-0.350.193.37 0.07
Attention0.030.110.07 0.79-0.060.110.28 0.60 0.530.244.83 0.03
Experimental designe 0.56 0.76 1.11 0.57 1.04 0.59
Between participants0.010.060.04 0.84 0.090.100.80 0.37 0.060.090.39 0.53
Within participants0.070.090.51 0.47-0.090.160.31 0.58 0.090.110.65 0.42
Random assignmentf -0.070.070.95 0.33-0.170.093.63 0.06 0.160.131.47 0.22
Age 0.000.030.0010.98 0.060.051.58 0.21 0.020.060.10 0.76
Meditative experience-0.050.041.81 0.18-0.060.042.04 0.15-0.100.081.81 0.18
Full sampleNonpartisan samplePartisan sample
Notes. aReference = Journals, bReference = TM, cReference = Trait, dReference = Fluid intelligence, eReference = Mixed, fFor tolerance reasons, the regression model that included random assignment did not include the "within-participants" effect-coded variable.