1

Wim Van den Noortgate Katholieke Universiteit Leuven, Belgium

Belgian Campbell GroupWim.VandenNoortgate@kuleuven-kortrijk.be

Workshop systematic reviews Leuven June 4-6, 2012

Advanced topics in meta-analysis

2

1. Modelling heterogeneity2. Publication bias

Content

3

1. Modelling heterogeneity

4

Growing popularity of evidence-based thinking:

Decisions in practice and policy should be based on scientific research about the effects of these decisions/interventions

But: conflicting results (failures to replicate), especially in social sciences!

5

1. The role of chance- in measuring variables- in sampling study participants

2. Study results may be systematically biased due to

- the way variables are measured- the way the study is set up

3. Studies differ from each other (e.g., in the kind of treatment, the duration of treatment, the dependent variable, the characteristics of the investigated population, …)

Explanation for failures to replicate?

6

Differences between observed effect sizes due to chance only

Population effect sizes all equal

Fixed effects model

1 2( ... )k

7

0 :H

Assessing heterogeneity

2

1

ˆ( )k

j jj

Q w g

2( 1)k

1 2 ... k :aH at least one differs from an otherj

0 : ~H Q

8

Measuring heterogeneity( )² *100%Q dfIQ

Rough guidelines:0% to 40%: might not be important 30% to 60%: may represent moderate heterogeneity 50% to 90%: may represent substantial heterogeneity 75% to 100%: considerable heterogeneity

Interpretation based on both I² and heterogeneity test!

= percentage of variability in effect estimates due to heterogeneity rather than chance

9

An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)

StudyWeeks prior

contact gj

1.2.3.4.5.6.7.8.9.

10.11.12.13.14.15.16.17.18.19.

Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)

2330033301001233123

0.030.12

-0.141.180.26

-0.06-0.02-0.320.270.800.540.18

-0.020.23

-0.18-0.060.300.07

-0.07

0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17

( )jg

10Q = 35,83, df = 18, I²= 50 %, p = .007

11

Not always wise: make set of studies more homogeneous!

Can help to say something about ‘fruit’ Can help to make detailed conclusions:

Does the effect depend on the kind of fruit?

Mixing apples and oranges?

12

13

Population effect size possibly depends on study category

Differences between observed effect sizes within the same category due to chance only

Fixed effects model with categorical moderator variable

14

An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)

StudyWeeks prior

contact gj

1.2.3.4.5.6.7.8.9.

10.11.12.13.14.15.16.17.18.19.

Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)

2330033301001233123

0.030.12

-0.141.180.26

-0.06-0.02-0.320.270.800.540.18

-0.020.23

-0.18-0.060.300.07

-0.07

0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17

( )jg

15

16

Total variabilityin observed ES’s

A weighted ANOVA

T B WQ Q Q

Variability within groups

Variabilitybetween groups= +

H0: QT ~²k-1

H0: QB ~²J-1

H0: QW ~²k-J

2

1

ˆ( )k

j jj

Q w g

QT : homogeneity test

QB : moderator test

QW : test for within group homogeneity

17

Q total = Q Between + Q within

² 35.83 20.38 15.45

df 18 3 15

p 0.007 0.0001 0.42

18

A second example(using a sorted caterpillar plot)

-1.5

-0.5

0.5

1.5

2.5

3.5

4.5

5.5

6.5

Observed effect sizes for the 3 tasks

Semantic categorization Lexical decision Naming

ES

= Mean ES REM

19

Population effect size possibly depends on continuous study characteristic

e.g.,

After taking into account this study characteristic, differences between observed effect sizes due to chance only

Fixed effects model with continuous moderator variable

0 1 1 ... j j p pjx x

20

Initial effect is moderate (0.41, p < .001), but decreases with increasing prior contact (with -0.16 per week, p <.001)

Conclusions for Raudenbush (1984):

21

Population effect size possibly varies randomly over studies

Differences between observed effect sizes are due to- chance- ‘true’ differences

Random effects model

22

23

Population effect size possibly depends on study category

Differences between observed effect sizes within the same category are due to- chance- ‘true’ differences

Random effects model with categorical moderator variable

24

Population effect size possibly depends on continuous study characteristic

e.g.,

After taking into account this study characteristics, differences between observed effect sizes are due to- chance- ‘true’ differences

Random effects model with continuous moderator variable

0 1 1 ... j j p pj jx x u

25

Random effects model with moderators:

◦ The least restrictive model: allows moderator variables & random variation

◦ Also called a ‘Mixed effects model’

26

FEM REM

Without moderator

Categorical moderator

Continuous moderator

27

1. Is there an overall effect?2. How large is this effect?3. Is the effect the same in all studies?4. How large is the variation over studies?5. Is this variation related to study

characteristics?6. Is there variation that remains unexplained?7. What is the effect in the specific studies?

Basic meta-analytic questions

28

An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)

StudyWeeks prior

contact gj

1.2.3.4.5.6.7.8.9.

10.11.12.13.14.15.16.17.18.19.

Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)

2330033301001233123

0.030.12

-0.141.180.26

-0.06-0.02-0.320.270.800.540.18

-0.020.23

-0.18-0.060.300.07

-0.07

0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17

( )jg

29

Parameter REM

Fixed

Intercept 0.084 (0.052)

Between study variance 0.019 (0.023)

0

2u

30

Parameter REM MEM

Fixed

Intercept 0.084 (0.052) 0.41 (0.087)

Weeks -0.16 (0.036)

Between study variance 0.019 (0.023) 0.00 (-)

0

1

2u

31

1. Models can include multiple moderators2. REM assumes randomly sampled studies3. REM requires enough studies4. Association (over studies) ≠ causation!

Be aware of potential confounding moderators (studies are not ‘RCT participants’!)

Remarks

Dependencies between studies◦ E.g., research group, country, …

Multiple effect sizes per study◦ Several samples◦ Same sample but, e.g., several indicator variables

32

Note: Sources of dependencies

Ignoring dependence? NO! Avoiding dependence

◦ (Randomly choosing one ES for each study)◦ Averaging ES’s within a study◦ Performing separate meta-analyses for each kind

of treatment or indicator Modelling dependence

◦ Performing a multivariate meta-analysis, accounting for sampling covariance.

◦ Performing a three level analysis

33

How to account for dependence?

34

2. Publication bias

35

Example 1: advanced ovarian cancer: monotherapy alkylating agent vs. combination chemotherapy? International Cancer Research Data Bank

(Egger, M. D., & Smith, G. (1998). Meta-analysis. Bias in location and selection of studies. British Medical Journal, 316, 61-66. http://www.bmj.com/cgi/content/full/316/7124/61).

36

Proportion of publication within 5 years after conference:

81 % (of 233 trials) for significant results 68 % (of 287 trials) for nonsignificant results

Example 2: 510 large trials presented at conferences of the American Society of Clinical Oncology (ASCO)

(Kryzanowska, M. K., Pintilie, M., & Tennock, I. F. (2003). Factors associated with failure to publish large randomized trials presented at an oncology meeting. Journal of the American Medical Association, 290, 495-501).

37

-1 -0.5 0 0.5 1 1.50

100

200

300

400

500

Observed effect sizes

Sam

ple

size

The funnel plot

38

-1 -0.5 0 0.5 1 1.50

100

200

300

400

500

Observed effect sizes

Sam

ple

size

Example: Raudenbush, 1984

39

Þ Thorough search for all relevant published and unpublished study results

a) Articlesb) Booksc) Conference papersd) Dissertationse) (Un)finished research reportsf) …

40

- outliers- detection using graphs (or tests)- conduct analysis with and without outliers

- calculation effect sizes : several analyses- publication bias: analysis with and without

unpublished results- design & quality: compare results from studies

with strong design or good quality, with those of all studies

- researcher: literature search, effect size calculation, coding quality, …, done by two researchers

- …

Note: sensitivity analysis: how robust are conclusions?

41

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131

-1.5

-0.5

0.5

1.5

2.5

3.5

4.5

5.5

6.5

Observed effect sizes

Experiment

ES

Outliers

42

Spreadsheets (e.g., MS Excel, …) Some general statistical software (note:

often not possible to fix the sampling variance)SAS Proc Mixed, Splus, R Metafor package, …

Software for meta-analysis (note: often not MEM; often only one moderator!)CMA (http://www.meta-analysis.com/), RevMan, …

Software for multilevel/mixed modelsHLM, MLwiN, …

Software for MA

43

Software Excel SAS R CMA RevManCalculation of effect sizes X X √ √√ XNumber of moderators X ∞ ∞ 1 1 (cat.)Funnel X X √ √ √Trim & Fill X X √ √ XForest X X √ √ √Max. nr of levels 2 ∞ 2 2 2Flexibility √ √√ √√ X XPrice Expensive Free Expensive Free

(but student (but limitedversion) trial vers.)

Complexity √ X X √√ √√

44

Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.) (2009). The handbook of research synthesis and meta-analysis. New York: The Russell Sage Foundation.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-

analysis. Thousand Oaks, CA: Sage.

Van den Noortgate, W., & Onghena, P. (2005). Meta-analysis. In B. S. Everittt, & D. C. Howell (Eds), Encyclopedia of Statistics in Behavioral Science (Vol. 3 pp. 1206-1217). Chichester, UK: John Wiley & Sons.

Recommended literature:

45

Site of David Wilsonhttp://mason.gmu.edu/~dwilsonb/ma.html

Site of William Shadishfaculty.ucmerced.edu/wshadish/

Recommended sites