Moving Beyond the Average

Treatment Effect: A Look at Power Analyses for Moderator Effects in Cluster

Randomized Trials Jessaca Spybrook Western Michigan University December 3, 2015 (Joint work with Ben Kelcey, University of Cincinnati, Nianbo Dong, University of Missouri) *This work was funded by a grant (DGE-1437692) from the National Science Foundation.

Overview

n Background n  Purpose n  2-level CRT n  3-level CRT n  Implications

Slide 2

Background

n What works?

Slide 3

Background

n What works?

n Under what conditions?

Slide 4

Background

n What works?

n Under what conditions?

n For whom?

Slide 5

Purpose

n What works?

n Under what conditions?

n For whom?

Slide 6

Purpose

n Designs ¨ 2-level cluster randomized trial

(2-level CRT)

¨ 3-level cluster randomized trial (3-level CRT)

Slide 7

2-level CRT

n  Suppose a team of researchers are planning an impact study to determine the effectiveness of a new 5th grade science curriculum. They are planning a 2-level CRT with students nested within schools and schools are randomly assigned to the new curriculum or the current curriculum.

Slide 8

2-level CRT – what works?

n What works?

What is the effect of the new science curriculum relative to the current curriculum on science achievement for fifth graders?

Slide 9

2-level CRT – what works?

n Model

Slide 10

( )( )τσγγ

,0~

,0~

0

200100

NrNeerTy

j

ijijjjij +++=

Main effect of Treatment

2-level CRT – what works?

n Model

Set

Slide 11

( )( )τσγγ

,0~

,0~

0

200100

NrNeerTy

j

ijijjjij +++=

Main effect of Treatment

2σττ

ρ+

=2

01

στ

γδ

+=

2-level CRT – what works?

n Minimum detectable effect size (MDES) (Bloom, 1995)

MJ-2 is multiplier that approaches 2.8 as J increases, two-tail test with 0.05 level of significance J is total number of sites n is number of individuals per site is the intraclass correlation

Slide 12

( )( )J

nMMDES J/14

2ρρ −+

= −

ρ

2-level CRT – what works?

n To increase precision ¨ Common practice to include a cluster-level

covariate ¨ School-level pretest

n Model:

Slide 13

ijjjjij erWTy ++++= 0020100 γγγ

Main effect of Treatment

2-level CRT

Question Effect of Interest

MDES

What works? Main effect of

treatment

Main effect of treatment w/covariate

Slide 14

( )( )J

nM J/14

2ρρ −+

−

( )( )J

nRM W

J

/1)1(4 2|

3

ρρ −+−−

2-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

J=30 n=10 0.39 n=50 0.29 n=100 0.27 J=60 n=10 0.27 n=50 0.20 n=100 0.19

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,

Slide 15 .75.0,23.0 2

| == WRρ

2-level CRT – under what conditions?

n Under what conditions?

Is there a difference in the mean effects by school type (urban vs. suburban)?

Slide 16

2-level CRT – under what conditions?

n Model:

Slide 17

ijjjjjjjij erWSTSTy ++++++= 00403020100 γγγγγ

Cluster level moderator effect

2-level CRT

Question Effect of Interest

MDES MDESD

What works? Main effect of

treatment

Main effect of treatment w/covariate

Under what conditions?

Cluster-level moderator

Slide 18

( )( )J

nM J/14

2ρρ −+

−

( )( )J

nRM W

J

/1)1(4 2|

3

ρρ −+−−

( )( )J

nRM SW

J

/1)1(16 2|

5

ρρ −+−−

2-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

Under what conditions?

CL Mod

J=30 n=10 0.39 0.75

n=50 0.29 0.53

n=100 0.27 0.49

J=60 n=10 0.27 0.52

n=50 0.20 0.37

n=100 0.19 0.34

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,

Slide 19 .80.0,75.0,23.0 2

|2| === SWW RRρ

2-level CRT – for whom?

n For whom?

Is there a difference in the mean effect for boys and girls?

Slide 20

2-level CRT – for whom?

n Model:

Slide 21

ijjijjijjij erXTXTy +++++= 011100100 γγγγ

Individual level moderator effect

2-level CRT

Question Effect of Interest

MDES MDESD

What works? Main effect of

treatment

Main effect of treatment w/covariate

Under what conditions?

Cluster-level moderator

For whom? Individual-level moderator

Slide 22

( )( )J

nM J/14

2ρρ −+

−

( )( )J

nRM W

J

/1)1(4 2|

3

ρρ −+−−

( )( )J

nRM SW

J

/1)1(16 2|

5

ρρ −+−−

( )( )J

nRM X

JJn

/1)1(16 2|

)2(*

ρ−−−−

2-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

Under what conditions?

CL Mod

For whom?

Ind Mod

J=30 n=10 0.39 0.75 0.51

n=50 0.29 0.53 0.23

n=100 0.27 0.49 0.16

J=60 n=10 0.27 0.52 0.36

n=50 0.20 0.37 0.16

n=100 0.19 0.34 0.11

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80,

Slide 23 .20.0,80.0,75.0,23.0 2

|2|

2| ==== XSWW RandRRρ

3-level CRT

n  Suppose a team of researchers are planning an impact study to determine the effectiveness of a school-wide science curriculum. They are planning a 3-level CRT with students nested within teachers nested within schools and schools are randomly assigned to the new curriculum or the current curriculum.

Slide 24

3-level CRT – what works?

n What works?

What is the effect of the new science curriculum relative to the current curriculum on science achievement for all students?

Slide 25

3-level CRT – under what conditions?

n Under what conditions?

Is there a difference in the mean effects by school type (urban vs. suburban)? Is there a difference in the mean effects by teacher experience level (0-3 years vs. more than 3 years)?

Slide 26

3-level CRT – for whom?

n For whom?

Is there a difference in the mean effect for boys and girls?

Slide 27

3-level CRT

Question Effect of Interest

MDES

What works?

Main effect of treatment

Slide 28

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

3-level CRT

Question Effect of Interest

MDES MDESD

What works?

Main effect of treatment

Under what conditions?

Cluster-level moderator

Slide 29

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLLSW

K

//1116 23232|

5

ρρρρ −−++−−

3-level CRT

Question Effect of Interest

MDES MDESD

What works?

Main effect of treatment

Under what conditions?

Cluster-level moderator

Under what conditions?

Teacher-level moderator

Note: u=J*K-(K-2) Slide 30

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLLSW

K

//1116 23232|

5

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLM

u

//1116 2322| ρρρ −−+−

3-level CRT

Question Effect of Interest

MDES MDESD

What works?

Main effect of treatment

Under what conditions?

Cluster-level moderator

Under what conditions?

Teacher-level moderator

For whom? Individual-level moderator

Note: u=J*K-(K-2), v=n*J*K-(J*K)-(K-2) Slide 31

( ) ( )[ ]{ }K

JnRM LLLLW

K

//114 23232|

3

ρρρρ −−++−−

( ) ( )[ ]{ }K

JnRM LLLM

u

//1116 2322| ρρρ −−+−

( )( )[ ]{ }K

JnRM LLX

v

//1116 232| ρρ −−−

( ) ( )[ ]{ }K

JnRM LLLLSW

K

//1116 23232|

5

ρρρρ −−++−−

3-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

J=5 n=10 0.24 n=30 0.22 J=30 n=10 0.19 n=30 0.18

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,

Slide 32 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|

2|23 ====== XPSWLL RandRRRρρ

3-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

Under what conditions?

CL Mod

J=5 n=10 0.24 0.45

n=30 0.22 0.41

J=30 n=10 0.19 0.34

n=30 0.18 0.33

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,

Slide 33 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|

2|23 ====== XPSWLL RandRRRρρ

3-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

Under what conditions?

CL Mod

Under what conditions?

TL Mod

J=5 n=10 0.24 0.45 0.29

n=30 0.22 0.41 0.23

J=30 n=10 0.19 0.34 0.12

n=30 0.18 0.33 0.09

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,

Slide 34 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|

2|23 ====== XPSWLL RandRRRρρ

3-level CRT – Minimum Detectable Effect Size

What Works?

Main Effect

Under what conditions?

CL Mod

Under what conditions?

TL Mod

For whom?

Ind Mod

J=5 n=10 0.24 0.45 0.29 0.18

n=30 0.22 0.41 0.23 0.11

J=30 n=10 0.19 0.34 0.12 0.08

n=30 0.18 0.33 0.09 0.04

Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools,

Slide 35 .20.0,30.0,80.0,75.0,08.0,15.0 2|

2|

2|

2|23 ====== XPSWLL RandRRRρρ

Implications

n Under what conditions? ¨ Cluster level moderator

n  Challenging given current study sizes n  Effectiveness studies rather than efficacy studies n  If priority, need to consider in design stage

60 60 60

30 S 30 U

15C 15C 15 T 15 T

30 T 30 C

40S 20U

20C 10C 10T 20T

30 T 30 C

40S 20U

30C 0C 20T 10T

30 T 30 C

Implications

n Under what conditions? ¨ Teacher level moderator

n  Number of teachers per school matters n  If priority, need to consider in design phase

Slide 37

Implications

n For whom? ¨ Individual level moderator

n  In many cases, reasonable given current study sizes n  If priority, need to consider in design phase

Slide 38

Implications

n Extensions ¨ Multisite studies ¨ R code available – jessaca.spybrook@wmich.edu ¨ Some parts implemented in PowerUP! -

http://web.missouri.edu/~dongn/ ¨ Add to Optimal Design Plus

Slide 39

Thank You!

Questions? Please email me: jessaca.spybrook@wmich.edu

Slide 40