Methods for Finding for Whom

an Intervention is Effective and under what Circumstance

Bengt Muthén

AMERICAN INSTITUTES FOR RESEARCH

AIR’s Center for Integrating Education and Prevention Research in Schools

2

Overview

Intervention effects: variation in impact, interactions

Longitudinal data: more than two time points

Hierarchical data: individuals within groups

Muthen: Analyses

3

Example 1: Baltimore reading treatment-baseline interaction

Sprin

g of

Gra

de 1

Rea

ding

Ach

ieve

men

t

Fall of Grade 1 Reading Achievement (Normal Curve Equivalent)

*Source: Ialongo LN, Werthamer S, Kellam SK, Brown CH, Wang S, Lin Y (1999). Proximal Impact of Two First Grade Preventive Interventions on the Early Risk Behaviors for Later Substance Abuse, Depression and Antisocial Behavior. American Journal of Community Psychology, 27, Vol, 5, 599-641.

Muthen: Analyses

4

Example 1B: Baltimore aggressiontreatment-baseline interaction

*Source: Khoo, S.T. (2001). Assessing program effects in the presence of treatment-baseline interactions: A latent curve approach. Psychological Methods, 6, 234-257.

Α

γ

γ

ρ

ε

σ

σ

ι

ο

ν

ι

ν

Γ

ρ

α

δ

ε

5

1 2 3 4 5

1

2

3

4

5

6

control treatment

1 2 3 4 5

1

2

3

4

5

6

control treatment

Baseline is EstimatedGrowth Model Initial Status

Baseline is Observed ScoreFall of Grade 1

Agg

ress

ion

in F

ifth

Gra

de

Aggression in Fall of Grade 1 (Baseline) Aggression in Fall of Grade 1 (Baseline)

Muthen: Analyses

5

Weaknesses of pretest-posttest analysis (GAM – Generalized Additive Modeling

ANCOVA – Analysis of Covariance)

Posttest at a single time point does not show the initiation and duration of impact

Pretest is a fallible baseline measure due to time-specific variation and measurement error

Avoid weaknesses by collecting longitudinal data(more than 2 time points) and using growth modeling

Muthen: Analyses

6

Example 1B: Aggression development control and treatment groups

Grades 1-7

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Grades 1-7

Control Group Treatment Group

Muthen: Analyses

7

Example 1B: Aggression developmentcontrol group sorted by trajectories

High Aggressive, Control Group

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Medium Aggressive, Control Group

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Late Starter, Control Group

Grades 1-7

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Low Aggressive, Control Group

Grades 1-7

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Muthen: Analyses

8

Example 1B: Aggression development trajectory classes for control and intervention groups

High Aggressive, Control Group

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

High Aggressive, Intervention Group

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Late Starter, Control Group

Grades 1-7

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Late Starter, Intervention Group

Grades 1-7

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Muthen: Analyses

9

Example 1B: Aggression development trajectory classes for control and intervention groups

Medium Aggressive, Control Group

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Medium Aggressive, Intervention Group

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Low Aggressive, Control Group

Grades 1-7

TOC

A-R

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Low Aggressive, Intervention Group

Grades 1-7

12

34

56

1F 1S 2F 2S 3S 4S 5S 6S 7S

Muthen: Analyses

10

Example 1B: Aggression development trajectory classes for control and intervention groups

1F 1S 2F 2S 3S 4S 5S 6S 7S

Grades 1-7

12

34

56

TOC

A-R

High Aggressive, 15%Medium Aggressive, 44%Late Starter, 22%Low Aggressive, 19%ControlIntervention

BIC=3394Entropy=0.80

Muthen: Analyses

11

Example 1B: Aggression developmenttrajectory classes for

control and intervention groups

1F 1S 2F 2S 3S 4S 5S 6S 7S

Grades 1-7

12

34

56

TOC

A-R

High Aggressive, 15%Medium Aggressive, 44%Late Starter, 22%Low Aggressive, 19%ControlIntervention

BIC=3394Entropy=0.80 Juvenile Court Record

Class Control Intervntn

High 80% 52%

Medium 33% 40%

LS 49% 20%

Low 25% 10%

Muthen: Analyses

12

Example 2: LSAY math achievement inSeventh through Tenth Grade and high school dropout

Dropouts

Grades 7-10 (20% Sample)

7 8 9 10

4060

8010

0

Mat

h A

chie

vem

ent

All Students

Grades 7-10 (5% Sample)

7 8 9 10

4060

8010

0

Muthen: Analyses

13

Example 2: LSAY math achievement trajectory classes predicting

high school dropout

Mat

h A

chie

vem

ent

Poor Development: 20% Moderate Development: 28% Good Development: 52%

69% 8% 1%Dropout:

7 8 9 10

4060

8010

0

Grades 7-107 8 9 10

4060

8010

0

Grades 7-107 8 9 10

4060

8010

0

Grades 7-10

Muthen: Analyses

14

Example 3: The New York School Choice Study*

Setting: Lottery for 20,000 applicants from low-income families attending First through Fifth Grade in NY public schools

Treatment: $1,400 dollar annual scholarships for 3 years in private schools

Sample: 1,960 families (controls and treatment; balanced)

Design: Propensity matched pairs design and randomized block

Design variable: low/high applicant school (test scores below/above city-wide median)

Measures: Spring 1987 pretest, Spring 1988 posttest; reading and math (ITBS)

_____________________________________________________________________________________________________ *Source: Barnard, J., Frangakis, C.E., Hill, J., Rubin, D.B. (in press). A principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Forthcoming in J. of the Am. Stat. Assoc.

Muthen: Analyses

15

Example 3 Continued: The New York School Choice Study

Complications

Adherence classes: 20-25% of those who won declined scholarship, 6-10% of those who did not win sent their children to private schools nevertheless

Missing data: on covariates and on posttest as a function of adherence classes

Results

Effect of private school attendance (CACE): 5 percentile points for math in low schools

Effect of winning the lottery (ITT): 3 percentile points for math in low schools

Muthen: Analyses

16

New analytical tools

Growth mixture modeling – see Slides 165, 166

Multilevel modeling – see Slides 167, 168

Missing data modeling – see Slide 169

Adherence class modeling – see Slide 170

Structural equation modeling – see Slide 171

Software and Literature – see Slide 172

Muthen: Analyses

17

Growth and growth mixture modeling

Captures intervention impact on trajectories in an efficient and flexible way

Captures intervention effects that vary across individuals

Muthen: Analyses

18

Weaknesses of pretest-posttest ANCOVA as compared to growth mixture modeling

02

46

810

time1 time2

TxCtrl

TxCtrl

TxCtrl

High Class

Middle Class

Low Class

Out

com

e

High ClassTx

02

46

810

Ctrl

Middle ClassTx

Ctrl

Low ClassTx

Ctrl

ANCOVA

Tx

Ctrl

Pos

ttest

Pretest

Muthen: Analyses

19

Multilevel modeling

Children in different classrooms (different teachers) and schools may benefit differently from an intervention (Aggression, LSAY, New York examples)

Individual-level relationships can vary across classrooms/schools

Variation can be explained by classroom- and school-level variables

Muthen: Analyses

20

Cross-level interactions

Adobe Systems

Adobe Systems

Muthen: Analyses

21

Missing data modeling

Attrition in longitudinal studies

Designed selection of children into treatment: cross-sectional and longitudinal screens

Muthen: Analyses

22

Latent adherence class modeling(CACE Analysis)

Adherers and non-adherers are often quite different

Latent class modeling where adherence is observed in intervention group and unobserved in control group

Muthen: Analyses

23

Structural equation modelinglatent variable modeling

Mediational modeling – for example, “path analysis”in a pretest-posttest design where intervention effect on outcome is mediated by implementation

General latent variable modeling – for example, longitudinal analysis where class size influences achievement development, which influences high school dropout (Tennessee STAR study)

Muthen: Analyses

24

Software and literature

Multilevel modeling including growth modeling (with missing data): GLLAMM, HLM, MIXOR, MLwiN, Mplus, SAS PROC MIXED

Growth mixture modeling: Mplus, SAS PROC TRAJ

Latent (adherence) class (CACE) modeling: Mplus

Structural equation modeling: Amos, EQS, LISREL, Mplus, Mx

Latent variable modeling: Mplus

Mplus-related references can be downloaded from www.statmodel.com (see home page, References, Randomized Trials)

Overview in Muthén (2002) Behaviormetrika

Muthen: AnalysesMuthen: Analyses

25

Key Points

Collect rich pre-intervention information to enable thorough investigation of treatment-baseline interactions

Collect longitudinal data at more than one post-intervention time point to enable investigation of intervention impact on trajectories

Use growth mixture modeling and multilevel modeling to find variation in impact

Muthen: Analyses