Multiple Comparisons & Factorial Designs
EdPsych 496
C.J. Anderson
Multiple Comparisons & Factorial Designs – p.1/38
Multiple Comparisons (part 2)
• When there is no interaction.. . . basically whatwe did with 1-Factor ANOVA• Example• Contrasts
• When there is an interaction.• Example• Contrasts
Multiple Comparisons & Factorial Designs – p.2/38
MC Procedures We’ll Cover
• Pairwise (simple):• Fisher LSD.• Tukey HSD.
• Simple or Complex:• Bonferroni• Scheffé
Multiple Comparisons & Factorial Designs – p.3/38
When There is NO Interaction
• Only need to perform multiple comparisonsfor main effects that were significant.
• Do not need to do them if a factor only has 2levels.
• Multiple comparison procedures are the sameas those used in 1-Factor ANOVA, exceptneed to make sure you’re using the correct n.
• To illustrate, need data with at least 1 maineffect and no interaction. . .
Multiple Comparisons & Factorial Designs – p.4/38
NELS88National Education Longitudinal Study:
• Conducted by National Center for Education Statisticsof the US department of Education. These dataconstitute the first in a series of longitudinalmeasurements of students starting in 8th grade. Thesedata were collected Spring 1988.
• We’ll just use data from a sample of N = 225 studentsfrom 23 schools.
• Response variable: math achievement test score.
• Factors: SES (Low, Middle, High) and Location ofschool (rural, urban, sub-urban).
Multiple Comparisons & Factorial Designs – p.5/38
NELS88: Mean Plots
Multiple Comparisons & Factorial Designs – p.6/38
NELS88: Summary StatisticsMeans and (standard deviations). njk = 25
Location
SES Rural Suburban Urban
Low 44.00 44.60 46.48 45.03
(9.36) (6.84) (9.32) (8.54)
Mid 46.52 49.64 51.20 49.45
(8.85) (10.71) (9.17) (9.60)
High 57.00 53.32 57.96 56.10
(9.17) (11.66) (8.86) (10.05)
49.51 49.19 51.88 50.19
(10.57) (10.46) (10.17) (10.42)Multiple Comparisons & Factorial Designs – p.7/38
NELS88: ANOVA Summary Table
Source df SS MS F p > F
SES 2 4653.90 2326.95 26.27 < .001
Urban 2 324.73 162.36 1.83 .16
SES*Urban 4 229.51 57.38 0.65 .63
Error 216 19130.78 88.57
Corrected Total 224 24338.78
• SES is significant.
• Urban and Interaction are not significant.
Multiple Comparisons & Factorial Designs – p.8/38
NELS88: Pairwise
• Since we have balanced design, we’ll useminimum significant differences.
•
Null Observed Differences
hypothesis between sample means Conclusion
µhigh = µmid 56.10 − 49.45 = 6.64
µmid = µlow 49.45 − 45.03 = 4.42
µhigh = µlow 56.10 − 45.03 = 11.07
Multiple Comparisons & Factorial Designs – p.9/38
NELS88: Fisher’s LSD• Controls type I error rate per comparison, α.
• Least significant difference is
LSD =(1−α/2) tνesy−y
• For α = .05 and our example, .975t216 = 1.971 and
sy−y =
√
MSe
(
1
nK+
1
nK
)
=
√
88.56778
(
2
25(3)
)
= 1.54
• So LSD = 1.971(1.54) = 3.03 −→ Reject all Ho’s.Multiple Comparisons & Factorial Designs – p.10/38
NELS88: Tukey’s HSD• Controls familywise type I error rate per, ασ — the
family being pairs of means for SES.
• Honest significant difference is HSD = qνe,ασ ,Jsy−y.
• For α = .05, νe = 216 and J = 3, q = 3.338.
• The standard error
sy−y =
√
MSe
2
(
1
nK+
1
nK
)
=
√
88.56778
2
(
2
25(3)
)
= 1.09
• So HSD = 3.338(1.087) = 3.63 −→ Reject all Ho’s.Multiple Comparisons & Factorial Designs – p.11/38
Complex Comparisons• For Complex (and simple) comparisons.• For a few planned comparisons −→
Bonferroni.• For post hoc and/or more than (J − 1) −→
Scheffé.• Consider the following two comparisons for
NELS data:
ψ1 = µlow + µmid − 2µhigh
ψ2 = µhigh + µmid − 2µlow
Multiple Comparisons & Factorial Designs – p.12/38
Complex Comparisons• The sums of squares for comparisons:
SSψ =nK(
∑Jj=1 cjYj·)
2
∑Jj=1 c2
j
=nK(ψ)2
∑Jj=1 c2
j
• For comparison 1:
ψ21 = (45.027 + 49.453 − 2(56.093))2 = 313.502
SSψ1=
25(3)(313.502)
12 + 12 + (−2)2=
25(3)(313.502)
6= 3, 919.07
• For comparison 2:
ψ22 = (−2(45.027) + 49.453 + 56.093)2 = 240.002
SSψ2=
25(3)(240.002)
6= 3, 000.03
Multiple Comparisons & Factorial Designs – p.13/38
Complex Comparisons• Test statistics:
Contrast df SS MS F
(low & middle) vs high 1 3,919.07 3919.07 44.25
low vs (middle & high) 1 3,000.03 3,000.03 33.87
Error 216 19,130.64 88.568
• For Bonferroni, αΣ = .05, (.975/2)t216 = 2.413, and for theF ratio, the critical value is (2.413)2 = 5.82
• For Scheffé, (J − 1)(1−αΣ)F(J−1),νe= 2(3.0376) = 6.08.
• For this NELS example, both comparisons aresignificant.
Multiple Comparisons & Factorial Designs – p.14/38
Complex Comparisons• For the complex comparison, we implicitly
tested (for example)Ho(1) : ψ1 = 0 vs Ha(1) : ψ1 < 0
• For directional alternatives, e.g.,Ha(1) : ψ1 < 0, use t-ratio,
t =ψ
sψ
=
∑
j cjYj·√
MSe
(
∑
j c2j/(nK)
)
• The critical value is the square root of theBonferroni or Scheffé (depending on whichyou’re doing).
Multiple Comparisons & Factorial Designs – p.15/38
Complex Comparisons• For our comparison 1 with Ha(1) : ψ1 6= 0
t =45.027 + 49.453 − 2(56.093)
√
88.568(
625(3)
)
=−17.705
2.662= −6.65
• Note: observed test statistic is
t2 = (−6.65)2 = 44.25.
• The (negative) square root of critical values
given previously.
• Bonferroni: tcrit = −2.41.
• Scheffé: −2.47Multiple Comparisons & Factorial Designs – p.16/38
When There is an Interaction
• The basic idea and methodology isessentially the same as multiple comparisonsfor main effects.
• Types of multiple comparisons:• Simple Effect• Mini-Factorials• Specific Cell Means
• We need another data set — a factorial withan interaction.. . .
Multiple Comparisons & Factorial Designs – p.17/38
When There is an InteractionModified hypothetical example from Tatsuoka & Lohnes(1988, p273-279):10 subjects were randomly assigned to one of twomethods (A and B) of teaching keyboarding to one of thefollowing 3 conditions:
• C1: 2 hours of instruction per day for 6 weeks.
• C2: 3 hours of instruction per day for 4 weeks.
• C3: 4 hours of instruction per day for 3 weeks.
At the end of the course, each of the 60 subjects was given
a standardized test measuring accuracy.
Multiple Comparisons & Factorial Designs – p.18/38
The DataThe “raw” data:
Method
Condition A B
C1 26, 22, 21, 23, 21, 25, 24, 29, 25, 26,
19, 25, 20, 21, 20 28, 25, 24, 24, 26
C2 17, 21, 17, 18, 23, 18, 19, 17, 17, 19,
19, 16, 19, 15, 14 19, 17, 21, 18, 20
C3 14, 14, 16, 16, 13, 11, 10, 10, 12, 11,
13, 12, 15, 14, 14 12, 14, 13, 12, 11
Multiple Comparisons & Factorial Designs – p.19/38
Summary Statistics
Condition
C1 C2 C3 Over Conditions
n11 = 10 n12 = 10 n13 = 10 n1• = 30
Method Y11 = 21.8 Y12 = 17.9 Y13 = 14.1 Y1· = 17.93
A s2
11= 5.07 s2
12= 7.43 s2
13= 1.66 s2
A= 14.62
n21 = 10 n22 = 10 n23 = 10 n2• = 30
Method Y21 = 25.6 Y22 = 18.5 Y23 = 11.6 Y2· = 18.57
B s2
21= 2.93 s2
22= 1.83 s2
23= 1.60 s2
B= 35.77
Over Methods n•1 = 20 n•2 = 20 n•2 = 20 n•• = 60
Y·1 = 23.7 Y·2 = 18.2 Y·3 = 12.85 Y·· = 18.25
s2
C1= 7.59 s2
C2= 4.48 s2
C3= 3.19 s2 = 24.87
Multiple Comparisons & Factorial Designs – p.20/38
Computing Sums of Squares• Total Sum of Squares: (Corrected for total mean)
SStot =J
∑
j=1
K∑
k=1
n∑
i=1
(Yijk − Y··)2 = (nJK − 1)s2
= (60 − 1)(24.87) = 1467.25
• Sum of Squares Within (error):
SSe =J
∑
j=1
K∑
k=1
n∑
i=1
(Yijk − Yjk)2 = (n − 1)
2∑
j=1
3∑
k=1
s2jk
= (10 − 1)(5.07 + 7.43 + 1.66 + 2.93 + 1.83 + 1.60)
= 184.70
Multiple Comparisons & Factorial Designs – p.21/38
Computing Sums of Squares• Sum of Squares Between Methods:
SSM = nK
J∑
j=1
(Yj· − Y··)2
= (10)(3)[(17.93 − 18.25)2 + (18.57 − 18.25)2]
= 6.02
• Sum of Squares Between Conditions:
SSC = nJK
∑
k=1
(Y·k − Y
··)2
= (10)(2)[(23.7 − 18.25)2 + (18.2 − 18.25)2 + (12.85 − 18.25)2]
= 1, 177.3
Multiple Comparisons & Factorial Designs – p.22/38
Computing Sums of Squares• Sum of Squares for Method × Condition Interaction:
SSMC = n
J∑
j=1
K∑
k=1
(Yjk − Yj· − Y·k + Y
··)2
= 10[(21.80 − 17.93 − 23.70 + 18.25)2 + . . .
+(11.60 − 18.57 − 12.85 + 18.25)2]
= 99.23
• or much easier
SSMC = SStot − SSM − SSC − SSe
= 1467.25 − 6.02 − 1177.30 − 184.70 = 99.23
Multiple Comparisons & Factorial Designs – p.23/38
ANOVA Summary Table
Source df SS MS F p–value
Method 1 6.02 6.02 1.76 .1903
Condition 2 1177.30 588.65 172.12 .0001
M × C Interaction 2 99.23 49.62 14.51 .0001
Within (error) 54 184.70 3.420
(corrected) total 59 1467.25
R2 =SSmodel
SStot
= (6.02 + 1177.30 + 99.23)/1467.25 = .8741
Multiple Comparisons & Factorial Designs – p.24/38
Let’s Take a Look. . .
Multiple Comparisons & Factorial Designs – p.25/38
MC of Simple Effects. . . by Example
Our design can be represented as
Practice
Conditions
C1 C2 C3
Teaching A
Method B
Multiple Comparisons & Factorial Designs – p.26/38
MC of Simple Effects (continued)
• To investigate simple effects of practice given teachingmethod A, test contrasts involving means in thesecells:
C1 C2 C3
A
• To investigate simple effects of practice given teachingmethod is B, test contrasts involving means in thesecells:
C1 C2 C3
B
Multiple Comparisons & Factorial Designs – p.27/38
Alternatively (or Additionally)
Investigate the simple effects of teaching method givenconditions are test comparisons of means from thefollowing cells:
C1 C2 C3
A A A
B B B
Ho : µAC1= µBC1
Ho : µAC2= µBC2
Ho : µAC3= µBC3
Multiple Comparisons & Factorial Designs – p.28/38
Mini-Factorials: (2 × 2) Sub-TablesOne possible way to break our 2 × 3 design into mini 2 × 2
factorials is as follows:
C1 C2 C2 C3
A A
B B
Sub-design on the left: Test whether there is an interactionfor teaching method and practice conditions C1 and C2 bytesting the hypothesis
Ho : (µAC1− µBC1
) = (µAC2− µBC2
)
or equivalently,
Ho : (µAC1− µAC2
) = (µBC1− µBC2
)
Recall that if there is no interaction, then “curves” shouldMultiple Comparisons & Factorial Designs – p.29/38
Mini-Factorials: (2 × 2) Sub-TablesGiven that pattern of the cell means. . .
C1 C3
A
B
Ho : (µAC1− µBC1
) = (µAC3− µBC3
)
or equivalently,
Ho : (µAC1− µAC3
) = (µBC1− µBC3
)
Recall that if there is no interaction, then “curves” shouldbe parallel; that is, the differences between pairs of meansshould be the same.
Multiple Comparisons & Factorial Designs – p.30/38
Mini-Factorials: (2 × 2) Sub-TablesWhat is the contrast for
Ho : (µAC1− µBC1
) = (µAC3− µBC3
)?
• Contrast
Ho : ψ = 1µAC1+ 0µAC2
− 1µAC3− 1µBC1
+ 1µBC3= 0
• Estimated value
ψ2 = (21.8 − 14.1 − 25.6 + 11.6)2 = (−6.3)2 = 39.69
• Needed for the sum of squares for the contrast∑
j
∑
k
c2jk/njk = 1/10 + 1/10 + 1/10 + 1/10 = .4
Multiple Comparisons & Factorial Designs – p.31/38
Mini-Factorials: (2 × 2) Sub-TablesTesting (continued)
Ho : (µAC1− µBC1
) = (µAC3− µBC3
)?
• Sum of squares,
SSψ =ψ2
∑
j
∑
k c2jk/njk
=39.69
.4= 99.225
• Test statistic,
F =SSψ/1
MSerror
=99.225
3.420= 29.01
• What should the critical value be?Multiple Comparisons & Factorial Designs – p.32/38
Mini-Factorials: (2 × 2) Sub-TablesTesting (continued)
• The critical value for our test. . .• Post hoc −→ Complex −→ Scheffé.• Scheffé critical value is (νAC)(1−αΣ)FνAC ,νerror
;that is,
(J − 1)(K − 1).95F(J−1)(K−1),N−JK
• Our example,Fcrit = 2(.95F4,54) = 2(2.54) = 5.08
• Since the test statistic F = 29.01, reject Ho.Multiple Comparisons & Factorial Designs – p.33/38
Testing Specific Means
• Contrasts and comparisons between any setof means that makes sense can beperformed.
• In our example, from the interaction plot itlooks like• µAC2
= µBC2
• µBC1> µAC1
• µAC3> µBC3
Multiple Comparisons & Factorial Designs – p.34/38
Testing Specific Means• Our example, our null and alternative
hypotheses would be• Ho(1) : µAC2
= µBC2versus Ha(1) : µAC2
6= µBC2
• Ho(2) : µBC1= µAC1
versus Ha(2) : µBC1> µAC1
• Ho(3) : µAC3= µBC3
versus Ha(3) : µAC3> µBC3
• We need t-ratios for Ho(2) and Ho(3) to test
directional hypotheses.
• For simplicity & a bit conservative, I’ll take the
alternatives to be non-directional.• Post hoc −→ Pairwise −→ Tukey HSD.
Multiple Comparisons & Factorial Designs – p.35/38
Testing Specific MeansResults for Tukey’s Studentized Range (HSD) Test for Y,αΣ = .05, νerror = 54, MSerror = 3.42, qcrit = 4.178 andMinimum significant difference = 2.44Means with the same letter are not significantly different.
Tukey Grouping Mean N group
A 25.6000 10 2 (B,C1)
B 21.8000 10 1 (A,C1)
C 18.5000 10 4 (B,C2)
C
C 17.9000 10 3 (A,C2)
D 14.1000 10 5 (A,C3)
E 11.6000 10 6 (B,C3)
Multiple Comparisons & Factorial Designs – p.36/38
Testing Specific Means• For Tukey, the HSD for a main effect
(marginal means), say factor A, use
HSDA = qα,J,N−JK
√
MSerror
nK
• For Tukey, the HSD for interaction(cells means) use
HSDAB = qα,JK,N−JK
√
MSerror
n
Multiple Comparisons & Factorial Designs – p.37/38
SAS & Testing Specific Means• SAS will compute pairwise tests for main effects by the
command
MEANS factorA / tukey;
• This doesn’t work for for cell means with factorialdesigns.
• Solutions:
• Re-run ANOVA but treat cells as a 1-Factor ANOVAand use the MEANS command.
• Run as factorial, but specify pairwise contrasts forall interaction effects using the CONSTRAST command.You need to then find the appropriate critical value.
Multiple Comparisons & Factorial Designs – p.38/38
Top Related