DOCUMENT RESUME

ED 064 343 TM 001 508

AUTHOR Draper, John For Jr.TITLE A Monte Carlo Investigation of the Analysis of

Variance Applied to Non-Independfunt BernoulliVariates.

PUB DATE Apr 72NOTE 40p.; Paper presented at the annual meeting of the

American Educational Research Association (Chicago,Illinois, April 1972)

EDRS PRICE MF-$0.65 HC-$3.29DESCRIPTORS *Analysis of Variance; *Factor Analysis;

*Mathematical Models; Research Methodology;*Statistical studies; *Tables (Data)

ABSTRACTThe applicability of the Analysis of Variance, ANOVA,

procedures to the analysis of dichotomous repeated measure data isdescribed. The design models for which data were simulated in thisinvestigation were chosen to represent simple cases of twoexperimental situations; situation one, in which subjects' responsesto a single randomly selected set of three stimuli or items wereevaluated dichotomously on four successive occasions, and situationtwo, in which subjects' responses ar3 evaluated dichotomously on foursuccessive occasions with respect to four separate randomly selectedset of three stimuli or items. The two experimental situations arerepresented by "three way factorial" designs with random factors forsubjects ani items and a fixed factor for occasions; the first withsubjects, items and occasions completely crossed and the second withitems and occasions crossed with subjects, but with items nestedwithin occasions. Results of the study are presented in 11 tables and7 figures. (Author/DB)

U.S. DEPARTMENT OF HEALTH.EDUCATION & WELFAREOFFICE OF EDUCATION

THIS DOCUMEAT HAS BEEN REPRO-DUCED EXACTLY AS RECEIVED FROMTHE PERSON OR ORGANIZATION ORIG-INATING IT POINTS OF VIEW OR OPIN-IONS STATED DO NOT NECESSARILYREPRESENT OFFICIAL OFFICE OF EDU-CATION POSITION OR POLICY.

A MONTE CARLO INVESTIGATION OF THE ANALYSIS OFVARIANCE APPLIED TO NON-INDEPENDENT BERNOULLI VARIATES

By

John F. Draper, Jr.CTB/McGraw-Hi11.

Paper presented at the annualmeeting of the American Educational Research Association

April, 1972

The problem with which this paper will be concerned is the applicability

of the Analysis of Variance, ANOVA, procedures to the analysis of dichotomous

repeated measure data. Recall, that in the employment of ANOVA procedures it

is assumed that the data can be reasonably modeled by a vector variate with

multi-variate normal density and diagonal or patterned "compound symmetric"

covariance matrix, that is with equal variances and equal covariances in

diagonal blocks for subjects luld zeros elsewhere, (Greenhouse dad Geisser,

1959). Can these familiar procedures still be employed when the data can

more reasonably be modeled by a vector variate with non-independent Bernoulli

variates as elements? Other investigators, Mandeville (1970) and Seegar"and

Gabrielson (1968) have employed Monte Carlo methods in order to achieve a

partial answer to the above question. Their results tend toward a cautious

but affirmative answer with respect to the design models and model parameter

configurations with which their simulations were concerned. The results re-

ported in this paper are less sanguine. That is, however, not.ta say that

the results reported in this paper are in disagreement with those of previous

investigatOrs, since the design models and model parameter configurations

with which this paper is concerned are somewhat different. Bradley (1968)

has indicated that the robustness of a parametric statistical test is idio-

syncratic rather than general. For example, the F-test has bean shown to be-

robust to heteroscedasiticity for balanced designs (equal cell n's), but

generally not rdbust to heteroscedasiticity for unbalanced Llesigns.

The design models for which datA were simulated in this investigation

were dhosen to represent simple cases of two experimental situations, situation

one, in which subjects responses to a single randomly selected set of three

stimuli or items were evaluated dichotomously on four successive occasions

and situation two, in which subjects' responses are evaluated dichotomously

on four successive occasions with respect to four separate randomly selected

sets of three stimuli or items. Table 1 contrasts these two experimental

situations.

Table 1. Contrast of experimental Situation 1 with Experimental Situation 2.

Situation 1

Occasion 01

02 03 04

Item 11 1213

111213 11 12 1 3

11 1213

Situation 2

Occasion 01

02 03 04

Item 111213 1415 16 17 18 19 1

10111

112

In terms of experimental design the two experimental situations can be represented

by "three way factorial" designs with random factors for subjects and items1

and a fixed factor for occasions; the first with subjects, items an4 occasions

completely crossed and the second with items and pccasions crossed with subjects,

but with items nested within occasions.

IMFIN.1Items were considered to represent a random source of variation, sinceexperimenters often wish to generalize their results to some domain ofitems or stimuli, rather than to restrict their findings to only thoseitems they actually employed.

3

These two experimental situations were considered to be of interest because

experimenters often form occasion measures for subjects as the sum of dichotomous

response evaluations to a series of items and then ignore items as a factor in

their experimental design. And because it may be shown, (Draper, 1971), that

for such experimental situations, there is a source of variation associated

with items2 which if nonrnull will be confounded-with the source of variation

for occasions. A situation which holds whether or not, items are included as

a factor in the experimental design and an ANOVA consistent with the design

is employed for analysis and which also holds whether the dependent measures,

or item scores, are dichotomous or have distributions which may be reasonably

approximated by a normal probability density.

For situation one an examination of expected mean squares, suggested

possible variance ratio tests for sources of variation due to subjects, occasions,

items, subjects by occasions interaction, and items by occasions interaction.

Further, since the items by occasions interaction could be nonnull and would

thus be confounded with occasions, a QuasiF variance ratio test for occasions

(Satterthwaite, 1941) suggested itself. Similarly for situation two possible

variance ratio tests were suggested for the sources of variation, subjects,

occasions, items, and subjects.by occasions interaction, where both a "regular"

variance ratio test and a "QuasiF" test were suggested to test for occasions

effects. Calculations with respect to all the suggested tests were made on

simulated data, but of major interest were those tests for occasions and the

subjects by occasions interaction.

2An item by occasion interaction effect in situation one or a main effectdue to items in situation VATO.

It was considered of interest to vary the following model parameters over

examples of the experimental situations to be investigated.

1) The base probability of one3, on item difficulty, in the data,

2) The number of subjects,

3) The degree of relative heterogenity of the subjects4,

4) The effect of items4,

5) The effect of occasions4,

and

6) The effect of subject by occasion interaction4.

Three different values of the base probability of a one, .5, .2, and .1, suggested

by results due to Lunney (1969), were investigated as were five values of the

number of subjects 4, 6, 8, 10, and 12. The four levels of relative subject

heterogenity, Levels 1, 2, 3, and 4, which were employed, correspond to four

ratios of subject to error variance, 2:8, 3:7, 4:6 and 5:5 respectively. Com-

binations of these parameter values in combination with the values for effects

due items, occasions and subject by occasion interaction resulted in 720 different

model parameter configurations for data were simulated. The data for

the model parameter configurations were gimulated as pseudo-random numbers

with distributions which could reasonably be approximated by a normal proba-

bility density function and all ANOVA statistics calculated, then the data

were dichotomized and the ANOVA calculations repeated. Determinations were

then made as to which statistics fell to various a rejection regions and counters

which had been initialized at zero were incremented by one where appropriate.

3The base probability of a one will refer to the general probability of aone in the absence of possible main and interaction effects.

4Whether null or non-null.

5

This process was repeated 999 times for each parameter configuration for a

total of 1,000 simulations.

As an example of the type of analysis which was done on the frequency

data obtained from the simulation runs consider the following. Table 2 contains

frequencies in a = .05, .025 and .01 rejection regions of variance ratio tests

of occasion effects under conditions in which the occasion effects were null,

items were crossed with occasions and there was no interaction between subjects

and occasions. Table 2 is laid out as a six-dimensional array with three left

margins and three upper margins. The left-most margin indicates the number of

subjects employed with respect to a given simulation of data. Proceeding from

left to right the next margin indicates the nature of the items, whether null

in effect or non-null, and the right-most indicates the level of subject

heterogeneity. The upper margins from top to bottom indicate: (1) the probability

of a one with respect to a given simulation df data, (2) one thousand times

nominal a or the expected frequency given a true F-variate, and (3) an indication

of whether the 1,000 variance ratios with respect to a given cell of the table

were calculated from the dependent variables when they were variates with

approximate normal density (N) or from the subsequently dichotomized "normals"

(D).

In order to test for trends in the data reported in tables, the margins

were considered as fixed sources of variaeon and a multivariate analysis of

variance, MANOVA, was performed on the frequency data three-tuples5 within the

table, employing the highest order interaction mean products as an estimate of

error under the assumption that the highest order interaction is truly null6.

5For a = .05, .025, and .01.

6An assumption checked by X2 tests separately on each interaction varianceestimate.

An initial analysis of the data in each of the tables indicated that the

frequencies for tests based on the dichotomous variates were in each case

significantly different from the frequencies for tests based on the normal

variates.

A more detailed account of the results may be found in the hand out. However.

without extensive reference to the results Figures 1. and 2. tell most of the

story with respect to the "regular" variance ratio test of occasions under

null occasion effect conditions. Note the interaction of the effects due to

the number of subjects and the probability of a one in both figures. Note also

the strong effzet due to nm-null items in Figure 2. The effect of the non-

null items condition confirms the previously indicated confounding of a non-

null items effect with occasions.

An examination of Tables 4. and 5. indicate something about the behavior

of the Quasi-F test under null occasions effects. Note that its behavior is not

particularly good, even when based on the "normal" data.. Tables 6. through 9.

contain the most startling results. Note that the Quasi-F tests based on

normal variates responded in at, appropriate manner to non-null occasion effects.

However, the Quasi-F tests based on dichotomous data had significantly fewer

frequencies in the a rejection regions when the effect being tested was non-null

than they did when the effect being tested was null! Thus it would appear

that Quasi-F tests based on dichotomous data, such as that simulated in this

study, are biased tests!

Tables 10. and 11. and Figures 6. and 7. axe with respect to.the variance

ratio tests of subject by occasion interactions. .Although the results concerning

the Quasi-F were the most startling of the results of this study, the most

disappointing results were those concerning the tests of subject by occasion

interactions. Examine in Figure 6. how the test becomes increasingly liberal

with increases in the number of subjects under conditions where the probability

of a one is not equal to .5.

In order to briefly examine the implications of this study, consider what

the results might suggest to an experimenter who would like to do a repeated

measures type of experiment and analyze his results with hypothesis testing in

mind. First an experimenter who is considering doing a repeated measures type

of experiment should consider the nature of the items or response evokeis that

will be employed in hir experiment, examine them for possible confounding,

and then employ a design and analysis which includes the items as a factor in it.

If an experimenter must have confounding in the analysis consi.stent with

the design of his repeated measures experiment, he is in a somewhat difficult

position with regard to analysis of variance testing of the source of variation

with which confounding is present. For even if he can expect his dependent

variables to have an approximately normal distribution, the results of this

study suggest that the Quasi-F test will not have particularly good properties.

If, on the other hand, the experimenter must evaluate responses in such a manner

that his dependent variables are dichotomous, the Quasi-F test appears to be

completely unacceptable.

On the other hand, if an experimenter finds that he can expect to have no

confounding such as that mentioned above, but must have dichotomous dependent

variables the results suggest, he can expect to appropriately employ the

ordinary analysis of variance, variance ratio test for the occasion effect, if

he employs enough subjects. The results also suggest that an experimenter

should expect the power of analysis of variance tests based on dichotomous

data to be between one third and one half the power he could expect if his

dependent variables could be considered normal in -distribution. The practical

suggestion implied by the results of this study is, that if the above experimenter

must have dichotomous dependent variables he should employ a still larger number

of subjects in order to obtain a reasonable power situation.

The results of this study with respect to the test of the subject by.

occasion interaction when based on dichotomous data, suggest that even a very

large variance ratio statistic may not indicate that the null hypothesis is

false if there are a large number of subjects and the probability of a one is

not close to .5 (see again Figure 6). For a probability of a one close to .5,

however, it would appear that the probability of a Type I error is only

approximately 1.2 times the nominal a level. Since the power of the test of

the subject by occasion interaction based on dichotomous data with a .5

probability of a one was greater than half the power of the test based on

normal data, the results suggest that when the probability of a one is close

to .5 the test may be appropriately employed. If the probability of a one

is not close to .5, however, the results suggest the above test may not be

appropriately employed. As a practical suggestion, an experimenter who could

expect possible subject by occasion interaction and who must have dichotomous

dependent variables should endeavor to employ items or stimuli which will give

him in general a .5 probability of a one in this data.

BIBLIOGRAPHY

Bradley, J. V. Distribution-Free Statistical Tests. New York:Prentice-Hall, 1968.

Donaldson, T. S. Power of the F-test for non-rcrmal distributionsand unequal error variances. Rand Corporation research memorandumRH-5072-PR, September, 1966.

Draper, J. F. A Monte Carlo investigation of the analysis of varianceapplied to non-independent Bernoulli variates. Unpublisheddissertation, Michigan State University, 1971.

Greenberger, M. Notes on a new pseudo-random number generator."Journal Assoc. Comp. Mach.," 8, 163-67.

Greenhouse, S. W. and Geisser, S. On methods in the analysis ofprofile data. "Psychometrika," 1959, 24, 95-112.

Hammersley, J. M. and Handscomb, D. C. Monte Carlo Method. New York:John Wiley, 1964.

Hsu, T. C. and Feldt, L. S. The effect of limitations on the number ofcriterion score values on the significance level of the F-test."American Educational Research Journal," 1969, 6, No. 4, 515-28.

Hudson, M. D. and Krutchkoff, R. G. A Monte Carlo investigation ofthe size and power of tests employing Satterthwaite's syntheticmean squares. "Biometrika," 1968, 55, 431-33.

Lunney, G. H. A Monte Carlo investigation of basic analysis ofvariance models when the dependent variable is a Bernoullivariable. Unpublished dissertation, Univ. of Minnesota, 1968.

Mandeville, G. K. An empirical investigation of repeated measuresanalysis of variance for binary data. Paper tnesented at theannual meeting of the American Educational Rz:earch Association,1970.

Marsaglia, G. and Bray, T. A. One line random number generators andtheir use in combinations. "Communications of ACM," 1968, 11,757-59.

Satterthwaite, F. E. Synthesis of variance. "Psychometrica," 1941,6 309-16.

Seeger, P. and Gabrielsson. Applicability of the Cochran Q test andthe F test for statistical analysis of dichotomous data fordwendent samples. "Psychol. Bull.," 1968, 69, 269-77.

10

TABLES AND FIGURES TO ACCOMPANY THE PAPER ENTITLED"A MONTE CARLO INVESTIGATION OF THE ANALYSIS OF

VARIANCE APPLIED TO NON-INDEPENDENT BERNOULLI VARIATES"

By

John F. Draper, Jr.CTB/McGraw-Hill

Paper presented at the annualmeeting of the American Educational Research Association

April, 1972

11

INDEX

TABLES PAGE

1. Contrast of Experimental Situation 1 with Experimental Situation 2. 1

2. The Empirical Frequencies, in a.1000.Rejection Regions, of theVariance Ratio Test for Occasions, Based on Normal and DichotomousData, Items Crossed with Occasions, Interaction and Occasion EffectsBoth Null.

3. The Empirical Frequencies, in a.1000 Rajection Regions, of theVariance Ratio Test for Occasions, Based on the Dichotomous Data,Items Nested within Occasions Interaction and Occasion Effects BothNull.

4. The Empirical Frequencies, in a.1000 Rejection Regions, of the Quasi-F for Occasions, Based on Normal and Dichotomous Data, Items Crossedwith Occasions, Interaction and Occasion Effects Both Null.

5. The Empirical Frequencies, in a.1000 Rejection Regions, of the Quasi-F for Occasions, Based on Dichotomous Data, Items Nested withinOccasions, Interaction and Occasion Effects Both Null.

6. The Empirical Frequencies, in a.1000 Rejection Region, of the Quasi-F for Occasions, Based on Normal and Dichotomous Data, Items Crossedwith Occasions, Occasion Effects Non-null, Interaction Null.

7. The Empirical Frequencies, in a.1000 Rejection Regions, of the Quasi-F for Occasions, Based on Dichotomous Data, Items Nested withinOccasions, Occasion Effects Non-null, Interaction Null.

8. The Empirical Frequencies, in a.1000 Rejection Regions, of the Vari-ance Ratio Test and Quasi-F for Occasions, Based on Normal andDichotomous Data, Items Crossed with Occasions, Twelve Subjects.

9. The Empirical Frequencies, in a.1000 Rejection Regions,ance Ratio Test and Quasi-F for Occasions, Based on theDichotomous Data, Items Nested within Occasions, Twelve

of the Vari-Normal andSubjects.

10. The Empirical Frequencies, in a.1000 Rejection Regions, of the Vari-ance Ratio Test for the Subjects by Occasions Interaction, Based on theNormal and Dichotomous Data, Items Crossed with Occasions, Interactionand Occasions Effects Both Null.

11. The Empirical Fi:equencies, in a.1000 Rejection Regions, of the Vari-ance Ratio Test for the Subjects by Occasions Interaction, Based on theNormal and Dichotomous Data, Items Nested within Occasions, Interactionand Occasions Effects Both Null.

3

4

10

11

13

14

19

20

21

22

FIGURES PAGE

1. The Interaction of the Probability of a One and the Number of Subjects,with Respect to the Data in Table 2., Data for the "Regular" OccasionsTest.

2. The Interaction of the Probability of a One, the Number of Subjects;and Items Null vs. Non-null, with Respect to the Data in Table 3., Data

for the "Regular" Occasions Test.

3. The Interactionwith Respect toOccasions.

4. The Interactionwith Respect toOccasions.

of the Probability of a One and the Number of Subjects,the Data in Table 4., Data for the Quasi-F Test of

of the Probability of a One and Items Null vs. Non-null,the Data in Table 4., Data for the Quasi-F Test of

5. The Interaction of the Probability of a One and the Number of Subjects,with Respect to the Data in Table 5., Data on the Quasi-F Test ofOccasions.

6. The Interaction of the Probability of a One and the Number of Subjects,with Respect to the Data in Table 10., Data on the Test of Subjects byOccasions Interaction.

7. The Interaction of the Probability of a One and Items Null vs. Non-null,with Respect to the Data in Table 10., Data on the Test of Subjects byOccasions Interaction.

BIBLIOGRAPHY

APPENDIX

ii

5

7

15

16

17

23

Table 1. Contrast of Experimental Situation 1 with Experimental Situation 2.

Situation 1

Occasion 01 02

03 04

Item 111213 111213 111213 111213

Situation 2

Occasion 01

02

03

04

Item 111213 141516 171819 110111112

"Re ular" Variance Ratio Test of Occasion Results

With respect only to the frequencies based on dichotomous variates

in Table 2, it was concluded that there were significant main effects due

to the probability of a one, Pr (1), and the number of subjects as well as

an interaction between the two significant main effects. The significant

interaction is represented in Figure 1. In the figure the two horizontal

lines represent .95 probability limits1 for mean frequencies such as those

graphed, given an expected value of 50. Observing Figure 1, it appears

that a favorable comparison of nominal and empirical probabilities of a

Type I error occurred when the probability of a one was .5 and there were

six or more subjects. Also a favorable camparison occurred when the

probability of a one was .2 and there were ten or more subjects. However,

when the probability of a one was .1 no favorable comparisons occurred,

although the graph suggests that a favorable comparison might occur given

more subjects.

Table 3 differs from Table 2 in only two aspects: (1) the values in

the table were obtained by simulating data which could have arisen from

situation 2 rather than situation 1, and (2) no frequencies appear with

respect to tests based on normal variates.

From the analysis of the frequencies in Table 3, it was concluded that

there were significant main effects due to: (1) the probability of a one,

(2) the number of subjects, and (3) items null in effect versus items non-

null in effect. There were also significant interactions between

1See Appendix A

Table 2.

The Empirical Frequencies, in 0'4.1000 Rejection Regions, of the

Variance Ratio Test for Occasions, Based on Normal and Dichotomous Data,

Items Crossed with Occasions, Interaction and Occasion Effects Both Null

piob

abili

ty o

f a

One

.2.1

Of

1000

5025

1050

2510

5025

10

Dic

hoto

mou

s or

Rom

a11

BD

IID

BD

110

8D

IIla

A13

1111

NN

o. o

fSu

b.*.

II

Item

sI

Lev

el o

fSu

b. B

et.

151

4426

207

841

4121

2412

1141

6715

427

15nu

ll2

4755

2827

99

2233

917

76

1053

525

414

339

4618

274

1614

566

332

1512

1.: C

211

27

44

4545

2223

010

1255

637

324

949

515

51

11

4750

2825

311

2756

930

119

2644

1420

712

non-

null

240

5220

332

1429

4*7

154

1015

44ls

:15

*9

327

5018

296

1127

5111

253

1413

561

141

64

3845

1530

717

2040

624

017

1249

419

09

151

4025

238

850

3216

157

642

:a0

1317

97

null

266

5031

226

630

3219

146

637

246

126

33

5251

3428

1014

3534

2012

105

2229

314

i)2

64

6157

3827

1412

3932

2115

28

lb25

612

05

148

5522

285

1241

2523

1415

522

2314

117

3no

n-nu

ll2

4649

2127

610

3029

1216

57

1529

912

74

353

3426

149

930

3117

196

1028

240

113

44

5138

1513

56

3328

2013

35

3222

114

01

148

3630

1911

834

3616

257

431

2919

141

4nu

ll2

4739

2227

89

4241

1831

104

3045

2230

C4

359

4729

2614

1335

4117

3212

640

388

300

0e

461

4638

2314

1129

4413

266

816

480

340

S1

5352

2630

1011

5234

1524

74

3945

1931

97

non-

null

247

6216

355

932

4316

283

216

5416

3d4

03

6651

4037

199

5532

727

33

2654

937

24

457

3830

2917

239

4422

336

728

45b

27i

S

146

5530

299

1536

4921

265

162*

*922

2513

10nu

ll2

4246

1621

510

4559

1928

413

4257

le30

64

355

6332

285

1644

6515

363

1425

513

230

1110

453

5216

287

1364

5316

294

1423

651

401

131

5160

2140

1222

3246

1922

19

4156

3335

12Id

non-

null

257

5530

2912

1747

6220

358

16 '

3643

1225

715

353

5529

348

1759

6423

276

1323

553

200

94

5259

2937

1221

3059

1840

514

2343

035

012

146

5328

2711

1344

6223

352

860

5319

364

9nu

ll2

4563

2539

1516

7049

2829

74

3947

1635

24

365

5523

278

837

5015

264

1136

4814

368

124

6353

2433

1012

5846

2129

95

1750

434

.4

91

5957

2933

7lb

4756

3537

1314

3145

2526

106

non-

null

246

SO18

224

671

6550

3925

1039

4126

29le

33

53B

O28

4312

1860

5433

3422

1845

3125

2211

5'4

4963

3340

148

484e

2130

107

6450

1930

44

.4 A

N, n

40 {

.4.,

1.44

,JA

kdat

-.

V11

1,11

4;M

:7,

4.1

Table 3.

The Empirical Frequencies, in e0,1000 Rejection Regions, of the

Variance Ratio Test for Occasions, Based on the Dichotomous Data, Items Nested

within Occasions, Interaction and Occasion Effects Both Null.

Prob

abili

ty o

f a

One

.5.2

A

or10

0058

2510

5025

1050

2510

No:

of

Lev

el o

fub

item

sSu

b. B

et.

151

26?

4121

1241

157

247

289

227

105

4

nul

l3

39re

414

612

122

2

44

4522

512

63

95

51

7338

1411

561

2244

176

256

3615

7247

1419

50

non-null

374

2314

5926

654

328

462

265

.69

289

2910

0

151

258

5016

742

139

266

316

3019

637

,6

6nul 1

352

3410

3520

1021

30

64

6138

1439

242

le6

01

100

6837

120

6326

6936

21.

299

5620

119

41ta

8751

'6non-null

312

567

2411

948

2410

456

454

134

6419

119

6331

7834

19

-

148

3011

3416

731

191

247

22a

4218

1030

220

null

359

2914

3517

124Q

80

84

6/38

1429

136

160

01

127

6339

131

8638

133

6433

212

570

3015

310

564

82 .

2311

non-null

312

276

3617

111

556

9047

244

115

5631

156

101

5719

571

21

146

309

3821

.5

1822

132

4216

545

194

it218

8nul 1

355

325

4415

325

30

204

5316

764

164

201

1

114

583

4215

511

077

200

122

602

172

106

5820

413

569

153

9649

non-null

316

289

4620

714

167

111

5030

417

310

449

211

148

9410

042

24

146

.20

1 1

4423

a60

194

nu 1

12

4525

1570

287

3916

23

6523

a37

154

3614

a12

463

2410

5621

917

44

120

614

380

250

160

101

177

129

50non-null

218

812

661

253

185

123

213

108

543

202

148

8120

615

912

117

497

484

174

120

8426

816

410

716

090

55

60

50

40

20

Number of Subjects

12 =

RI = 08 =6 7-1 04 .= *

1

0.5 0.2 0.1

Probability of a One

Figure 1. The interaction of the probability of a one and thenumber of subjects, with respect to the data in Table 2., datafor the "regular" occasions test..

'11

the probability of a one and items null vs. non-null and between the

number of subjects and items null vs. non-null. Both of these interactions

were represented on one graph, Figure 2. The horizontal lines in Figure 2

are .95 probability limits about 50, for means such as those graphed. An

inspection of Figure 2 indicates that a favorable comparison of nominal a

and empirical probabilities of a Type I error occurred when items were null

in effect and the probability of a one was .5. Also a favorable camparison

occurred when the items were null, the probability of a one was .2 and there

were 10 or more subjects. In the absence of the above conditions, however,

only unfavorable comparisons resulted.

The marginal mean vectors2 for items null in effect was [40, 18, 6] and

for items non-null it was [131, 78, 42]. The large mean frequencies for items

under non-null conditions confirm the contention that non-null item effects

are confounded with occasions and will cause the regular variance ratio test

for occasions to have too many Type I errors.

In order to examine the behavior of the "regular" variance ratio test

for occasions effects under non-null occasion conditions, an index of

relative pawer was formed. The relative power of the test statistic3 based

on dichamous data was defined to be, the frequency of the test statistic

based on normal non-null data which fell into an a mjection region divided

into the frequency in the same rejection region of that same statistic

based on the same data after it had been subsequently dichotomized. Relative

power values were obtained for the non-null occasion effect analogs of all

20rdered for frequencies in a = [.05, .025, .01] rejection regions.

3For each design model and parameter configuration.

240

200

yr-4lifte

fatio

01111/ord

do,

Oleo

de

or

o,or 144,dror

Oe

mawOOP

fr odogoo

nos or 41°.

Dr Or or OW MO

ow dr 41 olio

4144,111%

elldsallr IMP 4M. dal 0 1111.

4Ildb%el

4144,

or or dor goo or goo dr 44,46. ***.444

.11.1.11111.

Number of Subjects

12 = 6 = 0

10 =0 4=*8 = 0

S.

doe

%0

am? Non-Null

em"-"1 Null

0.5 0.2Probability of a One

Figure 2. The interaction of the probability of a one, the numberof subjects, and items null vs. nonvinull, mlth respect to the datain Table 3., data for the "regular" occasions test.

0.1

of the parameter configurations indicated in Table 2. These data were put

to an arc-sin transformation and then analyzed by means of a MANOVA. This

analysis indicated three significant main effects and no interactions. The

significant effects were (1) the probability of a one, Pr (1), (2) the number

of subjects, and (3) the level of subject heterogeneity. The overall mean

vector of relative power variables was [.45, .38, .32], which indicates that

the relative power of the variance ratio test for occasions effects decreased

as the nominal a level decreased from .05, to .025, to .01. An ANOVA revealed

no interaction between nominal a level and other sources of variation.

For probabilities of a one equal to .5, .2, and .1 the mean relative

powers for a = .05 ware .58, .49, and .28 respectively. For numbers of

subjects 4, 6, 8, 10, and 12, the means were .39, .41, .43, .50 and .54.

For the four levels of subject heterogeneity 1, 2, 3, and 4 the means were

.52, .46, .43, and .41 respectively.

The differences in mean relative power show clear trends. As the prob-

ability of a one becames smaller so does the relative power, which is the

opposite of the trend which might have been expected, since the degree of

non-null effect in the simulated data was selected to counter the effect

of decreased variance corresponding to a decreased pr6bability of a one.

Thus the power of tests based on the dichotamous variates should not have

changed across levels of a probability of a one, whereas the power of the

test based on normals should have and did decrease across levels of a

probability of a one (and decreasing non-null effects). The results

indicate however, that the power of tests based on the dichotomous variates

fell off more rapidly across levels of the probability of a one than did

the power of tests based on the normals. The explanation of the above may

be that as the probability of a one becomes smaller the point of dichoto-

mization is such that more of the "information" carried in the normals is

lost. Another clear trend is that as the number of subjects increases, the

relative power does so as well. The trend with respect to the number of

subjects is most likely a function of the effect of the central limit theorem.

The third trend indicates a loss in relative power with an increase in

subject heterogeneity, a trend for which this investigator has at present

no explanation.

Quasi-F Test of Occasion Results

The most startling result of this study concerned the empirical testing

the application of the Quasi-F test of occasion on dichotomous data.

Tables 4 and 5 contain the frequencies in a = .05, .025, and .01

rejection regions of the Quasi-F ratio t=.-st statistics for tests of

occasion effects when the data were simulated under null occasion effect

conditions and null subject by occasion interaction effect conditions.

That is Tables 4 and 5 are the Quasi-F analogs to Tables 2 and 3.

Again the data in which were with respect to normal variates were

significantly different from the data in Table 4 which were with respect

to dichotomous variates.

The analysis of the data with respect to the dichotomous variates in-

dicated significant effects due to: (1) the probability of a one, Pr (I)

(2) the number of subjects, the interaction of (1) and (2)0 and the inter-

action of (1) and items null vs nm-null respectively. The above two

significant interactions are reptesented in Figures 3 and 4 respectively.

Zs:

°Col

.!d!

.d

Table 4.

The Empirical Frequencies, in 04.1000 Rejection Regions,

of the

Quasi-F for Occasions, Based onNormal and Dichotomous Data, Items Crossed

with Occasions, Interaction and Occasion

Effects Both Null.

Probability of a One

.5

.2

.1

a.1.000

50

25

10

50

25

10

50

25

10

4iehotosous or dorsal

DODNDN

DVIDND

D

.;

No. of

Sub.s

11..svel

of

Items

Sub. Het

175

4537

2515

1245

5923

299

1520

606

332

22nu

l I2

6347

3326

1912

6142

3917

156

1654

42

92

143

6847

3630

128

3066

2027

FS14

646

32

00

d4

451

6426

3210

1727

5610

336

Id6

613

330

141

6857

4325

179

6171

2833

911

355

023

014

non-

null1

255

5728

2715

1234

4726

2313

IC8

595

332

173

7639

2512

128

3152

2322

1310

2252

1127

912

457

4426

2512

1223

6212

383

191

b50

1423

413

178

3945

1818

874

3536

1721

342

5519

2711

5nu

ll2

8134

4713

245

8337

5017

295

4338

1 b

213

3

373

4546

2518

971

. 32

3119

165

2652

82

91

46

487

5645

2619

749

4825

3014

512

487

210,

4i

7739

4120

137

6643

2319

147

4341

721

02

non-

nul li

260

4431

2113

240

4633

2716

650

4522

164

1

383

4349

2126

660

4944

2613

738

3715

130

04

6750

452?

276

5548

3019

123

2439

1610

20

110

224

5712

295

5717

4011

150

4814

306

110

null

281

3052

1730

771

1629

913

041

2621

149

03

7338

4419

148

7225

4815

140

3027

245

I 5

38

467

2347

1723

7dC

2551

1617

037

2520

136

01

6232

3018

16a

51la

348

190

-82

2753

942

0no

n-nu

ll2

7236

4321

2610

5419

385

160

9633

be1

235

03

7727

4210

172

5813

343

90

4634

3815

120

464

1440

516

060

20.

389

190

5321

305

00

198

3759

1925

1094

2976

631

643

2414

37

2nu

ll2

9327

6310

337

9436

538

328

110

3881

053

33

9238

5315

270

104

4251

1229

958

2129

612

310

480

3646

1423

478

3441

531

553

3935

118

41

9234

6212

267

7329

394

52

6231

3310

187

non-

nul I

289

2752

1125

775

3632

715

472

2442

610

33

8429

4412

246

9827

632

392

92:2

051

015

44

823?

5013

IT9

112

2476

953

581

4143

1221

7

113

225

6312

357

9025

59IC

271

137

3354

1641

5

null

283

4653

2027

574

2451

1026

289

2763

222

72

394

2050

923

291

2544

1023

292

2135

1417

64

8028

596

350

7913

397

110

7431

5319

2 7

5

193

3956

1934

366

3537

2426

412

133

7415

400

non-

null

2$4

2241

1024

167

3549

2019

110

117

4910

303

378

4533

2321

48

?37

4721

267

9519

5414

335

471

2147

1027

296

3457

1243

368

3438

1522

4.

11...

-1

Table 5.

The Empirical Frequencies,

0(41000 Rejection Regions, of the

Quasi. -F for Occasions, Based on Dichotomous Data, Items Nestedwithin Oc-

casions, Interaction and Occasion Effects Both Null.

Prob

abilL

q of

a O

ne.5

.2.1

1000

5025

10SO

2510

5025

10

No.

of

Sub.

sIt

ems

Lev

el o

fSu

b. H

et.

4 6 8

10 12

null

non

nti

nu 1

1

non-

nun

huII

bah.

hull

hon-

m41

/

non-

wa

If

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 1 3 4 1 2 3 4

61 59 62 47 59 68 64 58 77 62 68 85 108 86 84 72 92 73 66 67 102 92 140

119 82 100 87 es 106

102

113

87 101 '78

77 97 123

122

128

174

37 35 29 25 38 40 37 32 41 39 39 43 61 55 30 46 52 44 38 41 65 55 101

85 49 51 48 53 59 65 70 46 67 48 49 69 77 87 09 120

14 17 11 4 14 19 13 17 17 22 15 16 20 28 13 16 23 18 14 16 33 34 51 54 22 23 25 21 32 48 28 20 39 19 27 25 40 68 58 04

27 50 25 24 45 50 42 35 61 68 70 52 67 61 61 64 54 51 65 83 74 0511

3 86 90 78 BO 73 96

134

114

113

102 92 82 85 05 72 81 115

15 32 12 9 25 28 13 14 36 38 31 21 44 35 42 37 19 29 35 47 45 49 62 46 57 49 50 50 63 82 85 75 54 53 47 4$ 63 45 53 60

9 13 6 516 13

5 9 13 20 23 11 21 23 23 21 18 914 24 17 33 38 27 35 28 31 35 52 52 50 44 29 28 26 12 29 .28 37 31

a 6 4 618 27 32 24 51 47 35 16 46 35 34 47 70 54 53 37 44 46 85 64 71 93 66 67 50 56 61 45 94 96 10

5 7710

321

310

210

1

1 4 4 3 4 19 10 17 24 20 13 10 24 17 12 25 25 20 30 10 23 32 36 38 27 66 29 38 21 34 41 34 46 59 58 52 02 108 68 43

a 0 3 0 4 711

4 3 10 5 2 11 a 9 11 11 15 221 a 11 25

.20 13 41 12 11 12 24 24 10 31 33 30 16 41 54 44 31

Both figures indicate that the empirical probability of a Type I error is

not in general close to .05. Figure 3 indicates that although the frequency

in the rejection region is less affected by the probability of a one as the

number of subjects increases, the tests became rather liberal. Figure 4 is

self explanatory.

The analysis of the data in Table 4 with respect to the normal variates

was interesting in that it tends to contradict earlier findings. The analysis

indicated a strong significant effect due to the number of subjects and

indicates a general downward trend in the empirical probability of a Type

error with an increase in number of subjects.

The data in Table 5 were analyzed by means of a multivariate analysis

of variance and significant effects were found with respect to: (1) the

probability of a one, (2) the number of subjects, (3) items null vs. non-

null, and the interaction of (1) and (2). The significant interaction is

represented in Figure 5 which is somewhat similar to Figure 3 and which in

general lends itself to the same interpretation.

Tables 6 and 7 contain frequencies in a = .05, .025, and .01 rejection

regions of the Quasi-F test statistics for tests of occasion effects when

the data were simulated under non-null occasion effect conditions. The

results in Tables 6 and 7 are most startling for it is apparent that although

the Quasi-F tests based on normal variates responded in an appropriate manner

to non-null effects that the Quasi-F tests based on dichotomous variates

did not, The Quasi-F test based on dichotomous variates had significantly

fewer frequencies in rejection regions when the effect it was testing was

non-null Chan it did when the effect it was testing was null, and in all of

the cases investigated the empirical power of the Quasi-F test was consistently

less than the nominal a level:

-12-

25

Table 6.

The Empirical Frequencies, in the *1.1000 Rejection Region, of

the Quasi-F for Occasions, Based on Normal and Dichotomous Data, Items

Crossed mith Occasions, Occasion Effects Noo-null, Interaction Null.

irobabillty of

a O

ne.

..

0410

0050

2510

SO

25

10

50

25

10

Die

boto

sow

s or

Nor

mal

DNDN

DN

D.NDNDN

DNDNDM

1

No.

of

JI

Lev

el o

fMet

128

298

1318

96

107

2010

616

917

4329

111

1767

1234

null

220

328

922

53

129

2620

816

126

365

810

98

666

303

1138

47

258

415

510

244

1513

38

627

140

668

235

44

649

02

350

121

615

209

718

43

825

151

588

046

124

288

1020

5S

122

3318

319

101

1056

1411

34

551

32no

n-nu

ll2

1734

211

237

312

514

193

210

91

5512

118

361

232

312

346

423

21

126

2524

411

141

578

1313

69

675

2)4.

1346

35

345

121

611

280

119

40

945

151

110

01

47

114

412

627

52

150

2223

211

168

461

4314

225

6510

30nu

ll2

547

42

343

122

223

266

810

35

9345

166

1179

344

311

545

640

63

277

2828

97

202

712

325

177

1091

451

64

665

63

516

235

413

386

626

42

149

1421

99

114

164

19

403

328

60

182

2223

615

155

073

5112

719

6410

39no

n-nu

ll2

843

94

309

320

218

257

818

97

8448

142

2067

047

37

544

441

52

276

2926

524

197

1998

3017

810

0910

484

76C

21

472

131

923

401

1930

011

193

3422

127

126

1469

110

544

932

94

233

2130

810

219

811

246

218

1911

31

48nu

ll2

562

91

497

029

932

342

1724

16

145

4220

922

109

1358

34

697

256

70

393

2042

09

275

917

330

253

1217

65

928

42

792

169

01

529

2251

514

376

722

831

295

2523

020

131

15

582

444

31

273

3431

523

218

1411

841

207

2411

216

53no

n-nu

ll2

1161

45

487

331

418

353

424

13

147

5723

334

133

2476

315

672

957

97

399

2443

012

2 90

019

457

245

2CP

1 51

2091

48

790

265

30

512

1652

014

388

023

737

304

1422

30

109

19

696

655

94

371

937

95

291

220

756

234

1616

3.1

-677

null

25

732

458

24

436

1844

110

307

622

939

249

2118

914

107

32

915

270

21

525

1352

56

377

226

97

294

120

3a

122

104

291

00

804

(`65

811

620

247

50

299

2034

37

2 31

514

11

368

91

534

134

724

356

1828

72

197

4122

318

160

1182

non-

null

25'

760

362

00

440

IC44

910

314

123

043

266

301

5613

110

32

827

072

50

532

2152

612

357

425

530

330

1422

50

142

42

911

278

40

644

2261

15

465

031

123

343

132

315

151

-1

373

4.0

622

044

4.10

432

631

33

173

4635

332

1 19

2453

null

2I

902

167

90

49'3

'12

462

633

13

208

3825

425

1 39

2282

30

880

077

70

625

456

94

404

227

440

319

3122

110

115

124

093

70

878

0 na

066

60

519

036

747

330

3120

717

141

10

514

038

70

278

940

24

308

016

352

261

3813

121

63no

n-nu

ll2

152

20

420

027

913

494

636

00

221

4525

722

146

671

31

591

144

70

331

654

85

391

024

633

318

V2

397

110

42

603

247

92

364

661

62

497

234

535

329

352

7124

16,)

Table 7.

The Empirical Frequencies, in 0(41000 Rejection Regions, of the

Quasi-F for Occasions, Based on Dichotomous Data, Items Nested within Oc-

casions, Occasion Effects Non-null, Interaction Null.

Prob

abili

of a

One

.1

cc 1

000

5025

1050

2510

5025

10

No.

of

Lev

el o

fSu

b.s

item

sSu

b. H

et.

3.25

14a

1?a

415

10nu

lIl

2 312 a

4 42 2

28 1613 13

2 55 13

3 1

0 o4

4 14 30

2 17o 8

a38

4 25o 12

? 167 3

0 1

non-

,iuIS

2 336 30

13 126 5

47 2426 9

1? 915 21

9 a5 5

416

95

2313

623

73

113

64

2613

445

328

non

210

41

256

239

310

315

53

2916

328

236

410

33

114

218

121

127

177

4425

1240

2313

213

42

3831

1025

04

n O

n- h

ull

313

62

4433

1636

24a

414

63

2622

1150

204

19

72

178

359

207

23

1o

329

349

1711

34

42

226

133

114

84

21

o32

106

3?24

101

2414

260

4332

7234

12

non-

non

2 328 17

12 144 4

39 4534 29

16 1654 53

30 2010 15

422

1?9

4637

2479

2615

15

55

189

547

2114

noa

24

33

95

232

219

33

20

1?11

120

60

104

00

010

30

111

1412

1914

538

2914

5132

11

noil-

2 39 4

8 23 1

52 4239 31

23 1349 35

33 2622 17

46

31

1410

67

?3

13

o0

e3

33?

3211

nu2

21

06

33

6229

43

11

06

32

5531

612

4o

00

0o

043

2417

18

41

4120

1151

3927

non

-114

411

2 3a

105 10

3 638 33

19 2012 5

91 8556 57

40 374

108

469

400

9942

24

100

90

an

50

ariSUBJECTS

(pp 40

12=

a 30 10= 0i4P4

20

10

8 =

6 =0

4 =*

00.5 0.2Probability of a One

Figure 3. The interaction of the probability of a one and thenumber of subjects, with respect to the data in Table 4., datafor the quasi-F test of occasions..

0.1

90

"14..

1416 111,

ms,

Non-Null

11111110

Nu

l

0

0.5

0.2

Probability

of a One

Figure

4.

The

interaction

of

the

probability

of a one

and

items

null

vs.

non-null,

with

respect

to

the

data

in

Table

4.,

data

for

the

quasi-F

test

of

occasions.

114,

411.011114.ftimm

eivassio

0.1

II

110

100

90

80

70

60

0 50

40

30

Pei

20

Number of Subjecis

12 m

10 =0

8 =In

6 = 0

4 m*

0.5 0.2Probability of a One'

Figure 5. The interaction of the probability of a one and thenumber of subjects, with respect to the data in Table 5., dataon the quasi-F test of occasions.

0.1

4.3

7.3

The data in Table 6 with respect to normal variates has an overall mean

vector of [407, 300, 195] which is significantly less than the mean vector of

analogous power data based on the regular variate ratio test.

Table 8 is a rearrangement of data which has been previously presented.

The arrangement of data in Table 8 was established to allow easy contrast of

data with respect to regular F and Quasi-F tests based on normal and dichoto-

mous variates under null and non-null repeated measures conditions.

Table 9 is laid out in the same manner as Mlle 8 and includes some new

data, that with respect to normal variates under situation 2.

ect

Tables 10 and 11 present frequencies in a = .05, .025, and .01 rejection

regions for variance ratio tests of subject by occasion interaction effects,

when the data were simulated under null occasion and null subject by occasion

interaction effect conditions. The frequency data in Table 10 are with respect

to situatior. 1 and the frequency data in Table 11 with respect to situation 2.

Again MANOVA indicated a significant difference between the data in

Table 10 based on normal variates and the data in Table 10 based on dichotomous

data.

The multivariate analysis of the data in Table 10 based on dichotomous

variates indicated significant effects due to: (1) the probability of a one,

(2) the number of subjects, (3) items fixed vs. random, (4) the level of

subject heterogeneity the interaction (1) and (2) and the interaction of (1)

and (3). The two interactions are displayed graphically in Figures 6 and 7.

Inspection of Figure 6 indicates that a favorable comparison of empirical

probability of a Type I error to nominal occurred only for 6 subjects and a

Table 8.

The Empirical Frequencies, in ot.1.000 Rejection Regions, of the

Variance Ratio Test and Quasi-F for Occasions, Based on Normal and Dicho-

tomous Data, Items Crossed with Occasions, Twelve Subjects.

0g 1

009

5025

10SO

2520

5025

Nor

mal

or

Die

hotO

noua

D1

8V

8II

8I

811

811

0If

8II

00

FI

1cel

oft.

Sub.

fle

I1

4653

2027

1113

4462

2335

28

6053

1534

49

ol

nqn

245

6825

3915

1670

4929

297

439

4716

352

43

6555

2327

00

3750

1526

411

3648

1436

312

&ri

ft*

463

5324

3310

1258

4621

299

517

504

344

91

5957

2933

718

4756

3537

1314

3145

2520

106

naa-

nall

246

SO13

224

671

6550

3925

1339

4128

29Id

33

5380

2843

12la

6054

3334

22.1

845

3125

2211

54

4963

3340

140

4840

2130

107

6450

1930

44

Cen

tral

110

225

6312

357

9025

5910

271

8730

5416

415

nun

283

46SO

2027

574

2451

1026

269

2763

2227

2gu

llet-

394

2050

923

291

2544

1023

292

2135

1417

64

8028

596

350

7918

39T

110

7431

5319

273

190

3956

1934

366

3537

2428

412

1a3

7415

406

tios_

rolf

204

2241

1024

167

3549

2019

110

117

4910

303

378

4533

2321

487

3747

2126

795

1954

1433

54

7121

4710

272

9634

5712

403

6834

3815

224

161

589

649

482

035

169

936

560

926

350

717

434

915

834

385

244

4514

2no

II2

623

938

563

878

356

746

343

653

250

556

133

407

125

379

9425

244

158

366

197

753

292

939

705

239

677

126

965

619

151

214

746

475

357

3622

7P°

711.

475

198

863

696

944

393

645

485

432

075

418

661

417

648

910

240

554

291

161

191

547

283

533

371

936

555

525

546

215

734

915

035

011

725

554

.

171

-om

it2

563

931

495

874

349

766

376

664

267

558

185

409

146

380

115

294

6515

73

666

966

538

93C

387

858

405

752

296

642

176

435

157

407

119

34a

4122

04

696

992

582

97C

420

941

402

839

308

712

201

615

160

491

9243

545

214

Nor

:-ce

ntra

l1

373

40

622

044

410

432

631

33

173

4635

332

119

2453

non

21

002

267

90

497

1246

26

331

320

8 -

3025

425

139

.22

132

30

830

077

73

525

456

94

404

227

440

319

3122

118

115

Qua

si.-

40

937

087

83

720

066

60

519

036

747

330

3126

717

141

.1

051

40

387

027

39

402

430

50

163

5226

13d

131

2163

atal

-sd'

21

528

042

00

279

1349

48

360

022

145

247

2214

66

913

159

11

447

033

16

548

539

10

246

3331

89

239

711

04

260

31

479

236

46

616

249

72

345

3532

935

271

2415

0

!14

.11,

314,

4

.,4.

Table 9.

The Empirical Frequencies, in ota1000 Rejection Regions, of the

Variance Ratio Test and Quasi-F for Occasions, Based on the Normal and

Dichotomous Data, Items Nested within Occasions, Twelve Subjects.

,Probability of a On

.5

r

1311000

50

25

10

SO

25

10

SO

25

10

Normal or Diehotosbous

1:1

11D

alD

11D

110

11E

l11

011

011

D2

Oecautins

FItems

Lev

el o

fSub. Het. ....,

146

53

28

27

11

13

44

62

23

35

28

60

53

19

36

49

hitt/

245

63

25

39

15

16

70

49

28

29

74

39

47

16

35

24

365

55

23

27

aa

37

50

15

26

411

36

48

14

36

a12

00-

4 163

206

53

236

24

143

33

207

10

8Z

12

127

58,

258

46

445

21

160

29

339

9101

5

266

17

177

50

486

4129

34

369

450

9

291

608-00 0

2 ,188

202

3C3

334

126

143

217

245

61

81

154

181

253

206

484

534

185

159

410

476

123

121

313

352

213

114

515

546

10o

97

44 5

459

5446

353

360

4174

393

128

304

64

204

268

571

164

495

107

413

160

652

90

550

55

45

Central

1101

40

67

14

39

8102

34

54

14

29

494

46

46

20

31

9

pull

278

55

48

27

19

13

92

39

53

14

28

496

41

59

17

33

12

(Nast-

3 '4

77

29

49

727

382

38

47

13

26

4105

40

58

14

30

9

97

40

69

11

25

285

31

48

912

377

46

52

20

16

10

1123

54

77

44

40

20

85

104

63

89

29

51

103

93

82

71

41

SO

non-nut/

2122

75

87

52

68

17

72

113

45

82

26

48

213

119

166

1SO

54

66

3126

69

89

66

58

28

81

109

53

77

37

47

102

127

66

107

44

66

4174

95

128

73

84

36

115

109

60

88

31

49

ICI

144

43

101

31

59

.....,

1615

898

494

820

351

699

365

609

263

507

174

349

158

343

85

244

45

142

PIM

2623

933

500

878

356

746

363

653

250

556

133

407

125

379

94

292

44

150

3661

977

632

929

397

852

396

771

265

656

191

512

147

464

75

357

36

227

Parkt

4751

988

636

969

448

936

452

654

320

754

206

614

176

489

102

465

54

292

1655

835

550

827

447

744

492

764

404

631

293

593

311

650

265

564

199

400

2686

909

590

868

479

795

459

818

404

720

317

604

272

707

214

593

182

479

II0IF411111

3734

928

646

931

508

349

516

536

422

792

313

705

317

774

214

699.

170

574

Hoa-

4750

933

667

907

541

876

515

907

445

871

333

791

2.44

834

221

749

136

676

Central

13

791

06:19

0541

8499

3360

3250

37

259

32

206

11

99

hUn

22

649

1745

C600

6525

3413

3273

62

262

29

175

491

31

904

1828

0708

8612

3513

2341

55

317

31

254

i153

40

948

0914

3842

,0

698

0591

0466

43

366

24

306

17

197

Qua

si-

1.8

553

4446

1319

41

245

20

173

11

136

51

167

3/

134

27

113

MM441

23

577

5469

3354

30

278

12

189

12

137

91

179

56

124

44

66

310

593

10

524

6391

33

321

20

217

5144

'

85

209

57

153

37

111

410

632

8523

4416

69

321

48

229

0143

99

224

42

167

24

1)2

J

k'

Table 10.

The Empirical Frequencies, in 4;4)1000 Rejection Regions, of the

Variance Ratio Test for the Subjects by Occasions Interaction, Based on the

Normal and Dichotomous Data, Items Crossed with Occasions, Interaction and

Occasions Effects Both Null.

Piro

ba61

11of

a O

no.

..

.2.

ot .1

000

5025

1050

25.1

050

2510

Dic

hoto

mou

s or

dor

eal

01

01

08

0II

111

08

D11

011

718

No.

of

sub_

5462

3130

1610

4750

2428

1113

3040

2527

1714

null

254

4026

257

1062

6537

3324

2%.,

4657

3020

2713

371

4540

2225

771

3556

1631

1364

5647

1928

94

454

4126

2C12

865

3550

143E

,10

4735

3213

217

a55

6835

3514

1253

4530

2115

951

5433

1816

10no

n-nu

ll2

4962

2929

2213

4543

2417

138

5746

3619

2912

362

713?

3626

2144

4525

1813

961

4031

1827

114

.75

5146

3226

1847

5224

2521

1564

5337

2732

18

158

4329

2010

746

4653

2:12

556

6631

3912

18nu

ll2

6353

3020

185

6549

3?22

1211

101

5170

2757

143

6534

4119

173

8162

5729

238

116

55d3

2752

106

468

5739

2818

1587

5259

3123

1211

360

7234

3516

139

3723

1510

1061

4836

2425

852

4533

2917

20no

n-nu

ll2

5443

2?14

146

4858

3134

1921

8549

5127

2812

363

4543

2615

951

5141

3134

1181

5649

3337

204

6950

4427

117

4351

'37

3222

1475

3947

2337

13

152

5229

249

553

3639

20,

157

7833

3518

184

null

258

5831

3114

.11

6640

4116

287

111

437/

2767

73

4445

1525

512

6555

5222

2211

13J

29la

1951

2a

473

4736

1922

1090

5349

3028

1312

243

8322

497

155

3836

2111

1160

3944

2325

667

3146

2229

10no

n-nu

ll2

4942

2318

1010

7C33

4519

1911

7438

7025

364

256

5231

3314

1283

4245

2833

1375

3347

1426

34

6049

3522

1713

0438

'38

2625

876

3649

).9

154

168

6041

2915

1564

4945

1615

1593

5154

2625

22nu

ll1

5641

2317

1310

9349

4721

2219

115

3567

il51

93

6545

3123

1311

112

5257

2038

1714

247

109

2563

2010

467

4235

18la

1010

362

6724

4318

150

5511

627

8 /

201

4841

la21

a11

6661

2817

2014

112

6047

2930

26no

n-nu

ll2

4746

2215

98

5958

2424

1819

7755

4226

3918

349

4921

1810

1291

052

5926

2421

130

5264

2337

214

5343

2214

128

9940

6030

5226

124

5464

2151

17

145

5120

286

984

4247

2515

1113

046

9230

5012

null

240

5223

2811

1611

052

7235

3819

121

5892

3555

203

7456

3428

2011

6849

4530

2012

7951

7222

5711

466

4642

3112

710

850

6033

249

153

5(.

1d:9

3491

161

5445

3229

1012

5655

3038

1716

64.;

5636

2027

12no

n-nu

ll2

6259

2740

1519

6144

5134

268

104

5370

2650

103

4648

1936

1116

6247

3232

415

9240

5529

35-

104

6546

2924

1412

121

478

Z26

4214

100

4783

2445

15

Table 11.

The Empirical Frequencies, in cc+1000 Rejection Regions, of the

Variance Ratio Test for the Subjects by Occasions Interaction, Based on

the Normal and Dichotomous Data, Items Nested within Occasions, Interaction

and

Occ

asio

nsEffects Both Null.

Probability of a One

.5

.2

.1

Of

1000

SO

25

MW

25

20

SO

25

10

Oichotomous or Normal

OVANDODIMON

DNONEIN

No.

of

Sub

.s.1

/Will

iLe

vel o

fS

Ub.

Het

.1

5957

3222

1610

5053

2920

912

4636

3126

2514

,..

null

255

4427

168

765

4528

2014

5251

412

189

364

5234

2418

642

4718

247

7146

5526

3012

44

5951

3321

1510

4343

1132

663

2939

17ld

b.

155

4926

2811

1450

4213

199

854

4942

2223

10no

n-nu

ll2

6563

3226

910

5842

2018

B11

6834

4425

3211

354

5528

299

1376

5239

1914

461

395L

2621

144.

5760

3731

1717

5247

4215

256

7029

5121

)25

7

155

4924

2312

861

4344

1814

a65

6536

3519

18

null

269

5342

2822

870

4741

2321

1699

51o4

2957

lo3

'66

4539

2221

973

5259

2326

1110

057

7)38

63.L

76

485

6442

3221

1781

4869

3036

1710

654

ol31

5814

164

4430

2713

1484

3843

1921

1155

5237

2417

1-no

n-nu

ll2

6947

3925

1610

8044

5618

3510

7342

5123

519

379

5044

2818

1161

4453

2740

1283

505d

3748

174

7257

3733

2514

102

5273

3452

1586

5260

3339

13

161

4936

3012

851

4431

1410

255

5240

ld21

L

null

252

5535

3117

670

4438

1520

611

462

66lo

5610

357

4526

279

1360

5851

2126

512

348

0517

56d

84

8549

5528

314

106

5364

2429

312

651

7120

509

161

4330

229

769

4337

22lti

777

6844

1735

b

non-

null

261

5331

1713

781

4741

1525

3o3

55bl

16'id

23

5159

3623

182

7640

4614

224

9262

5113

350

462

4330

2117

7-

9549

5617

295

8143

526

430

169

5238

3012

1470

5655

2821

1288

5529

3229

lbnu

ll2

6750

2425

712

8549

4230

2815

112

4598

1759

7

368

6032

2513

1311

959

7037

3616

15d

5511

J31

7617

104

6946

3623

1813

104

5667

3448

1714

669

109

3673

ld1

6354

3426

1417

6465

4545

1915

7176

7137

2612

non-null

2 376 67

56 4651 36

33 2322

614 10

8810

062 67

61 5234 32

35 3415 17

106

128

73 6271 16

4.)

3545 61

lo 134

9954

6225

2413

9565

7737

2819

141

61llo

3892

Id

150

5321

2810

1482

4749

2819

1711

940

t-.7

3341

17

null

2 3

44 8459 72

19 4728 35

IC 1715 12

105 90

49 5074 61

39 3754 21

24 1810

8 9347 48

19 7.;

37 2656 41

28 1412

479

5245

2313

1210

639

7434

4114

143

5211

733

8621

162

7942

4225

1691

3941

3417

207

fjeo

l21

3646

132

4452

2534

1314

7646

4637

1620

103

3564

2651

12non-null

346

5033

3412

e5t

.56

2.;

403

2311

842

1C,;

g5

6..

144

6047

3529

1918

7628

4116

1914

101

33o6

2%;

347

,

120

1 1 0

Number of Subjects

1 00 12 =

10 = 0

90 8 =

6 = 0

80 4 = *

40

0.5 0.2 0.1

Probability or a One

Figure 6. The interaction of the probability of a one and tilenumber of subjects, with respect to the data in Table 10.1, 4ataon the test of subjects by occasions interaction.

probability of a one equal to .5, and for 4 subjects and a probability of

a one equal to .2 or .1. For all other conditions the test is too liberal.

Figure 7 is self explanatory.

A multivariate analysis of dhe data in Table 11 indicated the same trends

and significant effects that were found in Table 10.

Relative power variables were formed for both Table 10 and Table 11

and separate multivariate analyses of variance were performed on the relative

power variables for both tables. In both analyses significant effects were

found due to; (1) the probability of a one, (2) the number of subjects, the

interaction of (1) and (2), the interaction of (1) and items fixed vs.

random, and the interaction of (1) and the level of subject heterogeneity.

In addition the analysis of the relative power variables from Table 11

disclosed a significant main effect due to items fixed vs. random.

In general it may be observed that higher relative powers correspond to

greater "liberalness" in the test of a true null hypothesis. Thus the

general increases in relative power observed across levels of a probability

of a one .5 to .2 to .1 are in same sense specious.4

I 00

90

700 /1:3

60eel

a)

50k

40

0

"6" Non-Nullaserrore Null

0.5 0.2Probability of a One

Figure 7. The interaction of the probability of a one and itemsnull vs. non-null, with respect to the data in Tabl 10., dataon the test of subjects by occasions interaction.

0.1

BIBLIOGRAPHY

Bradley, J. V. Distribution-Free Statistical Tests. New York:

Prentice-Hall, 1968.

Donaldson, T. S. Power of the F-test for non-normal distributionsand unequal error variances. Rand Corporation research memorandumRM-5072-PR, September, 1966.

Draper, J. F. A Monte Carlo investigation of the analysis of vuriance

applied to non-independent Bernoulli variates. Unpublisheddissertation, Michigan State University, 1971.

Greenberger, M. Notes on a new pseudo-random number generator."Journal Assoc. Comp. Mach.," 8, 163-67.

Greenhouse, S. W. and Geisser, S. On methods in the analysis ofprofile data. "Psychometrika," 1959, 24, 95-112.

Hammersleyt J. M. and Handscomb, D. C. Monte Carlo Method. New York:

John Wiley, 1964.

Hsu, T. C. and Feldt, L. S. The effect of limitations on the number ofcriterion score values on the significance level of the F-test."American Educational Research Journal," 1969, 6, No. 4, 515-28.

Hudson, M. D. and Krutchkoff, R. G. A Monte Carlo investigation of

the size and power of tests employing Satterthwaite's syntheticmean squares. "Biometrika," 1968, 55, 431-33.

Lunney, G. H. A Monte Carlo investigation of basic analysis ofvariance models when the dependent variable is a Bernoullivariable. Unpublished dissertation, Univ. of Minnesota, 1968.

Mandeville, G. K. An empirical investigation of repeated measuresanalysis of variance for binary data. Paper presented at the

annual meeting of the American Educational Research Association,1970.

Marsaglia, G. and Bray, T. A. One line random number generators andtheir use in combinations. "Cammunications of ACM," 1968, 11,757-59.

Satterthwaite, F. E. Synthesis of variance. "Psychometrica," 1941,

6, 309-16.

Seeger, P. and Gabrielsson. Applicability of the Cochran Q test and

the F test for statistical analysis of dichotamous data fordependent samples. "Psychol. Bull.," 1968, 69, 269-77.

APPENDIX A

If 1,000 samples are simulated so that a null hypothesis is true

and so that the assumptions of an ANOVA are met, the number of F-statistics

testing the above hypothesis which have values which exceed the Fl_a

quantile of a corresponding F-distribution4 will be approximately 1,000a.

If 1,000 samples of data are simulated so that a null hypothesis is not

true and so that the assumptions of an ANOVA are met, the number of F-

statistics testing the above hypothesis which have values which exceed the

F1-a

quantile of a corresponding F-distribution will be approximately 1,000

times the nominal power (1-$) for the situation simulated.

Let X. be defined according to the rule

1, F FXi

= {

0, otherwise

where Fi

is the F-variate calculated on the ith

sample, i = 1, 2, ...,

1,000, and F1_01 is the 1-a quantile of the corresponding F-distribution.

The variate Xiis then an indicator variable, which takes on the value

one 'when F.is in the a rejection region and which takes on the value zero

when F does not occur in the rejection region. Note that the frequency

of Fi's which fall in the a rejection region,

1000f, may be represented by the expression f = E X.

i=1

Observe that f is a binominal variate with parameters p = a and n = 1000.

The variance of f is then np(1-p) or for n = 1000 and a = .05, the var

(f) = 47.5. Therefore by employing the normal approximation a, .95

probability interval may be formed about the expected value of f, E(f) =

1000 a, which for a = .05 is Pr(36.5 < f < 63.5) = .95.

4An F-distribution with the same degrees of freedom as the variance ratiofor the test.

40