AD-A125 099 THE USE OF THE GENERAL RASCH MODEL WI1TH /MULTIDIMENSIONAL'ITEM RESPONSE DATA(U) AMERICAN COLLTESTING PROGRAM IOWA CITY IR R L MCKINLEY ET AL.

UNCLASSIFIED AUG 82 RR-82-i-ONR N88@14-8i-K-98i7 F/G 12/1 N

IMMEM on MINEIEhIhI1hhhhhhhhhE111EhhhhhhhEND

-~ ~ ~~2 -- -- -

1.0 ki IL .I~632 ItI E _

"1.25 .-

MICROCOPY RESOLUTION TEST CHART

- PATIOW WRMf OF STANDARDS-1963-A

. -. * ..

The Use of the General Rasch Modelwith Multidimensional Item Response Data

Robert L. McKinleyand

Mark D. Reckase

Research Report ONR 82-1August 1982

The American College Testing ProgramResident Programs Department

Iowa City, Iowa 52243 DT1C

Prepared under Contract No. N00014-81-KA817

__ with the Personnel and Training Research ProgramsC" Psychological Sciences Division

, ¢Office of Naval Research

Approved for public release; distribution unlimited.-.J- Reproduction In whole or In part is permitted for

any purpose of the United States Government.

83 03 02 016

"'-. - "'Y':'., ', ,'% '-:'. :. ."_ . -. : " :-". : . :-""T . , ." , : -."•"" - ": :"A ,: : -

SECURITY CLASSIFICATION OF THIS PAGE ('Nmen Data nIterod.

REPORT DOCUMENTATION PAGE READ ISTRUCIONS' BEFORE COMPLETING FORM1. REPORT NUMBER 2. GOVT ACCESSION NO. S. RECIPIENT'S CATALOG NUMBER

4. TITLE (end Subtitle) AS. TYPE OF REPORT & PERIOD COVERED

The Use of the General Rasch Model with Technical ReportMultidimensional Item Response Data

6. PERFORMING ORG. REPORT NUMBER

3 7. AUTHOR(q) 8. CONTRACT OR GRANT NUMMER(a)

Robert L. McKinley andMark D. Reckase N00014-81-K0817

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT PROJECT. TASKAREA & WORK UNIT NUMBERSThe American College Testing Program P.E.: 61153N ProJ.: RR042-0

P.O. Box 168 T.A.: 042-04-01Iowa City, IA 52243 W.V.: NR150-474

11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATEPersonnel and Training Research Programs August 1982Office of Naval Research 13. NUMBER OF PAGES

Arlington, VA 22217 2314. MONITORING AGENCY NAME & ADDRESS(i! dilferent frog Coltrolling Office) IS. SECURITY CLASS. (of this reponj

UnlasifiedIS. OECL ASSI FIC ATION/DOWNGRADING

SCHEDULE

6IS. DISTRIBUTION STATEMENT (of thl Report)

Approval for public release; distribution unlimited. Reproduction inwhole or in part is permitted for any purpose of the United StatesGovernment.

17. DISTRIBUTION STATEMENT (of the ebstract ented In Block 20 it different Imam Repot)-i

IS. SUPPLEMENTARY NOTES

IS. KEY WORDS (Continue on reverse aide if necessary a Identify b bock number)

Latent Trait ModelsGeneral Rasch ModelMultidimensional Models

20. ABSTRACT (Continue on revese aide if necesiy mad identilt by blak mnbee)

-- Several special cases of the general Rasch model, varying in complexity,were investigated to determine whether they could successfully modelrealistic multidimensional item response data. Also investigated waswhether the parameters of the model could be readily interpreted. Ofthe formulations of the model investigated, all but two were found tobe incapable of modeling realistic multidimensional item responsedata. One of the remaining formulations of the model was found tohave limited applications. The version of the model found to be most

DD I A 1473 EDITION Ol I NOV I5 1S OBSOLETES/N 0102. LF- 014- 6601 SECURITY CLASSIFICATION OF THIS PAGE (km, Date Bitoo0

........ ..... ....-.-.. .-.. .......--.-.... .

SE[CUWTY CL.ASSr4ICAT@W OF THIS PAGIC t D. ~A

- usefu1 i8 an extension of the two-parameter logistic model.

S/ 00- F01-60 ECRIYCL~~llATO O HI P6[(li~ at hxm

-,'.V. , , - ., ,.,/:" ,-', ". . , .-. *, , ,., , . .,/.-.". ,. . .. /'- . . --- , .- . o

: :j CONTENTS

~Page

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I

Multidimensional Latent Trait Models ........ ................. 1

Summary ....... ..... ............................. 4

Method ....... ....... .... ........................... 5

Design ....... ...... ................ . .......... 5Analyses. ........ ..... ............................. 5

Results ........ ...... ................................ 6

Vector Model ........ .... ........................... 6Product Term Model ........ .. ........................ 9Vector and Product Term Model ...... .................. .10Reduced Vector and Product Term Model . .... ............. .12Item Cluster Model ......... ........................ .15

Discussion ....... .... .............................. ... 19

Vector Model ........ .. ........................... .19Product Term Model ......... ....................... .20Vector and Product Term Model ...... .................. .20Reduced Vector and Product Term Model .... .............. .. 20Item Cluster Model ....... .. ........................ .20

Summary and Conclusions ......... ........................ .21

References ........ ..... .............................. 22

jNTiS GP..&I

2 DO ~ ?1~TAB

. t- .at i

IAvsta1 iyCe

Av 11 it -( 1r

* *A f

, K o , , _.:.; % .,,. ,_' . . . ., . ,, .... , .. , . . . .., . ., . . . .. .. ,..;,%, ,. .. .. : .,..,

The Use of the General Rasch Modelwith Multidimensional Item Response Data

Latent trait theory has become an increasingly popular area for researchand application in recent years. Areas of application of latent traittheory have included tailored testing (McKinley and Reckase, 1980), equating(Marco, 1977; Rentz and Bashaw, 1977), test scoring (Woodcock, 1974), andcriterion-referenced measurement (Hambleton, Swaminathan, Cook, Eignor, andGifford, 1978). While many of these applications have been successful,they are limited to areas in which the tests used measure predominantly onetrait. This limitation is a result of the fact that most latent traitmodels that have been proposed assume unidimensionality. Because of thisrequirement, in some situations latent trait models have not been successfullyapplied. For example, in achievement testing the goal is not to measure asingle trait, but to sample the content covered by instruction. Therefore,most latent trait models are inappropriate since tests designed for thispurpose generally cannot be considered to be unidimensional. Even when thegoal is to measure a single trait, if dichotomously scored items are usedno generally accepted method exists for forming unidimensional item sets,for determining the dimensionality of existing item sets, or for testingthe fit of the unidimensional model to the data.

An alternative to trying either to construct unidimensional item setsor to fit a unidimensional model to already existing item sets is to developa multidimensional latent trait model. Several such models have beenproposed (Bock and Aitkin, 1981; Mulaik, 1972; Rasch, 1961; Samejima, 1974;Sympson, 1978; Whitely, 1980), but little research has been done usingthese models. Some work has been completed on the estimation of the parametersof the Bock and Aitkin model (Bock and Aitkin, 1981), the multidimensionalRasch model (Reckase, 1972), and the Whitely model (Whitely, 1980), but noextensive research has been completed on the characteristics and propertiesof any of these models. The purpose of this paper is to present the resultsof research on the characteristics and properties of the multidimensionalRasch model. Before presenting these results, however, the multidimensionalmodels that have been proposed will be briefly discussed, as will theresearch that has investigated the characteristics of these models.

Multidimensional Latent Trait Models

Three of the multidimensional latent trait models that have beenproposed have been extensions to the multidimensional case of the unidimen-sional Rasch model (Rasch, 1960). The unidimensional Rasch model is givenby P( jexp(x 11 (8j + a1 )) (1)

P(x 1 I8leJ' 1) 1+ exp(ej + ai )

where e. is the ability parameter for Person j, (. is the item easiness3 1

parameter for Item i, and P(xij t Ojloi ) is the probability of response xij

(0 or 1) to Item i by Person j. The multidimensional model proposed byRasch (1961) is given by

1 exp[ ) (x) - - + 6jX X + p(x)I (2)

P(ibjS) ~is)_

'ii)-2-

where j, a., and P(xij I eji) are as defined above; j, , x, and p arescoring functions which are functions of x only; and y(_.,o.) is a normal-

-J -Iizing factor necessary to make the probabilities of the response alternativessum to 1.0. The scoring functions t, t, and X act as weights for theparameters, while the p term is used to adjust the scale for differentscoring procedures. Both the scoring functions and the p term depend onthe score obtained by a person on the item. In order to apply this modelto multidimensional data, 8, C-, t, and t are defined as column vectors,

t(x)'8 and t(x)a i are inner products of vectors, and X(x) is defined as a

matrix. The terms t and t now represent vectors of weights for the differentelements in the a- and 6-vectors. The [ matrix is a matrix of weights.Rasch never attempted to apply this model.

Reckase (1972) tried to apply the generalized Rasch model to real andsimulated item response data with limited success. In this study, the

0 multidimensional model fit multidimensional data no better than did theunidimensional Rasch model. However, Reckase did not include the 8. X(x)a.

term in the model, which may have resulted in the poor fit of the model tomultidimensional data. In addition, several methodological problems mayhave contributed to the poor results of the study. First, the sample sizeused to estimate the parameters of the model was relatively small for thenumber of parameters estimated. Second, in addition to estimating theparameters of the model, Reckase also estimated the values of the t andscoring functions. Finally, in order to estimate the parameter vectors,the dichotomously scored items used in the study were combined into clustersto form nominal response patterns. The most appropriate way to form theclusters was not known, which may have caused problems in the estimation ofthe parameters. Despite these difficulties, a least squares estimationprocedure was developed which did yield somewhat reasonable parameterestimates.

Hulaik (1972) also proposed a multidimensional model that is ageneralization of the Rasch model. The model proposed by Mulaik is givenby

klexp[( jk + ark)Xij]P(xi 18J'-i k- (3)

1 + Cexp(e jk + 'ik)

where 8.k is the ability parameter for Person j on Dimension k, and aik is

the difficulty parameter for Item i on Dimension k. Although lulaik neverapplied this model, he did suggest procedures for estimating the parametersof the model for three separate cases: when item responses are normallydistributed and have a common variance for all items and subjects; whenitem reponses follow a Poisson distribution; and when item responses aredichotomous.

-3-

Samejima (1974) proposed a multidimensional latent trait model that isa generalization of a different unidimensional model. The model proposedby Samejima is based on the two-parameter normal ogive model. This modelis given by

P(xle a j bl [a' (6j - bi ) 1 (4)

where O(x) is the normal distribution function, ai is a column vector of

item discrimination parameters, bi is a column vector of item difficulty

parameters, and e. is a column vector of ability parameters for Person j.-JUnfortunately, the basic derivation of this model used the continuousresponse version of the normal ogive model. Therefore, its use withdichotomous data requires that item scores be translated to the continuousscale. Since no procedure for translating item scores to the continuousscale is available, the model cannot at present be applied to dichotomousdata. Like Rasch and Mulaik, Samejima never applied this model, but onlysuggested how the parameters might be estimated.

Sympson (1978) proposed a multidimensional model based on the three-parameter logistic model. The Sympson model postulates that the probabilityof a correct response is determined by the product of the conditionalprobabilities of a correct response on each of the dimensions being measured.The three-parameter logistic model is given by

P(xij~l [0~lb~ i =-c + (1 - c i ) (5)~)1 + exp[Da (e - b )]

where 0. is the ability parameter for Person j, a. is the item discriminationJ1 1

parameter for Item i, b. is the item difficulty parameter for Item i, c. isi. i

a pseudo-guessing parameter for Item i, and D = 1.7. The three-parameterlogistic model is used to model the conditional probability for each dimension,although the ci parameter does not have a separate value for each dimension,

but rather is a scalar parameter related to the item as a whole. Themultidimensional model is given by

PM) b exp[aik(9 k - bik)xijlP(x- x 0,1 (6)

k-l I + exp[aik(0jk - bik)]

where the parameters are as defined above and m is the number of dimensions.Although Sympson has done some work on estimating the parameters of thismodel, no application of the model to multidimensional data has yet beenattempted.

The model proposed by Whitely (1980) is somewhat similar to Sympson'smodel. This model, called the multicomponent latent trait model, definesthe probability of a correct response to an item as the product of theprobabilities of performing successfully on each cognitive component of theitem. The Whitely model is given by

~-4-

m exp[(ejk - b ik)x i01P1 x, 16_ , i b f 7T , - 0 ,1 (7 )

ii ;j Zi k=1

I + exp[6jk - blk]

where all the parameters are as previously defined. It can be seen thatthis model is essentially another extension of the Rasch model. The modelfocuses on the different cognitive skills required to perform on an itemrather than the global dimensions hypothesized by Sympson. Estimationprocedures have been developed for the model and some applications havebeen made to real data. However, because of the emphasis placed on identi-fying the different cognitive skills required by an item, the applicationof this model is limited to data collected under very restricted experimentalconditions.

Bock and Aitkin (1981) have proposed a multidimensional two-parameter

normal ogive model for use with dichotomously scored response data. Thismodel is given by

P (X -± I !J,yf,aji - 0[(y, + Q -e ) / C) (8)

where Yi is the difficulty parameter for Item i, gi is a column vector of

discrimination parameters for Item i, ej is a column vector of ability

parameters for Person J, *(x) is the normal distribution function, and (Iis given by M

CT e (i E a. ak 2 ) (9)

k-l

Bock and Aitkin described a method for estimating the parameters ofthis model, and presented the results of the application of the model tothe data for the Law School Admissions Test (LSAT) presented in Bock andLieberman (1970). The results of the application of the model to the LSATindicated that a two-dimensional solution fit the data better than a one-dimensional solution. Fit was assessed using a likelihood ratio chi-squaretest.

Summary

Six different latent trait models have been proposed for use with multi-dimensional item response data. Of these six models, two are of littleinterest here. The Samejima model is uot designed for use with dichotomouslyscored item response data, and the Whitely model is appropriate only forspecial experimental conditions. Of the remaining four models, only theBock and Aitkin model and a special case of the Rasch model have beenapplied, and no attempt has been made to extensively investigate thecharacteristics and properties of any of the models. The purpose of thisresearch is to extensively investigate the characteristics of one of thosemodels, the generalized Rasch model.

-5-

Method

Design

The design of this research was to start with the most simple formulationof the multidimensional Rasch model, investigate its ability to describemultidimensional item response data, and if necessary to investigate increasinglymore complex versions of the model until good model/data fit was obtained.At each level of complexity the properties of the model were investigated,and the reasonableness and usefulness of the model were explored. This wasdone by generating test data to fit the particular form of the model beinginvestigated, and analyzing that data in an attempt to assess how well thecharacteristics of the data matched the characteristics of real data withmultidimensional characteristics.

The most general formulation of the model investigated in this researchis the model described by Rasch (1961), given by Equation 2. The simplerformulations of the model used in this research were obtained by eliminatingdifferent terms from the model statement by setting the appropriate scoringfunctions equal to zero for all item scores.

For each model statement that was obtained, simulated test data weregenerated to fit the model. Using the known parameters and model statement,predictions were made as to the dimensionality of the generated data andthe characteristics of the hypothetical items. Analyses were then performedon the simulation data in order to test the predictions. If it were foundthat a model statement could not be used to generate realistic data, interms of either dimensionality or item characteristics, then the model wasrejected, and a different model statement was investigated. This involvedaltering the terms of the general model (Equation 2) that were included andthose that were zeroed out. In some cases, all of the terms in a particularrejected model statement were retained, and one or more additional termsfrom the general model were added.

Analyses

The first analysis performed on the simulation data generated usingthe models was a factor analysis. Factor analysis, in this case, is notbeing used as a means of validating the models, but as a procedure fordetermining whether the data generated from the models have characteristicssimilar to those of real test data. All of the factor analyses performedin this research were performed using the principal components method onphi coefficients. When the obtained and expected factor structures of thedata did not match, follow-up analyses were performed in an attempt todetermine why the obtained factor structure was different from what wasexpected.

° Follow-up analyses included plotting the true item parameters againstthe factor loadings and against traditional item statistics such as proportion-correct difficulty values and point biserial discrimination values. Theseanalyses were performed using both the unrotated factor loading matrix andthe factor loading matrix rotated to the varimax criterion. The purposesof these analyses were three-fold. One purpose was to determine whether

e ' i . l',. i -. *% " , .-- ,. - N-C , '.?. '. .-..- .-... ,. ',

-6-

the obtained factor structure was a result of the model statement, thevalues used for the model parameters, or both. The second purpose was tofacilitate interpretation of the model parameters, and the third purposewas to determine whether the model yielded items with reasonable characteris-tics. In many cases it was necessary to generate additional data, usingdifferent values for the parameters of the model, in order to answer specificquestions about a particular model statement.

Using the results of all of the analyses performed for a particularmodel, a decision was made as to whether the model adequately generateddata similar to real test data. If a model statement were rejected, anattempt was made to determine from the results of the analyses what changesin the model would yield a more acceptable model.

Results

Vector Model

The first model that was investigated was a simple vector parameter

model. The j 1"X(x)ai and p(x) terms were eliminated, yielding the modelgiven by

P(xj_ = exp(q(x)'8. + (x) ) (10)Z Y(ej~) - -1 - -

- -

where all the terms are as defined for Equation 1, and 0, a., j and t are

vectors. For this model the scoring functions all took on values of onefor a correct response and zero for an incorrect response. This model wasselected first because it appeared to be a straightforward extension of theunidimensional Rasch model (Rasch, 1960) to the multidimensional case. Theexpectation was that data generated according to this model would have adimensionality that would vary with the number of elements in the parametervectors. For instance, when data were generated using two elements in boththe item and person parameter vectors, it was expected that the data wouldyield a two-factor solution when factor analyzed. This was not the case,however. Regardless of the number of elements in the parameter vectors,this model yielded one predominant factor. This was true regardless of theactual values of the parameters or the values that were used in the scoringfunctions.

Table 1 shows the first two eigenvalues from a typical principalcomponent solution for the vector model. As can be seen, there is a dominantfirst factor, with one minor factor. Table I also shows the unrotatedfactor liading matrix obtained for these particular data, as well as theproportion-correct difficulty and the inner product of the item parametervector and scoring function for each item (sum of the item parameters). Ascan be seen, there is little variation in the loadings on the first factor,while the minor factor is related to item difficulty. Factor II generallyhas positive loadings for easy items and negative loadings for hard items.Once it was ascertained that the vector model would not yield multi-factordata, it was not difficult to determine why. Equation 8 can be written as

P(xi 1 , Y(.e ,R) exP(Mj + i) (11)

" -7-

TABLE 1

IPrincipal Component Factor Loadings Based onPhi Coefficients with the Sums of the Item Parameters

and Observed Proportion Correct forTwo-Dimensional Vector Rasch Model

Item Sum of Item Observed FactorI FactorIlItem Parameters Difficulty

1 .89 .65 .57 .28

2 -.89 .33 .52 -.24

3 .43 .58 .56 .04

4 2.02 .80 .47 .29

5 .59 .60 .59 .27

6 -.91 .33 .52 -.14

7 -1.44 .24 .51 -.27

8 .47 .58 .56 .13

9 -1.05 .30 .51 -.24

10 -1.76 .20 .48 -.19

11 .98 .64 .58 .07

* 12 2.58 .88 .43 .35

13 -1.31 .25 .50 -.34

14 1.22 .71 .53 .29

15 -.64 .39 .55 -.09

16 .05 .48 .56 .06

17 -2.33 .15 .46 -.36

18 -.54 .39 .54 -.05

19 -.60 .38 .53 -.26

20 2.26 .82 .43 .39

Eigenvalues 5.42 1.18

7-8-

where a. = t(x)'. and Pi = (x)'Y" Equation 11 is the unidimensional

Rascb model, with inner products of vectors as parameters. Therefore,regardless of the values of the model parameters, as long as the innerproduct remains the same, the probability of a response is the same.Therefore, the dimensionality of the vectors is unimportant, only theproduct is critical. The model is still a unidimensional model. Thefactor analysis results typified by the solution shown in Table 1 serve asan empirical demonstration that the vector model is a unidimensional model.It can also be empirically demonstrated that the inner products of thescoring function and parameter vectors serve as parameters for the model.Figure 1 shows a plot of proportion-correct difficulty by the inner productof the scoring function and item parameter vectors, which for this case isjust the sum of the item paramaters. As can be seen, there is an almostperfect relationship between the inner products and the proportion-correctscores. When data were generated using the unidimensional Rasch model,with the inner products from the two-dimensional model as parameters,exactly the same plot was obtained.

Figure 1

Relationship Between the Proportion Correct and the Sumof the Item Parameters for the Two-Dimensional

Vector Model

0

2

0

It

0 °1

o5

-c3

% 1

o.m

01

.~

. 17

( SUM OF ITEM PRAaMETERSii Note: The numbers in the plot are the item numbers.

~-9-

Product Term Model

It was clear from the results just reported that using parametervectors in an otherwise unidimensional model did not make it a multidimensional

11 model. Therefore, the vector model was rejected as a multidimensionalmodel. The next model that was investigated contained only the 8.'X(x)a.

- - -term. This was the next model investigated because it involved more thansimple inner products of scoring and parameter vectors, but was simplerthan using both inner products and the 8jX'(x)o. term.

J

When 8. and u. are vectors, x(x) must be a matrix. The product e.a."-J -1 -J -'

represents a matrix of products of all possible pairs of the elements inthe B.- and 0.-vectors. For two-dimensional 8.- and 0.-vectors,

J 1 2 1

= (12)"'2 1° 2_2j

The X(x) matrix is a scoring matrix having an element for each elementof the 8.o. matrix. If the X(x) matrix for the matrix in Equation 12 were

x(x) = (13)0 1for a particular response x, then the numerator of the model statement forthat response would be exp(01o I + 020 2). The nonzero elements of the X(x)matrix indicate which elements of the O.o. matrix are included in the

exponent. It is clear from this that by selectively using zeros in the

_X(x) matrix, various products of . and o. elements can be selected.

Varying the values of the nonzero elements in X(x) assigns different weightsto different combinations. Thus, the product term model, given by

P(xij Ij,£1 ) a 1 exp[e 4 (x)ci] (14)J~~ ~ ~ _J -iY(1 5i

is a very rich model in terms of the number of alternative formulations ofthe exponent of the model that are available. Unfortunately, when datawere simulated using some of these alternatives, it was discovered thatthis model had an inconvenient property. Regardless of which formulationof this model was used, and regariless of what values were taken on by theitem parameters, the item proportion-correct difficulties were all approx-imately equal to each other. A closer examination of the product termmodel indicates why this occurred. Using the item parameters shown inTable 2, data were generated using

X(X 1) = (15)0. -

-10-

for a correct response, and

0)4(40) = (16)-0 0

for an incorrect response. This yields a model given by

P (X l ) exp(8 1 l + OJ2'i 2 (17),. P~xij = ll ) 1+ exp(Ojl',, + 'j2 aU)

where the 9jk and Ojk terms are elements in the 0- and o-vectors. From

Equation 17 it can be seen that the item parameters are similar to thediscrimination parameter in the unidimensional two-parameter logistic (2PL)model presented by Birnbaum (1968) since they are multipliers of the personparameters. In fact, if written as

P(xij = IO6jia) - 1 ) exP[o 1 (0jl + 0) + a 12 (a 2 + 0)] (18)

the model is essentially a two-dimensional two-parameter logistic modelwith both of the difficulty parameters equal to zero for all items. Becausethe data used for Table 2 were generated using a bivariate N(0,1) withp = 0 distribution of ability, difficulty parameters of zero yielded apredicted proportion-correct difficulty of .5.

A principal components analysis of phi coefficients yielded evidencethat the use of two item parameters resulted in a two-dimensional model.The first three eigenvalues obtained for the data generated using the itemparameter values in Table 2 were 4.0, 2.4, and .9. The role of the itemparameters as discrimination parameters in this model is indicated bycomparing the item parameters shown in Table 2 with the rotated factorloading matrix, also shown in Table 2. The correlations between the itemparameters and the factor loadings indicated that there was a strong linear

relationship between the item parameters and factor loadings (r = .98 for01 with Factor II, r = .99 for 02 with Factor I), supporting the conclusionthat 01 and 02 are acting as discrimination parameters.

Vector and Product Term Model

The vector model that was investigated first was essentially a uni-dimensional model that contained a difficulty parameter (the inner product.(x)'oi) as the only item parameter. The product term model is a multidimensional

model that contains discrimination parameters as the only item parameters.In order to obtain a multidimensional model which contained a difficultyparameter, the vector and product term models were combined. A combinationof these two models is given by

,4

4j

4 -' , , -- - . .- , . .- - . . . . ,. . ':; ,) ..' ,/ / . . ''''" . "' -,,.. ' .- ' ' .. ,-. .. ' ,

[C TABLE 2

Item Parameters, Proportion Correct Item Difficulty and Factor Loadingsfrom a Varimax Rotated Principal Components Solution on

Phi Coefficients for the Product Term Model

Item a1 02 p Factor I Factor II

1 0.150 1.150 .48 .52 .11

2 1.280 0.200 .50 .08 .56

A 3 0.260 1.350 .51 .57 .09

4 1.000 0.300 .52 .14 .52

5 0.250 1.050 .49 .47 .09

6 1.040 0.100 .51 .03 .57

7 0.110 1.150 .47 .52 -.01

8 0.200 1.200 .49 .57 .09

9 1.400 0.300 .50 .12 .58

10 0.300 1.200 .48 .54 .07

11 1..50 0.150 .51 .08 .59

12 0.400 1.200 .50 .53 .14

13 1.150 0.250 .50 .01 .52

14 0.150 1.300 .49 .61 .04

15 1.000 0.250 .51 .11 .49

16 0.100 1.400 .46 .61 .04

17 1.350 0.150 .50 -.01 .59

18 1.250 0.100 .52 -.03 .59

19 0.200 1.500 .48 .62 .03

20 1.150 0.500 .51 .25 .46

;:?~~~~......... ....... -.......... ..-.............-............ ............ .. .. ..... .

-12-

P (x ij 2i jg ) " xp W, (x'i + j '(x)W a l" (19)

As can be seen in Equation 19, the (x)'8. term was eliminated when the twomodels were combined. -3

Table 3 shows the item parameters used to generate data to fit thevector and product term model. These data were generated using two-dimensionalparameters. The scoring functions were also two-dimensional and werevectors of ones for a correct response and vectors of zeros for an incorrectresponse for all elements. Table 3 also shows the rotated factor loadingsobtained for the first two factors from the principal components analysisof phi coefficients obtained for that data. The first three eigenvaluesfrom the solution are 5.26, 2.28, and 1.07. Initial analyses indicatedthat this model could be used to model multidimensional data, and that itemdifficulties were not constant (see Table 3). However, these analyses alsoindicated that it was not realistic to use the same item parameters in boththe parameter vectors and the product term. The problem is indicated bythe magnitude of the correlation of the item proportion-correct difficultieswith the item point biserials. Because of the double role played by theitem parameters, the proportion-correct scores and point biserials had acorrelation of r = .94. That is not a very realistic situation. Therefore,this model was also rejected as a reasonable method for describing multi-dimensional item response data.

Reduced Vector and Product Term Model

Since the analyses of the vector and product term model indicated that

the same item parameters should not appear in both the parameter vectorsand the product term, the item parameter vector and the scoring functionswere altered so that parameters appeared in one or the other, but not both.In order to facilitate this, two additional elements were inserted into theitem parameter vector. For a correct response the first two elements werezeroed out of the product term, while the last two were elements werezeroed out of the vector term. This procedure results in the first twoitem parameters acting as difficulty parameters and the last two parametersacting as discrimination parameters. Although four item parameters wereused, only two dimensions were modelled in this case. For an incorrectresponse all of the parameters were zeroed out. All nonzero elements inthe scoring functions were set equal to one. The resulting model is givenby

P~x = l(ex 1J2) ( 1 O , exp[a 1l + a12 + aDa 1 + c14ej 4 ] (20)

where the 8 and u terms are elements of the corresponding vectors.

The first three eigenvalues obtained from the principal componentsanalysis for this model are 5.39, 1.30, and .99. Table 4 shows the itemparameters that were used to generate the data, as well as the factor

........................................,

-13-

TABLE 3

Item Parameters, Proportion Correct Item Difficulty,and Rotated Factor Loadings forthe Vector and Product Term Model

Item a p Factor I Factor I

1 .230 .190 .56 .63 .12

2 .880 2.180 .77 .73 -.10

3 .900 -1.920 .35 .08 .71

4 -.900 -.110 .32 .39 -.22

5 -.640 .830 .52 .58 -.35

6 -.540 -1.040 .16 .10 .31

7 1.730 -1.350 .54 .34 .65

8 .940 .630 .69 .69 .21

9 .030 -.110 .46 .56 .11

10 -1.610 -.570 .13 .01 -.37

11 -1.170 1.260 .74 .75 .05

12 -.550 -1.070 .17 .04 .22

13 -.420 -.480 .31 .32 -.01

14 .220 -.070 .53 .60 .15

15 -.020 -1.670 .21 -.04 .55

16 2.420 .370 .78 .61 .40

17 1.230 .400 .69 .68 .28

18 .250 .410 .58 .65 .08

19 .140 .760 .64 .67 -.10

20 -1.770 .550 .30 .27 -.60

.-

-14-

TABLE 4

Item Parameters and Rotated Factor Loadingsfor the Reduced Vector and Product Model

Item 2 aa3 a4 Factor I Factor II

1 .206 -.503 .373 .997 .51 .01

2 -.164 .888 1.205 1.832 .60 .32

3 .448 .261 .766 .876 .34 .36

4 .814 -.008 1.321 1.714 .55 .34

5 .111 -.908 1.344 1.216 .41 .42

6 -.947 .044 1.758 1.694 .46 .51

) 7 -.490 .111 .687 .738 .40 .24

8 .553 -.502 .347 1.454 .61 .07

- 9 -.344 .639 1.307 .127 -.06 .64

-. 10 -.257 .303 .851 .824 .26 .39

U U -.069 -.542 .472 .404 .22 .25

12 .779 .432 .392 .656 .30 .13

13 -.611 .571 .578 1.252 .59 .13

14 -.140 -1.032 .334 1.066 .60 -. 04

15 -.705 .081 .821 .480 .07 .44

16 -.386 -.164 1.912 .244 .03 .71

17 -.154 .044 1.193 .537 .16 .56

18 .474 .249 1.385 1.287 .49 .44

19 .438 -.210 1.320 1.110 .42 .45

20 .294 .190 1.634 1.492 .45 .49

-15-

,3 loadings obtained from a varimax rotation of the first two principal components.The item parameters that were used as multipliers (oY and 04) were allpositive in order to avoid having items with negative discriminations.

The results of the factor analysis of these data indicate that adominant first factor is present. However, there was a second componentpresent in the data which was strongly related to the item parameters(r = .87 for 03 and Factor II, r = .87 for 04 and Factor I). The itemparameters in the product term were related to the factor loadings, whilethe sum of the item parameters in the vector term were found to be relatedto the proportion correct difficulty. The correlation between the sum of

the parameters in the vector term and the proportion correct difficulty wasr = .98, indicating that the sum of the vector parameters act as difficultyparameters. There was not a significant correlation between the itemdifficulty and point biserial values (r = .12). The sum of 03 and 04 had acorrelation of r = .96 with the item point biserials.

The analyses of the model set out in Equation 20 indicate that it has* many desired characteristics. The rotated factor loadings are highly

related to the item parameters in the product term, the item difficulty ishighly correlated with the sum of the item parameter vector elements, andthere is no correlation between item difficulty and item discrimination.

One problem that does exist with the data that were generated is thatthe factor analysis results indicated that the data had only one predominantfactor. One possible reason for this is that so many of the items hadlarge values for both of the item parameters in the product term. In orderto test this, data were generated for the set of item parameters shown inTable 5. The eigenvalues from the principal components analysis for thesedata are 2.49, 2.28, 1.05, and 1.03. As can be seen, when using the itemparameters from Table 5 to generate data, there are two factors of approxi-mately equal magnitude present in the data.

Item Cluster Model

Although the reduced vector and product term model appears to adequatelymodel multidimensional data, the presence of the product term complicatesparameter estimation, since the separation of the item and person parametersis not possible through techniques of conditional estimation. Because ofthis, one more model that does not have a product term was investigated.This model is the item cluster model.

One of the reasons the item vector model, given by Equation 10, doesnot adequately model multidimensional data is that no information about thedifferent dimensions is preserved in the item score when the item is dichoto-mously scored. The elements for the different dimensions are sumed, andthe sums are treated as parameters. If it were possible to score thedimensions separately, then the vector model might be able to model multi-dimensional data. This requires, however, polychotomous item scoring.

.. ,.4. --:. .

5-~~~~ ~ ~ ~ ~ -CST r- 27 ..-'g 7-"- 70p. - -- -- "

-16-

TABLE 5

Item Parameters for the ReducedVector and Product Model

a i

item 0 2 03 4

1 .977 .258 1.000 .000

2 .359 .728 .000 1.000

3 -.322 .377 1.000 .000

* 4 -1.289 1.128 .000 1.000

5 -.613 .219 1.000 .000

6 1.299 .797 .000 1.000

7 .029 -.213 1.000 .000

8 -.360 -.862 .000 1.000

9 .769 -.487 1.000 .000

10 -1.447 2.092 .000 1.000

11 -1.252 -.243 1.000 .000

12 -.778 -1.426 .000 1.000

13 .668 -1.860 1.000 .000

14 2.102 -.025 .000 1.000

15 -.724 .968 1.000 .000

16 1.230 -.535 .000 1.000

17 .260 -1.216 1.000 .000

18 -1.092 -.432 .000 1.000

19 -.994 1.479 1.000 .000

20 -.206 -.525 .000 1.000

4 " " " ' 5*.. " ." ." - -, ' v "

' " " . . . . "/ -" • ' " " " - . ' " - ,- .- ," " -' i. - . .-- -

-17-

.4

Scoring an item on each dimension would require 2n response categories,where n is the number of dimensions. Unfortunately, most test data are notscored polychotomously.

An alternative to having polychotomous item scoring is to considermore than one item at a time. If two dichotomously scored items are clusteredtogether, and the cluster is treated as a single unit, then the cluster has22 or 4 response categories - (0,0), (0,1), (1,0), and (1,1). The modelgiven by Equation 10 can then be applied, with the exception that theoG-vector now represents a cluster rather than a single item, the scoringfunctions now take on values for 4 response categories instead of 2 and the

* response x is a vector with two elements. Further, the number of elementsin the o vector need not be the same as the number of items in the cluster,but rather should reflect the dimensionality of the cluster.

The procedure by which this model was investigated is as follows. For thetwo-dimensional case, item parameters were selected for 20 items. The

' items were paired so that Items 1 and 2 formed Cluster 1, Items 3 and 4

nformed Cluster 2, and so on until 10 clusters were formed. For each clusterthere were four response categories, which were scored as follows:

a) t(x) = t(x) = 181 for k equal to both items incorrect;

b) t(x) = t(x) = [?] for x equal to the first item incorrect, thesecond item correct;

c) (x) = !(x) = [ ] for x equal to the first item correct, the seconditem incorrect;

and d) t(x) = t(x) = [11 for x equal to both items correct.

For any one cluster two responses were generated, one for each dimension,using the parameters shown in Table 6. Table 6 also contains the unrotatedfactor loadings for the first two factors from a principal componentsanalysis of phi coefficients obtained for these data. The first foureigenvalues were 3.61, 3.06, 1.33, and 1.21.

As can be seen, for the factor analysis the simulation data weretreated as 20 items, rather than as 10 clusters. The eigenvalues listedabove indicate that there were two roughly equal components in the data.Table 6 shows that the first component was defined by the items that wereplaced first in the cluster, and the second component was defined by theitems that were in the second position in the cluster. Consistent with thescoring functions, there were two equal independent factors.

s In order to demonstrate that the factors need not be independent, thesame item parameters were used to generate data using the following scoringfunctions:

-18-

TABLE 6

Unrotated Factor Loadings on First Two Principal Componentsfor the Two-Dimensional Item Cluster Model

Independent Model Dependent Model

IteItem Factor I Factor II Factor I Factor IIParameters

1 .893 .56 .00 .40 .21

2 -.850 .02 .65 .33 -.36

3 -.892 .66 -.02 .38 .24

4 .690 .01 .66 .33 -.33

5 .430 .64 .00 .41 .20

6 3.200 .02 .19 .25 -.24

7 2.016 .36 -.04 .36 .16

. 8 -3.310 .07 .22 .10 -.35

9 .594 .61 -.05 .40 .33

10 .470 .00 .69 .26 -.39

11 -.913 .66 .01 .47 .30

12 1.220 .06 .55 .32 -.38

13 -1.437 .58 -.01 .37 .20

14 -1.260 .00 .58 .32 -.43

15 .467 .65 -.07 .45 .28

,, 16 .880 .04 .62 .28 -.36

17 -1.048 .66 -.04 .44 .28

18 -.970 -.02 .64 .28 -.37

19 -1.760 .56 .07 .43 .10

20 -2.140 .01 .42 .20 -.31

.J. .., JL. P. .7. P . r .. . ... , .. ... -.6. ,.-7.; -. ,--. '.r-, -- ,.-. .. , .......

-19-

a) t(x) = (x) = 181 for x equal to both items incorrect;

b) t(x) = j(x) = [ 1] for x equal to the first item incorrect,the second item'2orrect;

c) V(x) = t(x) = [>] for x equal to the first item correct,the second item'incorrect;

and d) ( (x) = [1] for x equal to both items correct.

The principal components analysis of phi coefficients for the datagenerated according to this model yielded six factors with eigenvaluesgreater than one [2.46, 1.83, 1.09, 1.08, 1.01, 1.00]. Table 6 shows theunrotated factor loadings. As can be seen, there are still two factorspresent in the data. However, the factors are no longer defined only bythe items in the corresponding position in the cluster. The first componentis a general factor, while the second component indicates the position ofthe item in the cluster. Clearly these two sets of items are not independentin this case.

Discussion

The use of simulation data to study the characteristics of a modelbefore selecting it for application is perhaps atypical of research onlatent trait models. Usually a model is adopted, estimation procedures arederived, and the model is applied without ever going through the processthis study has employed. In this study this approach has been taken fortwo main reasons. First, it was felt that when dealing with multidimensionallatent trait models much of the acquired wisdom concerning latent traitmodels might no longer apply. It was felt that considerable research wasnecessary in order to gain an understanding of how these models work andwhat the model parameters represent before they could be applied. Thisbelief has been borne out several times in this study by findings indicatingthat the models were not behaving in the anticipated manner.

A second reason for taking this approach was that it seemed impracticalto attempt to develop estimation procedures for some of these models.Specifically, the general model set out by Rasch has a very large number ofparameters. It seemed impractical to try to estimate all of them, and itwas hoped that research on the model could help simplify the estimationprocess by eliminating some terms from the model and by discovering restrictionson the range of values for the parameters. With these considerations inmind, the results of this study will now be discussed.

Vector Model

The simplest formulation of the general model that was investigatedwas the vector model. This model is simply the unidimensional Rasch model,but with vectors for parameters instead of scalars. This model was foundto be totally inadequate for modelling multidimensional data. When datawere generated according to this model, the resulting data were unidimen-sional, with item characteristics determined by the inner product of theitem parameter vectors and scoring functions. From this it follows thatthis model would fit multidimensional data no better than a unidimensionalmodel having parameters equal to the inner products from the vector model.

. .- t . .,

-20-

Product Term Model

Because of its slight similarity to Birnbaum's two-parameter logisticmodel, it was felt that the product term model would be better able tomodel multidimensional data. It was anticipated that the item parametersin the product term would behave as discrimination indices, and that is howthey did behave. Unfortunately, without the vector terms in the modelthere were no terms playing the role of difficulty parameters. The datagenerated for this model had items of constant difficulty. From this itwas concluded that this model would be useful only for modelling items ofconstant difficulty, and when items have varying difficulties this model isinappropriate.

Vector and Product Term Model

Based on the findings for the vector model and the product term model,it was hypothesized that a combination of the two models would be necessary

to model items that were both multidimensional and of varying difficulty.Analyses of the vector and product term model indicated that it would modelmultidimensional data, and that it would model items of varying difficulty.However, it was also found that, as long as the item parameter vectorelements appeared both in the vector terms and in the product term, theitem difficulties and disciminations would be highly correlated. Sincethis is rarely the case in real test data, it was concluded that this modelwould be useful only in a very limited number of circumstances.

*t Reduced Vector and Product Term Model

In order to overcome the deficiencies of the vector and product termmodel, it was clear that a given item parameter vector element shouldappear only in the vector term or the product term, but not both. It wasanticipated that similar problems might exist if the person parametervector elements occurred in both the vector term and the product term, sothe same restriction was placed on the person parameters as was placed onthe item parameters.

The resulting model appears to be quite successful at modelling realisticmultidimensional data. It is capable of modelling correlated as well asindependent dimensions, and the item parameters are readily interpretable.The only real problem there seems to be with this model is with the estimationof the parameters. Although there are fewer parameters to estimate than isthe case with the general model, there are still a fair number to estimate.Moreover, there are no observable sufficient statistics for the parametersin the product term. These problems do not make estimation of the modelparameters impossible, and probably not even impractical. However, they domake estimation more difficult.

Item Cluster Model

The item cluster model was proposed as an alternative to the vectormodel. This model does not involve a product term, but it still cansuccessfully model multidimensional data. However, it % oes involve clustering

,..

-

: : . . , . ', N , '-.''o ''"',. -,'." ". ""-. " .-.- " .. "' ..

-21-

items, which gives rise to a number of new problems. For instance, as yetit is unclear what the effect is of forming different combinations ofitems, or whether all items should be clustered with the same item.Preliminary investigations seem to indicate that the optimal clusteringprocedure is to cluster all items on a subtest with one item taken from adifferent subtest. Another alternative, which has not been explored, is to

%! apply the model only in situations where items are already clustered, suchas is the case with passage units. Clearly more research is needed on this

type of application of the item cluster model.

Summary and Conclusions

The purpose of this study was to investigate the application of thegeneral Rasch model to multidimensional data. Several formulations of themodel, varying in complexity, were investigated to determine whether theycould successfully model realistic multidimensional data. Also investigatedwas whether the parameters of the models could be readily interpreted. Themodels investigated included: a) the vector model; b) the product termmodel; c) the vector and product term model; d) the reduced vector andproduct term model; and, e) the item cluster model.

Of the models investigated, all but the reduced vector and productterm model and the item cluster model were rejected as being incapable ofmodelling realistic multidimensional data. The item cluster model appearsto be a useful model, but its applications may be limited in scope. Of themodels studied, the reduced vector and product term model was found to bethe most capable of modelling realistic multidimensional data. Althoughthe estimation of the parameters of the reduced vector and product term

* model may be more difficult than it would be for other models, this modelappears to be the model that is most worth pursuing.

.4

A..

4.

-22-

REFERENCES

,irnbaum, A. Some latent trait models and their use in inferring anexaminee's ability. In F.M. Lord and M.R. Novick, Statisticaltheories of mental test scores. Reading, MA: Addison-Wesley, 1968.

* . Bock, R.D., and Aitkin, M. Marginal maximum likelihood estimation of item

paramaters: An application of an EM algorithm. Psychometrika,1981, 46, 443-459.

Bock, R.D., and Lieberman, M. Fitting a response model for n dichotomoslyscored items. Psychometrika, 1970, 35, 179-197.

Hambleton, R.K., Swaminathan, H., Cook, L.L., Eignor, D.R., andGifford, J.A. Developments in latent trait theory, models, technicalissues, and applications. Review of Educational Research, 1978,48, 467-510.

Marco, G.L. Item characteristic curve solutions to three intractabletesting problems. Journal of Educational Measurement, 1977, 14,139-160.

McKinley, R.L., and Reckase, M.D. A successful application of latenttrait theory to tailored achievement testing (Research Report80-1). Columbia: University of Missouri, Department of EducationalPsychology, February 1980.

Mulaik, S.A. A mathematical investigation of some multidimensional Raschmodels for psychological tests. Paper presented at the annual meetingof the Psychometric Society, Princeton, NJ, March, 1972.

Rasch, G. On general laws and the meaning of measurement in psychology.Proceedings of the Fourth Berkeley Symposium on MathematicalStatistics and Probability, Berkeley: University of California Press,1961, 4, 321-334.

Rasch, G. Probabilistic models for some intelligence and attainment tests.Copenhagen: Danish Institute for Educational Research, 1960.

Reckase, M.D. Development and application of a multivariate logisticlatent trait model (Doctoral dissertation, Syracuse University,1972). Dissertation Abstracts International, 1973, 33. (UniversityMicrofilms No. 73-7762)

Rentz, R.R., and Bashaw, W.L. The National Reference Scale for Reading:An application of the Rasch model. Journal of Educational Measurement,1977, 14(2), 161-180.

Samejima, F. Normal ogive model for the continuous response level inthe multidimensional latent space. Psychometrika, 1974, 39,111-121.

'. a-t. .....- . ... -* .-. .. .- . .

-23-

Sympson, J.B. A model for testing with multidimensional items. InD.J. Weiss (Ed.), Proceedings of the 1977 Computerized AdaptiveTesting Conference. Minneapolis: University of Minnesota, 1978.

Whitely, S.E. Measuring aptitude processes with multicomponent latenttrait models (Technical Report No. NIE-80-5). Lawrence: Universityof Kansas, Department of Psychology, July 1980.

Woodcock, R.W. Woodcock Reading Mastery Tests. Circle Pines, Minnesota:American Guidance Service, 1974.

.,. ... . . .

. .... +. ... -. . -. -. -

Navy Navy

Dr. Ed Aiken 1 Dr. Jim HollanNavy Personnel R&D Center Code 304San Diego, CA 92152 Navy Personnel R & D Center

San Diego, CA' 921521 Meryl S. Baker

NPRDC 1 Dr. Norman J. KerrCode P309 Chief of Naval Technical TrainingSan Diego, CA 92152 Naval Air Station Memphis (75)

. Millington, TN 380541.Dr. Jack R. Borsting

' Provost & Academic Dean 1 Dr. William L. MaloyU.S. Naval Postgraduate School Principal Civilian Advisor forMonterey, CA 93940 Education and Training

Naval Training Command, Code OOADr. Robert Breaux Pensacola, FL 32508Code N-711NAVTRAEQUIPCEN 1 CAPT Richard L. Martin, USNOrlando, FL 32813 Prospective Commanding Officer

USS Carl Vinson (CVN-70)Chief of Naval Education and Training Newport News Shipbuilding and Drydock Co

Liason Office Newport News, VA 23607Air Force Human Resource LaboratoryFlying Training Division 1 Dr. James McBrideWILLIAMS AFB, AZ 85224 Navy Personnel R&D Center

San Diego, CA 92152CDR Mike CurranOffice of Naval Research 1 Dr William Montague800 N. Quincy St. Navy Personnel R&D CenterCode 270 San Diego, CA 92152Arlington, VA 22217

1 Mr. William NordbrockDR. PAT FEDERICO Instructional Program DevelopmentNAVY PERSONNEL R&D CENTER Bldg. 90SAN DIEGO, CA 92152 NET-PDCD

Great Lakes Naval Training Center,Mr. Paul Foley IL 60088Navy Personnel R&D CenterSan Diego, CA 92152 1 Ted M. I. Yellen

Technical Information Office, Code 2011 Dr. John Ford NAVY PERSONNEL R&D CENTER

Navy Personnel R&D Center SAN DIEGO, CA 92152San Diego, CA 92152

1 Library, Code P201LDr. Patrick R. Harrison Navy Personnel R&D CenterPsychology Course Director San Diego, CA 92152LEADERSHIP & LAW DEPT. (7b)DIV. OF PROFESSIONAL DEVELOPHMENT 1 Technical DirectorU.S. NAVAL ACADEMY Navy Personnel R&D CenterANNAPOLIS, MD 21402 San Diego, CA 92152

','

Navy Navy

6 Com. anding Officer 1 Dr. Worth Scanland, DirectorNaval Research Laboretory Research. Development, Test & FvaluationCode 2627 N-5Washington, DC 20390 Nava] Education and Training Command

NAS, Pensacola, FL 32508. Psychologist

ONR Branch Office 1 Dr. Robert G. SmithBldg 114, Section D Office of Chief of Naval Operations666 Summer Street OP-987HBoston, 14A 02210 Washington, DC 20350

1 Office of Naval Research 1 Dr. Alfred F. SmodeCode 437 Training Analysis & Evaluation Group800 N. Quincy SStreet (TAEG)Arlington, VA 22217 Dept. of the Navy

Orlando, FL 328131-Office of Naval Research

Code 441 1 Dr. Richard Sorensen800 N. Quincy Street Navy Personnel R&D CenterArlington, VA 22217 San Diego, CA 92152

5-Personnel & Training Research Programs 1 W. Gary Thomson(Code 458) Naval Ccean Systems Center

Office of Naval Research Code 7132Arlington, VA 22217 San Diego, CA 92152

Psychologist 1 Roger Weissinger-BaylonONR Branch Office Department of Administrative Sciences1030 East Green Street Naval Postgraduate SchoolPasadena, CA 91101 Monterey, CA 93940

1 Office of the Chief of Naval Operations 1 Dr. Ronald WeitzmanResearch Development & Studies Branch Code 54 WZ

(OP-115) Department of Administrative SciencesWashington, DC 20350 U. S. Naval Postgraduate School

Monterey, CA 93940LT Frank C. Petho, MSC, USN (Ph.D)Selection and Training Research Division 1 Dr. Robert WisherHuman Performance Sciences Dept. Code 309Naval Aerospace Medical Research Laborat Navy Personnel R&D CenterPensacola, FL 22508 San Diego, CA 92152

Dr. Bernard Rimland (03B) 1 Mr John H. WolfeNavy Personnel R&D Center Code P310San Diego, CA 92152 U. S. Navy Personnel Research and

Development CenterSan Diego, CA 92152

Army Army

1 Technical Director 1 Dr. Robert SasmorU. S. Army Research Institute for the U. S. Army Research Institute for the

Behavioral and Social Sciences Behavioral and Social Sciences5001 Eisenhower Avenue 5001 Eisenhower Avenue

. Alexandria, VA 22333 Alexandria, VA 22333

'.1,Mr. James Baker 1 CommandantSystems Manning Technical Area US Army Institute of AdministrationArmy Research Institute Attn: Dr. Sherrill

5001 Eisenhower Ave. FT Benjamin Harrison, IN 46256

Alexandria, VA 22333 1. Dr. Joseph Ward

1 Dr. Beatrice J. Farr U.S. Army Research Institute

U. S. Army Research Institute 5001 Eisenhower Avenue

5001 Eisenhower Avenue Alexandria, VA 22333Alexandria, VA 22333

1 Dr. Myron Fischl Air ForceU.S. Army Research Institute for the

Social and Behavioral Sciences.-. 5001 Eisenhower Avenue- w1 Air Force Human Resources Lab

Alexandria, VA 22333,., AFHRL/tPPD

. 1 Dr. Michael Kaplan Brooks AFB, TX 78235

U.S. ARMY RESEARCH INSTITUTE

5001 EISENHOWER AVENUE 1 U.S. Air Force Office of Scientific

ALEXANDRIA, VA 22333 ResearchLife Sciences Directorote, NL

" 1 Dr. Milton S. Katz Bolling Air Force Base

* Training Technical Area Washington, DC 20332

, U.S. Army Research Institute 1 Dr. Earl A. Alluisi5001 Eisenhower Avenue HQ, AFHRL (AFSC)Alexandria, VA 22333 Brooks AFB. TX 78235

1 Dr. Harold F. O'Neil, Jr. 1 Dr. Alfred R. FreglyAttn: PERI-OK AFOSR/NL, Bldg. 410]Army Research Institute Rolling AFB5001 Eisenhower Avenue Washng AFBAlexandria, VA 22333 Washington, DC 20332

1 DR. JAMES L. RANEY 1 Dr. Genevieve Haddad

U.S. ARMY RESEARCH INSTITUTE Program Manager

5001 EISENHOWER AVENUE Life Sciences Directorate

ALEXANDRIA, VA 22333 AFOSRBolling AFB, DC 2033:2

ir. Robert Ross 1 David R. Hunter" U.S. Army Research Institute for the AFHRL/11OAM

Social Pnd Behavioral Sciences Brooks AFB, TX 78235;' 5001 Eisenhower Avenue

Alexandria, VA 22333

*t*.. . .. .. .......

Air Force CoastGuard

1 Research and Measurment Division Chief, Psychological Reserch BranchResearch Branch, AFMPC/MPCYPR U. S. Coast Guard (G-P-1/2/TP42)Randolph AFB, TX 78148 Washington, DC 20593

1 Dr. Malcolm Ree Mr. Thomas A. Warm

AFHRL/MP U. S. Coast Guard InstituteBrooks AFB, TX 78235 P. 0. Substation 18BOklahoma City, OK 73169

2 3700 TCHTW/TTGH Stop 32Sheppard AFB, TX 76311

Other DoD1 Dr. Joe Ward, Jr.

AFHRL/MPMDBrooks AFB, TX 78235 12 Defense Technical Information Center

Cameron Station, Bldg 5*' Alexandria, VA 22?14

Attn: TCMarines

1 Dr. William GrahamTesting Directorate

1 H. William Greenup MEPCOM/MEPCT-PEducation Advisor (E031) Ft. Sheridan, IL 60037Education Center, MCDECQuantico, VA 22134 1 Director, Research and Data

OASD(MRA&L)1 Director, Office of Manpower Utilization 3B919, The Pentagon

HO, Marine Corps (MPU) Washington, DC 20301BCB, Bldg. 2009

SQuantico, VA 22134 1 Military Assistant for Training and

1 Headquarters, U. S. Marine Corps Personnel TechnologyOffice of the Under Secretary of Defense

Code MPI.20 for Research & EngineeringWashington, DC 20380 Room 3D129, The Pentagon

Washington, DC 203011 Special Assistant for Marine

Corps Matters 1 Dr. Wayne SellmanCode lOOM Office of the Assistant SecretaryOffice of Naval Research of Defense (MRA & L)800 N. Quincy St. 2B269 The PentagonArlington, Vh 22217 Washington, DC 20301

1 Major Michael L. Patrow, USMC 1 DARPASHeadquarters, Marine Corps 1400 Wilson Blvd.

(Code MPI-20) Arlington, VA 22209Washington, DC 20380

1 DR. A.L. SLAFKOSKY

* SCIENTIFIC ADVISOR (CODE RD-1)HQ, U.S. MARINE CORPSWASHINGTON, DC 20380

1

" -~~~~~~~~~~~~~~~~~~. . ...... .... ... .,•, .- .- .- .- .- ...............-.... ,

.A~~~ ~.. .. .. . .. . . . .. . . .

Civil Govt Non Govt

1 Dr. Paul G. Chapin 1 Dr. James AlginaLinguistics Program University of FloridaNational Science Foundation Gainesville, FL 32611Washington, DC 20550

S1 Dr. Erling P. Andersen1 Dr. Susan Chipman Department of Statistics

Learning and Development Studiestraede 6National Institute of Education 1455 Copenhagen1200 19th Street NW DENMARKWashington, DC 20208

1 Anderson, Thomas H., Ph.D.I Dr. John Mays Center for the Study of Reading

National Institute of Education 174 Children's Research Center1200 19th Street NW 51 Gerty DriveWashington, DC 20208 Champiagn, IL 61820

1 Dr. Arthur Melmed 1 Dr. John AnnettNational tntitute of Education Department of Psychology1200 19th Street NW University of WarwickWashington, DC 20208 Coventry CV4 7AL

ENGLANDDr. Andrew R. MolnarScience Education Dev. 1 1 psychological research unit

and Research Dept. of Defense (Army Office)National Science Foundation Campbell Park OfficesWashington, DC 20550 Canberra ACT 2600, Australia

Dr. H. Wallace Sinaiko 1 Dr. Alan BaddeleyProgram Director Medical Research CouncilManpower Research and Advisory Services Applied Psychology UnitSmithsonian Institution 15 Chaucer Road801 North Pitt Street Cambridge CB2 2EFAlexandria, VA 22314 ENGLAND

" Dr. Vern W. Urry 1 Ms. Carole A. BagleyPersonnel R&D Center Minnesota Educational ComputingOffice of Personnel Management Consortium1900 E Street NW 2354 Hidden Valley Lane

* Washington, DC 20415 Stillwater, MN 55082

1 Dr. Frank Withrow 1 Dr. Isaac BejarU. S. Office of Education Educational Testing Service400 Maryland Ave. SW Princeton, NJ 08450Washington, DC 20202

1 Capt. J. Jean BelangerDr. Joseph L. Young, Director Training Development DivisionMemory & Cognitive Processes Canadian Forces Training SystemNational Science Foundation CFTSHQ, CFB TrentonWashington, DC 20550 Astra, Ontario KOK 1B

Mon Govt Non Govt

1 Dr. Fenucha Birenbaum 1 Dr. Norman Cliff

School of Education Dept. of PsychologyTel Aviv University Univ. of So. CaliforniaTel Aviv, Ramat Aviv 69978 University ParkIsrael Los Angeles, CA 90107

Dr. Werner Pirke 1 Dr. William E. CoffmanDezWPs im Streitkraefteamt Director, !owa Testing ProgramsPostfach 20 50 03 334 Lindquist CenterD-5300 Bonn 2 University of IowaWEST GERMANY Iowa City, IA 52242

1 Dr. R. Darrel Bock 1 Dr. Meredith P. Crawford

Department of Education American Psychological AssociationUniversity of Chicago 1200 17th Street, N.W.Chicago, IL 60637 Washington, DC 20036

1 Liaison Scientists 1 Dr. Hans Crombag

Office of Naval Research, Education Research CenterBranch Office , London University of LeydenBox 39 FPO New York 09510 Boerhaavelaan 2

2334 EN LeydenI Dr. Robert Brennan The NETHERLANDS

American College Testing ProgramsP. 0. Box 168 1 Dr.,Fritz DrasgowIowa City, IA 52240 Yale School of Organization and Manageme

". Yale University1 DR. JOHN F. BROCK Box 1A

Honeywell Systems & Research Center New Haven, CT 06520(MN 17-2318)

2600 Ridgeway Parkway 1 Mike DurmeyerMinneapolis, MN 55413 Instructional Program Development

Building 90"1 DR. C. VICTOR BUNDERSON NET-PDCD

WICAT INC. Great Lakes NTC, IL 60098UNIVERSITY PLAZA, SUITE 101160 SO. STATE ST. 1 ERIC Facility-Acquisitions

OREM, UT 84057 4833 Rugby AvenueBethesda, MD 20014

1 Dr. John B. Carroll

Psychometric Lab 1 Dr. Benjamin A. Fairbank, Jr.Univ. of No. Carolina McFann-Gray & Associates, Tnc.Davie Hall 013A 5825 CallaghanChapel Hill, NC 27514 Suite 225

San Antonio, Texas 7822F

.4

.

• ...

Non Govt Non Govt

Dr. Leonard Feldt 1 Dr. Delwyn HarnischLindquist Center for Measurment University of IllinoisUniversity of Iowa 242b EducationIowa City, IA 52242 Urbana, IL 61801

Dr. Richard L. Ferguson 1 Dr. Chester HarrisThe American College Testing Program School of EducationP.O. Box 168 University of CalifornipIowa City. IA 52240 Santa Barbara, CA 93106

Dr. Victor Fields 1 Dr. Dustin H. HeustonDept. of Psychology Wicat, Inc.Montgomery College Box 986Rockville, MD 20850 Orem, UT 84057

Univ. Prof. Dr. Gerhard Fischer 1 Dr. Lloyd HumphreysLiebiggasse 5/3 Department of PsychologyA 1010 Vienna University of IllinoisAUSTRIA Champaign, IL 61820

Professor Donald Fitzgerald 1 Dr. Steven HunkaUniversity of New England Department of EducationArmidale, New South Wales 2351 University of AlbertaAUSTRALIA Edmonton, Alberta

CANADADR. ROBERT GLASERLRDC 1 Dr. Earl HuntUNIVERSITY OF PITTSBURGH Dept. of Psychology3939 O'HARA STREET University of WashingtonPITTSBURGH, PA 15213 Seattle, WA 98105

Dr. Daniel Gopher 1 Dr. Jack HunterIndustrial & Management Engineering 2122 Coolidge St.Technion-Israel Institute of Technology Lansing, MI 48906HaifaISRAEL 1 Dr. Huynh Huynh

College of EducationDr. Bert Green University of South CarolinaJohns Hopkins University Columbia, SC 29208Department of PsychologyCharles & 34th Street 1 Professor John A. KeatsBaltimore, MD 21218 University of Newcastle'-: AUSTRALIA 2308

Dr. Ron Hambleton

School of Education 1 Mr. Jeff KeletyUniversity of Massechusetts Department of Instructional TechnologyAmherst, MA 01002 University of Southern California

Los Angeles, CA 92007

Non Govt Non Govt

1 Dr. Stephen Kosslyn 1 Dr. Scott Maxwell

Harvard University Department of PsychologyDepartment of Psychology University of Houston33 Kirkland Street Houston. TX 77004Cambridge, MA 021?6 1. Dr. Samuel T. Mayo

1 Dr. Marcy Lansman Loyola University of Chicago

Department of Psychology. I 25 820 North Michigan Avenue

University of Washington Chicago, IL 60611Seattle, WA 98195

1 Professor Jason Millman1 Dr. Alan Lesgold Department of Education

Learning R&D Center Stone HallUniversity of Pittsburgh Cornell UniversityPittsburgh, PA 15260 Ithaca, NY 14853

1 Dr. Michael Levine 1 Dr. Allen Munro

Department of Educational Psychology Behavioral Technology Laboratories

210 Education Bldg. 1845 Elena Ave., Fourth Floor

University of Illinois Redondo Beach, CA 90277

Champaign, IL 61801 1 Dr. Melvin R. Novick

1 Dr. Charles Lewis 356 Lindquist Center for Measurment

Faculteit Sociale Wetenschappen University of IowaRijksuniversiteit Groningen Iowa City, IA 52242

Oude Boteringestraat 239712GC Groningen 1 Dr. Jesse Orlansky

Netherlands Institute for Defense Analyses400 Army Navy Drive

1 Dr. Robert Linn Arlington, VA 22202College of EducationUniversity of Illinois 1 Wayne M. Patience

Urbana, IL 61801 American Council on EducationGED Testing Service, Suite 20

1 Dr. Frederick M. Lord One Dupont Cirle, NW

Educational Testing Service Washington, DC 20036*Princeton, NJ 08540

1 Dr. James A. Paulson

1 Dr. James Lumsden Portland State University

Department of Psychology P.O. Box 751University of Western Australia Portland, OR 97207Nedlands W.A. 6009AUSTRALIA 1 MR. LUIGI PETRULLO

2431 N. EDGEWOOD STREET

1 Dr. Gary Marco ARLINGTON, VA 22207

Educational Testing ServicePrinceton, NJ 08450

t4

,%

Non Govt Non Govt

Dr. Steven E. Poltrock 1 Dr. J. RyanDepartment of Psychology Department of EducationUniversity of Denver University of South CarolinaDenver,CO 80208 Columbia, SC 2920q

DR. DIANE M. RAMSEY-KLEE I PROF. FUMIKO SAMEJIMAR-K RESEARCH & SYSTEM DESIGN DEPT. OF PSYCHOLOGY3947 RIDGEMONT DRIVE UNIVERSITY OF TENNESSEEMALIBU, CA 90265 KNOXVILLE, TN 37916

MINRAT M. L. RAUCH 1 Frank L. SchmidtPII 4 Department of PsychologyBUNDESMINISTERIUM DER VERTEIDIGUNG Bldg. GGPOSTFACH 1328 George Washington UniversityD-53 BONN 1, GERMANY Washington, DC 20052

1-Dr. Mark D. Reckase 1 Dr. Kazuo Shigemasu

Educational Psychology Dept. University of Tohoku

University of Mi3souri-Columbia Department of Educational Psychology4 Hill Hall Kawauchi, Sendai 980Columbia, MO 65211 JAPAN

1 Dr. Lauren Resnick 1 Dr. Edwin ShirkeyLRDC Department of PsychologyUniversity of Pittsburgh University of Central Florida3939 O'Hara Street Orlando, FL 32816Pittsburgh, PA 15213

1 Dr. Richard SnowMary Riley School of EducationLRDC Stanford UniversityUniversity of Pittsburgh Stanford, CA 943053939 O'Hara StreetPittsburgh, PA 15213 1 Dr. Kathryn T. Spoehr

Pscyhology DepartmentDr. Leonard L. Rosenbaum, Chairman Brown UniversityDepartment of Psychology Providence, RI 02912Montgomery CollegeRockville, MD 20850 1 DR. PATRICK SUPPES

INSTITUTE FOR MATHEMATICAL STUDIES INDr. Ernst Z. Rothkopf THE SOCIAL SCIENCESBell Laboratories STANFORD UNIVERSITY600 Mountain Avenue STANFORD, CA 94305Murray Hill, NJ 07974

1 Dr. Hariharan Swaminathan1 Dr. Lawrence Rudner Laboratory of Psychometric and

403 Elm Avenue Evaluation ResearchTakoma Park, MD 20012 School of Education

University of MassachusettsAmherst, MA 01003

.4

% w'. . ."• ."" -" -, .". " .. ''' .-.... , .. ,-.-.-. ..-.-.-... .- -.... ". ...- ,- ...

Non Govt

Dr. Brad SympsonPsychometric Research GroupEducationnl Testing ServicePrinceton, NJ 08541

*- 1 Dr. Kikumi T3tsuoka

Computer Based Education ResearchLaboratory

252 Engineering Research LaboratoryUniversity of IllinoisUrbana, IL 61801

Dr. David Thissen

Department of PsychologyUniversity of KansasLawrence, KS 66044

* 1. Dr. Douglas Towne

Univ. of So. CaliforniaBehavioral Technology Labs184I5 S. Elena Ave.

" Redondo Beach, CA 90277

Dr. Robert TautakawaDepartment of StatisticsUniversity of MissouriColumbia, MO 65201

Dr. David ValeAssessment Systems Corporation2395 University AvenueSuite 306St. Paul, MN 55114

* 1. Dr. Howard WainerDivision of Psychological StudiesEducational Testing ServicePrinceton, NJ 08540

DR. SUSAN E. WHITELYPSYCHOLOGY DEPARTMENTUNIVERSITY OF KANSASLAWRENCE, KANSAS 660441

* 1 Wolfgang WildgrubeStreitkraefteamtBox 20 50 03D-5300 Bonn 2

"S

:. ,:''G .' ,',T h' ' "., ' . .-. .'.- '.- "J ," ".$".-.. ,. .

. . . . . . ..,. ...._ .. .- . .*,, , , , -,..,', *, ,.-.. . . . . *: , . .. .. >* .,-;,.