RD-Ri47 OCT UNCLSSIFIED NL mhhhhhhmhmml · 46 @M*14IW ~ WAtA mm-mmSil ifme blm CAnmIM MOMS 116...

RD-Ri47 9i5 CONJUNCTIVE AND DISJUNCTIVE ITEMI RESPONSE FUNCTIONS(U) i/iEDUCATIONAL TESTING SERVICE PRINCETON NJ F MI LORDOCT 84 N@@@i4-83-C-8457

UNCLSSIFIED F/0 12/ NL

mhhhhhhmhmml

I u A 2.8 2

'a.0

I1.25 MIA11- i l

MICROCOPY RESOLUTION4 TEST CHART

*hIOhi VW V S?7-S %

% %sIL.

;el

NEW

-- Z:, ~ ~ 7 17 *i. .7 _ 7 .77-1

UNCLASSIFIEDSa:jn*vv Z;.ASSVrCA-Q*Of 01 040 Oftsm Dhat as~.I~

REPOR DOCM~t4T~l~REPAG 24 IRhUCflOSREPRT OCMENATON AG SUCRE COMPLETDCG ?owd

1L11111Smg GOYr AccasSION Z6~ 2. *cM~tI s CAA4.OO aUUS*

Cojncieand Disjunctive Item Response Technical ReportFunctions 5 sraaa u.uormml

7. AUTIOG~fe. CWNTRA6T Olt GRANT 160,11410)Frdei M. Lord

Fredeic N.Lord 00014-83-C-0457

S. 109@NAVO1110 4044AMI&AION HMS AUGO AOOMIS I P1 GAMSLS4g. 0 :UI?. A=

Educational Testing ServicePrinceton, NJ 08541 NR 150-520

I1. CMOGmOI.6l3MS OFFCE NMAAlsO oC IL S. RIT OATS

Personnel and Training Research Programs Otobner 1QRI.

Off ice of Naval Research 25MUI O ~I46 @M*14IW ~ WAtA mm-mmSil ifme blm CAnmIM MOMS 116 MOUUMt?? CLASS. (ad Wa ie

Unclassified

Is.S GINOUTlON SAFIUU (of meAmp

Approved for public release; distribution unlimited.

M IT ol sUSSft SVAT2111"? Wq 4164M11 a""" so a" M If toen 1MM aesaw

Is. SUPOI.6"IMsANY NOT"

it. 49V WOOM (CenMil 41 ....... sft. M 01eg0OaPI G- 8~Vl or Wll& own"

Item Response TheoryFunctional Equations

MO LOSYRAC? CMUSUOt sus "WOMad MI UfeW NE OW Moe or OlAMMGiven that the examinee knows the answer to item i if and only if he

knows the answer to both item g and item h , a 'conjunctive' item responsemodel is found such that items g , h , and i all have the same mathemati-cal form of response function. Since such items may occur in practice, it isdesirable that item response models satisfy this condition. For models withtwo parameters per item, the most general functional form satisfying thiscondition is found. A third, 'guessing' parameter may be added. The cor-

reanoninm diiunc~iva moel1is lso drived.00j~31473 com030' oM03 1 o otoeon UNCLASSIFIED

SN ofa103. 6% *1*. &do I 561,MIVpAIIICAVION OF 7365 -paS (l e.-

Ites Response Functions "

I

Conjunctive and Disjunctive Item Response Functions

Frederic M. Lord

Educational Testing Service

October 1984

Accession For

NTIS GRA&IDTIC TABUnannounced 0Justificatio

ByDistribution/

Availability CodesAvail and/or

D Dist Special

"" ',ll -

.1%I'

%'.4,'

.. ::

* Item Response nctions

2

Conjunctive and Disjunctive Item Response Functions

Abstract

Given that the examinee knows the answer to Item i if and

only if he knows the answer to both Item g and item h , a

'conjunctive' Item response model Is found such that items g , h

and i all have the smine mathematical form of response function.

Since such items may occur in practice, it is desirable that item

response models satisfy this condition. For models with two

parameters per item, the most general functional form satisfying

this condition is found. A third, 'guessing' parameter may be

added. The corresponding disjunctive model is also derived.

' .''.

• .1

5-.

."' ft.;z),'o

item Response Functions

3

Conjunctive and Disjunctive Item Response Functions*

Consider two free-response spelling items that ask the

examinee to spell, respectively, the word wife and the word house.

Now consider a free-response spelling item that asks the examinee

to spell the word housewife. Assume that an examinee will answer

the third item correctly if and only If the examinee would answer

both of the first two items correctly. If so, then

P3 (6) = PO)P2() (1)

where 8 is the ability of the examinee and P1(0) is the item

response function for (probability of giving a correct answer to)

item I •

Kristof (1968) pointed out that it is highly desirable that

P , and P3 should all have the same mathematical form,

but that this condition is not satisfied by the usual logistic or

normal ogive models In Item response theory (IRT). He derived one-

parameter families of Item response function, P(6) , satisfying

(1), stating that this result "is attainable If and only If all

Item (response] functions are powers of each other.... The model

*This work was supported in part by contract H00014-83-C-0457,

project designation NR 150-520 between the Office of Naval Researchand Educational Testing Service. Reproduction in whole or in partIs permitted for any purpose of the United States Government.

Z. ".' -Z Z %.CZZz

F'".... . . . . .

Item Response Functions C

4

rules out normal ogives as well as logistic functions as possible

item [response] functions.*.60C.

more, recently Yen (1964) discussed certain difficulties

encountered In vertical equating usIng a logistic model. She

"hypothesized that [these difficulties I occur because the items

Increase In complexity as they Increase In difficulty." Such

problems might be avoided If the model P(O) satisfying (1) were

used.

In Kristof' a study,* each item is characterized by a single

item parameter. Here, Kristof's result Is generalized, using a

different approach, so as to allow Items to differ in each of two

paramters.

Necessary Condition for Solving the Functional Equation-S*A

Suppose the Item response function for any itemn has the form

P(a*,b 59 6) : it is a function of 0 characterized by two iLemn

parameters, &* and b* . Denote the values of a* and b* for

iteml1by a and b ; for Item 2, by A and 3; for Ite 3 by

a and p*Then (1) becomes

* P(a,b,O)?(A,3,9) a P(*,O,S) .(2)

If such an idenity isto hold for all 0, a, b A, 3,

a ,and pit is necessary that the Item paramters a and

..........

Z!g tZ-' Y;

Item Rsponse Functions

5

be functions of a , b A A, and B (item parmeters by

definition do not depend on 0 ):

a Z (a,bA,B) and 0 0(a~b,AB) .(3)

Define L(a*,b*,O) log P(a*.b*,O) .From (2) .

L(a,b,8) + L(A,BO)B L(a,0,8) . (4)

This is a functional equation In five variables and three unknown

functions, L( a ( ), and 0()

Take the derivative of this with respect to a ,to b ,and

to A:

La(a,b,6) L0 LO,)~ (a L 0(SAP,)0a ,(5)

Lb(a,b,O) Lo,.ab+ LO(CL.S0)0

L A(A,D,O) L(,P,8)A + JP(G,O,9)OA

where each superscript denotes a derivative: for example,

a a a(ab,A,B)/Oa . Continuity and differentiability of

functions are assumed as needed here and in the following derivations.

Given any fixed set of values of the three variables a , 0

and 0 , both L(,0,0) and LO(m,0,6) are fixed. Equations (5)

are then three nonhomogeneous linear equations relating L

I t Ct W,4& l%% Z " -'Z e Z Z Z

Item Response Functions

6

and LO . These three equations will be Inconsistent unless the

augmented atrix

[ (a.b,A,) b ,(a.b.A) Le(ab,6)

b bba (a,b,A,B) 0 (abA,B) Lb(a.b,O)

LA(abAB) 0 A(abA,3) LA(ab.e)

Is siglar. Theefore the corresponding determinant mset vanish:

€',S'- .bSa)LA +. .A)La (b+ ,. _ .A)Lb a o

Reurite this

b,,.b.AB)e a f(a.b.A,2)L. 4 s(+.b&.a5, B)L . (6)

This remalt esesiUisbs the elanatm L nd LO fram

Immediate esieideratLon.

Given any fixed set of values of a, b ,d , both

L8 sad Lb are fixed. ition (6) amproooe a linear relatiouship

betm L and Lb . This relationship mat held as (A,B)

tas" em different values (A1,i), (A2,32). (A3.33). .. . Thus

(6) repreems a (Infinite) set of linar W equatlons

rQlt ng La sd Lb .Tee equatin will be inensistent, if

the 6sa ted istriz,

-? :. - -. . :;,¢- ;;%%.%% %- .;% , %'%.. V', . :..A%.. . %


7

fa,hAA0A ) 'aafh.At gy h(&,b.AI 3dLA, 1.0)

f(a,,,A 2 12 ) g(a,b,,A2 '3) h(a,b, A2 , 2 )LA( , 2 0e)

A~ ~f(a'b,%,913) S(a,b,A,.3) hlAb,%,13 )LA(A 3 , 13, e )

Is of rank 3. Thus (using an obvious notation) the determinant

(192 A fg) 39~ + (f3gl flg 3)YiL + "f293 -f 39 2 )hi1t1 0

(7) a..

This result accomplishes the elimination of La and Lb from

Imediate consideration.

Given any fed set ofvaluesof a, b, AI ,B1 A2 2 .

2 A3 , and 13 , the f '. g 'a. nd h'a in (7) are fied.

Equation (7) must still hold for S - e19029e3,..* . Thus (7)

represents an (infinite) set of homogeneous linear equations in the

three fixed quantities (f192 - f2gl)h3 , (f391 - f 1 93 )h 2 , end

(f 2 S3 - f3 A2)hl • Such equations can be consistent only if the . *

matrix

i"

* a

Xtem Response Functions ,"

8

A A AL (Al 13 1 -01 ) L (A2 '3 2 101 ) L (A3 953 901 )

A LAL (A 9. 1 , 62) L (A2 ,2 9a 2 ) LA(A 3 , 0 2 )

LA( A ,e 3) LA(% A2 ,e 3) LA(A 3 B,3e 3) :'

Is of less than full rank. If the equations are consistent, the ".

third row of the matrix must be linearly dependent on the first to ,"

A AA

rove: in other words L A(A,3,0 3 ) ust be a linear function of

LL(A,B, 2 ) and LL((,B, .

This conclusion is a contradiction, since 3 is arbitrary and

cannot be expressed as a function of 0 1 and 0 2 * Thus the prmise

ol tfthat the equations are 'consistent' must be false. This proves that

then varies, (7) represents a set of 'inconsistent' equations,

": according to cossion temtnology. This does not masn, howver thn

(7) is nelrd: the 'inconsistent' boogeneou linear equations

will be satsfied provided

(fig - ~lh fll-f )h2 -(f 3 -f2)hl- 0 . (8) '..

12 YI ' 31 f9 23 f9

This proves that If d afferentiable functions satisfying (4) exist e

4--

then the f og ,and h must Tstsf ( n)o According to (8) and

(7), either t ism e ni.5,

%.k1% - k~J*' . (f."'g" .- ,.°' f%. g1 )h . %3 * (f g1 - f % %)h (fg %- f g2" • 0" *%. (8)- e ' '* ,


9

(Case 1) h(a,b,A,B) 3 0 ,or

(Case 2) h(ab,A,B) is nonzero but g(a,b,A,B) I0 and/or

f(a,b,AB) 3 0 , or

(Case 3) f/g Is independent of A,B:

f(a,b,A,B,) -f(a,b, A23B2) F(ab

gj(a,b,A 1 9B1 g(a~b.A, 2 I

where F (a,b) denotes a (more or less) arbitrary function

of a and b only. ~ '

If Case 1 were to hold, then, from the definition of h in

(6)v

30

But for fixed A ,B ,this is the Jacobian of the transformation

a I (a,b) , 0 0(a,b) . When the Jacobian Is zero, a and 0

do not vary independently. In the present problem, this would

mean that for fixed A , B , LUa ,B ,) is only a one-parameter

family, and thus that P(a ,B ,) is only a one-parameter family

of Item response functions. This is contrary to the original

requirements, so Case 1 In Inappropriate.

Q9.


10

Case 2 will be treated later. The remaining possibility is

given by (9). Substitute (9) into (6), obtaining

h(a,b,AB)LA(A,B,O) S g(a,b,A,B)[F (a,b)La(a,b,0 )

+ Lb(a,b,6) .

Denote the quantity in brackets by F2(a,b,O) . Take the2V

logarithm of the absolute value of both sides:

logjhj + IA-- log gl f2 (10)

where IA- logiLA, and f2 logIF2 1 " Differentiate (10) on e :A ( . 2 21"

1 A,B,O) -- (a,b,8) . (11) "'-

Now the left side of (11) is not a function of a or b , the

right side is not a function of A or B * It follows that each

side must be independent of all four variables a , b , A , B .

Thus -S..

IA ,Be) (12)

• " ',,.a (sore or less) arbitrary function of e only. .

-f-. .

16%

-. 4,.


Integrate (12) on 6 :

A (A,BO) 02(e) + F3 (A,B)

where *2(6) f 1 (O)de and F3 (AB) is the 'constant of

integration.' Exponentiate, obtaining

LA(A,B,8) = xp[ (O) + F (A-B)-

L(A,B, ) - 4 (A,B)*4e )

where F4 and *3 are (more or less) arbitrary nonnegative functions.

Integrate on A to find, finally

L(A,B,O) BF 5(AB)*3(6) + Gl(Be) , (13)

where F5 (A,B) S fF4 (A,B) dA and GI(B,e) is the constant of

integration, both (more or less) arbitrary functions.

The only remaining case, Case 2, also leads to (13), as will now

be shown. If both g(a,b,A,B) and g(a,b,A,B) were zero in Case 2,

(6) would become

h(a,b,A,B)LA(A,B,e) = 0• *

- - • -y . a. .... 9-.- .- T -


12

Since h( ) is nonzero in Case 2, this would imply LA - 0 ° But

LA 0 means that the log of the item repsonse function does not

vary with A , a special limiting situation of no general interest

here. The only interesting alternatives under Case 2 is that

either f 0 or else g= 0 , but not both.

When g 0, h* 0 ,and f* 0, then (16) becomes

hLA fLa

Take the logarithm of the absolute value of both sides and then

differentiate on 8 to find

e £e i

A- a

This is equivalent to (11). Thus Case 2 with g - 0 also leads

to (13).

The remaining possibility, that f 0 , hO , and g 0p.. I

(in Case 2) also leads to (13), except that a and b , also A and

B, and also a and 0 are interchanged. Since the original problem-. %

is invariant under this interchange, (13) will still apply, given an

appropriate initial choice of parameter assignment. Thus with

suitable parameter assignment, (13) determines a specific form of

( , ) that is necessary (but not yet sufficient) for satisfying (4).p.'-..

_X~~~~ fce.r,1.

* Item Response Functions .

13

Necessary and Sufficient Condition for a Solution

Using (13), dropping numerical subscripts and rearranging, (4)

can nov be written

[F (a,b) + F (A,B) -F(,)J() 5 G1(A,e) - G1(b,e) - GI(B,O)

(14)

No,*(e) is not identically zero, since by (13) this would make

make L(A,B ,6) and hence L( ,0 ,0) independent of A *Divide by

(14) by 43(6):

F,(a~b) +- 15 (A,B) -FS(aO) IG 2 (0,8) -G 2 (b,e) - G2 (B,O)

(15)

where G2( ) 1( )/4 3 (e) .Differentiate on e

G6 ( 8)=(b,O) + G2-(BO (1)

Since the right side is independent of a and A , (a,b,A,B) must

be also. Hereafter the notation O(b,B) will be used.

Differentiate (16) on b to find:

%2 '' 2

10

I.I

Itm Response Functions

14

Take the logarithim of the absolute value of both sides:

1elogb + ge(5,e) - £O(b.0)

here 0b - *G and go likewise. Differentiate on

6

Repeat the foregoing operations on (15), switching the roles

of b and B to find

8 0go ,o() - gB(B,0)

Elaimnate g0 from the last two equations, obtaining

o ego %b(b,O) S93 (3,0)

Since the left side i not a function of B and the right side

is not a function of b , it is seen that each side is a function

of 0 only; so

Sb

Ste . . . . . e . - e • . .. .


15

say*

Integrate this on 60

S(b,) S + k (b)

wehere *2 (0) S * 1i(O)dO and k1 (b) is the constant of

integration. Exponent late:

ObjG2 (b,O)I as expL)*3() +k

Integrate on b and then on 6

G2-(b,O) k3 (b)* 3 (8) + I)

G2 (b,6) ik 3 (b)* 4 (e) + X2 (e) + k4 (b)

Thus, finally, by the relation of G2toG 1 ,

Itm Response Functions

'4o

16

G (bO) a k3 (b)* 5 (e) + k4 (b)# 3 () + Y1) (17)

From (13) and (17),

L(ABO) - 51A B)#3(9) + k3(3)*5() + k4()#31e) + *6(6)

a F6 1A)# 4 (e) + k3 (3)*s(e) + Y1)

From this and (4)

(F6 (ab) + F6 (A,B)L* 4 (e) + [k 3 (b) + k3 (B)]* 5 (0)

- F6 1a&)4 4 (e) + k3 ()* 5 (e) - Y6) •

This shows that *6(6) Is a linear function of 44 and 5(6) :

YO(e) S %4 () + K*5(e)

where C and K are constants (since 6) does not depend on

a , b , A , B ). Substitute this into each of the two preceding

equations, replace 41 ) by *(0), 5(0) by 1(6), and

k ( ) by G( ) , dropping numerical subscripts to obtain finally3

.

-. Item Response Functions

17

L(A,B,8) *F(A,3)4(O) + G(B)Y(O) *(18)

[F(a,b) + F(A,B)]#(S) + [G(b) + G(3)IY(O)

F(*,O)#O) + G(O)Y(9) .(19)

For equality to bold in (19), it is necessary that O(bj)

G ([G(b) + G(B)I where G71( )Is the Inverse function of G()

Also that

*(a,b,A,B) Fb)[F(a~b) + UT(A,I)J

where -1 Is the Inverse function defined for each given b

by F1 IF(a b)I a a

The solution to (2) is found by exponentiating (18):

P(A,B,O) 3 .zp[F(A,B)9(O) + G(B)Y(6)j

3 [f(A,B)j (0)[g(B)j"(0)

.5. UL,(,)F(A) 1 ()G(B) ,(20)

where 0(6) log #(8) , (8) Plog *(6) ,F(A,B) Ulog f(A,B),

and G(B) a log g(B)

. kY -


18

The Conjunctive Item Rsponse Function

A reparemsteriuation of the IRT model will simplify (20) for

present purposes. Define new Item parameters by b' s G(B) *

a' a F(A,B) . The general conjunctive Item response function

(20) is om simply

P(a',b',O) i [*(e)J('[#(e)j (21)

Without loss of generality, since #(6) and #(O) remain

to be chosen, take a' ) 0 and b' ) 0 . To be an Item,

response function (lF), it is necessary that P(a',b',e) be a

monotonic increasing function of 6 and that P(a' ,b' ,-..) - 0 ,

P(a',b',e) - 1 * If it is possible to have a' + 0 , then *(0)

must satisfy corresponding conditions. If it Is possible to have

b* + 0 , then #(0) must satisfy corresponding condition.

It will be assumed hereafter that both #(9) and +(0) satisfy

such conditions.

For convenience, the primes In (21) will now be dropped. If

#(e) # *(8) , it would follow that #(0) * #(8) , since otherwise

*(m) u *(.) . I could not hold. If #(8) s #(0) , then

CCbP(a,b,e) . [+(e)])b . In this case, the reparameterization

a" - a + b would reduce P( ) to a one-parameter family. Thus

it is essential that #(0) and #(8) should not be proportional.

r.q

7_ I.4 4'o


19

A plausible moel is obtailed by choosing *(0) and #(0)

to be sefamiUar two-parMIter IM,. If the t-parmter

logistic fmection is gmea, for exampl.e, the IRY for Item I Is

-- b?(a:1 ,b1 ,O) (1 + ')(I + e4 (22

where a and pi are arbitrary constants. These constants mst

be the sam for all Items In a given test, buat they can vary

from one test to-another. ideally a and p should be constant9 ,

f or all Items of a single type: they are test or item-type

parameters, not Item parameters.*i~

Up to this point, the argument has dealt With the probability

that a given examines knows the correct aswr to an item or to

a component part of an iten. If the final Item Is presented in

multiple-choice form, however, the examine amy answer correctly

simply by random gueening. To deal with this situation, a

'guessing' paramter ci can plausibly he Introduced into (22),

so that nom

P(&Iqbi~qci,6) C a ci)(l ~"-' ~ + e*o

(23) '5

t5 ;'A


20

This IRF does not satisfy the functional equation (2).

Nevertheless, (23) uay be quite appropriate; for example, when

guessing does not contribute to the probability that an examinee

knows the correct answer to the component parts of the Item.

The Disjunctive Item Response Function

The original reasoning dealt with situations where an

examinee would know the correct answer to the actual item only if

he knew the correct answer to two separate subproblems In the

actual item. One can also Imagine situations where the exainee .

will know the answer to the actual item whenever he knows the

answer to either one of two separate subproblems: In other

words, there are two independent routes to knowing the answer

to the actual item. This is the disjunctive case.

The corresponding functional equation is the same as (2)

except that (ab,e) a - P(a,b,O) Is substituted for

P(a,b,8)

Q(ab,O)Q(A,B,O) 5 Q(,,0) • (24)

The corresponding in Is thus

a biQ(ab,O) 11 [1 - (8)] , (25)

. Nor;

P:.

S!


21

where #(O) and #(e) satisfy the sae conditions as before. In

the logistic case with guessing,

Q(ai,bici.) 0 1 - ci)1 + eoe+)-0 + a..-

(26)

Practical Implementation and Conclusion

The conjunctive (disjunctive) IRa's have one convenient

feature: When there is no guessing ( c I 0) , the probability of

answering all of a set of a items correctly (incorrectly) has

a such simpler mathematical form than is usually the case. For

conjunctive items, -

Prob(u1 - 1, u2 - ',...,u3 - 1) - i[,( 8 ) b .

(27) .

The parameters aI and bi of conjunctive or disjunctive

Items are not at all readily Interpreted in terms of itemdifficulty and Item discriminating power. It is hard to find even

a complicated function, let alone a simple function, of aI and

and bi that can be comfortably interpreted as a satisfactory

measure of item difficulty or of Item discriminating power.

,P 0

*e

::_",....,,.


22

Figure 1 shows six plots of IRF's from (22) illustrating

various degrees of skewness. One can, of course, obtain a pure

logistic curve from (22) by setting either ai or bi equal to

zero. It is also possible to obtain double ogives with three

points of inflexion (not Illustrated here).

A new computer program has been written for simultaneous

eaximutm likelihood estimation of the item parameters a1 and bi

the test parameters a and V , and the ability parameters 8 for

the conjunctive model (23). The new program uses fixed cI

values determined in advance by a run of the data on the computer

program LOGIST (Vingersky, 1983). Although the estimates converge

in any one computer run from a given set of trial values, the result

is apparently not yet useful because from different sets of trial

values the computer reaches different local maxima corresponding

to different assignments of zero values to ai or to bI for

different subsets of items.

The fit of the model to the responses of a group of examinees

could probably be such improved by substituting (26) for (23)

whenever an item Is estimated to have a zero value of a1 or bi

thus assuming that such items are disjunctive items. Subsequent

iterations of the estimation process might then lead to a

.. 1*

.1.


23

0.6-0.

4..

1.0 .

6.4- 5.4

-I 0.6 i A

NL Fiur 1. I Illsrtv cojntv ite repos funct I I I ions

wi.h c5(2.*

A.

V" Item Response Functions

24

reassignment of some of these items to the conjunctive model while

other items might be transferred to the disjunctive model.

Ultimately, such a process might successfully classify all items as

conjunctive or disjunctive in such a way as to find a global

maximum of the likelihood function and an optimal fit of the models

to the data. It is not at all clear, however, how much computer

time such a process might require.

It is possible that the conjunctive and disjunctive models,

based as they are on relevant psychological considerations, may

provide a better fit to real data than the usual logistic or normal

ogive models.

5, A, S I* AO '.

4¢

4%.o

1'oo

4.


References

Kristof, U. (1968). On the varallelization of trace lines for a

certain test model (11-68-56). Princeton, NJ: Educational

*Testing Service.

Wingeraky, M. S. (1983). LOGIST: A program for computing maximum

likelihood procedures for logiatic test models. In R. K.

Hambleton, (Ed.), Applications of item response theory.

Vancouver: Educational Research Institute of British

Columbia.

Yen, W. M. (To be published). Increasing item complexity: A

posilecause of scale shikg for unidlisensional item

response theory. Monterey, CA: CTB/McGraw 1H111.

.17

.'

'.4

p.4

.4.

4...

4**

V.

t "4

,. *T.

'~ 4.

.4. f a.'4 .4.

'4..4'.4.

.4%

.44.

I..-4.

4%*4 4

4.

* I...4*49

* .4.9

4.4

.4

'.4..

4%~*4

1 4

-. 9i#..~ *7~ ~* ~ ~ ~ ~;4. ~ ~ # .PA,.\4-4

~ .~.' '.~ ~ ' ~" '~ 'I'.~

;N.

ETS/Lard 18-Sup- 4 Page I

Navy Navy

I Dr. Nick Dond 1 Dr. Norman J. KerrOffice of Naval Research Chief of Naval Education and Training

Liaison Office, Far East Code OCA2

APO San Francisco, CA 96503 Naval Air StationPensacola, FL 32508

i Lt. Alexander DoryApplied Psychology I Dr. Leonard KroekerMeasurement Division Navy Personnel R& CenterNANRL San Diego, CA 92152

NAS Pensacola, FL 32508I Daryll Lang

Dr. Robert Breaux Navy Personnel R&D CenterNAVTRAEGUIPCEN San Diego, CA 92152Code N-95ROrlando, FL 32813 1 Dr. William L. Maloy (02)

Chief of Naval Education and Training

I Dr. Robert Carroll Naval Air Station

NAVOP 115 Pensacola, FL 32508"ashington , DC 20370

1 Dr. James McBride

I Dr. Stanley Collyer Navy Personnel R&D Center

Office of Naval Technology San Diego, CA 92152800 N. Quincy StreetArlington, VA 22217 1 Dr William Montague

NPIDC Code 13

1 CDR Mike Curran San Diego, CA 92152

Office of Naval Research800 N. Quincy St. I Library, Code P23D.

Code 270 Navy Personnel R&D Center

Arlington, VA 22217 San Diego, CA 92152

1 Dr. John Ellis 1 Technical DirectorNavy Personnel R&D Center Navy Personnel R&D CenterSan Diego, CA 92252 San Diego, CA 92152

I DR. PAT FEDERICO 6 Personnel t Training Research Group

Code P13 Code 442PTWRDC Office of Naval ResearchSan Diego, CA 92152 Arlington, VA 22217

I Mr. Paul Foley 1 Dr. Carl RcssNavy Personnel R&D Center CNET-PDCD

San Diego, CA 92152 Building 90

Great Lakes NTC, IL 60088I Mr. lick Hosham

NAYOP-135 1r. Dr- SandsArlington Annex NPRDC Code 62Rooe 2834 San Diego, CA 92152Washington ,C 20350

I ary Schratz

Navy Personnel R&D Center

San Diego, CA 92152

%J6.

+ "'I

ETS/Lord 18-Sep-64 Page 2

Navy Navy

I Dr. Robert 6. Smith 1 Dr. Mallace Mulfeck, IIIOffice of Chief of Naval Operations Navy Personnel R&D CenterOP-97H Sn Diego, CA 92152Washington, DC 20350

1 Dr. Alfred F. StodeSenior ScientistCode 71Naval Training Equipment CenterOrlando, FL 32913

I Dr. Richard SnowLiaison ScientistOffice of Naval Research

Branch Office, LondonBox 39FPO Nn York, NY 09510

I Dr. Richard SorensenNavy Personnel R&U CenterSan Diego, CA 92152

I Dr. Frederick SteinheisrCMO - OPII5Navy AnnexArlington, VA 20370

I Hr. Brad SympsonNavy Personnel RU1 CenterSan Diego, CA 92152

1 Dr. Frank icinoNavy Personnel R&D CenterSan Diego, CA 92152

1 Dr. Ronald WeitzmanNaval Postgraduate SchoolDepartment of Administrative

SciencesMonterey, CA 9.940

I Dr. Douglas WetzelCode 12Navy Personnel R&D CenterSan liego, CA 92152

I DR. NARTIN F. MISKOFFNAVY PERSMONNE R6 D CENTERSAN DIEDO, CA 92152

I Mr John H. WolfeNavy Personnel RIO CenterSan Diego, CA 92152

'..-,- ,,+ ' - ,' ,'. . ,,,,',-t" t'.,".,,"_---_ *_.."+r':. ".__. ' " *,_,, ,e*_. .j ._ /_ .,',-..._s .. _' .+. "..+' ".... ",,,' < *..+." ' l._'.,,"r _'." ° ,-

ETS/Lard 18-Sep-84 Page 3

Marine Corps Army

I Jerry Lehnus 1 Dr. Kent EatonCAT Project Off ice Ary Research InstituteHD irine Corps 5001 Eisenhower Blvd.Washington , C 203890 Alexandria , VA 22333

I Headquarters, U. S. Marine Corps I Dr. Clessen Martin .-.'Code NPI-20 Arey Research InstituteWashington, K 20390 5001 Eisenhower Blvd.

Alexandria, VA 22333I Special Assistant for Marine

Corps Hatters 1 Dr. Willia E. NordbrockCode lOON FMC-ADCO So 25Office of Naval Research APO, Wif 09710800 N. Quincy St.Arlington, VA 22217 1 Dr. Harold F. O'Nail, Jr.

Director, Training Research LabI Major Frank Yohannan, USHC Arty Research InstituteHeadquarters, arine Corps 5001 Eisenhower Avenue(Code fPI-20) Alexandria, VA 22333Washington, BC 20380

I Coamander, U.S. kuy Research Institutefor the Behavioral I Social Sciences

ATTN: PERI-BR (Dr. Judith Orasanu)5001 Eisenhower AvenueAlexandria, VA 22333

1 Mr. Robert RossU.S. Ary Research Institute for the

Social and Behavioral Sciences

5001 Eisenhower AvenueAlexandria, VA 22333

1 Dr. Robert SaserU. S. Arey Research Institute for theBehavioral and Social Sciences -,

5001 Eisenhower Avenue

Alexandria, VA 22333

I Dr. Joyce ShieldsArey Research Institute for the

Behavioral and Social Sciences5001 Eisenhower AvenueAlexandria, VA 22333

1 Dr. Hilda Wing

eAry Research Institute5001 Eisenhower Ave. .'-

Alexandria, VA 22333

..d %

-: l~' _ ,'~g'.' T .)'.-. , :_,_ ,,. .'j _ _% '. %, ., _, 'L'.-_ ,_% ',.,_% ," o' , ,.-. .? .. , __-. ' _* - , , _'--%' '_ .'-',',-,:_, '1 '--L,,' . *3* %* .

*,--

ETS/Lord 1S-Sep-If Page 4

Air Force Department of Defese

I Dr. Earl A. Allsisi 12 Defnse Technical Information Center

HO, AFIL (AFSC) Cameron Station, Bldg 5Brooks AI, TI 78235 Alexandria, VA 22314

Attn: TCI Hr. Raymond E. Christal

AFNIL/NOE I Dr. Clarence McCormick,. kooks AFS, TI 7235 HO, NEPCOH

IIPCT-PI Dr. Alfred A. Frgly 2500 Drme Day RAnd

AFOSR/lI bnprth Chicago, IL MBoling AFI, DC 20332

I Military Assistunt for Training andI Dr. Patrick Kylloooo Personnel Technology

AFNRLIOE Office of the Under Secretary of Defensfor Research % Engineering

Brooks AI, TI 79235 Room 33129, The PentagonVashington, DC 20301

I Dr. Randolph ParkAFHRL/NIAN I Dr. I. Steve SollmunBrooks AFD, TI 79235 Office of the Assistant Secretary

of Defense (NRA & L)I Dr. Roger Pennell 23269 The Pentagon

Air Force Humn Resources Laboratory 11ashington, DC 20301Lowry AI, CC 80230

I Dr. Robert A. isherI Dr. Malcolm Rn OUSIRE (ELS)

AFRL/WP The Pentagon, Room 31129Brooks AFI, TI 78235 ashington, C 20301

I HMaor John IelshAFHRL/NIAN

Brooks FI ,TI 79223

'V.

• *4

.

,e.

a. .U.

ETSILord 19-Sep-94 Fage 5A.4'i Civilian Agencies Private Sector

I Dr. Patricia A. Butler I Dr. Erling . AndersenNIE-DRN Bldg, Stop 7 Department of Statistics

1200 19th St., NN Studiestraede 6Washington, DC 20208 1435 Copenhagen

DENMARKI Dr. Vern N. Urry

Personnel R&D Center 1 Dr. Isaac bujarOffice of Personnel Nanageent Educational Testing Service1900 E Stret NN Princeton, NJ 0B450Washington, K 20415

I Dr. Mnucha irenbausI Mr. Thomas A. Marl School of EducationU. S. Coast Snard Institute Tel Aviv UniversityP. 0. Substation 18 Tel Aviv, Ramat Aviv 6978Oklahoma City, OK 73169 Israel

I Dr. Joseph L. Young, Director I Dr. Verner Dirkee nory & Cognitive Processes Personalstamsaat der Bundiswhr

National Science Foundation 0-3000 Keln n0Washington, DC 20550 NEST SER NY

, Dr. . Darrell BckDkpartnent of EducationUniversity of ChicagoChicago, I. 60637

I Mr. Arnold DobrerSection of Psychological ResearchCawne Petits Chateau

1000 DruiselsBelgium

I Dr. Robert BrennanAmerican College Testing ProgramsP.O. Box 16"Issa City, IA 52243

I Dr. lIenn Bryan40 Pce Roadlathesda, NO 20917

1 Dr. Ernest R. Cadotte307 StokelyUniversity of TennesseeKnoxville, TN 37916

I Dr. John 3. Carroll409 Elliott Rd.Chapel Hill, NC 27514

• €'.r

::-..'9

ETS/Lord 1-Sip-P Pag 6

Private Sector Private Sector

I Dr. Norman Cliff I Dr. John A. EddinsDept. of Psychology University of IllinoisUniv. of So. California 252 Engineering Rsearch LaboratoryUniversity Park 103 South fathees StreetLos Angeles, CA 90007 Urbana, IL 61801

I Dr. Hans Crombag I Dr. Susan EnbertsonEducation Research Center PSrCHOLDBY DEPARTMENTUniversity of Leydeo UlIVESITY OF KAWSBorhaavoln 2 Lasrence, KS 6M0452334 EN LeydenThe NETHERLIS I E3I1 Focility-Acquisitions

4133 Rugby Avnue1 Lee Cronbach Bethesda, NO 20014

16 Laburnue RoadAtherton, CA 94205 1 Dr. Benjamin A. Fairbank, Jr.

Performance Netrics, Inc.I CTBINcGraw-Hill Library 52 Callaghan 7...'

2500 Barden Road Suite 225Nonterey, CA 93940 San Antonio, TX 76229

I Mr. Timothy Davey I Dr. Leonard FeldtUniversity of Illinhois Lindquist Center for MeasuraentDepartment of Educational Psychology University of Ioa

Urbana, IL 61801 low City, IA 52242

I Dr. Dattpradad Divgi I Univ. Prof. Dr. Gerhard FischerSyracuse University Libliggasse 513Department of Psychology A 1010 YiennaSyracuse, ME 33210 AUSTRIA

I Dr. Emmanuel Donchin I Professor Donald FitzgeraldDepartment of Psychology University of lh EnglandUniversity of Illinois Armidale, Now South Vales 251Champaign, IL 61820 AUSTRALIA

I Dr. Hei-Ki Dong I Or, Dexter Fletcher

Ball Foundation University of Oregon800 Roosevelt Road Department of Computer ScienceBuilding C, Suite 206 Eugene, OR 97403Gen Ellyn, IL 60137

Dr. John R. Frederikuen

I Dr. Fritz Drasgon kit Berauek I N-wDepartment of Psychology 50 Moulton StreetUniversity of Illinois Cambridge, MA 02133603 E. Daniel St.Champaign, IL 61820 1 Dr. Janice Gifford

University of assachusetts

1 Dr. Stephen Dunbar School of EducationLlndquist Center for Measurefmt Amherst, NA 01002University of IowaIowa City, IA 52242

1%%

*, .,,

£ThI~ord 10e-84 Pape 7

Pri vate Sector Private Sector -:

I Dr. Robert Slawe I Dr. Steven IlunkaLearning Research I Devlopmt Center Department of EducationUniversity of Pittsburgh University of Alberta3939 O'Mara Street Edmonton, AlbertaP1TSUhl, PA 15260 CNADA

1 Dr. Marvin 3. Block I Dr. Jack Hunter V217 Stone Hall 2122 Coolidge St.Cornell University Lansing, MI. 48906Ithaca, NY 1U53

I Dr. Jbiynb Nuynh1 Dr. Bert Green College of Education

Johns Hopkins University University of South CarolinaDepart wnt of Psychology Columbia, SC 29208Charles I 34th StreetBaltimors, RD 21218 1 Dr. Douglas H. Jones

Advanced Statistical TechnologiesDR. JAMES B. GREENO corporationLAMC 10 Trafalgar CourtUNIVERSITY OF PITTSBURGH Laurenceville, NJ 091493939 0' HARA STREETPITTSBURGH, PA 15213 1 Professor John A. Keats

Departent of PsychologyI Dipl. Pad. Michael U. Habon The University of NeNcastle

Universitat Dusseldorf N.S.N. 2308 -

Erzithanlssenshaftliches Inst. 11 AUSTRALIAUlnivesitatsstr. ID-400 Dusseldorf I I Dr. William KochNEST GERMWN University of Texas-Austin

Meuremt and Evaluation Center1 Dr. Ron Hambleton Austin, TI 79703

School of EducationUniversity of Nassachustts 1 Dr. Thomas LeonardAmerst, MA 0100 University Of isconsin

Department of StatisticsI Prof. Lutz F. Hornke 1210 West Dayton Street

Universitat Dusseldorf Maiso, WI 535Erziehungsuissenschaftliches lnst. 11Universitatsstr. 1 1 Dr. Alan LesgoldDusseldorf 1 Learning R&D CenterWEST GERMANY University of Pittsburgh

3939 O'Hara Street1 Dr. Paul Horst Pittsburgh, PA 15260b716S Street, 1194 .**

Chula Vista, CA 90010 1 Dr. Michael LevineDepartment of Educational Psychology

I Dr. Lloyd Nuephroys 210 Education Bldg.Department of Psychology University of IllinoisUniversity of Illinois Champailg IL 61901603 East Daniel StreetChampagn, IL 61320

"C" or l

ETS/Lord 18-Sep-4 Page 9


I Dr. Charles Levis I Dr. Melvin A. NovickFKulteit SOCiale etenuChappe 35 Lindquist Center for NeasursentRijkswniveriteit Groninge University of Io4aDude oterlngestraat 23 low City, IA 5224297126C Groning.nNetherlands I Dr. June Olson

VICAT, Inc.I Dr. Robert Lion lm South state Street

Collee of Education Ore. UT 94057 University of IllinoisUrbana, IL 61301 1 Uawe N. Patience

America Cncil on EducationI Dr. Robert Lockean E0 Testing Service, Sits 20

Center for Naval Analysis One Dupont Cirle, NN200 North beareard St. ashington, DC 20036Alexandria, VA 22311

I Dr. Jame Paul son "I Dr. Frederic N. Lord Dept. of PsychologyEducational Testing Service Portland State UniversityPrinceton, NJ 08541 P.O. Sox 731

Portland, OR 972071 r. James Luuiden,-.

Department of Psychology I Dr. Mark 0. Reckae.University of Western Australia ACTNedlands N.A. 6009 P. 6. 116 148AUSTRALIA low City, IA 52243

I Dr. Gary Marco I Dr. Laernce RudnerStop 31-E 403 Elm AvenueEducational Testing Service Takeum Park, ND 20012Princeton, NJ 06451

I Dr. J. Ryan "* "1 Hr. Robert McKinley Department of Education

American Collage Testing Programs University of South CarolinaP.O. lox 169 Columbia, SC 29209Iowa City, IA 52243

1 PROF. FUNIKO SAMEJINAI Dr. Barbara Means DEPT. OF PSYCHOLOGY

Human Resources Aesearch Organization UNIVERSITY OF TENNESSEE300 North Washington KNIVILLE, TN 37916Alexandria, VA 22314 1-rn...cmd

I Frank L. Scheidt ...-.

I Dr. Robert Misl$vy Department of Psychology711 Illinois Street Bldg. 66 "Geneva, IL 60134 Oorge Washington University

Washington, DC 200521 Dr. N. Alm Nicewander

University of Oklahoma I Lowe I SchorDepartment of Psychology Psychological I QuantitativeOklahoma City, OK 73069 Foundations 0 .*

College of EducationUniversity of Iowalow City, IA 52242

ETSILord 18-Sep-34 Page 9


I Dr. Kazuo Shigosu IMr. Bey nomasson7-9-24 Kugenusa-Kaigan University of IllinoisFuJusava 251 Department of Educational Psychology

JPN Champaign, IL 61320

1 Dr. illiam Sin I Dr. Robert TsatakaCenter for Naval Analysis Departeont of Statistics200 North Beaurejard Street University of MissouriAlexandria, VA 22311 Columbia, NO 65201

1 Dr. H. Nallace Sinaiko I Dr. Ledyard TuckerProgram Director University of IllinoisManpower Research and Advisory Services Department of Psychology ISmithsonian Institution 403 E. Daniel Street901 North Pitt Street Champaign, IL 61920Alexandria, VA 22314

Dr.. . . R. pulur-1 Martha Stocking Union Carbide Corporation

Educational Testing Service Nuclear DivisionPrinceton, NJ 08541 P.O. Box Y

Oak Ridge, TN 370301 Dr. Peter Stolof,.

Center for Naval Analysis I Dr. David Vale200 North Stauregard Street Assessment Systems CorporationAlexandria, VA 22311 2233 University Avenue

suite 3101 Dr. William Stout St. Paul, HN 55114University of IllinoisDepartment of Mathematics I Dr. Howard NaierUrbana, IL 61001 Division of Psychological Studies

Educational Testing ServiceI Dr. Hariharan Swau athan Princeton, NJ 08540

Laboratory of Psychometric andEvaluation Research 1 Dr. Ning-Nei langSchool of Education Lindquist Center for MeasurmentUniversity of Massachusetts University of lovaAmherst, NA 01003 Iona City , IA 52242

1 Dr. Kikuni Tatsuoka I Dr. Brian Waters* Computer Based Education Research Lab HUMAN

252 Engineering Research Laboratory 300 North ashingtonUrbana, IL 61801 Alexandria, VA 22314

I Dr. Rawrice Tatsuoka I Dr. David J. Weiss220 Education Bldg N"0 Elliott Hall1310 S. Sixth St. University of Minnesota ,.IChampaign, IL 61320 75 E. River Road

Minneapolisl MN 55.I Dr. David Thsan

Department of Psychology I Dr. Rand A. Vilcox

University of Kansas University of Southern CaliforniaLawence, KS 4044 Department of Psychology

Los Angeles, CA 90007

ETS/Lord PagSe--B4 eae 10

Private Sector

!I germ lflitary Rapresotatlve

ATTN: Wolfgang ildeqrubStreitkraefteat0-5300 Bonn 2 :

4000 Draatdyuine Street, NIHashington , VC 20016

I Dr. Druce WilliaumDupartsmst of Educational PsychologyUniversity of IllinoisUrban&, IL 61801

Educatilyn ingerekyEducational Testing ServicePrinceton, MJ 01541

I Dr. George Hngiosttistics Laboratory

,umorial Sloa-Kettering Cancer CenterIM York AvenueNew York, NY 10021

I Dr. Hendy YenCTSlI/c~rao Hill%l Nonte Research Park

Snnterey, CA 1,"940

Y. *.

,I.,

" i~

**

*4.S

C. .

,. o.

'~ 9 S "

16

2 8

....

RD-Ri47 OCT UNCLSSIFIED NL mhhhhhhmhmml · 46 @M*14IW ~ WAtA mm-mmSil ifme blm CAnmIM MOMS 116...

Documents

Transcript of RD-Ri47 OCT UNCLSSIFIED NL mhhhhhhmhmml · 46 @M*14IW ~ WAtA mm-mmSil ifme blm CAnmIM MOMS 116...