GATBDEVE

8/14/2019 GATBDEVE

1/98

Development of General Aptitude Test Battery

(GATB) Forms E and F

Steven J. Mellon Jr., Mi!elle Da""ett, #ine MaManus

and Brian Morits!

Su$mitted To%

Division of S&ills Assessment and Analysis'ffie of oliy and )esear!

Employment and Trainin" Administration

*.S. Department of +a$or

Su$mitted By%

aifi Assessment esear! and Development ,enter

-- +at!rop /ay, Suite A

Saramento, A .01-0

-..2

8/14/2019 GATBDEVE

2/98

Addendum

Please note that the General Aptitude Test Battery (Forms E & F) referred to within this report

has been renamed the Ability Profiler (Forms 1 & 2) The name of the assessment was !han"edto refle!t# 1) the fo!us on reportin" a profile of s!ore results from the instrument for !areer

e$ploration purposes% 2) the te!hni!al improements made to the assessment !ompared topreious forms of the instrument% and ') the !apa!ity to use the Ability Profiler in !onun!tion

with other instruments to promote whole person assessment for !areer e$ploration

This material is in!luded as hapter 2 in the unpublished report# * A +!loy, T - *ussell, -- .ise (Eds), GATB improvement project final report .ashin"ton, /# 0 /epartment of-abor

8/14/2019 GATBDEVE

3/98

Table of Contents

CHAPTER 2 DEVELOPMENT OF GATB FORMS E AND F 1han"es to pe!ifi!ations for Test -en"th and Format and to upportin" +aterials 2

on!erns and be!ties 2

*edu!in" peededness 2!orin" Pro!edures 3

4nstru!tions to E$aminees 3

*esear!h on Test Aestheti!s 5/eelopment and *eiew of 6ew 4tems 11

4tem .ritin" 11

Editorial *eiew and !reenin" 124tem Tryout and tatisti!al !reenin" 17

4tem Tryout Boo8let /esi"n 17

4tem Tryout ample 13

/ata olle!tion Pro!edures 19

/ata Analysis 1:alibration and !reenin" of Power Test 4tems 2;

alibration and !reenin" of peeded Test 4tems 22ele!tion of 4tems for Final

8/14/2019 GATBDEVE

4/98

Table 2?12 Psy!homotor /ata olle!tion /esi"n '5

Table 2?1' Group i@es for the Edited 4ndependent GroupsE>uatin" ample ':

Table 2?1= Group i@es for the Edited *epeated?+easures ample ':Table 2?17 /emo"raphi! omposition of the 4ndependent?GroupsE>uatin" ample =;

Table 2?13 /emo"raphi! omposition of the *epeated?+easures ample =1

Table 2?19 /emo"raphi! omposition of the Psy!homotor ample =2

Table 2?15 /emo"raphi! omposition of the A""re"ate ample ='Table 2?1: Aptitude !ore omposition ==

Table 2?2; Alternate?Form *eliability Estimates, 6ormal /eiates, and p?

8/14/2019 GATBDEVE

5/981

CHAPTER 2DEVELOPMENT OF AT! FORM" E AND F teen +ellon, r, +i!helle /a""ett, uatin" will be ne!essary This re>uires the attentionto pro!edures and normatie "roups des!ribed in the pre!edin" !hapter

2 A!!ess to operational test forms must be seerely limited to only those /epartment

of -abor and Employment eri!e personnel inoled in the testin" pro"ram and to

those proidin" te!hni!al reiew tri!t test a!!ess pro!edures must be implemented

' eparate but parallel forms of the GATB should be made aailable for !ounselin"and "uidan!e purposes

= A resear!h and deelopment proe!t should be put in pla!e to redu!e the

speededness of the GATB A hi"hly speeded test, one that no one !an hope to

!omplete, is eminently !oa!hable For e$ample, s!ores !an be improed by tea!hin"

test ta8ers to fill in all remainin" blan8s in the last minute of the test period 4f this!hara!teristi! of the GATB is not altered, the test will not retain its alidity when

"ien a widely re!o"ni@ed "ate8eepin" fun!tion (Carti"an & .i"dor, 1:5:, p 113)

The primary purpose of the GATB Forms E and F /eelopment Proe!t was to deelopalternate forms of the !o"nitie portion of the GATB (Parts 1?9) followin" pro!edures that fulfill

the hi"hest professional standards The proe!t was initiated prior to the 6A reiew and in!luded a

reiew of test len"ths and s!orin" pro!edures ubse>uent to the 6A reiew, the fo!us of the

proe!t was e$panded to in!lude other obe!ties e$pli!itly re!ommended by the 6A or otherwiseimpli!it in its findin"s

8/14/2019 GATBDEVE

6/982

Chapter 2. Development of GATB Forms E and F

The e$panded obe!ties were

D /eelop new forms of the GATB that are less speeded and less sus!eptible to !oa!hin"

by redu!in" the number of items and inesti"atin" the feasibility of in!reasin" test time

limits

D 4nesti"ate and in!orporate into the test the most appropriate s!orin" pro!edures, anddeelop instru!tions to e$aminees that !learly des!ribe those s!orin" pro!edures

D /eelop test items free from bias, in terms of both the ethni! and "ender sensitiity ofthe lan"ua"e and the statisti!al fun!tionin" of the items for different "roups

D Assemble test forms as parallel to ea!h other as possible and lin8 s!ores on these formsto s!ores from earlier forms

D 4mproe the aestheti!s of the test boo8lets and test items

D *eise answer sheets and other related GATB materials to be !onsistent with !han"es inthe test format and to proide the opportunity for e$aminees to ma$imi@e their tests!ores

The A*/P met the e$panded obe!ties for the new GATB forms throu"h a series ofresear!h steps First, to address the speededness and !oa!hin" issues, the A*/P made !han"es to

spe!ifi!ations for test len"th and format and to supportin" materials n!e new items were written,the A*/P !ondu!ted item reiews, an item tryout, and statisti!al s!reenin" of items Based on thedata, the A*/P deeloped new final forms Finally, a study to lin8 new Forms E and F to base

Form A was underta8en +ore detailed information about this resear!h is !ontained in the

Te!hni!al *eport on the /eelopment of GATB Forms E and F (+ellon, /a""ett, +a!+anus, &+orits!h, 1::3)

C#anges to "$e%ifi%ations for Test Lengt# and Format and to"u$$orting Materials

Con%erns and Ob&e%ti'es

4n its report, Fairness in Employment Testin", the 6A !on!luded that the seen paper?and?pen!il tests of the GATB !ontained many more items than most e$aminees !ould possibly !omplete

in the amount of time allotted for ea!h test !ores on items at the end of the test were more li8elyto be an indi!ation of whether the e$aminee was !oa!hed on a rapid respondin" strate"y than of theaptitude that the test is intended to measure onse>uently, the in!lusion of items that few, if any,e$aminees rea!h if they seriously attempt to answer ea!h >uestion detra!ts from the alidity of thetest ptions for redu!in" the speededness of most of the GATB tests in!luded in!reasin" the timeallotted andor redu!in" the number of items for ea!h test An analysis of these options was the firststep ta8en in reisin" the test spe!ifi!ations Additional steps to redu!e the impa!t of test?ta8in"strate"y, in!ludin" !han"es to test instru!tions and the s!orin" pro!edures used with the tests thatremain speeded, were also !onsidered

The 6A ommittee and GATB users also e$pressed !on!erns related to the tests format,the oerall aestheti! appeal of test items, and the format of the answer sheets A!tions ta8en to

improe appearan!e and format were addressed in detail in the Test Aestheti!s Proe!t (/a""ett,

1::7), whi!h !onstituted the se!ond maor step in reisin" the test spe!ifi!ations

Redu%ing "$eededness

The A*/P addressed issues pertainin" to the GATBs speededness in three steps First,

A*/P staff analy@ed s!ore distributions for the !urrent forms and deeloped initialre!ommendations for redu!in" the number of items in ea!h test The se!ond step inoled new

8/14/2019 GATBDEVE

7/98'

resear!h on test speededness !ondu!ted by the Ameri!an 4nstitutes for *esear!h (A4*) under

!ontra!t to /- (a"er, Peterson, & ppler, 1::=) The final step was a reiew of the aboe wor8

by an e$pert panel, who deeloped reised re!ommendations !on!ernin" both the number of itemsand the time to be used for ea!h test Ea!h step is des!ribed briefly here, with more !omplete

information proided in the te!hni!al report (+ellon et al, 1::3) and in a"er et al (1::=)

(nitial Re%ommendations) A*/ staff performed an initial analysis of possible test

len"th redu!tion (A*/, 1::2) They used normatie data in the GATB deelopment manual ands!ore !onersion tables in the administration manuals for Forms A throu"h / to determine the

number of !orre!t responses re>uired to a!hiee a ::th per!entile raw s!ore for ea!h test The intentwas to appro$imate a pra!ti!al limit to the number of items used to differentiate amon" nearly all of

the !urrent e$aminees olumn four of Table 2?1 !ontains ran"es a!ross the four forms in the ::th

per!entile s!ores

Table 2*+Re%ommended C#anges in AT! Test Order, Lengt#, and Time Limits

Form A?/*eised Test rder For m s A ? / :: ile 4nitial *eised P ro p o sal

(urrent rder) -en"th Time *an"es Proposal -en"th Time

Arithmeti! *easonin" 27 4tems 9 +in 1:?21 4tems 2= 4tems 15 4tems 2; +in(Part 3)

8/14/2019 GATBDEVE

8/98


Ne- Resear%# on Test Lengt#s) A4* reiewed prior literature on issues of testspeededness (Peterson, 1::') and desi"ned and e$e!uted a study to proide further data on themost releant issues found in the literature reiew The reiew !oered (1) methods of assessin"

test speededness, (2) relatie merits of power and speeded power tests, (') relationships between

speeded and power tests of similar items, (=) differential effe!ts of speededness, and (7) adersepsy!holo"i!al rea!tions

A 8ey findin" from this reiew was that seeral of the !onstru!ts measured by GATB tests(parti!ularly Arithmeti! *easonin", uen!es for redu!in" speededness (s!ores would be less sus!eptible to !han"es inadministration !onditions, whether intended or not and adaptin" tests for disabled e$aminees is

more easily a!!omplished)M (p =7)

After !ompletin" the literature reiew, A4* staff desi"ned a study to address the issues

ud"ed most salientNspe!ifi!ally, whether speeded and non?speeded ersions of four of the GATBtests measured the same !onstru!ts% the e$tent to whi!h e$aminee sub"roups defined by ra!e,

ethni!ity, a"e, or "ender showed "reater differen!es in one form or the other% and the e$tent to

whi!h !orrelations amon" the tests in ea!h form were the same for different e$aminee "roups Anadditional issue addressed was whether !han"es in instru!tions, item formats, and answer sheets

had a si"nifi!ant impa!t on test s!ores A non?speeded test battery in!ludin" omputation, Three?

/imensional pa!e,

8/14/2019 GATBDEVE

9/98

D 6onspeeded ersions of the Three?/imensional pa!e, Arithmeti! *easonin", anduantitatie, spatial, and per!eptual

!onstru!ts should !ontinue to be measured but that three of the tests (ie, uate measure of theper!eptual speed and a!!ura!y !onstru!t, and this !onstru!t was ud"ed to be somewhat less

important than measures of erbal, >uantitatie, and spatial s8ills ne other re!ommendationNthat

the power tests should be administered as a "roup before the speeded testsNled to the proposedreorderin" of the remainin" tests shown in Table 2?1

1

1Althou"h Form +at!hin" was eentually dropped, the final de!ision !ame late in the proe!t onse>uently, it

was in!luded in many of the deelopment steps des!ribed in this !hapter Also, Tool +at!hin" was eentuallyrenamed be!t +at!hin"

8/14/2019 GATBDEVE

10/98


"%oring Pro%edures

The A*/P altered s!orin" pro!edures to redu!e the effe!ts of test?ta8in" strate"ies

Preious GATB forms used number?!orre!t s!orin", in whi!h the final s!ore is simply the totalnumber of >uestions answered !orre!tly, with no penalties for in!orre!t answers E$aminees who

were willin" to "uess, een to the point of respondin" randomly (but rapidly) to items in the more

speeded tests, were able to in!rease their total s!ores Efforts to redu!e the speededness of the

power tests were desi"ned to redu!e the influen!e of this type of test?ta8in" strate"y

After reiewin" alternatie approa!hes, the A*/P sele!ted a !onentional formula s!orin"

pro!edure (ie, one infli!tin" a penalty for ea!h in!orre!t response) for use with the three remainin"

speeded tests The penalty for in!orre!t responses is based on the number of response alternatiesfor ea!h item The !on!ept is that, if there are 8 alternaties, an e$aminee who responds randomly

will hae 8?1 in!orre!t responses for eery !orre!t response *espondin" randomly does not re>uire

any 8nowled"e of the !onstru!t bein" measured The !onentional formula introdu!es a penalty forin!orre!t responses that will !an!el out the number of !orre!t responses e$pe!ted by !han!e

throu"h random respondin" The "eneral form of the formula is *?.(8?1), where * is the number

of !orre!t (ri"ht) responses, . is the number of in!orre!t (wron") responses, and 8 is the number ofoptions for ea!h item The spe!ifi! s!orin" formulas for the three GATB speeded tests in Forms E

and F are as follows#

D omputation# * ? .= (a redu!tion of 1=th point for in!orre!t responses)%

D be!t +at!hin"# * ? .' (a redu!tion of 1'rd point for in!orre!t responses)%

D 6ame omparison# * ? . (a redu!tion of one point for in!orre!t responses)

The A*/P !onsidered alternatie approa!hes that in!luded a lar"er penalty for "uessin"+ost speeded tests !ontain items that all e$aminees should answer !orre!tly "ien suffi!ient time

+ore seere penalties for in!orre!t responses are sometimes introdu!ed in an attempt to for!e

e$aminees to ta8e enou"h time with ea!h item to answer it !orre!tly This approa!h was notre!ommended for use with GATB Forms E and F for two reasons First, it introdu!es effe!ts of

test?ta8in" strate"y, albeit in the opposite dire!tion Guessin" would lead to lower s!ores in

!omparison to omittin" e!ond, pla!in" a "reater emphasis on a!!ura!y relatie to speed !han"esthe !onstru!t bein" measured to some e$tent, ma8in" "enerali@ations from prior alidity studies

more tenuous

The A*/P de!ided that use of number?!orre!t s!orin" should be !ontinued for the power

tests To the e$tent that e$aminees answer all items in a test, the number !orre!t and formula s!oresare linearly related E$aminees are ordered in e$a!tly the same way in both !ases The number?

!orre!t s!ore is simpler to e$plain to e$aminees 4nstru!tions to e$aminees on how to ma$imi@e

their s!oresNby attemptin" to answer eery itemNare also simpler when usin" number?!orre!t

s!orin"

(nstru%tions to E.amineesFor GATB Forms A throu"h /, both the "eneral instru!tions and the test?spe!ifi!

instru!tions proide limited information re"ardin" test?ta8in" strate"ies, and neither dis!usses

s!orin" pro!edures Test standards deeloped by the AE*AAPA6+E oint !ommittee on test

standards (1:57) re>uire that e$aminees be told how tests will be s!ored and "ien spe!ifi!instru!tions that allow them to ma$imi@e their s!ores For this reason, !han"es to test s!orin"

pro!edures were a!!ompanied by !onsideration and reision of e$aminee instru!tions

8/14/2019 GATBDEVE

11/98

(tem Pretest) /urin" the item pretest, both "eneral and test?spe!ifi! instru!tions weremodified to improe the information proided to e$aminees The releant portion of the "eneralinstru!tions used durin" the pretest stated

.ou pro)a)ly will not )e a)le to finish all the ,uestions in the first

three parts$ Each part has so many ,uestions that very few peoplecan finish in the time allowed$ /owever' answer as many as you

can$

For ea!h of the speeded tests, the instru!tions "ien after !ompletion of the pra!ti!e items

in!luded the followin"#

0or- as FA&T and as 1A*EF233. as you can$ +n this eercise&PEED is very important$ 4f you have some idea of the answer to a

,uestion' even if you are not a)solutely positive' it is to your

advantage to ta-e your BE&T G2E&&$ For eample' if you caneliminate one or more of the choices to a ,uestion' ta-e your BE&T

G2E&&$ /owever' if you have no idea what the correct answer is'

don5t spend time guessing$ 6ove on to the net ,uestion$

For the power tests, e$aminees were simply instru!ted to Lwor8 as A0*ATE-O and FAT asyou !anM

Test Tr/out) /urin" the test tryout, instru!tions were modified to proide information ons!orin" pro!edures as well as adi!e on test ta8in" strate"y At this time, the tests were reorderedso that all of the power tests were administered first, followed by the three speeded tests eparatesets of "eneral instru!tions were proided for the power and speeded tests The "eneral instru!tions

for the power tests (Parts 1, 2, and ') stated

+n the net three parts wor- 1A*EF233.$ .ou should haveenough time to answer each ,uestion$ 4t is to your advantage to

A%&0E* E7E*. 82E&T4+%$ Even if you5re not sure of ananswer' ma-e your BE&T G2E&&' fill in your answer' then go tothe net ,uestion$ .our score for each part will )e the num)er of

,uestions you answer correctly$ There is no penalty for answering

incorrectly$

This information was repeated in the spe!ifi! instru!tions followin" the pra!ti!e items for ea!h of

the power tests

After the power tests were !ompleted, "eneral instru!tions for the speeded tests were

proided These instru!tions stated#

The net three parts are different from the parts you5ve alreadyta-en$ +n these parts' &PEED is 7E*. 46P+*TA%T$ .ou won5t

have time to answer every ,uestion$ .ou must wor- as FA&T as

you can )ut don5t )e careless$

4f you have even the slightest idea of the answer' it is to your

advantage to ma-e your BE&T G2E&&$ 4f you can eliminate one or

more wrong choices to the ,uestion' then ma-e your BE&T G2E&&

8/14/2019 GATBDEVE

12/98


from the remaining choices$ /owever' if you have no idea of thecorrect answer' don5t spend time guessing9 go to the net ,uestion$

.ou will receive one point for each correct answer$ .ou5ll )e

penali:ed for wrong answers$ Points will not )e su)tracted for

,uestions you don5t answer$

This information was also repeated in the spe!ifi! instru!tions followin" the pra!ti!e itemsfor ea!h of the speeded tests At that point, e$aminees were told the spe!ifi! penalty for in!orre!tresponses on that test For the omputation test, for e$ample, e$aminees were told#

.ou will receive one point for each correct answer$ .ou5ll lose one

,uarter (;

8/14/2019 GATBDEVE

13/98

omputation and Arithmeti! *easonin" items to !onform to the amount of spa!e needed by ea!h

item

Test (tems) 4n "eneral, a number of !han"es were made to improe the appearan!e anduser friendliness of the test items themseles pe!ifi! !han"es for the indiidual items are

dis!ussed below for ea!h test

Arithmetic *easoning 4ndiidual items were pla!ed in !ells !ontainin" one double erti!alline (ie, two erti!al lines ada!ent to ea!h other) and one sin"le hori@ontal line Althou"hthe two?!olumn format was maintained, the number of items on ea!h pa"e was redu!ed

nly Arabi! numerals (instead of words) were used to e$press numbers E$!ept for

monetary alues, a @ero was used as the first di"it for de!imal alues less than one

1omputation The same item format deeloped for the Arithmeti! *easonin" items wasused for the omputation items Also, there were no more than ei"ht omputation items on

ea!h pa"e onsisten!y in pun!tuation, response ali"nment, and use of monetary symbols

was maintained throu"hout the test Finally, the arithmeti! operation symbol for ea!h itemwas pla!ed within the item (4n prior forms, the operation symbol was pla!ed aboe the

item)

Form 6atching The se!ond item blo!8 was redu!ed from '7 to 27 items, and the number

of response options was redu!ed from 1; to 7

%ame 1omparison The number of items on ea!h pa"e was redu!ed from 7; to '; Thehori@ontal line after eery fifth item was repla!ed by a blan8 line, and a blan8 spa!epre!eded and followed the dash that separated the two names within ea!h item

+)ject (Tool# 6atching The print >uality was improed, and the test title was !han"ed

from Tool +at!hin" to be!t +at!hin"

Three!Dimensional &pace The print >uality and resolution were improed throu"h the useof the orel/*A.Q = "raphi!s pa!8a"e (orel orporation, 1::') to deelop the

indiidual items

7oca)ulary The item format was !han"ed from hori@ontal to erti!al The number of

items on ea!h pa"e was redu!ed from to '; to no more than 1; The 1: items were arran"edin three !olumns of fie items and one four?item !olumn, ea!h separated by a double

erti!al line and one hori@ontal line

Ans-er "#eet) eeral !han"es were made to the answer sheets used in the Forms E andF deelopment resear!h 4n "eneral, response formats were reised to !onform to those made in the

test items *esponse bubbles were redu!ed in si@e by eliminatin" the top and bottom portions of

ea!h bubble and addin" small hori@ontal lines to form oals This !han"e was made in response to a!on!ern raised in the 6A reiew that diffi!ulty in fillin" in lar"e !ir!les mi"ht impede

performan!e on the speeded tests pe!ifi! !han"es in!orporated at ea!h sta"e in the deelopment

resear!h are des!ribed below

4tem Pretest Phase$ Four s!annable answer sheet formats were used durin" this phase fordata !olle!tion The answer sheets were modified to# (1) !ontain four se!tions% (2) support the

e$perimental test !onfi"uration% (') a!!ommodate format !han"es made to Form +at!hin" and

8/14/2019 GATBDEVE

14/98


Test Tryout Phase$ The test tryout answer sheet was used with both Forms E and F 4t was

printed in purple to differentiate it isually from Form A and Form B answer sheets Thenumber of se!tions was in!reased from four to seen, and they were rearran"ed to !oin!ide with

the new test arran"ement (Part 9?Form +at!hin" was later eliminated) The shaded pra!ti!e

bo$es were pla!ed at the top left of ea!h se!tion The !orre!t number of item responses waspla!ed in ea!h se!tion and reformatted into !olumns of e>ual numbers with the tops of ea!h

!olumn startin" on the same line .ith the elimination of the wraparound !olumns, the phraseLBe"in CereM be!ame unne!essary and was eliminated The demo"raphi! se!tion was modifiedto !olle!t resear!h?spe!ifi! information The most notable modifi!ation, howeer, was repla!in"

the lar"e response !ir!les with oals

+perational Phase$ The operational answer sheet differs from the test tryout answer sheet inthe followin" ways#

D *esear!h?spe!ifi! demo"raphi! information was eliminated

D The Form +at!hin" se!tion (Part 9) was eliminated

D The response identifi!ation letter was pla!ed inside ea!h response oal

DAlternate blo!8s of fie item responses were shaded

Administration Manual) The spe!ifi! reisions made to date in!lude

D reformattin" as a te!hni!al manual%

D in!orporatin" a !onersational business tone (ie, employin" simple, !on!rete, !learlan"ua"e)%

D writin" in a manner to address readers needs first%

D !onsolidatin" pro!edures and instru!tions into distin!t sube!t areas (ie, eliminatin"unne!essary lan"ua"e, in!onsisten!ies, repetition, and redundan!y)%

D subtitlin" ea!h sube!t area%

D

updatin" and rewritin" information so that the readin" leel and detail are appropriate forall users%

D formattin" manual into a lo"i!al se>uen!e for better understandin" of administrationpro!edures and instru!tions%

D addin" !uein" "raphi!s and bullets%

D usin" !olored paper as a !uein" te!hni>ue%

D in!reasin" white spa!e and redu!in" line len"th%

D !han"in" to 12?point serif typefa!e with in!reased use of headers%

D usin" hi"her "rade paper with improed print >uality%

D deelopin" a standardi@ed introdu!tion s!ript% and

D bindin" the manual into an 5 12 in by 11 in hardba!8 binder (be!ause of on"oin"

modifi!ations)

The followin" !han"es hae been proposed for the final ersion of the administration

manual#

D different !olors of print to sere as !uein" dei!es for the test administrator (e", phrasesto be read aloud) at different lo!ations in the manual%

D redu!in" the si@e of the pa"es to 9 in by 1; in with wire spiral bindin"% and

D in!ludin" demonstration models for the Three?/imensional pa!e pra!ti!e items

8/14/2019 GATBDEVE

15/98

The final administration manual for GATB Forms E and F will in!lude s!orin" pro!edures

and !onersion tables The use of administration aids su!h as !olor, !he!8lists, and forms will be

in!reased Additional aestheti! and format !han"es will be made to further enhan!e readability andusability, su!h as redu!in" the si@e of the manual, tabbin" the se!tions, addin" more !uein"

"raphi!s, and in!orporatin" a two?!olumn format with shorter lines

De'elo$ment and Re'ie- of Ne- (temsAn e$tensie effort was underta8en to deelop new items for GATB Forms E and F 6ote

that many more items were deeloped than used in the final forms Efforts to spe!ify !ontent anddiffi!ulty !ate"ories, write items for ea!h of these !ate"ories, and then reiew items for editorial

and sensitiity !onsiderations are des!ribed here 4tem tryouts, statisti!al s!reenin", and !alibration

pro!edures are des!ribed in the followin" se!tion

(tem riting

A brief des!ription of the deelopment of e$perimental items for Parts 1?9 of Forms E and F

is "ien below For ea!h test, items from preious forms were analy@ed and sorted into !ate"oriespotentially related to item diffi!ulty our!es for item material also were identified +ore detailed

information on spe!ifi!ations and item types!ontent !ate"ories for ea!h test are presented in the

Forms E and F proe!t te!hni!al report (+ellon et al, 1::3) Brief summaries of the itemdeelopment pro!edures used for ea!h test are presented here

Name Com$arison) The =;; 6ame omparison items were deeloped to be parallel toForm A items and representatie in terms of "ender and ethni!ity The number of items with names

that were the same was e>ual to the number of items with different names 4tem sour!es in!ludeddire!tories, di!tionaries, and item deeloper !reatiity Analyses were then performed to deelop

preliminary estimates of item diffi!ulty Based on these analyses, the number of !hara!ters in the

left?hand !olumn of the two?!olumn format used for this test was sele!ted as the item diffi!ultymeasure The 2;; items for ea!h form were diided into four 7;?item >uarters of appro$imately

e>ual estimated oerall diffi!ulty The item order was then randomi@ed within ea!h >uarterCom$utation) The 1'3 omputation items were deeloped to be parallel to Forms A?/

The ori"inal items were deeloped and reiewed to ealuate diffi!ulty The number of di"its a!rossnumbers within ea!h type of operation was used as the item diffi!ulty measure The 35 items forea!h form were diided into four 19?item >uarters of e>ual estimated oerall diffi!ulty Type ofarithmeti! operation and response options were balan!ed within ea!h >uarter A low?diffi!ulty itemwas assi"ned to the first position within ea!h >uarter with the remainin" items ordered randomly

T#ree*Dimensional "$a%e) The 1'; Three?/imensional pa!e items were deeloped tobe similar in !ontent to prior forms The number of folds was used as a measure of item diffi!ulty%

it had si$ leels 6ewly deeloped items were "rouped a!!ordin" to the number of folds so that ane>ual number of items would be deeloped for ea!h of the si$ diffi!ulty leels 4tems were then

drawn on a !omputer, usin" the A//?' software pa!8a"e 4tems were !ontinually reiewed for

!larity and !orre!tness, and shadin" was added ompleted items were transferred to +ylar paper

and redu!ed in si@e photo"raphi!ally, then plates were made for printin" 4tems were reieweda"ain and reised when ne!essary 4tems were then assi"ned to forms on the basis of diffi!ulty, and

response options were !he!8ed and tallied ption positions were !han"ed as ne!essary The items

were rephoto"raphed and printed

The A*/P used the orel/*A.Q = "raphi!s pa!8a"e (orel orporation, 1::') to redrawall of the items to ma8e them !onsistent in appearan!e amera?ready !opies of the reformatted

items were prepared and sent to a "raphi! artist for proofin" ome of the items were later reised

8/14/2019 GATBDEVE

16/98


to !orre!t the problems identified by the "raphi! artist Three diffi!ulty leels were identified based

on the number of folds andor rolls made in ea!h item These diffi!ulty alues were then used toform three 13?item >uartiles and one 19?item >uartile of appro$imately e>ual estimated oerall

diffi!ulty within ea!h form .ithin ea!h >uartile a low?diffi!ulty item was assi"ned to the first

position with the order of the remainin" items randomi@ed The !orre!t response option fre>uen!ieswere balan!ed within ea!h >uartile

Vo%abular/) The 13; uen!y distributions were balan!edwithin >uartiles and forms

Ob&e%t Mat%#ing) The 13' ori"inal be!t +at!hin" items were deeloped to be parallelto Forms A?/ The A*/P used the number of shaded areas in the four response alternaties for

ea!h item to estimate diffi!ulty leel /iffi!ulty leel, !ontent !onsiderations, and lo!ation of the!orre!t response were used to form four 2;?item >uartiles of similar oerall diffi!ulty for ea!h

form (Three items were deleted) The item order was randomi@ed within ea!h >uartile A surplus

item was then added to ea!h >uartile to form three seen?item pa"es that !ould be shifted to meetthe re>uirements of the resear!h desi"n

Arit#meti% Reasoning) The 33 Arithmeti! *easonin" items were deeloped to be

parallel to Form A 6ew situations, !ontemporary monetary alues, "ender representation,e$!lusion of e$traneous information, and a si$th?"rade readin" leel were additional !onsiderationsin item deelopment The A*/P reiewed and reised the items so they !onformed more !losely to

the "uidelines for deelopment 4tem diffi!ulty was estimated by the number of operations needed

to sole the problem, the type(s) of operations, and the number of di"its in!luded in the terms usedin the operation(s) ne of the two least diffi!ult items was assi"ned to the first item position in

Form E and the other item assi"ned to Form F The remainin" 3= items were then assi"ned to four

ei"ht?item >uartiles for ea!h form on the basis of diffi!ulty, type (s) of operation(s), !orre!tresponse 8ey, and !ontent The items in ea!h >uartile were ordered from least to most diffi!ult with

the item order then randomi@ed within ea!h >uartile

Form Mat%#ing) The 2;; Form +at!hin" items were deeloped to be parallel to Forms

A?/ items in terms of !ontent and parallel to Form A item si@e and arran"ement Ei"ht 27?itemblo!8s were deeloped by modifyin" ea!h of the ei"ht blo!8s of items in Forms A?/ The number

of response options for ea!h item was redu!ed from 1; to fie

Editorial Re'ie- and "%reening

Literature Re'ie-) The deelopment of the item reiew pro!edures be"an with aliterature reiew fo!usin" on the pro!ess for !ondu!tin" item reiews and sele!tin" the parti!ipantsin the reiew pro!ess Two types of reiew pro!edures were identified# (1) pro!edures used for

resear!h purposes, and (2) pro!edures used in on"oin" testin" pro"rams

8/14/2019 GATBDEVE

17/98

(tem Re'ie- (nstruments) Throu"h the literature reiew, it was determined that (1)most preious item reiew pro!edures were desi"ned for use with edu!ational a!hieement tests,and (2) reiew pro!edures used in most preious studies were not hi"hly stru!tured and appeared to

be deeloped independently with limited "uidan!e from the edu!ational measurement literature

Coweer, the literature reiew did un!oer ei"ht referen!es (Boldt, 1:5'% Cambleton & *o"ers,1:55% Carms, 1:95% -o!8heed?Kat@, 1:9=% +adaus, Airasian, Cambleton, onsalo, & rlandi,

1:9:% lson & moyer, 1:55% !hrat@ & .ellens, 1:51% Tittle, 1:52) that proided the foundationfor the instruments and pro!edures that were used in the GATB Forms E and F item reiew These

ei"ht referen!es proided information in three areas# bias "uidelines, pro!edural issues, and ratin">uestions

Preliminar/ Re'ie-) /raft ersions of item sensitiity reiew >uestions, instru!tions, andan answer form were sent to A*/ !enters for reiew Based on the !omments, A*/P staff reiseddraft ersions of the sensitiity reiew materials and sent them to A*/s for further reiew The

only reision was a minor !han"e in the answer form

Pilot Test) A pilot test was !ondu!ted in?house with three ooperatie Personnel eri!es(P) staff members, enablin" indiiduals who were not inoled in the A*/P test resear!hpro"ram to proide input to the reiew pro!ess The results led to a number of modifi!ations in

pro!edures, instru!tions, and do!uments that would be used for the item reiew

(tem Re'ie- Materials)6ine do!uments were used in the item reiew pro!ess# (1) a listof the !riteria to sele!t panel members, (2) a !onfidentiality a"reement, (') a des!ription of theGATB tests and aptitudes, (=) written instru!tions for panel members, (7) the administrators

ersion of the written instru!tions for panel members, (3) a list of !hara!teristi!s of unbiased test

items, (9) a list of the reiew >uestions with e$planations, (5) an answer form, and (:) an answerform supplement

Panel Member C#ara%teristi%s) een panel members parti!ipated in the reiew Thepanel in!luded two Afri!an Ameri!ans, three Cispani!s, and two whites Three members were maleand four female Three members were personnel analysts, two were uniersity professors in

!ounselor edu!ation, one was a personnel !onsultant, and one was a postdo!toral fellow ine!onomi!s

Pro%edures) At an orientation meetin" held at ea!h of the three parti!ipatin" A*/s,!onfidentiality a"reements were si"ned, GATB items and instru!tions were "ien to panel

members, and seeral items in ea!h test were reiewed and dis!ussed Panel members reiewed the

remainin" items at their !onenien!e After all items were reiewed, a follow?up meetin" was heldat ea!h !enter to resole any problems and to dis!uss the reiew pro!ess

"ummar/ of Results) The answer forms of ea!h panel member were reiewedummaries of the !omments for ea!h test are presented below

%ame 1omparison = omments fo!used on ra!ial, ethni!, and "ender stereotypin" and

representation pe!ifi! !on!erns in!luded the la!8 of female and minority businesses, andthe need for more females in nontraditional professions, obs, and businesses

1omputation = omments primarily dealt with item !hara!teristi!s pe!ifi! !on!erns

in!luded diffi!ult and time?!onsumin" problems that mi"ht be s8ipped by testwise

appli!ants, poor distra!tors, and un!lear instru!tions

Three!Dimensional &pace omments !on!erned possible "ender bias and item!hara!teristi!s omments in!luded the presen!e of male?oriented items and abstra!t items

8/14/2019 GATBDEVE

18/98


that mi"ht be unfamiliar to females, diffi!ult and time?!onsumin" items that !ould be

s8ipped by testwise appli!ants, "ender?biased instru!tions, and oerly !ompli!ated items

7oca)ulary omments !on!erned hi"h readin" "rade leel, oerly diffi!ult words% wordswith different meanin"s for different "roups% and in!lusion of forei"n?lan"ua"e words and

te!hni!al, biolo"i!al, and s!ientifi! terms

Tool 6atching omments fo!used mainly on possible "ender bias due to differen!es infamiliarity and the presen!e of male?oriented items Coweer, !on!erns were also e$pressedthat items with ele!tri!al and me!hani!al !omponents mi"ht !ause problems for minorities

due to la!8 of familiarity and opportunity to learn ther !omments !on!erned !larity of

instru!tions and positionin" of the response letters for the item alternaties

Arithmetic *easoning +ost !omments were dire!ted toward two areas# (1) ra!ial, ethni!,and "ender representation, and (2) "ender o!!upational and a!tiity stereotypin" ther

!omments !on!erned time?!onsumin" items that mi"ht be s8ipped by testwise appli!ants,

!onfusin" and in!omplete instru!tions, the presen!e of items that were oerly !ompli!atedor inoled too many steps, and some "roups not hain" the opportunity to learn how to

perform the operations needed to answer the !omple$ items

Form 6atching omments in!luded a possible pra!ti!e effe!t for the test and un!lear

instru!tions be!ause of readin" leel omments that were dire!ted toward spe!ifi! itemsin!luded linear illustrations bein" per!eied as Lhostile,M minute differen!es amon" shapes,

and possible !onfusion due to shape similarity and lo!ation

(tem Content Re'ision) Based on results from the panel ealuation, the !ontent ofspe!ifi! items was reised A summary of the types of !han"es introdu!ed for ea!h test is presentedhere

%ame 1omparison The reisions addressed the ra!ial, ethni!, and "ender stereotypin" and

representation !riti!isms Guidelines based on the 1::; 0 ensus were used to in!rease

ra!ialethni! and "ender representation tereotypin" was addressed by in!ludin" items with

minorities and females in nontraditional o!!upations and businesses% more professionalo!!upations and businesses were in!luded Fewer items with Germani! names were used

Format !han"es in!luded separatin" the items into blo!8s of fie, eliminatin" hori@ontallines, and in!reasin" the hori@ontal and erti!al spa!e within and between items Finally, the

instru!tions were reworded to in!rease !larity% bold and itali!i@ed types were used for

emphasis

1omputation /istra!tors were reised to ma8e them more plausible based on fie errortypes +inor format !han"es in!luded addin" !ommas to numbers with at least four di"itsand pla!in" the operation si"n within the item Finally, the instru!tions were rewordedsli"htly to in!rease !larity, and bold and itali!i@ed types were used for emphasis

Three!Dimensional &pace 4ndiidual items were reised when needed to in!rease !larity*eisions were reiewed by a "raphi!s e$pert familiar with the test format and the drawin"

software to ensure that the items were free of errors 4nstru!tions were reworded sli"htly toin!rease !larity and eliminate possible "ender bias% bold and itali!i@ed types were used for

emphasis

7oca)ulary .ords were repla!ed on the basis of the item reiew panel member !omments

and on an analysis of word diffi!ulty in /ale and *our8e (1:51) 4tems were modified asneeded to ensure that ea!h items leel of word diffi!ulty was appropriate, word forms

within items were identi!al, and the same type of !orre!t response (ie, synonym or

8/14/2019 GATBDEVE

19/98

8/14/2019 GATBDEVE

20/98


items were printed in ea!h boo8let pe!ifi!ally, for Form +at!hin", two sets of 27 items ea!h

were printed in ea!h boo8let% for Tool (be!t) +at!hin", two sets of 2; items ea!h were printed inea!h boo8let% and for 6ame omparison, three sets of 7; items ea!h were printed in ea!h boo8let

The intention was that the items within ea!h blo!8 would be analy@ed for the boo8let where the

blo!8 appeared in the first position 4n this way, enou"h e$aminees would !omplete (rea!h) theitem to allow assessment of item diffi!ulty, item?total !orrelation, and sub"roup differen!es in item

performan!eFor the speeded tests, there was no attempt to e>uate a!ross boo8lets at the item leel 4tem

results were e$pe!ted to ary widely a!!ordin" to ea!h items position in the test, so e>uatin"would be handled at the test leel in the form !alibration study For the power tests, an attempt was

made to e>uate item diffi!ulties and 4*T parameter estimates not only between the Forms E and F

item sets, but also with the item parameter estimates from an operational form, Form Aonse>uently, all items from the Form A ersion of a "ien power test were in!luded alon" with

half of the new items for that test (ie, all items for either Form E or F) in Part = of a tryout boo8let

The A*/P determined the order of items within ea!h power test by diidin" Form A

(an!hor) items into dis!rete blo!8s and then spa!in" these blo!8s throu"hout the tryout boo8letThe remainin" item positions were filled with new items For power tests, an items position within

a form should not affe!t its diffi!ulty (or dis!rimination) The tryout boo8let desi"n in!luded

proision for testin" this assumption pe!ifi!ally, two ersions of ea!h power test were !reated

with the order of the new items reersed in the een?numbered tryout boo8lets relatie to theirpositions in the odd?numbered boo8lets The (Form A) an!hor items were printed in the same

position in ea!h of these boo8let pairs Table 2?2 summari@es the desi"n of the 13 boo8lets

deeloped for the item tryout study

Table 2*2(tem Tr/out !oo1let Design

Tryout Booklet

(Booklet pair arei!e"ti#al e$#ept %orre&ere! or!er o%Part ' "e ite)*

Part +For)Mat#,

-. ite)/+. )i"0

Part 2Tool

Mat#,'1 ite)/

)i"0

Part 3Na)eCo)p0

+-. ite)/++ Mi"

Part 'O"e o% t,e Poer Tet

(For) A a"#,or ite) ere 4roupe! i"%i$e! 5lo#k6 "e ite) %ille! re)ai"i"4

poitio"0*

For) E For) F 7te) 8 7te) 8 7te) 8 Tet A"#,or Ne Ti)e

1,2 :,1; 1?7; 1?=: 1?17; omputation 7; 35 394tems 4tems min

',= 11,12 23?97 21?3: 71?2;; '/ pa!e =; 4tems 37 7; min

4tems

7,3 1',1= 71?1;; =1?5;, 1;1?2;;,

8/14/2019 GATBDEVE

21/98

ample si@e tar"ets were set to support a!!urate estimation of item statisti!s and analyses of

differential item fun!tionin" (/4F) a!ross "ender and ra!eethni! "roups An oerall tar"et of 1,;;;

e$aminees per item was set to support item response theory (4*T) and !lassi!al item analyses.ithin the oerall tar"et, a rou"hly e>ual split of males and females, and a minimum of 2;;

members from ea!h of the three ra!eethni! "roups to be !ompared (.hites, Afri!an Ameri!ans,

and Cispani!s) were desired amplin" at ea!h site was !ondu!ted to assure an e>ual "ender split

within ea!h ra!e"ender "roup insofar as possible These tar"et sample si@es applied to pairs ofboo8lets that differed only in the orderin" of the Part = (power test) items E$!ept for analyses of

item position effe!ts in Part =, the data for ea!h pair of boo8lets were pooled in the item analysesA tar"et sample si@e of 7;; e$aminees per boo8let was set to a!hiee samples of 1,;;; for ea!h

pair of boo8lets

The sube!ts for the 4tem Pretest phase were appli!ants of Employment eri!e lo!al offi!es

in the fie A*/P "eo"raphi! re"ions Ea!h A*/ tested appro$imately the same number ofe$aminees, and to the e$tent allowed by re"ional demo"raphi! !hara!teristi!s, ea!h !enter obtained

the same minimum subsample si@es for the re>uired ethni! and "ender "roups as spe!ified in the

resear!h desi"n The total number of indiiduals tested for this study was :,2'9 The sample was

appro$imately =7 per!ent female and 77 per!ent male ample ethni! !omposition in per!enta"es

was as follows# Afri!an Ameri!an, =; per!ent% Asian, ' per!ent% Cispani!, 15 per!ent% white, '7per!ent% 6atie Ameri!an, 2 per!ent% and sube!ts !hoosin" the LotherM !ate"ory, 2 per!ent The

mean a"e and edu!ation of the total sample were '=:1 and 1232 years, respe!tiely The standarddeiation (years) was 1229 for a"e and 292 for edu!ation Table 2?' shows the sample

!omposition by boo8let, "ender, and ra!e for sube!ts with usable data on the Form +at!hin" test

(Part 1)

Data Colle%tion Pro%edures

Ea!h sube!t !ompleted 1 of 13 item tryout boo8lets Time limits were set to enable thesube!ts to !omplete all items in one power test, and 27 per!ent of the items in one form of thethree speeded tests Althou"h the time limits for the speeded tests were in!reased sli"htly aboe

operational time limits, the speeded nature of these tests was presered ne half of the sube!ts!ompleted Form E, and the other half !ompleted Form F All sube!ts !ompleted one of the threespeeded tests (Parts 1, 7, and 9), one of the four power tests (Parts 2, ', =, or 3), and an an!hor test(GATB Form A) for the power test The oerall time limit ran"ed from 99 to 1;; minutes for ea!hof the ei"ht test boo8lets for ea!h test form as shown in Table 2?2

Four answer sheets were prepared to !onform to the four tests within ea!h boo8let All data

were re!orded on an opti!al s!anner and read to dis8ettes The answer sheets and dis8ettes were

submitted to PA*/ for analysis PA*/ also prepared data !olle!tion and submissioninstru!tions

8/14/2019 GATBDEVE

22/98


Table 2*3(tem Tr/out "am$le "i4e for Ea%# Test !oo1let

Booklet Total Male Fe)0 9,ite Bla#k Hip01R2 1,2'' 333 739 7=5 ':' 211

1 79= 2:: 297 2== 157 11;2 37: '39 2:2 ';= 2;5 1;1

'R= 1,13: 3== 727 '53 =9' 213' 3;= '1: 257 1:= 2=1 127

= 737 '27 2=; 1:2 2'2 :1

7R3 1,2;1 335 7'' '1: 75: 2;;

7 777 '12 2=' 1=7 275 1113 3=3 '73 2:; 19= ''1 5:

9R5 1,1:' 331 7'2 =9: ==3 2;=9 3;= '': 237 2== 215 1;9

5 75: '22 239 2'7 225 :9

:R1; 1,1;2 3;7 =:9 =;' =;' 2;9

: 799 '23 271 22' 1:7 1121; 727 29: 2=3 15; 2;5 :7

11R12 1,112 32' =5: '97 =79 2;9

11 775 ';; 275 1:9 21: 1;=12 77= '2' 2'1 195 2'5 1;'

1'R1= 1,;13 793 ==; 299 =3= 1:31' 77' '1= 2': 173 2== 1;9

1= =3' 232 2;1 121 22; 5:

17R13 1,1=; 3;2 7'5 =1' ='7 21=

17 731 ';2 27: 211 2;7 1;=

13 79: ';; 29: 2;2 2'; 11;

Test Administrator Training) Trainin" for !olle!tin" item pretest data wasa!!omplished in two sta"es First, A*/P staff !ondu!ted a two?day train?the?trainer session to

proide proe!t lead staff from ea!h of the fie A*/s with a !omprehensie oeriew of theproe!t and detailed instru!tions for trainin" staff who would !olle!t the data The lead staff were

then responsible for !ondu!tin" similar trainin" sessions in their respe!tie "eo"raphi!al re"ions

Trainin" was diided into ei"ht modules#

+odule 1 An eriew of the E and F /eelopment Proe!t

+odule 2 E and F Pretest /ata olle!tion eriew

+odule ' Administration eriew

+odule = Administerin" the GATB

+odule 7 Before, /urin", and After the Testin" ession

+odule 3 pe!ifi! 4nstru!tions for GATB Tests

+odule 9 Administerin" the GATB ? A Pra!ti!al E$er!ise

+odule 5 he!8in" in with the /ata olle!tion oordinator

8/14/2019 GATBDEVE

23/98

+aterials needed for !ompletin" the trainin" modules in!luded# Trainer5s Guide' GATB

4tem Pretest Administration 6anual, sample test boo8lets, and sample answer sheets The

Administration +anual was desi"ned to !ontain all information needed by data !olle!tion staff for!ompletin" trainin" and a!tual data !olle!tion opies of the Trainer5s Guide and the GATB 4tem

Pretest Administration 6anual are in!luded in Appendi$ E of the Technical *eport on the

Development of GATB Forms E and F (+ellon et al, 1::3)

i$teen test boo8lets and four separate s!annable answer sheet formats were re>uired to!olle!t data for the 13 test ersions 4nstru!tions for mat!hin" ea!h of the four answer sheets to the

appropriate test boo8let were proided in the GATB 4tem Pretest Administration +anual The

Administration +anual instru!tions also in!luded a !he!8list of all supplies and materials neededfor data !olle!tion PA*/ proided all materials e$!ept s!rap paper, pen!ils, and stop wat!hes to

ea!h A*/ 6A*/ deeloped and distributed s!annin" software and instru!tions

/ata were !olle!ted se>uentially for ea!h of the 13 test boo8lets, be"innin" with Boo8let

6o 1 .here possible, within ea!h re"ion, tar"eted sample si@es for the si$ sub"roups (Afri!anAmeri!an female, Afri!an Ameri!an male, Cispani! male, Cispani! female, white female, and

white male) and the total sample si@e for ea!h boo8let were obtained before be"innin" data

!olle!tion on ea!h su!!eedin" boo8let This was done to allow for preliminary reiews of databefore the entire data !olle!tion effort was !ompleted pe!ifi! instru!tions for !olle!tin",

re!ordin", and submittin" data were proided in the Administration +anual

Data Anal/sis

Analyses of the power and speeded test items pro!eeded somewhat independently due to

differen!es in the nature of the data to be analy@ed and the information re>uirements for s!reenin"

and form !onstru!tion The analysis of items for GATB power tests was done under separate!ontra!ts with +easured Pro"ress, C*trate"ies, and Ooun" and Asso!iates .innie Ooun"

performed preliminary edits and produ!ed !lassi!al item analysis statisti!s /r 6eal Kin"ston of

+easured Pro"ress ealuated the dimensionality of the power tests and performed !omputer

analyses to estimate 4tem *esponse Theory (4*T) parameters, in addition to !ondu!tin" apreliminary sele!tion of items C*trate"ies used this information, in !onun!tion with their own

analyses for assessin" differential item fun!tionin" (/4F), to sele!t the final items for the newforms (C*trate"ies, 1::=) /r Frit@ /ras"ow proided te!hni!al adi!e to C*trate"ies staff

The Cuman *esour!es *esear!h r"ani@ation (Cum**) analy@ed speeded test data and sele!ted

items for the four speeded tests (+!loy, *ussell, Brown, /iFa@io, & Green, 1::=) /r Bert Green

proided te!hni!al adi!e to Cum** staff 6ote that the omputation test items were in!luded inboth sets of analyses As noted earlier, this test was ori"inally in!luded with the power tests but

subse>uently treated as a speeded test

The pro!edures used to estimate item statisti!s for (ie, !alibrate) the power test items and

to s!reen them for fairness to different e$aminee "roups are des!ribed in the ne$t se!tion This is

followed by a des!ription of the pro!edures used with the speeded test items and a dis!ussion ofhow item statisti!s were used to sele!t items for the final ersion of ea!h Form E and Form F test

8/14/2019 GATBDEVE

24/98


Calibration and "%reening of Po-er Test (tems

After the raw data were edited, analysis of the tryout data for the power test items be"an

with attention to two methodolo"i!al issues The first issue was whether there were si"nifi!antdifferen!es in apparent item diffi!ulty (and to a lesser de"ree, item dis!rimination) as a fun!tion of

the items position in the tryout boo8let *e!all that ea!h new item was tried out in two boo8lets,

with the orderin" of the new items in the een?numbered boo8lets bein" reersed from their

orderin" in the ne$t lower odd?numbered boo8let The >uestion was whether it was reasonable to!ombine the data from the two boo8lets into a sin"le analysis of item statisti!s The se!ond issue

was whether all of the items for a test measured a sin"le underlyin" !onstru!t Be!ause 4*T models(whi!h are !ommonly used in item s!reenin" and in sele!tin" items for in!lusion in a test) assume

unidimensionality, it was important to !he!8 this assumption before pro!eedin"

n!e these two issues were addressed, PA*/, +easured Pro"ress, and C*trate"ies

estimated item statisti!s and analy@ed /4F a!ross ethni! and "ender "roups At the !on!lusion ofthese steps, a few items had been dropped be!ause of /4F% the remainder of the items were

!alibrated and ready for sele!tion into the final forms Ea!h of these steps is des!ribed briefly here

and dis!ussed in more detail in +ellon et al (1::3)

(tem Position Effe%ts) The effe!t of an items position on its apparent diffi!ulty wasassessed by !omparin" diffi!ulty estimates from the forward and reersed orderin" of the new

items for ea!h test Table 2?= summari@es the distribution of differen!es in item diffi!ulty

(proportion !orre!t) a!ross the two boo8lets for ea!h test Althou"h the results showed a si"nifi!ant!orrelation between item position and proportion passin", the si@e of the differen!es was "enerally

modest The omputation test showed the lar"est item position effe!ts, whi!h was !onsistent with

!on!erns that the test was partially speeded omputation was subse>uently treated as a speeded

test +easured Pro"ress and PA*/ de!ided that !ombinin" data from the two item orderin"s wasthe best approa!h to minimi@e the effe!ts of item position The pooled data were used in the

remainin" analyses

Test Dimensionalit/) /r 6eal Kin"ston of +easured Pro"ress analy@ed thedimensionality of ea!h power test usin" the TETFAT pro"ram (.ilson, .ood, & Gibbons,1::1) Appli!ation of ordinary linear fa!tor analysis pro!edures to di!hotomously s!ored itemariables typi!ally yields e$traneous diffi!ulty fa!tors TETFAT lar"ely !orre!ts this problem byin!orporatin" a lo"isti! model of the relationship between the underlyin" fa!tors and the itemresponses

Table 2*5Effe%t of C#anging (tem Position on (tem Diffi%ult/

Tet

Statiti#Arit,)eti#Reao"i"4

Co)putatio" 3:D Spa#e Vo#a5ulary

+inimum /ifferen!e :0+' :0+; :02+ :0+-

1st Suartile /ifferen!e :0.' :0.' :0.' :0.3+edian /ifferen!e :0.+ 0.+ 0.. :0.+'rd Suartile /ifferen!e 0.- 0.< 0.3 0.3+a$imum /ifferen!e 0+' 023 0+< 03uently treated as a speeded test, 4*Tanalyses were not used with this test For the remainin" three tests, the first fa!tor a!!ounted for

rou"hly half of the total arian!e and the remainin" fa!tors were small and lar"ely uninterpretable

ne of the '?/ pa!e fa!tors appeared to be related to diffi!ulty, but this was most li8ely an

artifi!ial result not fully eliminated by the use of TETFAT onse>uently, the Arithmeti!*easonin", '?/ pa!e, and

8/14/2019 GATBDEVE

26/98


The +antel?Caens@el estimates /4F as a ratio of the probability of passin" the item for

members of the two "roups that is assumed to be !onstant a!ross different ability leels 4tems hae/4F to the e$tent that this odds ratio differs from one (or the lo" of the ratio differs from @ero)

bsered number !orre!t s!ores are typi!ally used to e>uate ability leels for indiiduals within

ea!h ethni! or "ender "roup, and proportion passin" statisti!s are analy@ed within ea!h s!ore leeloftware written by Cambleton and *o"ers (1::') was used to "enerate +antel?Caens@el statisti!s

for ea!h of the power test itemsThe primary "oal of the /4F analyses was to identify those items hain" !learly aberrant

/4F 4tems were fla""ed if /4F statisti!s were si"nifi!ant at p;1 usin" either of the twopro!edures After inspe!tion of the fla""ed items, 17 items were dropped for e$treme /4F alues

ne of these items, a Form A o!abulary item, had been fla""ed for /4F in the Cispani! sample by

the 4*T pro!edure but not the +antel?Caens@el pro!edure All other items had been fla""ed byboth pro!edures Table 2?3 shows the number of items s!reened out by the /4F analyses as well as

by other fa!tors and the number of items that remained for possible use in the final forms

That only items with e$treme /4F were dropped refle!ted a strate"y of !ontrollin" /4F at

the test leel by balan!in" items with positie and ne"atie /4F, rather than !ontrollin" ti"htly atthe item leel Eliminatin" items with mar"inal /4F !ould inadertently narrow the !ontent of the

test, perhaps redu!in" its alidity

Table 2*9(tem "%reening Results for t#e Po-er Tests

Arit,)eti#

Tet

3:D Spa#eS#ree" Reao"i"4 Vo#a5ulary

Total E & F items tried out 33 13; 1';

U dropped for biserial 17 1 ' 5

U e$!luded from 4*T /4F analyses 1 1; 9

U dropped for /4F (Afri!an Ameri!an?.hite) ; = ;

U dropped for /4F (Cispani!?.hite) ; ; ;U dropped for /4F (Female?+ale) 1 ' ;

Total new items remainin" 3' 1=; 122

Calibration and "%reening of "$eeded Test (tems

4tem analysis pro!edures for speeded tests are !onsiderably less standardi@ed relatie to

power test item analysis pro!edures 4n some models, where all of the emphasis is on assessin"speed rather than a!!ura!y, traditional !on!epts of diffi!ulty and dis!rimination do not apply

.ithout !omputer administration, howeer, it is not possible to re!ord response laten!ies on an

item?by?item basis and so analyses of speed at the item leel are not feasible

For the speeded tests on the GATB, there is some eiden!e that items do ary in terms of

diffi!ulty, with response errors bein" more !ommon for some items than for others 4t was thereforeessential that measures of item diffi!ulty and dis!rimination be analy@ed so the new forms !ould be

balan!ed with respe!t to these statisti!s Gien the importan!e pla!ed on test fairness, analyses of/4F a!ross "ender and ethni! "roups were also deemed essential

A spe!ial item analysis pro"ram was deeloped to !ompute the proportion passin"

(diffi!ulty) and point biserial !orrelation (dis!rimination) for ea!h item usin" only those e$aminees

who rea!hed the item 4f the proportion passin" an item were based on all e$aminees, items towardthe end of a speeded test would automati!ally hae low alues, be!ause ery few e$aminees rea!h

that point in the test and attempt to answer the item Althou"h only items in the first blo!8 of ea!h

8/14/2019 GATBDEVE

27/98

speeded test were analy@ed, the relationship of item position to the number passin" the item was

substantial omputin" the proportion passin" only for those e$aminees who rea!hed the item

"reatly redu!ed the artifi!ial !onfoundin" of an items diffi!ulty estimate and its position in thetryout boo8let

The relationship between item position and point biserials !omputed on all e$aminees was

e>ually substantial For e$ample, for the be!t +at!hin" test, the !orrelation between item

position and point biserials aera"ed :1 a!ross the ei"ht item blo!8s when all e$aminees werein!luded in the !omputation of the point biserials .hen e$aminees who did not rea!h the item in

>uestion were e$!luded, howeer, the aera"e !orrelation de!reased to 37

The spe!ifi! item and test statisti!s !omputed for the speeded tests in!luded#

D 6ean and standard deviation of the total score on the analy:ed items$

D 6ean and standard deviation of the item!corrected total test scores$ The item?!orre!tedtotal test s!ore is the total test s!ore e3ludin" the item in >uestion

D 4tem difficulty The diffi!ulty of an item is assessed in !lassi!al test theory by the

proportion of e$aminees who answered the item !orre!tly Be!ause this !omputation was

based only on those e$aminees who rea!hed the item in >uestion, the results are referred

to as L!orre!ted diffi!ulty aluesMD Point )iserial correlations Point biserial !orrelations are an inde$ of item dis!rimination

in !lassi!al test theory Point biserials were !omputed based on all e$aminees and alsobased only on the e$aminees rea!hin" the item in >uestion (!orre!ted point biserials)nly the !orre!ted point biserials were used in the item s!reenin" and form !onstru!tionanalyses

D *esponse!alternative information For ea!h item response alternatie, the proportion ofe$aminees who endorsed that alternatie, the mean !orre!ted total test s!ore for thosee$aminees, and the point biserial !orrelation between the response alternatie and the!orre!ted total test s!ore were !omputed

The statisti!s were also !omputed separately for separate e$aminee "roups, in!ludin"

females, males, Afri!an Ameri!ans, Cispani!s, and whites Be!ause total s!ores are based more onspeed than on a!!ura!y, the models for /4F that "roup e$aminees on the basis of total s!ore and

loo8 for differen!es in item passin" rates within total s!ore leels do not appear appli!able 4n the

present analyses, /4F was analy@ed in terms of effe!t si@es !omputed as the standardi@ed differen!ein !orre!ted passin" rates (ie, proportion !orre!t !al!ulated only on those indiiduals who rea!hed

the item) between ea!h fo!al "roup (females, Afri!an Ameri!ans, or Cispani!s) and the

!orrespondin" referen!e "roup (males or whites) Althou"h this strate"y !an !ause problems when

applied to power tests (e", lower reliability and alidity of the resultin" test), the approa!h isreasonable when sele!tin" items for hi"hly speeded tests, be!ause few if any substantial "roup

differen!es are typi!ally en!ountered for su!h tests 4tem sele!tion for the omputation subtest was

supplemented by e$aminin" +antel?Caens@el /4F statisti!s !omputed by C*trate"ies

4tem response alternatie data were also e$amined, primarily to identify ery poor items(e", those with positie point biserials for the distra!tor% a distra!tor endorsed by e$aminees

whose mean test s!ore e$!ludin" that item was hi"her than the mean s!ore for those who !orre!tly

responded to the item) 6o items were e$!luded for this reason

E$perimental items for the omputation test were tried out in two item orders# forward andreersed orrelations between item diffi!ulty estimates from the odd numbered boo8lets (forward

dire!tion) were !orrelated with the diffi!ulty estimates obtained from the !orrespondin" een

numbered boo8lets (reerse dire!tion) The !orrelations were 9: (for boo8lets 1 and 2) and 51 (forboo8lets : and 1;), su""estin" that item diffi!ulties are hi"hly similar, re"ardless of order of

8/14/2019 GATBDEVE

28/98


presentation onse>uently, item data were pooled a!ross the !orrespondin" boo8lets in the

!omputation of item statisti!s

To minimi@e the ne"atie impa!t on sub"roups, items eiden!in" a sub"roup effe!t si@e"reater than ;27 (that is, one?>uarter standard deiation) were remoed from further !onsideration

4tem response alternatie data were also e$amined for the presen!e of distra!tors with positie

point biserials, or mean item?!orre!ted total test s!ores that were (a) based upon a sample si@e of at

least ten or more, and (b) hi"her than the mean item?!orre!ted total test s!ores for those whoresponded !orre!tly to the item 6o items were remoed from the be!t +at!hin" and 6ame

omparison tests for these reasons Ei"hteen e$perimental omputation items were fla""ed by the/4F analysis f these, ei"ht were in!luded in the proposed Forms E and F (si$ on Form E, two on

Form F) 4n no instan!e did the sele!ted items possess effe!t si@es of "reater than one?half standard

deiation Table 2?9 details the !hara!teristi!s of the items fla""ed by the /4F analysis

"ele%tion of (tems for Final Versions of Forms E and F

4tems for Forms E and F power tests were sele!ted by first !reatin" information "raphs for

the base form (A) and draft new Forms E and F 4nformation "raphs were !al!ulated for the thetas

between ?2 and 2 4nformation for Form A was !al!ulated by summin" the information for allitems, and then s!alin" down to the len"th of the new forms For e$ample, Form A

8/14/2019 GATBDEVE

29/98

"ele%tion of Po-er (tems

The analyses and report of power test item sele!tion were !ondu!ted by C*trate"ies

E$!erpts from their report of pro!edures and results (C*trate"ies, 1::=) are in!luded herePreious wor8 !ondu!ted by +easured Pro"ress had produ!ed draft forms for all three power tests

These forms were preliminary, howeer, in that /4F analyses had not been !ompleted These draft

forms were used as a startin" point in draftin" the new Forms E and F of the power tests usin" the

full pool of items that passed the s!reenin" for /4F

4tems were sele!ted in a!!ordan!e with professionally a!!epted pra!ti!es (see Cambleton &

waminathan, 1:57% -ord, 1:99) to meet the followin" obe!ties#

D The new Forms E and F power tests must be fair'

for !andidates of all ba!8"rounds

D The new Forms E and F power tests must be as parallel as possible

D The new Forms E and F power tests should improe upon the measurement properties of

Form A

D To the e$tent possible, the new Forms E and F power tests should proide measurementthat is uniformly pre!ise oer a broad ran"e of abilities

4tem sele!tion pro!eeded as follows#D The draft forms !reated by +easured Pro"ress sered as a startin" point These initial

forms were based on a desire to hae informatie (ie, dis!riminatin") items thatspanned the ran"e of item diffi!ulties 4n a!tuality, there were many informatie, diffi!ultitems and few informatie, easy items Therefore, item sele!tion fo!used on the need to

proide ade>uate pre!ision below theta J ?1

D Any item dis!arded durin" the /4F s!reenin" was deleted from the form

D 0sin" software written in Pas!al, item information fun!tions were !omputed at 1' thetapoints for all items in the pool, separately by "ender and ethni! "roup The number oftheta points used appeared more than ade>uate for !omparin" the smooth information!ures obsered in the present resear!h

D 4nformatie items were sou"ht to repla!e those deleted as a result of the /4F analysesD The test information fun!tions were then !omputed for Forms A, E, and F as the sum of

the item information fun!tions The use of a spreadsheet fa!ilitated !omputation and

"raphin" of the test information fun!tions for arious !hoi!es of items

The aboe steps were iterated when !omparisons between forms su""ested a la!8 of!orresponden!e between E and F, or when the measurement properties of A were superior to those

of E andor F for some ran"e of abilities

ome of the items in!luded in the forms may hae been asso!iated with small to moderately

lar"e /4F statisti!s durin" s!reenin" Therefore as a final !he!8, measurement e>uialen!e forethni! and "ender "roups was !he!8ed at the test leel

Assessment of Test*Le'el E;ui'alen%e) The final step in forms deelopment was toassess the e>uialen!e of the forms on a total test?s!ore leel to ensure that e$pe!ted s!ores of

e>ually able persons were essentially the same 4t was !ru!ial to ma8e this last, test?leel!omparison be!ause de!isions will be made on the basis of the total test s!ore

The pro!edure for this !omparison was to

'Fair, as it is used here, means that indiiduals with the same leel of ability hae the same e$pe!ted s!ore

8/14/2019 GATBDEVE

30/98

Chapter 2. Development of GATBFormsE and F

D 0se the final linear lin8in" parameters !omputed preiously to mat!h ea!h fo!al "roup

ability metri! to the referen!e "roup metri! (to!8in" & -ord, 1:5') usin" ES0ATE

(Ba8er, Al?Karni, & Al?/osary, 1::1)% and

D Plot the Ts for ea!h "roup on a !ommon metri!

The plots of the e$pe!ted number !orre!t differen!es for the final forms are presented in

Fi"ures 2?1 throu"h 2?' for the final Arithmeti! *easonin",

8/14/2019 GATBDEVE

31/98

+Di%%ere"#e 5etee" t,e e$pe#te! "u)5er #orre#t #ore o% %o#al

a"! re%ere"#e 4roup )e)5er0 Poi"t a5o&e=ero i"!i#ate a!&a"ta4e %ort,e %o#al 4roup0

Figure 2*+) Test C#ara%teristi% Cur'e 7TCC8 Differen%es for Arit#meti% Reasoning

8/14/2019 GATBDEVE

32/98



Figure 2*2) Test C#ara%teristi% Cur'e 7TCC8 Differen%es for Vo%abular/

8/14/2019 GATBDEVE

33/982:



Figure 2*3) Test C#ara%teristi% Cur'e 7TCC8 Differen%es for T#ree*Dimensional"$a%e

8/14/2019 GATBDEVE

34/98';

The resultin" forms proided less test information than (ie, were not as pre!ise as) the

ori"inal forms and in some re"ions were somewhat less pre!ise than Form A Thus, a se!ond round

of modifi!ations was made to ma8e the forms more pre!ise without sa!rifi!in" fairness This

re>uired addin" a small number of Form A items to the final ersions of Forms E and F ForArithmeti! *easonin", the new Forms E and F !ontain two and one items from an!hor Form A,

respe!tiely For Three?/imensional pa!e, Forms E and F !ontain two items ea!h from Form A

6o Form A items were added to the uitable forms ompared to the "ains ine>uity, howeer, the loss of pre!ision was small, and the new forms are "enerally !omparable to

Form A The de"ree of differen!e between referen!e and fo!al "roups is rou"hly !omparable to that

found by /ras"ow (1:59) for the AT En"lish 0sa"e and +athemati!s 0sa"e tests

Measurement Pro$erties of t#e Final Forms) The measurement properties of ea!hiteration of Forms E and F, as well as the an!hor Form A, were e$amined from seeral

perspe!ties, in!ludin" inspe!tion of three types of "raphs based on parameters estimated in thetotal sample First, !ures displayin" test information at ea!h of 1' theta leels were ealuated

e!ond, the test information !ures of Forms E and F were "raphed alon" with "raphs showin" the

ratio of Form E or F information relatie to Form A information at ea!h of these points 4n addition,

be!ause test information may not be easily interpretable to all readers, a third "raphi! based on aheuristi! pro!edure was used to proide estimates on a reliability s!ale

For the absolute and relatie information "raphs, the information for a hypotheti!al, shorter

Form A was estimated by multiplyin" by the number of items on Forms E and F diided by the

number of items on Form A For e$ample, Form A of the Three?/imensional pa!e test had =;items, whereas Forms E and F had 2; To ma8e the forms !omparable, the test information plotted

for Form A was multiplied by 2;=; J ;7; at ea!h theta point prior to plottin"

The absolute information "raphs for the final forms are proided here as Fi"ures 2?=

throu"h 2?3 The "raphs "enerally show that the new forms are about as informatie as Form A,e$!ept for the

8/14/2019 GATBDEVE

35/98'1

Figure 2*5) Arit#meti% Reasoning Absolute (nformation ra$#

Figure 2*6) Vo%abular/ Absolute (nformation ra$#

Figure 2*9) T#ree*Dimensional "$a%e Absolute (nformation ra$#

4n sum, the information "raphs demonstrate that the new Forms E and F are about asinformatie as Form A For Arithmeti! *easonin", the new forms are not >uite as pre!ise as Form

A, parti!ularly oer the ran"e ; theta 27

8/14/2019 GATBDEVE

36/98

8/14/2019 GATBDEVE

37/98''

number of shaded areas in the stimulus) as spe!ified in Test &pecifications for GATB Forms E and

F (PA*/, 1::7)

4tems were sorted by item dis!rimination alues Be!ause the !orrelation between item

order and point biserial !orrelation did not anish, items were sele!ted only if the ori"inal tar"et of:7 per!ent of the e$aminees rea!hed the item Cen!e, those items at the end of the test boo8lets

often were not !onsidered, een thou"h they demonstrated si@able point biserials The possible

influen!e of !areless responders on the point biserial alues for these items !ould hae !ontributedto these lar"e alues Therefore, the additional sele!tion !riterion was used

C#ara%teristi%s of t#e Pro$osed Ob&e%t Mat%#ing Test) The proposed Forms E andF for the be!t +at!hin" test and their su""ested administration order are "ien in Appendi$ of+ellon et al (1::3) The forms are irtually identi!al in their mean diffi!ulty (5:= and 5:',

respe!tiely) and dis!rimination (299 and 23', respe!tiely) 4n addition, the effe!t si@e statisti!s

indi!ated that the sele!ted items proide little a""re"ate differential impa!t a!ross sub"roups Thesealues were all low and similar a!ross test forms# For malefemale, whiteAfri!an Ameri!an, and

whiteCispani! !omparisons, the alues were ?;;32, ;1;:, ;;;5, for Form E% and ?;;25, ;1;1,

and ;;17 for Form F% respe!tiely

(tem "ele%tion for t#e Name Com$arison Test) 0sin" the item statisti!s as a "uide,:; items were sele!ted for ea!h new form 4tems were !hosen to ma$imi@e dis!riminability andminimi@e differential impa!t on sub"roups Cen!e, items with lar"e point biserial !orrelations and

small effe!t si@es were tar"eted 4n addition to these psy!hometri! !on!erns, the e$perimental items

were e$amined to ensure they represent the arious !ontent !onsiderations (e", male and femalenames, Cispani! names, number of !hara!ters in the stimuli) as spe!ified in Test &pecifications forGATB Forms E and F (PA*/, 1::7)

n!e sele!ted, the items were sorted by 8eyed response and !orre!ted diffi!ulty alues, and

the !ontent spe!ifi!ations of the items were tallied Althou"h the psy!hometri! >ualities of theinitial proposed Forms E and F were satisfa!tory, the !ontent spe!ifi!ation tally indi!ated 2' male

items appeared on Form E, whereas Form F !omprised only 1= Thus, items were shifted between

forms, repla!in" the male items from Form F with indeterminant items from Form E n!e sortedinto the final proposed forms, the items were a"ain sorted for ea!h form by 8eyed response and

!orre!ted diffi!ulty 4tems were sele!ted to obtain =7 LameM items and =7 L/ifferentM items

To determine the order of administration within ea!h proposed form, a table of random

units was used to proide a 8eyed response pattern for ea!h form +erely orderin" items fromeasiest to most diffi!ult was infeasible for the 6ame omparison test, be!ause the ame items were

far easier than the /ifferent items rderin" by diffi!ulty alone would hae resulted in all the ame

items appearin" at the front of the test onse>uently, the se>uen!in" of ame and /ifferent itemswas randomi@ed After "eneratin" the response pattern, the items were ordered so that the easiest

item hain" the re>uired 8eyed response was sele!ted Thus, the item position represents an

orderin" by item diffi!ulty (from easiest to most diffi!ult), !onditional upon the randomly"enerated 8eyed response pattern After e$amination of the test mo!8ups, howeer, it be!ame

apparent that some adustments to item orderin" were re>uired (e", four !onse!utie Form F items

!ontained the abbreiation LoM) Two items on Form E and three items on Form F were

e$!han"ed to eliminate this type of !ontent repetition

*e"ardin" the !ontent !hara!teristi!s of the sele!ted 6ame omparison items, !onsideration

was "ien to the number of !hara!ters in the stimulus name (the name on the left side of the pair)

ran"in" from 3 to 27, the ethni! "roup the name represents (A J Asian, B J Afri!an Ameri!an, C J

8/14/2019 GATBDEVE

38/98'=


Cispani!, J ther), the "ender of the name (+ J male, F J female), whether it was a business

(B) or personal (P) name, and the type of differen!e that o!!urred for dissimilar items (1 J inertedletters, 2 J same sounddifferent meanin", ' J different sound and letter, = J addition or deletion of

a sin"le letter, and 7 J different word forms but similar meanin") +!loy et al (1::=) and +ellon

et al (1::3) in!lude more detailed information on the distribution of items a!ross these !ate"oriesin the final forms

An attempt was made to sele!t items that represent a fair distribution of the releant !ontent!hara!teristi!s a!ross forms Althou"h there is less balan!e for !ertain !hara!teristi!s, "reater

emphasis was pla!ed upon assurin" balan!e of the "enderethni! name representation and theBusinessPersonal !hara!teristi!s, in parti!ular The Type of /ifferen!e !hara!teristi! is less

balan!ed a!ross proposed test forms, but the items fun!tion e>uialently from a psy!hometri!

perspe!tie 6o items with LType of /ifferen!eM e>ual to L7M (ie, different word forms but similarmeanin") were !hosen for the new test forms These items demonstrated lar"e differential impa!t to

Asians pe!ifi!ally, Asians were disproportionately li8ely to miss these items that, althou"h

stru!turally ery different, mean the same thin" For e$ample, item =; from boo8s 9 and 5 has a

whiteAsian effe!t si@e of 1125 The item is

Natuire addin" fie or si$ multi?di"it

numbers The final ersion of both Form E and Form F was to in!lude =; items, 1; of ea!h type ofoperation, and within ea!h type of operation, items should represent the ran"e of !ontent

!hara!teristi!s as spe!ified in Test &pecifications for GATB Forms E and F (PA*/, 1::7)

4tems were !hosen to ma$imi@e dis!riminability and minimi@e differential impa!t on

sub"roups Cen!e, items with lar"e point biserial !orrelations and small effe!t si@es were tar"eted

8/14/2019 GATBDEVE

39/98'7

Anal/sis of Form A An%#or (tems) The data for Form A items (whi!h were an!horitems on all forms) were !ompared with the e$perimental item data to see how !omparable theywere in terms of item diffi!ulty and dis!riminability 4n "eneral, the entire pool of e$perimental

items was sli"htly more diffi!ult than the pool of Form A items +ean diffi!ulties for Form A and

e$perimental items respe!tiely were 5; and 95 for addition, 57 and 9: for subtra!tion, 9: and97 for multipli!ation, and 91 and 33 for diision (+ellon et al, 1::3, Appendi$ -) The

e$perimental items were somewhat less dis!riminatin" than the Form A items +ean !orre!tedpoint biserials for Form A and e$perimental items respe!tiely were# '1 and 2' for addition, '9

and '' for subtra!tion, =2 and '7 for multipli!ation, and =' and '5 for diision

4tems were sele!ted based on their psy!hometri! properties, and ra!e and se$ differen!es

4tems were pooled a!ross forms and or"ani@ed a!!ordin" to the type of arithmeti! operation 4tems

with point biserials one or more standard deiations below the mean point biserial for that type of

operation were identified As mentioned earlier, +antel?Caen@el !hi?s>uare analyses !ondu!ted byC*trate"ies (1::=) were also used to e$amine /4F 4tems that (a) were fla""ed by the /4F

analysis and (b) showed more than half a standard deiation differen!e between sub"roups, were

not sele!ted Ei"hteen e$perimental items were fla""ed by the /4F analysis f these, ei"ht wereretained (si$ on Form E, two on Form F) 4n no instan!e did the sele!ted items possess effe!t si@es

of "reater than one half standard deiation

The pool of remainin" "ood items was sorted a!!ordin" to !ontent !hara!teristi!s, and theitems were or"ani@ed a!ross forms After ma8in" an initial !ut a!ross forms, the mean itemstatisti!s were !omputed for ea!h new form To ma8e the forms as similar as possible (statisti!allyas well as in !ontent), some items were shifted from one form to the other, yieldin" the final set ofproposed items for Forms E and F Table 2?5 summari@es the statisti!s for the sele!ted items ofea!h of the proposed forms

4tems were ordered as follows# (1) an addition item, (2) a subtra!tion item, (') amultipli!ation item, and (=) a diision item This se>uen!e of item types was repeated throu"hout

the test 4tems were ordered a!!ordin" to diffi!ulty (within type of operation) For e$ample# 4tem U1

!ontained the easiest addition item, 4tem U7 !ontained the ne$t easiest addition item, , and 4temU'9 !ontained the most diffi!ult addition item This pro!ess was repeated for ea!h type of item until

all items were sele!ted

Table 2*=Com$utation Test (tem "ele%tion> "ummar/ "tatisti%s for "ele%ted Form E and Form F (tems

(tem Corre%ted Diffi%ult/ Point !iserials Effe%t "i4eForm T/$e Mean "D All (tems ?O (tem M*F *! *H

E Addition ;9: ;'5 '1 2: ?;;5 ;21 ;;2

ubtra!tion ;9: ;': '7 '' ?;1= ;'; ;12

+ultipli!ation ;93 ;=; =; '9 ?;;: ;2= ;17

/iision ;3: ;== =1 '5 ?;;9 ;25 ;2;

Grand +ean ;93 ;=1 '9 '= ?;;: ;23 ;12

F Addition ;5; ;'5 '2 2: ?;;9 ;2' ;;:

ubtra!tion ;99 ;': '5 '3 ?;11 ;29 ;12

+ultipli!ation ;97 ;=1 =; '9 ?;;: ;2= ;17

/iision ;3: ;=' =3 =' ?;;: ;'7 ;1:

Grand +ean ;97 ;=; ': '3 ?;;: ;29 ;1=

8/14/2019 GATBDEVE

40/98

8/14/2019 GATBDEVE

41/98'9

Table 2*@(nde$endent*rou$s "am$le "i4es

ARDC Form A Form E Form F Total

EA*/ ==9 '9; '5: 1,2;36A*/ ='3 '9; =;1 1,2;9

A*/ ';1 ''; ''= :37

PA*/ =;2 '92 ':2 1,133.A*/ =77 =73 ='9 1,'=5

Total 2,;=1 1,5:5 1,:7' 7,5:2

Re$eated*Measures "am$le) E$aminees in the repeated?measures sample wereadministered two forms of the GATB These data were used primarily for e$aminin" the reliability

and !onstru!t alidity of the GATB Coweer, a portion of these data were also used to supplementthe e>uatin" data These data were used to perform a detailed !omparison of measurement

properties between the old and new forms

Ea!h e$aminee parti!ipatin" in the repeated?measures portion of the study was randomly

assi"ned to one of ei"ht !onditions These !onditions and the numbers of e$aminees in ea!h!ondition are presented in Table 2?1; 6ote that !onditions 1, 2, 9, and 5 !onsist of samples of

appro$imately e>ual si@e (6 V =';) The remainin" !onditions listed in Table 2?1; also !onsist of

appro$imately e>ual sample si@es (6 V 215) The rationale for the sample si@e re>uirement isoutlined in the data analysis se!tion The numbers of e$aminees tested in ea!h !ondition at ea!h site

are proided in Table 2?11

Table 2*+Re$eated Measures Design and "am$le "i4es

"e%ond Test

First Test A ! E F

ABEF

2 (='2)

=(213)

1 (=11)

3 (217)

7 (2;:)

5 (==3)

' (2'3)

9 (==3)

Table 2*++Re$eated*Measures "am$le "i4es b/ Test "ite

Condition

ARDC + 2 3 5 6 9 : = Total

EA*/ =' =3 29 22 2= 27 7' =5 2556A*/ 51 52 =1 =7 =; =; 55 :' 71;

A*/ := 1;; 72 71 =9 =7 93 99 7=2PA*/ 1;9 :5 3' =: 7' =: 1;' 1;5 3';

.A*/ 53 1;3 7' =: =7 73 123 12; 3=1

Total =11 ='2 2'3 213 2;: 217 ==3 ==3 2,311

At two?room sites, e$aminees were randomly assi"ned to the ei"ht !onditions upon arrialAt one?room sites, all e$aminees were s!heduled for testin" prior to arrial This pro!edure ensuredrandom assi"nment of e$aminees to !ondition

8/14/2019 GATBDEVE

42/98'5


Ps/%#omotor "am$le) This sample of 7'5 e$aminees re!eied the fie psy!homotortests alon" with the non?psy!homotor portions of Forms A and F The desi"n is presented in Table2?12 E$aminees were randomly assi"ned to one of two "roups Ea!h "roup re!eied three se!tions#

(1) Form A (non?psy!homotor), (2) Form A (psy!homotor), and (') Form F (non?psy!homotor),

with the order of presentation !ounterbalan!ed a!ross the two "roups As indi!ated in Table 2?12,Group 1 re!eied Form A (non?psy!homotor) and Form A (psy!homotor) portions in the mornin"

session, and Form F (non?psy!homotor) in the afternoon Group 2 re!eied the same battery of testswith the order of the non?psy!homotor se!tions of Forms A and F reersed

At two?room sites, e$aminees were randomly assi"ned to the two !onditions upon arrialAt one?room sites, all e$aminees were s!heduled for testin" prior to arrial This pro!edure ensured

random assi"nment of e$aminee to !ondition

"am$le C#ara%teristi%s

This se!tion proides an ealuation of the demo"raphi! !hara!teristi!s of ea!h of the three

samples in!luded in this study# (1) the independent?"roups (4G) sample, (2) the repeated?measures

(*+) sample, and (') the psy!homotor (P+) sample An ealuation of the random e>uialen!e ofsele!ted "roups within ea!h of the three samples is also proided, be!ause the random e>uialen!e

of these "roups is a 8ey assumption made in the e>uatin", reliability, and alidity analyses

Table 2*+2Ps/%#omotor Data Colle%tion Design

rou$ + 7N B 2968 rou$ 2 7N B 2:38

+ornin"

Afternoon

1 Form A (non?pmotor) 1 Form F (non?pmotor)

2 Form A (pmotor) 2 Form A (pmotor)

' Form F (non?pmotor) ' Form A (non?pmotor)

This se!tion also summari@es the data editin" pro!edure used to remoe unmotiatede$aminees and other hi"hly influential !ases from ea!h of the three samples The numbers of !ases

remoed from ea!h sample are reported

Demogra$#i%s and rou$ E;ui'alen%e) For ea!h of the three primary samples,statisti!al tests of the differen!es amon" randomly e>uialent "roups were !ondu!ted by "ender,

ra!e, a"e, and edu!ation 6on?si"nifi!ant results are !onsistent with the e$pe!tation based on

random assi"nment of e$aminees to !ondition and support the assumption of e>uialent "roupsmade in the e>uatin", reliability, and alidity analyses All si"nifi!an!e tests of differen!es a!ross

"roups by form (for the 4G sample) and by !ondition (*+ and P+ samples) yielded non?si"nifi!ant

results with W J ;7 (!f e"all & +on@on, 1::7, for detailed results)

The results of these analyses indi!ate dierse samples with respe!t to "ender, ra!e, a"e, andedu!ation Furthermore, the si"nifi!an!e tests performed on the three samples (independent?"roups,

repeated?measures, and psy!homotor) proide reassuran!e that the assi"nment pro!edures wor8ed

as intended, produ!in" "roups that are randomly e>uialent with respe!t to demo"raphi!!hara!teristi!s Althou"h the e>uialen!e of the "roups on !o"nitie and psy!homotor abilities

!annot be tested with e$istin" data, the results based on the demo"raphi! ariables proide

additional !onfiden!e in this assumption, sin!e in some instan!es demo"raphi! and!o"nitiepsy!homotor ariables tend to be !orrelated

Outlier Anal/sis) Prior to data analysis, a small number of !ases with unli8ely s!oreswere deleted from the database ases for deletion were identified usin" a pro!edure su""ested by

8/14/2019 GATBDEVE

43/98':

Cotellin" (1:'1), whi!h identifies !ases that are unli8ely "ien that the obserations are sampled

from multiariate ellipti!al?shaped distribution

eparate outlier analyses were performed for the three samples (independent?"roups,

repeated?measures, and psy!homotor) Furthermore, separate analyses were performed for ea!h"roup, within ea!h sample 6ote that it is possible for two types of patterns to be fla""ed and

deleted by the !hosen pro!edure ne type o!!urs when an e$aminee re!eies e$treme s!ores on

many tests (e", all low s!ores) Another type of unli8ely pattern o!!urs when an e$aminee s!oreshi"h on one test and low on a se!ond that is hi"hly !orrelated with the first (ie, alternate forms of

the same test) A small number of !ases with @ero number?ri"ht s!ores on one or more tests were

also deleted, be!ause /- poli!y di!tates that s!ores should not be proided to su!h e$aminees

(nde$endent*rou$s "am$le) Table 2?1' proides the editin" results for theindependent?"roups sample Cere, sele!ted !ases from the *+ sample were !ombined with the 4G

sample pe!ifi!ally, data from the first test administered (Forms A, E, or F) of onditions 1, ', =,3, 9, and 5 were !ombined with the 4G data to in!rease the sample si@es for the e>uatin" study /ata

editin" was performed in ea!h of the three 4G*+ "roups# LA,M LE,M and LFM A few !ases were

rem

GATBDEVE

Documents

Transcript of GATBDEVE