The cloze test as a measure ofachievement in educational measurement

Item Type text; Thesis-Reproduction (electronic)

Authors McLean, David Richard, 1944-

Publisher The University of Arizona.

Rights Copyright © is held by the author. Digital access to this materialis made possible by the University Libraries, University of Arizona.Further transmission, reproduction or presentation (such aspublic display or performance) of protected items is prohibitedexcept with permission of the author.

Download date 24/06/2018 06:44:14

Link to Item http://hdl.handle.net/10150/554589

THE CLOZE TEST AS A MEASURE OF ACHIEVEMENT

IN EDUCATIONAL MEASUREMENT

by

David Richard M cLean

A T h es is Submitted to the Facu lty of the

DEPARTMENT OF EDUCATIONAL PSYCHOLOGY

In Partia l Fulfillm ent of the Requirements For the D egree of

MASTER OF ARTS

In the G raduate C ollege

THE UNIVERSITY OF ARIZONA

1 9 7 3

STATEMENT BY AUTHOR

This th e s is has been subm itted in pa r t ia l fu lf illm ent of re ­quirem ents for an advanced degree at The U n ivers ity of Arizona and is d ep o s ited in the U n iv ers i ty Library to be made av a ila b le to borrowers under ru les of the Library.

Brief quo ta tions from this th e s e s are a llow able without sp e c ia l p e rm iss io n , provided th a t accu ra te acknow ledgm ent of source is m a d e „ Requests for pe rm iss io n for ex tended quo ta tion from or reproduction of th is m anuscrip t in whole or in part may be granted by the head of the major departm ent or the D ean of the G raduate C o llege when in his ju d g ­ment the proposed use of the m ateria l is in the in te re s ts of s ch o la r ­sh ip . In a ll o ther i n s t a n c e s , how ever, perm ission m ust be obta ined from the au thor.

SIGNED:

APPROVAL BY THESIS DIRECTOR

-This th e s is has been approved on the d a te shown below:

kflA/iety/., O fM A ll i n •/DARRELL L. SABERS 7 D ate

A sso c ia te P ro fesso r of Psychology

ACKNOWLEDGMENTS

The author w ish es to acknow ledge full coo p era tio n and

a s s i s ta n c e of D r. D arre ll S a b e rs , who not only gave much of his t im e,

but a lso he lped in the developm ent of the d es ig n and a n a ly s is of the

d a ta . Also g ra tefu lly acknow ledged are D r. W ilbur S. Ames, who gave

many helpfu l su g g e s t io n s in the d e s ig n and scoring of the c loze te s t

and D r. W illiam L. Logan, who a s s i s t e d in the f ina l w riting of th is

thesis:..

TABLE OF CONTENTS

Page

LIST OF TABLES v

ABSTRACT ....................... v i

I . INTRODUCTION ................... 1

D efin ition and Previous U se of C loze . . . . . . . . . . . . 1Problem . . . . . . . . . . . . . . . . . . . . „ . . . . . . . 3

II . RELATED LITERATURE . . . . . . . . . . . . . . . . . . . . . . 5

III . METHOD . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Type of C loze T est U sed 9S e lec tio n of the C loze P a s sa g e s . . . . . . . . . . . . . . . 10D ele tio n Procedure and Scoring . . . . . . . . . . . . . . . . 11

IV. RESULTS ............................ 13

V. DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Validity . . . . . . . . . . . . . . . . . . . . . . . . . . i, . . 16R eliab ility . . . . . . . . . . . . . . . . . . . . . . . . . . 18S ugges tions for Further Study . . . . . . . . . . . . . . . . 20

VI. CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . 22

APPENDIX A: CLOZE TEST . . . . . . . . . . . . . . . . . . . . 23

REFERENCES CITED . . . . . . . . . . . . . . . . . . . . . . . 30

iv

LIST OF TABLES

Table • Page

1. C orre la tions Among All T e s t s — KR-20 R e liab i l i t ie s onD iag o n a l, and Number of S ub jec ts Below the D iagonal . „ 14

2. C orre la tions Among All T ests C orrected for A ttenuation . . 14

3 „ C orre la tions Among All S u b tes ts of the C loze--K R -20R eliab il i t ie s on D iag o n a l, Number of Subjects = 5 1 . „ 19

v

ABSTRACT

In the p a s t the c loze t e s t has been used only to m easure

read ing com prehension and re a d a b i l i ty . This study in v e s t ig a te d the

c loze as a m easure of c o u rse -o r ie n te d ach iev em en t. The su b je c ts for

th is s tudy were from two. ed u ca t io n a l m easurem ent c l a s s e s and the c loze

p a s s a g e s were s e le c te d at random from a tex t s im ilar to th a t being u sed

by the c l a s s e s . S ign ifican t po s it iv e co rre la tio n s were ob ta ined be tw een

in s truc to r-m ade ach ievem en t t e s t s and the c loze t e s t . The Kuder-

R ichardson re l ia b il i ty of the c loze t e s t was com parable to those

ob ta ined with in s tru c to r-m ad e t e s t s .

v i

CHAPTER I

INTRODUCTION ■

D efin ition and Previous Use of C loze

The word c lo ze comes from the G e s ta l t law of c lo su re w hich

s ta te s . th a t c lo se d a reas more read ily form u n i t s . H ill (197.1, p . 100)

notes tha t n o n g es ta lt p sy ch o lo g is ts u se the term '“c lo su re " to refer to

the " se n s e of com pletion" ob ta ined when a ta sk is f in is h e d .

W ilso n L. Taylor was among the f irs t to in v e s t ig a te the c loze

p rocedure . His d e f in it io n of c loze (Taylor, 1953, p . 416) has been

accep ted by most re se a rc h e rs working w ith th is p rocedure: C lgze is

"a method of in te rcep tin g a m essag e from a 't ra n sm it te r ' (writer or

s p e a k e r ) , m utila ting i ts language pa tte rn s by d e le t in g p a r t s , and so

adm in istering it to ' r e c e iv e r s ' (readers and l is ten e rs ) th a t the ir attem pts

to make the pa tte rns whole again p o te n tia l ly y ie ld a c o n s id e rab le number

of c lo ze u n i t s . " In .i ts most common form, a c loze p a s s a g e c o n s is ts of a

s e t of paragraphs w ith every fifth word d e le te d . The s u b je c t ’s ta sk is to

f il l in the e x a c t d e le ted words by u se of con tex t c u e s .

R esearch on the c loze techn ique has been prim arily concerned

with the m easurem ent of com prehension . R esearchers u s in g the c loze in

the fie ld of reading have d esc rib ed com prehension as th a t w hich is

1

m easured by s tan d a rd ized reading t e s t s . S ta ted le s s o p e ra t io n a l ly ,

Bormuth (1968, p . 50) say s : "C om prehension ab il i ty is thought to be a

se t of g en e ra l ized k n o w le d g e -a c q u is i t io n sk i l l s w hich perm it people to

acquire and exh ib it information ga ined as a co n seq u en ce of reading

prin ted la n g u a g e ." Because a com prehension and ach ievem en t t e s t may

con ta in the same i te m s , the most important d is t in c t io n is score in te rp re ­

ta t io n . The score on an ach ievem ent t e s t is in te rp re ted as the amount of

knowledge a p a r t ic u la r s tuden t has been able to acquire from reading a

p a s s a g e . A com prehension t e s t sco re re p re se n ts the ab i l i ty of a s tu d en t

to acquire knowledge from w ritten language of the type rep resen ted by

tha t p a s s a g e .

Another way of looking at the re la t io n sh ip of the c loze t e s t to

com prehension and ach ievem ent is to v iew te s ts as forming a con tinuum .

A n as tas i (1972, p . 392) makes the following s ta tem en t concern ing

sp e c if ic i ty of t e s t s : "Existing te s t s of developed a b i l i t ie s may be o rdered

along a continuum in terms of s p e c if ic i ty of e x p e r ien t ia l background they

p r e s u p p o s e . " The following l i s t , ordered from s p e c if ic to genera l

ach iev em en t, exem plif ies th is continuum:

1. C o u rse -o r ie n te d achievem ent t e s t s

2. B road ly-orien ted achievem ent t e s t s

3 . V erbal-type in te l l ig en ce and ap titude te s t s

4 . N on-language and performance t e s t s

5 . "C u ltu re -fa ir" te s t s

D is tin g u ish in g be tw een ap titude and ach ievem ent t e s t s ,

A nas tas i s ta te s th a t ach ievem ent t e s t s m easure e ffec ts of learn ing under

more con tro lled and known conditions th an do ap titude t e s t s . Since th is

is s im ila r to Bormuth's d is t in c t io n b e tw een com prehension and a c h ie v e ­

m ent, i t is ev iden t tha t com prehension and aptitude are h ighly re la te d .

Therefore , if com prehension is roughly equ iv a len t to v e rb a l- ty p e

in te l l ig en ce t e s t s , then the c loze t e s t , as trad it io n a lly u s e d , would fa l l

a t roughly the same p lace on A n a s ta s i 's continuum .

Problem

Taylor (1957, p . 25), after in v es t ig a t in g the u se of a c loze t e s t

as a m easure of com prehension , concluded : “Although c lo ze and com pre­

h en s io n t e s t s were gen era lly s im ilar in the k inds of r e su l ts they y ie ld e d ,

the two kinds of t e s t s were very d ifferen t in the c o s t , e ffo r t , and time

required for c o n s tru c t io n . The ad v an tag es seem to lie w ith c loze pro­

cedure in g enera l . . . " Because the method of developm ent of an

achievem ent c loze t e s t is roughly the sam e as th a t for a com prehension

c loze t e s t , the ad v an tag es of the former should be the sam e in terms of

c o s t , effort and time required for co n s tru c t io n .

Although the c loze techn ique has been used to m easure com­

p reh en sio n , it has not been used to m easure c o u rse -o r ie n te d a c h ie v e ­

m ent. Therefore ev id en ce for the v a l id i ty of the c loze as a m easure of

c o u rse -o r ie n te d ach ievem en t is n e e d e d . If the c loze t e s t is indeed a

4valid m easure of c o u rse -o r ie n te d a c h ie v e m e n t , then the fo llow ing

h y p o th ese s should be true:

1. The co rre la tio n s betw een in s tru c to r-m ad e ach ievem ent

t e s t s and the c loze t e s t are s ig n if ican tly g rea te r than zero .

2. The in te rc o rre la tio n s among the c loze and in s tru c to r-m ad e

t e s t s are com parab le .

3. The Kuder-R ichardson re l ia b i l i ty of the c loze t e s t is

s ig n if ic a n t ly g rea te r than ze ro .

4 . The K uder-R ichardson re l ia b il i ty of the c lo ze t e s t is

com parable to those ob ta ined from in s tru c to r-m ad e a c h ie v e ­

ment t e s t s .

... C orre la tions correc ted for a t ten u a tio n are often preferred for

comparing v a l id i t ie s of t e s t s (N unnally , 1967, p . 204). Therefore , w ith

the information ob ta ined from the in te rco rre la tio n and r e l ia b i l i ty a n a ly s is ,

the co rrec tion for a t ten u a tio n will be ap p l ie d . Assuming th a t the r e l ia b i l -

i t ie s ob ta ined are good e s t im a te s of the true r e l ia b i l i t i e s , the co rrec tion

for a t ten u a tio n is an e s tim a te of the co rre la t io n s th a t would be ob ta ined

if the t e s t s were perfec tly re l ia b le .

CHAPTER II

RELATED LITERATURE

Although T ay lo r 's (1956, 1957) experim ent m ost c lo se ly

approxim ates the u se of a c loze t e s t to m easure ach iev em en t, none of

the s tu d ie s thus far have u sed the c loze to d irec tly m easure c o u rse -

o rien ted ach iev em en t.

C hall (1948) has dem onstra ted th a t reading com prehension can

be in fluenced more by s p e c if ic knowledge (achievem ent) than by read ing

ab il i ty in g enera l (com prehension). After adm in iste ring a com prehension

t e s t in a s p e c if ic s u b je c t a re a , she ranked the s tu d e n ts accord ing to

sco re s on a sp e c if ic knowledge t e s t and accord ing to s c o re s on the

Stanford Reading T e s t . Grouping the upper and low er 27% in e a c h c a s e ,

she found the m eans on the com prehension t e s t d iffered s ig n if ican tly

be tw een the groups in the sp e c if ic knowledge ranking but did not d iffer

in the Stanford Reading T e s t rank ing .

In a study corre la ting c loze w ith m u lt ip le -c h o ic e com prehehsion

sco res before and af te r a w eek-long s tudy period , Taylor (1956, 1957)

found co rre la tio n s of .70 for the p re te s t and .80 for the p o s t - t e s t . The

t e s t - r e t e s t r e l ia b i l i ty of th a t c loze t e s t w a s ... 88. That s tu d y , more than

any o th e r , invo lves the c loze as a m easure of ach ievem ent b e c a u se the

s u b je c ts were a llow ed to s tudy the p a s s a g e s during a o n e -w e e k in te rv a l

and only a sam ple of th e se p a s s a g e s w as u sed for the c lo ze t e s t .

Bormuth (1962, 1963) co n s tru c ted m u lt ip le -ch o ice item s from

p a s s a g e s us ing com prehension sk i l l s including: v o c a b u la ry , f a c t ,

s e q u e n c e , c a u s a l r e la t io n sh ip , main id e a , in fe ren ces from f a c t s , and

a u th o r 's pu rp o se . The re su l ts of th a t s tudy y ie lded Cloze vs m u ltip le -

cho ice t e s t co rre la tio n s ranging from .73 to .8 3 . Bormuth a lso noted

th a t the c lo ze t e s t s were highly re l ia b le as the r e l ia b i l i t ie s ranged from

.84 to .8 8 .

U sing the same com prehension s k i l l s as Bormuth (1962, 1963)

in the m u lt ip le -c h o ic e i te m s , Rankin and C ulhane (1969) found co rre la ­

tions ranging from .54 to .7 1 .

Repeating h is s tudy with the same com prehension s k i l l s , Bormuth

(1969) ran a fac to r a n a ly s is on the com prehension s k i l l s . The re su l ts

y ie ld ed only one fac to r from the various com prehension s k i l l s . This

would lead ev id en ce to the notion th a t the type of . q u e s tio n included in

the m u lt ip le -ch o ice t e s t has l i t t le e f fec t upon the com prehension s c o re .

Therefore it is co n ce iv ab le tha t the type of ques tio n s a sk e d on an 1

ach ievem ent t e s t would s im ilar ly have l i t t le e ffec t on the ach ievem ent

s c o re . . -

The c loze t e s t , th e n , has been e s ta b l is h e d as an e ffec tive too l

for the m easurem ent of reading com prehension . Since C h a ll has shown

th a t ach ievem ent is co rre la ted w ith co n ven tiona l com prehension t e s t s and

7

o thers have shown th a t conven tional com prehension t e s t s are co rre la ted

w ith c lo ze t e s t s c o r e s , what rem ains is to dem onstra te the d irec t

re la t io n sh ip be tw een ach ievem ent and c lo ze t e s t s c o re s . E s tab lish in g

th is d ire c t r e la t io n sh ip is the su b je c t of the p resen t in v e s t ig a t io n .

CHAPTER III

METHOD

The su b je c ts for th is experim ent were ob ta ined from an

ed u ca tio n a l m easurem ent c la s s taught at The U n ivers ity of Arizona.

These include two c l a s s e s , spring 197.3 (N = 37) and f ir s t summer,

s e s s io n 1973 (N - 3 0 ) , The seq u en ce of t e s t s w as: (1) midterm, (2)

p rac tice w ith a c loze t e s t , (3) c loze t e s t , (4) exam over te x t used in

c la s s (book exam ), and (5) f ina l exam . All t e s t s were adm in iste red w ith in

a th re e -w e e k in te rv a l .

Because the s tu d en ts were only required to take the book and

fina l exam , not a ll the su b je c ts took a ll of the o ther t e s t s . Of the 67

s tu d en ts who took the book and fina l exam , 51 took the c loze t e s t and

60 took the midterm exam .

The midterm and fina l exams e a c h c o n s is te d of a 75-item

m u lt ip le -ch o ice t e s t , and the book exam c o n s is te d o f a 60-item m u lt ip le -

cho ice t e s t , a l l developed by the in s tru c to r . The bulk of the items in

th e se exams were in fe ren tia l and problem so lv ing w ith the remaining being

l i t e r a l . The book exam tended to have more l i te ra l item s than e ither of

the o ther ex am s , i ts co n ten ts inc luded m ateria l covered in the tex t u sed

8

in c l a s s . The midterm and fina l exams were p a ra l le l forms concerning

m ateria l inc luded during the lec tu re as w ell as in the t e x t .

Type of C loze T est U sed

The d e le t io n schem e ch o sen for th is study is the same as tha t

u sed by the m ajority of re se a rc h e rs u s ing the c loze p ro c e d u re .

M acG in itie (1961) has show n tha t d e le t io n schem es of every 6th, 12th,

or 14th word are eq ua lly d iff icu lt to re s to re in a c loze p a s sa g e w hereas

an every 3rd word d e le t io n schem e is more d if f ic u l t . This is important,

b e c a u se more d iff icu lt p a s s a g e s re su l t in t e s t anx ie ty , w hich according

to Brown (1970, p . 59), y ie ld s u n re liab le r e s u l t s . Therefore a good

compromise be tw een d iff icu lty due to high frequency of d e le t io n on one

hand and the advantage of high frequency of da ta on the o ther hand is

found when the ty p ica l d e le t io n schem e of every 5th word is u se d .

The time a llo ted for the c loze t e s t was equa l to th a t a l lo tted for

o ther ex am s. This a l lo t ted time was the lim iting fac to r in determ ining

the c loze t e s t len g th . The experim enter found tha t five p a s s a g e s e ach

lim ited to 20 to 35 items (deletions) can be com pleted w ith in the hour

a l lo t te d .

A random s e le c t io n of m ateria l for c loze used was chosen over

in ten t io n a l s e le c t io n s in ce in ten tiona l s e le c t io n , involv ing scann ing

and ev a lu a tin g large numbers of p o s s ib le p a s s a g e s , is very su b jec tiv e

and time consum ing .

10

The c lo ze t e s t used in th is study was c o n s tru c ted from

p a s s a g e s s e le c te d at random from a tex t s im ilar to th a t be ing used by

the s tu d e n t s . The re a so n for th is was to avoid s p e c if ic r e c a l l on the

part of the s u b j e c t s . The book u sed by the s tu d en ts w as P sycho log ica l

Testing by Anne A n as tas i (1972) „ The book u sed for c lo ze s e le c t io n s

was P rinc ip les of Educationa l and P sycho log ica l T es ting by Frederic G .

Brown (1970) „

S e lec t io n of the C loze P a ssa g e s

Based upon the knowledge of the con ten t of the in s tru c to r-m ad e

t e s t s , the following con ten t a reas were sam pled for c lo ze t e s t d ev e lo p ­

ment:

1. Score in te rp re ta t io n

2. V alidity

3. R eliab ility

4. Item a n a ly s is

5. T ests in genera l (including s p e c if ic t e s t s such as IQ,

ach ievem en t, p e rso n a l i ty , e t c . ) - '

S e lec tio n of th e se co n ten t a reas was accom plished by the following

ru les :

1. C hap te rs d ea lin g with con ten t a reas were noted from

tab le of c o n te n t s .

2. These chap te rs were then opened a t random.

11

3. The f irs t com plete paragraph(s) on the right hand page was

s e le c te d which does (is) not:

A. have le s s than 100 or more than 175 words be tw een

f irs t and l a s t s e n te n ce (If there was more than one

paragraph needed to meet th is c rite rion then the para ­

graphs must be in s u c c e s s i o n . ) ,

B. refer to a previous s e c t io n in the te x t ,

C . refer to a figure or p l a t e ,

D . part of an enum eration o r . l i s t in g .

4. If any of the above s i tu a t io n s a p p l ie d , the paragraph

im m ediately af te r th is cho ice w as exam ined . If there was

s t i l l a p rob lem , the paragraph preceding th is cho ice was

exam ined , and a con tinua l s e a rc h was made in th is a l te rn a t­

ing manner un ti l a paragraph(s) was found th a t met the

above c r i te r ia . >

D ele tion Procedure and Scoring

Beginning with the second se n te n c e in each p a s s a g e , every

fifth word w as d e le te d un til the l a s t s e n te n c e was re a c h e d . The f irs t

and l a s t s e n te n c e s were kept in tac t and the word s e le c te d for f irs t

d e le t io n was determ ined random ly. Follow ing a method s e t by Taylor

(1957, p . 21), counted words inc luded " . . . any group of language

symbols ( le t te r s , f ig u re s , and re la ted punctuation) s e p a ra te d from

12

neighboring groups by empty s p a c e s , . . Punctuation su ch as

h y p h en s , apos trophes and commas in numbers were a lso su b je c t to

d e le t io n . The d e le ted words were rep laced by underlined b lanks of

equal length which were never s p l i t from the end of one line to the

beginning of ano ther . The t e s t w as th en mimeographed and c o lla ted in

the following order of co n ten t ca te g o r ie s : score in te rp re ta t io n , v a l id i ty ,

t e s t s in g e n e ra l , re l ia b i l i ty , and item a n a ly s is .

In scoring the c loze t e s t , only the e x ac t word w as scored

c o r re c t . Thus if the su b je c t responded " te s t" in s te a d of " te s t s " his

response w as marked wrong. This e lim ina ted the e f fec t of su b jec tiv e

sco r in g .

CHAPTER IV

RESULTS

The re su l ts of the in te rco rre la tio n and K uder-R ichardson

re l ia b il i ty a n a ly se s appear in Table 1. The co rre la tions be tw een the

c loze t e s t and book, midterm and fina l exams were .5 0 , .52 and .6 0 ,

r e sp e c t iv e ly and each is s ig n if ican tly g rea te r than z e ro . The Kuder-

R ichardson r e l ia b i l i t ie s ob ta ined range from .75 to .86 and e a c h is

s ig n if ican tly g rea te r than ze ro .

D ependent F isher Z te s t s showed th a t the m agnitudes of the

in te rco rre la tio n s involving the c loze t e s t are a ll c o m p arab le . T ests of

s ig n if ican ce be tw een re l ia b il i ty co e ff ic ien ts showed th a t e a c h Kuder-

Richardson re l ia b il i ty ob ta ined is com parable to th a t ob ta ined for the

c loze ex cep t in the c a se of the book exam (Feldt, 1969). ̂ The

re l ia b il i ty of the c loze t e s t is s ig n if ican tly (P<(.05) g rea te r than that of

the book exam .

Table 2 show s the in te rco rre la tio n s ob ta ined when the co rrec tion

for a t ten u a tio n is a p p l ie d .. These v a lu e s of r be tw een the c loze and o ther

t e s t s range from .62 to .7 0 . The fac t th a t th e se v a lu e s are not c lo se r

•'•In the p re se n t c a s e , with hypotheses, = p ^ , the s ta t i s t i c reduced to: W = (1 - ^ / ( l - r^ ) , w hich is d is tr ib u ted approxim ately as F w ith (n2 - 1) and (n^ - 1) degrees of freedom (n = number of s u b je c t s ) .

13

Table 1. C orre la tions Among All T e s t s — KR-20 R e liab il i t ie s on D iagona l, and Number of Subjects Below the D iagonal

C loze Book Midterm Final

C loze (.86*) .50* .52* .60*

Book 51 (.75*) .73* .68*

M idterm 49 60 (.83*) .63*

Final 51 67 60 , (.84*)

*P< .01

Table 2. C orre la tions Among All T es ts C orrected for A ttenuation

C loze Book Midterm Final

C loze .62 .62 .70

Book .92 .85

Midterm .76

15

to o n e , in the c a s e s involving the c lo ze t e s t , is an in d ica tio n tha t the

c loze t e s t is m easuring som ething d iffe ren t than e a c h of the o ther t e s t s .

CHAPTER V

DISCUSSION

Validity

The co rre la tio n s be tw een the c loze and o ther t e s t s are som e­

what low er in th is s tudy than those reported by Taylor (1956, 1957),

Bormuth (1962, 1963) and Rankin and Culhane (1969) for com prehension

vs c lo se s c o r e s «, This is unders tan d ab le in v iew of the fac t th a t the

s u b je c ts in th is s tudy never read the in tac t c loze p a s s a g e s . However,

the lower co rre la tio n s found in th is s tudy may mean tha t the c loze t e s t

m easu res com prehension but not c o u rse -o r ie n te d ach iev em en t.

An im portant part of e s ta b l is h in g the v a l id i ty of a t e s t is in

noting the v a l id i ty of the c r ite rion m e a s u re s . The in s truc to r-m ade

ach ievem ent t e s t s were co n s tru c ted from a tab le of s p e c if ic a t io n s

sam pling course m ateria l p re sen ted in c la s s and the tex t u sed by the

c l a s s . In th is way the con ten t v a l id i ty of the c r ite rion m easu res has

been e s ta b l i s h e d .

The following is a l i s t of problem areas w hich may have con­

tribu ted to the lower co rre la tio n s found in th is s tudy:

1. The tex t u sed to s e le c t the c lo ze p a s sa g e s may have em pha­

s iz e d d ifferen t e lem ents of the content a reas s e le c te d than the

tex t u sed in c l a s s .

17

2. The tex t used to s e le c t the c loze p a s sa g e s may have varied

' in s ty le , vocabu la ry and g en era l d iff icu lty le v e l as com­

pared to the te x t used in c l a s s .

3. The re la t iv e ly sm all sam ple of c loze p a s s a g e s as compared

with the en tire book lowers the con ten t v a l id i ty of the c loze

t e s t .

4 . The leng th of time betw een the c lo z e , b o o k , midterm , and

f ina l exam s w as su ff ic ien t for some learn ing to take p la c e .

This is p a r t ic u la r ly a problem when s tu d en ts cram for ex am s .

5 . The inheren t assum ption w as made tha t the s tu d e n ts were

equa lly m otivated to perform on each exam .

In re fe rence to point number tw o, it would be in te re s t in g to

know the re la t io n sh ip be tw een d iff icu lty lev e l and in te rco rre la t io n s

betw een c loze and conven tiona l ach ievem ent t e s t s . If the c loze t e s t is

too e a s y or too hard th is would r e su l t in the lowering of the to ta l v a r i­

ance w ith a correspond ing lowering of the c o r re la t io n s .

For the pu rposes of future com parison , the m ean , s tandard

d e v ia t io n , and d iff icu lty lev e l of th is c loze t e s t will be no ted h ere . The

mean and s tandard dev ia tio n of the co rrec t re sp o n se s o b ta ined for the

p resen t c lo ze t e s t are 63 .4 (47.3%) and 11 .3 (8.4% ), r e sp e c t iv e ly .

Rankin and C ulhane (1969) have rep ea ted Bormuth's experim ents

e s ta b l is h in g the read ab il i ty le v e ls of c loze p a s s a g e s . Their re su l ts a re ;

Percent C loze PercentItems C orrec t C om prehension

41 75

61 90

The 75% and 90% com prehension sco re s correspond to the in s tru c t io n a l

and independen t reading le v e ls u se d in read ing te s t s . Of the to ta l of 51

s tu d en ts who took the c lo ze t e s t in th is s tu d y , s ix rece iv ed to ta l c lo ze

sco re s l e s s than 41% and only one rece iv ed a score over 61%. There­

fore , accord ing to the Rankin and C ulhane c r i te r ia r the d iff icu lty le v e l o f

the c loze p a s s a g e s could be c a l le d in s tru c t io n a l for the m ajority of the

su b je c ts in th is s tu d y . -

R eliab ility

The re l ia b i l i ty of the c loze te s t in th is s tudy is com parable to

those ob ta ined by Taylor (1957) and Bormuth (1962, 1963). Table 3 show s

in te rco rre la tio n s among the s u b te s ts of the c loze and the Kuder-

R ichardson re l ia b il i ty of e a c h . Although the number of items per s u b te s t

is sm all (20-35), a l l ex cep t the con ten t a rea of r e l ia b i l i ty have Kuder-

R ichardson r e l ia b i l i t ie s s ig n if ican tly g rea te r than ze ro . The s iz e of the

in te rco rre la tio n s are roughly tha t ex p ec ted in v iew of the re l ia b i l i t ie s

ob ta ined ex cep t in the c a s e of the re l ia b i l i ty s u b te s t . Although the

re l ia b i l i ty of th is s u b te s t is low , it was included in, the a n a ly s is to

in c re a se the con ten t v a l id i ty of the en tire c loze t e s t .

19

Table 3 . C orre la tions Among All S u b te s ts of the C loze--K R -20 R e liab il i t ie s on D iagona l, Number of S ub jec ts = 5 1

Number of Item s: 35

ScoreIn te rp re ta tio n

25V alidity

28T ests in G eneral

26

R eliab ili ty

20Item

A nalysis

ScoreIn te rp re ta tion (.67*) .42* .47* .31* .58*

V alidity (.68*) .52* .44* .50*

T e s ts in G eneral (.55*) .50* .68*

R eliab ility (.33) .54*

Item A nalysis (.78*)

*P< .01

20

S ugges tions for Further Study

It is th is exper im en te r 's opinion th a t the p re se n t techn ique of

co rre la ting sco re s on com prehensive ach ievem ent t e s t s w ith a c loze

t e s t over the same large con ten t area le a v e s much to be d e s ire d due to

the re la t iv e ly sm all con ten t sam ple inc luded in the c loze s e le c t io n . In

th is w ay , the c loze t e s t is somew hat like a sp e c if ic e s s a y t e s t over a

large con ten t a re a . Therefore a techn ique of co rre la ting more sp e c if ic .

con ten t a reas w ith one ano ther should prove more s a t i s f a c to ry . The

u se fu ln e s s of the c lo ze techn ique as a form ative (d iagnostic) tool in

more sp e c if ic co n ten t a reas of a course could a lso be e s ta b l i s h e d by th is

tec h n iq u e . A more sp e c if ic c r ite rion t e s t could e a s i ly be cons truc ted

by the s e le c t io n of item s of a sp e c if ic con ten t area from a ready-m ade

ach ievem ent t e s t and in c lu s io n of only th e se items on the c r ite rion t e s t .

As for the c loze t e s t , Bormuth (1965) has shown th a t c lo ze t e s t s of le s s

than 50 item s tend to be u n re l ia b le , (The p resen t s tudy has shown tha t

th is is not a lw ays the c a s e b e c a u se a re l ia b i l i ty c o e ff ic ien t of .78 was

ob ta ined for a 20 item s u b te s t . ) Therefore , the number of item s on both

the cr ite rion and c loze t e s t should approach 50 i te m s .

Furtherm ore, there is a confounding var iab le w hich w as uncon­

tro lled in th is s tu d y . An in d iv id u a l 's sco re on a c loze t e s t can be thought

of as a com posite of two so u rces of v a r ian ce : com prehension and p rev i­

ous knowledge (co u rse -o r ien ted a c h ie v e m e n t) . Of p a r t ic u la r in te re s t for

contro l is tha t portion of com prehension w hich re p re se n ts an in d iv id u a l 's

21

sk i l l in u s in g sy n tax for word id en tif ica t io n s in ce th is is not d ire c t ly

re la ted to th a t w hich is m easured by trad it io n a l c o u rse -o r ie n te d

ach ievem ent t e s t s .

CHAPTER VI

CONCLUSION

Although the re l ia b il i ty of th is c lo ze t e s t is h igh and

com parable to th a t ob ta ined by o ther r e s e a r c h e r s , i t s co rre la tio n with

c r ite rion m easu res of ach ievem ent is not as high as d e s i r e d . Therefore

a rep lic a t io n of th is s tudy u s in g more sp e c i f ic con ten t a reas for

c r i te r ion m easu res to in c re a se the co n ten t v a l id i ty of the c loze t e s t

and c r ite rion m easu res is n e e d e d . Control of the com prehension

variab le is a lso d e s i r a b le .

22

APPENDIX A

CLOZE TEST*

Score In te rp re ta tio n

As with a ll s tanda rd score s c a l e s , any arb itrary mean and

s tandard d ev ia tio n can be u s e d . The recommended procedure is to

use ______ a mean of 50 and s tandard d ev ia tio n of 10.

H owever, norm alized s tandard sc o re s w ith o ther b a se s

can be u sed if the_________ s itu a t io n w arran ts . B ecause norm alized

s tandard sc o re s and l in ea r ly transform ed s tan d a rd •

sc o re s (Z scores) are often________reported on s c a le s with

the_________ sam e c o n s ta n ts (X = ________ 50_______, s = 10), they

c a n _________ e a s i ly be co n fu sed . I f the________ d is tr ib u tio n of

raw sco re s is______norm al, both s e ts of transform ed s c o re s

will have t h e same________ v a lu e s ; the farther the d is tr ib u tio n

of raw sco re s d e v ia te s from _______ norm ality , the g rea te r the

d isc re p a n c y in the two s e t s ______ of__________ derived s c o re s .

W hen norm alized s tandard sco res are reported on

_a______ transform ed s c a le w hich u s e d a mean of

*Perm ission to u se ex ce rp ts from tex t was ob ta ined from c o p y - write owner. F rederick G . Brown, P rinc ip les of E ducationa l and P sycho­log ica l T e s t in g , Dryden P ress In c . , 1970 , pp. 59, 120, 173 , 278, 349,

23

24

50 a n d ________Q s tandard d ev ia t io n of 10, they______ are

frequently c a l le d T sco res This usage c a u s e s some

confusion as T sc o re s were o rig inally defined (M cC all,

1922) with reference to a p a r t ic u la r norm group and

not in a generic s e n s e . M oreover, some p e rso n s use

the T -sco re labe l when referring to unnorm alized s tandard score

sys tem s w hich u se the 5 0 - 1 0 b ase ■ However, as

u sed to d ay , the T -sc o re d es ig n a tio n gen era lly ap p lie s to any norm alized

s tandard score system with X = 50 and s = 10.

25

V alidity , '

In our d is c u s s io n we have assum ed tha t the poin t on the t e s t

score sc a le , that d iv ided the a c c e p ta b le (p red ic tio n -su ccessfu l) , and

u n accep tab le (p re d ic t io n -u n su c c e s s fu l) group was know n. For p u rposes

of determ ining the v a l id i ty of a t e s t__________ we could ch o o se .

an arb itrary s e r ie s of sco re s and determ ine the d e c i s io n ­

making accu racy at ea c h ________ p o in t . For the t e s t u se r . ,

however , a more im p o r ta n t ta sk __ may be to determ ine

the_______ __optim al cu tting po in t , the point_________ on the t e s t

(predictor) sco re tha t w ill s e p a ra te the groups________so as

to p ro d u c e _______the_____ maximum number of co rrec t d e c is io n s

In se t t in g a cu tting score______ _ and determ ining i ts

v a l id i ty , three________ s ta t i s t i c s are of p a r t ic u la r re lev an ce :

the s e le c t io n r a t io , . th e to ta l________ correc t d e c i s io n s , and the

p o s it iv e h its . The s e le c t io n r a t i o ______ i s __________ the propor­

tion o f a p p lic a n ts s e le c te d ______ , i . e . , the ra tio of persons

s e le c te d to a p p l ic a n ts . A s the s e le c t io n reguirem ents

become more s tr in g en t ( i . e . , a s a higher cu tting________sco re

is u s e d ) , the s e le c t io n ra tio becom es low er. The other two terms

have been defined p rev io u s ly . .

T ests in G eneral

Although the con ten t Of s c h o la s t ic ap titude t e s t s is b a s ic a l ly

u n l im ite d , in point of fact con ten t is genera lly quite s ta n d a rd iz e d ,

which only a few item types appearing with any re g u la r i ty . Among the

most w idely _____ used item types are: vocabu la ry or a

v a r ia t io n such as an a lo g ie s or sen te n ce com pletion

(se lec tin g the proper word to com plete a sen te n c e ) : reading

com prehension ( where________ the s tuden t reads a paragraph

and then answ ers q u e s tio n s b ased upon the information

c o n ta in e d in ________ the paragraph); num erical a b i l i ty ,

te s te d through s tra ig h t com putation a n d / o r ________word

problems; t e s t s of a b s tra c t reaso n in g (such as number se r ie s

problems_____ or nonverbal an a ly q ies ) ; g enera l information .

item s; and te s t s of______ too ls______ s k i l l s such as use- o f ________

the d ic t io n a ry , map read in g , ________ and the in te rp re ta tio n of

t a b l e s ______ and_____ __g ra p h s . The e x a c t con ten t w il l_________ ,

of c o u r s e , vary w i t h the age leve l of the s tu d en ts

For e x a m p le , in the primary g ra d e s , before ch ild ren have

acguired any fluency in read in g , vocabu lary will be

te s te d by p ic ture vocabulary i te m s , but at o lder

ag es more com plex i t e m ______forms _, such as a n a lo g ie s , can

be in tro d u ced . S im ilarly , items m easuring sk il l in u s ing

27

the d ic t ionary and reading sim ple maps may be inc luded at the e lem entary

le v e l , while at the high schoo l and co lleg e lev e ls s tu d e n ts w ill be\

required to in terp re t com plex ta b le s and g rap h s .

28

R eliab ili ty

A re la ted problem is that of t e s t an x ie ty . Some peoplei

get very t e n s e , n e rv o u s , a n d u p se t a t the

p ro sp ec t of_____ tak ing a t e s t or exam ina tion . Freguentlv

th e se are people who do________ not have an ad eq u a te

unders tand ing of te s t in g procedures a n d / o r a r e _________ unsure

of th e ir own a b i l i t ie s ______. B ecause e x c e s s anx ie ty has a

de tr im en ta l e f fec t on the lev e l of perform ance and often

produces e r ra t ic p erfo rm ance , t e s t a n x i e ty _______ can

contribu te to m easurem ent error. M inor c a s e s of t e s t an x ie ty

can often be a l le v ia te d b y _______ a__________ thorough

ex p lan a tio n of the purposes of the te s t in g a n d ______ the______ _

u s e s of the t e s t s c o re s______ , and by insuring th a t the

su b je c t is fam iliar w ith the________te s t in g p ro c e d u re s . The la t te r

goal can be accom plished by e x te n s iv e p rac tice on

problems s im ila r to th o se to be found on_________

the t e s t . Both approaches d e c re a se anxie ty by reducing unknown

cond itions In more s e v e re 1 c a s e s , s p e c ia l te s t in g arrangem ents

or counse ling .m igh t be n ee d e d .

Item A nalysis

A second fundam ental index is the d isc r im in a tin g power of the

item , the a b i l i ty of the item to d if fe ren tia te betw een subgroups of

s tu d e n t s „ Id e a l ly , we would have an index of item

v a lid i ty w hich •_______showed the re la t io n o f item _______

.re sp o n se s to an e x t r a te s t c r i te r io n m easure; in p ra c t ic e , su ch

e x t e r n a l m easu res are not g en era lly a v a i la b le „ Thus

the d isc r im in a tin g power of the item is com puted

by ■________ comparing ind iv idual item r e s p o n s e s ________to

the to ta l score o n _______ the________ t e s t , b e c u a se the to ta l score_____

provides the b e s t e s t im a te j_ of the s tu d n e t 's overa ll

k n o w le d g e ______ of_________ the con ten t dom ain . N o t e tha t_________

when comparing the re sp o n se on_________an in d iv id u a l item to

the________ (total) t e s t s c o r e , w e _______ are assum ing th a t

a ll the t e s t_________item s are m easuring t h e sam e_________sk il l

or s e t of sk i l l s . Although th is a ssum ption may seem u n ju s t i ­

fied , the em pirica l d a ta u su a lly support th is g e n e ra l iz a t io n .

REFERENCES CITED

A n a s ta s i , Anne. P sycho log ica l T e s t in g . 3rd ed . M acM illa rd C o. ,1972.

Bormuth, John R. "Com parable C loze and M u lt ip le -C h o ic e Com prehen­s ion Test S c o re s ," Journal of R ead ing , Vol. X (February,1962), pp . 291-299 ,

'______. "C loze as a M easu re of R ea d ab il i ty , " P roceed ings ofthe In te rn a tio n a l Reading A s s o c ia t io n . Vol. VIII (19 63), pp . 1 3 1 -1 3 4 .. •

. "Optimum Sample Size and C loze Test Length in R e a d a b i l i ty ."Journal of Educational M e a su re m e n t , Vol. II (June 1965), p p . 111-116 .

___________ "An O pera tiona l D efin ition of Com prehension In s tru c t io n , “P sy c h o lin g u is t ic s and the Teaching of R ead in g , Boston: IRA Pre-C onven tion In s t i tu te , April 196.8, pp . 4 8 -6 0 .

_. "Factor Validity of C loze T es ts as M ea su re s of ReadingCom prehension A bility . " Reading R esearch Q u a r te r ly , Vol. IV (Spring, 1969), pp . 358-365).

Brown, F rederick G . P rinc ip les of Educationa l and P sych o lo g ica l T e s t in g , Dryden P re s s , I n c . , 19.70.

C h a l l , Jeanne S . "The In fluence of Previous Knowledge on Reading A b ili ty ," Educational R esearch B u lle t in , Vol. XXVII (January 1948), pp . 11 -20 .

F e ld t , Leonard S. "A T es t of the H ypo thes is that C h ro n b ach 's Alpha or Kuder-R ichardson C oeff ic ien t Twenty is the Same for Two T e s ts , " P sy c h o m etr ik a , Vol. XXXIV (September 19 69). pp . 363-373 .

H ill , W infred F . Learning A Survey of P sycho log ica l In te rp re ta t io n s . Revised e d . London: C hand ler Publish ing C o . , 1971.

30

31

M acG in itie ; W alte r H. "C ontex tual C o n s tra in t in English Prose P a ra g ra p h s ," The Journal of P sy c h o lo g y , Vol. LI (1961), pp. 121-130 .

M cC a ll , W . A. How to M easure in E d u c a tio n , Crowe 11-C o l l ie r and M acm illan , 1922.

N unnally , J u m C . Psycholm etric Theory . M cG raw -H ill Book C o . , 1967.

Rankin, Earl P . , J r . , and Joseph W . C u lh an e . "Com parable C loze and M u lt ip le -C h o ice Com prehension Test S c o re s ," Journal of R ead ing , Vol. XIII (December 1969), pp. 193-198 .

T a y lo r , W ilson L. "C loze Procedure: A New Tool for M easuringR ea d ab il i ty ," Journalism Q u a r te r ly , Vol. XXX (1953), pp. 415- 438.

_________ . "Recent D evelopm ents in the U se of C loze P ro ced u re ,"Journalism Q u a r te r ly , Vol. XXXIII (Winter, 1956), pp. 42 -28 .

__________. "C loze Readability Scores as Ind ices of Ind iv idual D iffer­en c e s in Com prehension and A p titu d e ," Journal of Applied P sy c h o lo g y , Vol. XLI (February 1957), pp. 19 -26 .