Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

1rsmichael

Mea

sure

men

t T

ypes

of I

nstr

umen

ts

Y52

0 S

trat

egie

s fo

r E

duca

tiona

l Inq

uiry

Rob

ert S

Mic

hael

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

2rsmichael

Firs

t Que

stio

ns

■W

hat i

s th

e st

ated

pur

pose

of t

he in

stru

men

t? Is

it to

: •

Col

lect

info

rmat

ion?

•P

redi

ct a

spe

cifie

d pe

rfor

man

ce (

e.g.

, The

pur

pose

of

the

SA

T is

to p

redi

ct fi

rst c

olle

ge s

emes

ter

GP

A).

•M

easu

re a

chie

vem

ent?

•M

easu

re p

erso

nalit

y?•

Mea

sure

atti

tude

tow

ard

[an

attit

udin

al o

bjec

t]?■

The

ade

quac

y of

a te

st c

an b

e ju

dged

onl

y if

you

know

its

sta

ted

purp

ose.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

3rsmichael

Bac

kgro

und

Ter

ms

& Id

eas

■In

stru

men

ts: C

olle

ctio

ns o

f ite

ms

or q

uest

ions

inte

nded

to

mea

sure

, col

lect

ivel

y (t

otal

sco

re)

leve

ls o

f th

eore

tical

var

iabl

es (

cons

truc

ts)

not e

asily

obs

erva

ble

di

rect

ly. E

xam

ples

:•

inte

llige

nce

•op

timis

m•

qual

ity o

f life

•jo

b sa

tisfa

ctio

n•

depr

essi

on•

anxi

ety

•po

litic

al o

rient

atio

n

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

4rsmichael

Inst

rum

ents

vs

Indi

ces

■In

stru

men

t(a

ka, t

est,

scal

e): T

ype

of r

espo

nse

to

item

s on

an

inst

rum

ent a

re p

resu

med

to b

e ca

used

by

the

unde

rlyin

g co

nstr

uct y

ou a

re tr

ying

to m

easu

re.

■In

dex

:In

con

tras

t, an

inde

x co

nsis

ts o

f ite

ms

that

de

fine

the

cons

truc

t.

■In

stru

men

t exa

mpl

e: G

iven

a s

cale

that

mea

sure

s op

timis

m, w

e be

lieve

that

mor

e po

sitiv

e re

spon

ses

(i.e.

, hig

her

tota

l sco

res)

are

cau

sed

by s

omeo

ne w

ho

has

high

er le

vels

of o

ptim

ism

.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

5rsmichael

Inst

rum

ents

vs

Indi

ces

■In

dex

exam

ple:

A s

ocio

econ

omic

sta

tus

(SE

S)

inde

x m

ight

incl

ude

a m

easu

rem

ent o

f edu

catio

n le

vel.

But

ha

ving

mor

e ed

ucat

ion

is n

ot c

ause

d by

hig

her

SE

S;

inst

ead

havi

ng m

ore

educ

atio

n in

fluen

ces

SE

S.

■A

n S

ES

inde

x m

easu

res

SE

S, b

y de

finiti

on. N

o at

tem

pt is

mad

e to

mea

sure

a la

tent

con

stru

ct.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

6rsmichael

Thr

ee ty

pes

of In

stru

men

ts

■Q

ues

tio

nn

aire

s. G

oal:

Col

lect

ion

of in

form

atio

n. N

o at

tem

pt to

mea

sure

und

erly

ing

late

nt c

onst

ruct

.•

Tea

cher

Tur

nove

r S

urve

y•

Ear

ly L

itera

cy In

terv

entio

n G

rant

Pro

g. S

urve

y

■C

og

nit

ive

do

mai

nin

stru

men

ts. G

oal:

Mea

sure

ac

hiev

emen

t or

aptit

ude.

■A

ffec

tive

do

mai

nin

stru

men

ts. G

oal:

Mea

sure

pe

rson

ality

con

stru

cts,

atti

tude

s, b

elie

fs, i

nter

ests

, etc

.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

7rsmichael

Que

stio

nnai

re D

evel

opm

ent G

uide

lines

■C

lear

, con

cise

, una

mbi

guou

s st

atem

ents

.■

Be

awar

e of

res

pond

ent r

eadi

ng le

vels

.

■P

ilot t

est,

pilo

t tes

t, pi

lot t

est.

■F

rary

, Rob

ert (

nd).

A b

rief g

uide

to q

uest

ionn

aire

deve

lopm

ent.

[Ret

rieve

from

http

://er

icae

.net

/ft/ta

mu/

vpiq

ues3

.htm

]

■D

illm

an, D

. A. (

1978

). M

ail a

nd te

leph

one

surv

eys:

The

tota

l des

ign

met

hod.

New

Yor

k: J

ohn

Wile

y.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

8rsmichael

Exa

mpl

e of

Wor

ding

•W

hen

pres

ente

d w

ith d

iffic

ult p

rofe

ssio

nal s

ituat

ions

w

here

a s

uper

ior

cens

ures

you

for

an a

ct fo

r w

hich

yo

u ar

e no

t res

pons

ible

, how

freq

uent

ly d

o yo

u re

spon

d in

an

asse

rtiv

e w

ay?

–A

ll of

the

time

–S

ome

of th

e tim

e–

Non

e of

the

time

■N

otic

e th

e vo

cabu

lary

leve

l and

sen

tenc

e co

mpl

exity

.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

9rsmichael

Alte

rnat

e W

ordi

ng

•W

hen

your

bos

s bl

ames

you

for

som

ethi

ng y

ou d

id

not d

o, h

ow o

ften

do y

ou s

tick

up fo

r yo

urse

lf?–

All

of th

e tim

e–

Som

e of

the

time

–N

one

of th

e tim

e

■W

hich

wor

ding

is p

refe

rabl

e? W

hy?

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

10rsmichael

Exa

mpl

e 2:

Alte

rnat

e W

ordi

ng

•D

urin

g th

e pa

st 4

wee

ks, h

ow o

ften

have

you

felt

a se

nse

of lo

nelin

ess?

–A

ll of

the

time

–S

omet

imes

–F

rom

tim

e to

tim

e–

Nev

er

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

11rsmichael

Exa

mpl

e 2:

Alte

rnat

e W

ordi

ng

•H

ow o

ften

in th

e pa

st m

onth

hav

e yo

u fe

lt al

one

in

the

wor

ld?

–E

very

da

y–

Som

e da

ys–

Occ

asio

nally

–N

ever

■Is

this

an

impr

ovem

ent?

Mig

ht s

ome

prob

lem

s st

ill

exis

t?

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

12rsmichael

Ste

ps in

Con

stru

ctin

g an

Inst

rum

ent

■D

eter

min

e cl

early

wha

t it i

s yo

u w

ant t

o m

easu

re.

■G

ener

ate

a po

ol o

f pos

sibl

e ite

ms

(abo

ut 6

0 fo

r a

20 it

em in

stru

men

t).

■D

ecid

e on

que

stio

n fo

rmat

(ye

s/no

, str

ongl

y ag

ree,

agr

ee, e

tc)

■D

ecid

e ho

w to

sco

re th

e in

stru

men

t (e.

g. o

n an

opt

imis

m s

cale

, all

resp

onse

s th

at in

dica

te o

ptim

ism

sho

uld

get h

ighe

r sc

ores

, so

that

the

mor

e op

timis

tic y

ou a

re, t

he h

ighe

r yo

ur to

tal s

core

).■

Adm

inis

ter

the

inst

rum

ent t

o a

pilo

t sam

ple

of p

eopl

e: 1

00 -

300.

■S

cale

pur

ifica

tion:

Eva

luat

e ea

ch it

em u

sing

the

pilo

t dat

a:•

The

goa

l is

to s

elec

t the

sm

alle

st p

ossi

ble

subs

et o

f ite

ms

that

is in

tern

ally

con

sist

ent.

Thi

s re

duce

s re

spon

dent

bur

den

(few

que

stio

ns)

and

high

rel

iabi

lity

(inte

rnal

co

nsis

tenc

y). U

se R

elia

bilit

y pr

oced

ure

in S

pss.

•C

alcu

late

Cro

nbac

h’s

alph

a fo

r va

rious

sub

sets

of i

tem

s an

d ca

lcul

ate

the

“item

-re

mai

nder

” co

effic

ient

s (t

he c

orre

latio

n of

an

item

with

the

sum

of th

e re

mai

ning

item

s).

•D

rop

item

s th

at lo

wer

Cro

nbac

h’s

alph

a•

If al

pha

rem

ains

low

, use

Spe

arm

an B

row

n P

roph

ecy

form

ula

to te

ll yo

u ho

w m

any

mor

e ite

ms

you

need

.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

13rsmichael

Con

stru

ctin

g In

stru

men

ts

■P

rece

ding

ste

ps w

ork

wel

l for

que

stio

nnai

res,

atti

tude

m

easu

res,

per

sona

lity

scal

es.

■S

ome

wha

t diff

eren

t pro

cedu

res

are

need

ed fo

r co

gniti

ve d

omai

n in

stru

men

ts.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

14rsmichael

Cog

nitiv

e D

omai

n In

stru

men

ts

■A

chie

vem

ent I

nstr

umen

ts•

Per

form

ance

lim

ited

to s

peci

fic d

omai

n (e

.g.,

mus

ic)

•C

lass

room

Tes

ts (

teac

her

deve

lope

d)•

“Hig

h st

akes

” te

sts

–Io

wa

Tes

t of B

asic

Ski

lls–

IST

EP

■A

ptitu

de In

stru

men

ts

•P

erfo

rman

ce “

gene

raliz

ed”

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

15rsmichael

Ach

ieve

men

t Tes

ts

■C

lass

room

test

s: m

easu

re s

tude

nt a

chie

vem

ent i

n sp

ellin

g, m

ath,

Eng

lish,

his

tory

, sci

ence

, etc

.■

The

late

nt c

onst

ruct

is “

achi

evem

ent”

■T

he o

pera

tiona

l def

initi

on is

“nu

mbe

r co

rrec

t on

a pa

rtic

ular

test

.”■

Item

s on

a te

st a

re a

sam

ple

from

all

poss

ible

qu

estio

ns th

at c

ould

be

aske

d fr

om a

par

ticul

ar

acad

emic

dom

ain.

■F

or te

st r

esul

ts to

be

inte

rpre

tabl

e, te

st (

or s

ubsc

ale)

sh

ould

incl

ude

item

s fr

om o

nly

one

dom

ain.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

16rsmichael

Ach

ieve

men

t Tes

ts (

cont

inue

d)

■C

lass

room

test

s (c

ontin

ued)

: If a

test

con

tain

ed b

oth

spel

ling

and

mat

h ite

ms

and

num

ber

corr

ect w

ere

sum

med

, wha

t wou

ld th

e sc

ore

mea

n?■

Inst

ead,

sum

the

corr

ect s

pelli

ng it

ems

sepa

rate

ly fr

om

the

corr

ect m

ath

item

s.■

Ach

ieve

men

t tes

ts c

an b

e ei

ther

nor

m-b

ased

or

crite

rion-

base

d.■

Can

be

obje

ctiv

ely

scor

ed it

ems

(tru

e/fa

lse,

mul

tiple

ch

oice

, mat

chin

g, o

rder

ing)

or

perf

orm

ance

as

sess

men

ts (

aka,

ess

ay),

or

auth

entic

ass

essm

ents

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

17rsmichael

Ach

ieve

men

t Tes

t Ite

ms

■H

ighl

y so

phis

ticat

ed g

uide

lines

for

item

writ

ing

exis

t. S

ee [e

nter

ref

eren

ce h

ere]

■Ite

ms

diffe

r in

the

leve

l of c

ogni

tive

func

tioni

ng

requ

ired

to a

nsw

er th

em:

•K

now

ledg

e•

Com

preh

ensi

on•

App

licat

ion

•A

naly

sis

•S

ynth

esis

/Eva

luat

ion

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

18rsmichael

Blo

om’s

Tax

onom

y: T

est I

tem

s

■B

loom

’s T

axon

omy

desc

ribes

diff

eren

t lev

els

of

cogn

itive

func

tioni

ng. I

tem

s ca

n be

cla

ssifi

ed

depe

ndin

g on

the

leve

l of c

ogni

tive

func

tioni

ng

requ

ired

to a

nsw

er.

■O

bjec

tivel

y sc

ored

item

s (e

.g.,

mul

tiple

cho

ice,

tr

ue/fa

lse,

mat

chin

g, s

hort

ans

wer

) ca

n be

writ

ten

to

requ

ire e

ither

kno

wle

dge,

com

preh

ensi

on, a

pplic

atio

n,

or a

naly

sis.

■C

aref

ully

con

stru

cted

ess

ay q

uest

ions

are

use

d to

re

quire

syn

thes

is/e

valu

atio

n an

swer

s.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

19rsmichael

Blo

om’s

Tax

onom

y: L

evel

s of

Kno

wle

dge

■K

no

wle

dg

e:R

ecog

nize

or

reca

ll ke

y fa

cts,

def

initi

ons,

co

ncep

ts, r

ules

, prin

cipl

es. (

“low

est l

evel

”)•

Key

wor

ds: D

efin

e, id

entif

y, la

bel,

list,

nam

e■

Co

mp

reh

ensi

on

:U

nder

stan

ding

of a

bstr

act t

erm

s.

Rec

ogni

ze o

r ex

plai

n si

mila

ritie

s an

d di

ffere

nces

.•

Key

wor

ds: T

rans

late

, Com

pare

, con

tras

t, re

late

, di

ffere

ntia

te, d

istin

guis

h.■

Ap

plic

atio

n:

App

ly a

con

cept

to a

con

cret

e ex

ampl

e.•

Key

wor

ds: S

olve

, app

ly,

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

20rsmichael

Blo

om’s

Tax

onom

y: L

evel

s of

Kno

wle

dge

■A

nal

ysis

:D

ivid

e a

who

le in

to it

s co

mpo

nent

s.•

Key

wor

ds: C

ateg

oriz

e, s

ort,

subd

ivid

e, s

epar

ate

■S

ynth

esis

/Infe

ren

ce: R

ecog

nize

or

expl

ain

the

evid

ence

rel

atin

g to

a g

ener

aliz

atio

n.•

Key

wor

ds: I

nfer

, gen

eral

ize,

ded

uce,

pre

dict

, sp

ecul

ate,

app

ly, c

oncl

ude

■E

valu

atio

n: J

udge

qua

lity,

cre

dibi

lity,

wor

th o

r pr

actic

ality

, gen

eral

ly u

se e

stab

lishe

d cr

iteria

and

ex

plai

n ho

w th

ose

crite

ria a

re o

r ar

e no

t met

.•

Key

wor

ds: E

valu

ate,

judg

e

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

21rsmichael

Con

stru

ctin

g a

Cla

ssro

om T

est

■T

able

of S

peci

ficat

ions

•C

ross

es c

onte

nt a

rea

(row

s) w

ith le

vel o

f cog

nitiv

e fu

nctio

ning

(co

lum

ns).

•In

ters

ectin

g ce

lls s

how

eith

er n

umbe

r or

per

cent

of

item

s to

be

writ

ten

on a

par

ticul

ar s

ubje

ct a

nd le

vel

of c

ogni

tive

func

tioni

ng.

•In

sure

s th

at te

st c

over

age

refle

cts

emph

asis

of

topi

cs in

cla

ss a

nd a

void

s in

clud

ing

too

man

y ite

ms

that

func

tion

at lo

wer

cog

nitiv

e le

vels

.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

22rsmichael

Tab

le o

f Spe

cific

atio

ns: E

xam

ple

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

23rsmichael

Ach

ieve

men

t Tes

t Val

idity

■V

alid

ity: D

oes

the

inst

rum

ent m

easu

re w

hat w

e in

tend

to

mea

sure

(an

d w

hat i

t was

des

igne

d to

mea

sure

)?■

“I a

m g

oing

to m

easu

re k

now

ledg

e of

res

earc

h m

etho

ds b

y pl

acin

g ea

ch s

tude

nt’s

term

pap

er o

n a

post

age

scal

e an

d re

cord

ing

the

num

ber

of o

unce

s.”

■T

he s

cale

rea

ding

is r

elia

ble

(abl

e to

be

repe

ated

) bu

t ou

nces

is n

ot r

elat

ed to

kno

wle

dge

of r

esea

rch

met

hods

.■

How

ever

, if a

n in

stru

men

t is

valid

, it m

ust a

lso

be

relia

ble.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

24rsmichael

Ach

ieve

men

t Tes

t Val

idity

■C

on

ten

t V

alid

ity

is o

ur c

once

rned

for

achi

evem

ent

test

s.■

Val

idity

is n

ot “

all o

r no

thin

g.”

Inst

ead

we

judg

e ba

sed

on th

e ac

cum

ulat

ion

of p

iece

s of

evi

denc

e.■

Pie

ces

of v

alid

ity e

vide

nce:

•V

alid

ity is

“bu

ilt in

”•

Tab

le o

f spe

cific

atio

ns s

erve

s as

“au

dit t

rail”

•Ite

ms

judg

ed b

y ex

pert

s as

rep

rese

ntin

g th

e do

mai

n•

Per

form

ance

of e

xper

ts v

sno

vice

s•

Fac

tor

anal

ysis

for

dim

ensi

onal

ity

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

25rsmichael

Ach

ieve

men

t Tes

t Rel

iabi

lity

■R

elia

bilit

y:•

mea

ns c

onsi

sten

cy, s

tabi

lity,

rep

eata

bilit

y.•

is e

stab

lishe

d be

fore

val

idity

.

•se

ts a

n up

per

limit

on v

alid

ity.

•al

way

s a

corr

elat

ion

coef

ficie

nt.

•a

test

can

be

high

ly r

elia

ble

but l

ack

valid

ity.

•a

valid

test

is a

lso

relia

ble.

•sa

crifi

ce s

ome

relia

bilit

y fo

r va

lidity

.

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

26rsmichael

Typ

es o

f Ach

ieve

men

t Tes

t Rel

iabi

lity

■S

tab

ility

:T

est p

rodu

ces

sam

e re

sults

with

rep

eate

d te

stin

g.•

Tes

t -re

test

•P

aral

lel f

orm

s

•A

ltern

ate

(equ

ival

ent)

form

s

■In

tern

al c

on

sist

ency

or h

omog

enei

ty: D

egre

e to

w

hich

all

item

s in

the

test

mea

sure

the

sam

e co

nstr

uct

(ach

ieve

men

t).

•Ite

m -

tota

l cor

rela

tion

•S

plit

-ha

lf re

liabi

lity

•C

ronb

ach’

sA

lpha

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

27rsmichael

Typ

es o

f Ach

ieve

men

t Tes

t Rel

iabi

lity

■E

qu

ival

ence

:T

est p

rodu

ces

the

sam

e re

sult

as d

oes

an e

quiv

alen

t ins

trum

ent.

•P

aral

lel i

tem

s on

alte

rnat

e fo

rms

Mea

sure

men

t -T

ypes

of I

nstr

umen

ts -

28rsmichael

Indi

cato

rs o

f Ite

m P

erfo

rman

ce

■It

em D

iffi

cult

y: P

erce

nt o

f tes

t tak

ers

who

ans

wer

ed

item

cor

rect

ly.

•T

oo e

asy

and

too

diffi

cult

item

s pr

ovid

e no

in

form

atio

n. “

Idea

l” di

fficu

lty d

epen

ds o

n ite

m ty

pe.

■It

em D

iscr

imin

atio

n:

•Id

entif

y 27

% o

f tes

t tak

ers

with

the

low

est t

otal

test

sc

ore;

like

wis

e, to

p 27

%.

•F

or e

ach

item

(pr

opor

tion

of to

p w

ho g

ot it

em r

ight

) –

(pro

port

ion

botto

m w

ho g

ot it

em r

ight

).•

Goo

d ite

ms:

few

in th

e bo

ttom

ans

wer

cor

rect

ly;

man

y in

the

top

grou

p do

.