Slide 1

Slide 2

Technical Issues Two concerns Validity Reliability

Slide 3

Data Collection Quiz 1 Answer the five questions on Quiz 1.Quiz 1

Slide 4

Data Collection Quiz 1 Answers Score you paper using the following key 1. A 2. B 3. A 4. B 5. B

Slide 5

Data Collection Quiz 1 How well did you do? Should I use this score as a part of your grade? Does this score indicate your level as a graduate student? What we have here is a serious lack of communication! Most students object strongly to using this score as a part of their grade because it isnt fair. Most students object strongly to being labeled bright or challenged on the basis of their grade. Their reasoning is the test isnt fair it doesnt cover material relevant to this course. Welcome to the technical world of instrumentation.

Slide 6

Technical Issues Validity extent to which interpretations made from a test score are appropriate Characteristics The most important technical characteristic Situation specific Does not refer to the instrument but to the interpretations of scores on the instrument Best thought of in terms of degree

Slide 7

Technical Issues Validity (continued) Four types Content to what extent does the test measure what it is supposed to measure Item validity Sampling validity Determined by expert judgment

Slide 8

Technical Issues Validity (continued) Construct the extent to which a test measures the construct it represents Underlying difficulty defining constructs Estimated in many ways Criterion-related Predictive to what extent does the test predict a future performance Concurrent - to what extent does the test predict a performance measured at the same time Estimated by correlations between two tests

Slide 9

Technical Issues Validity (continued) Consequential to what extent are the consequences that occur from the test harmful Estimated by empirical and expert judgment Factors affecting validity Unclear test directions Confusing and ambiguous test items Vocabulary that is too difficult for test takers

Slide 10

Technical Issues Factors affecting validity (continued) Overly difficult and complex sentence structure Inconsistent and subjective scoring Untaught items Failure to follow standardized administration procedures Cheating by the participants or someone teaching to the test items

Slide 11

Technical Issues Reliability the degree to which a test consistently measures whatever it is measuring Characteristics Expressed as a coefficient ranging from 0 to 1 A necessary but not sufficient characteristic of a test

Slide 12

Technical Issues Test reliability Stability consistency over time with the same instrument Test retest Estimated by a correlation between the two administrations of the same test Equivalence consistency with two parallel tests administered at the same time Parallel forms Estimated by a correlation between the parallel tests

Slide 13

Technical Issues Test reliability (continued) Internal consistency artificially splitting the test into halves Several coefficients split halves, KR 20, KR 21, Cronbach alpha All coefficients provide estimates ranging from 0 to 1