Barry Williams 1

Developing Assessment Developing Assessment InstrumentsInstruments

Dick & Carey

Chap. 7

Barry Williams 2

Criterion-Referenced TestsCriterion-Referenced Tests

Designed to measure explicit behavioral objectives

Used to evaluate:– learner performance– effectiveness of the instruction

Barry Williams 3

Criterion-ReferencedCriterion-Referenced

Also called objective-referenced Refers directly to explicit “criterion” or

specified performance “Criterion-Referenced Test” must:

– match test item and performance objective– stipulate degree of mastery of the skill

Barry Williams 4

Types of Criterion TestsTypes of Criterion Tests

Pretest: 1. Consists of items that:

- measure entry behavior skills- test skills to be taught- draw from skills below the entry behavior line

2. Helps determine appropriateness of required entry skills.

3. Used during formative evaluation process. May be discarded in final version of instruction.

Barry Williams 5

Types of Criterion TestsTypes of Criterion Tests

Posttest: 1. Assesses all the objectives, focusing on

terminal objectives 2. Helps identify ineffective instructional

segments 3. Used during the design process and may

be eventually modified to measure only terminal objectives

Barry Williams 6

Designing Tests for Learning Designing Tests for Learning DomainsDomains

Intellectual & Verbal Information– paper & pencil

Attitudinal– state a preference or choose an option

Psychomotor– performance quantified on checklist– subordinate skills tested in paper-and-pencil

format

Barry Williams 7

Determining Mastery LevelsDetermining Mastery Levels

Approach # 1– mastery defined as level of performance

normally expected from the best learners– arbitrary (norm-referenced)

Approach # 2– defined in statistical terms, beyond mere chance– mastery varies with critical nature of task

example: nuclear work Vs. paint a house

Barry Williams 8

Writing Test ItemsWriting Test Items

What should test items do? Match the behavior of the objective

– Use the correct “verb” to specify the behavior Match the conditions of the objective

Barry Williams 9

Writing Test ItemsWriting Test Items

How many test items do you need?Determined by learning domains

Intellectual requires three or more– Wide range use random sample

Barry Williams 10

Writing Items Writing Items (continued)(continued)

What types (true / false, multiple choice, etc..) to use?clues provided by the behavior listed in the

objectivereview “Types of Test Items” this chap. p 148

Barry Williams 11

Writing Items Writing Items (continued)(continued)

Item types tempered by: amount of testing time ease of scoring amount of time to grade probability of guessing ease of cheating, etc. availability of simulations

Barry Williams 12

Writing Items Writing Items (continued)(continued)

What types are inappropriate?– true / false for definition

discrimination, not definition

Acceptable alternatives from “best possible”– for simulations

list steps

Barry Williams 13

Constructing Test ItemsConstructing Test Items

Consider: vocabulary setting of test item (familiar Vs. unfamiliar) clarity

– all necessary information trick questions

– double negatives, misleading information, etc.

Barry Williams 14

Other FactorsOther Factors

Sequencing Items– Consider clustering by objective

Test Directions– Clear and concise– General– Section specific

Evaluating Tests / Test Items

Barry Williams 15

Measuring Performance, Measuring Performance, Products, & AttitudesProducts, & Attitudes

Write directions to guide learner activities and

Barry Williams 16

Evaluating Performance, Evaluating Performance, Products, & AttitudesProducts, & Attitudes

Construct an instrument to evaluate these activities– a product, performance, or attitude

Sometimes includes both process and a product – For example -- TRDEV 518

Barry Williams 17

Test Directions for Test Directions for Performance, Products, & Performance, Products, &

AttitudesAttitudes Determine the

– Amount of guidance?– Special conditions

time limits, special steps, etc.– Nature of the task (i.e., complexity)– Sophistication level of the audience

Barry Williams 18

Assessment Instruments for Assessment Instruments for Performance, Products, & Performance, Products, &

AttitudesAttitudes

Identify what elements are to be evaluated– cleanliness, finish, tolerance of item, etc.

Paraphrase each element Sequence items on the instrument Select the type of judgment for rater Determine instrument scoring

Barry Williams 19

Formats for Assessments of Formats for Assessments of Performance, Products, & Performance, Products, &

AttitudesAttitudes

Checklist Rating Scale Frequency Counts Etc.

Barry Williams 20

Performance, Products, & Performance, Products, & Attitudes -- ScoringAttitudes -- Scoring

Guidelines?

Barry Williams 21

Evaluating CongruencyEvaluating Congruency

Skills, Objectives, & Assessments should refer to the same behaviors

To check for congruency– Construct an Congruency Evaluation Chart

include: Subskills, Behavioral Objectives, & Test Items