How Does CRTDSupport Software Learning?

Criterion Referenced Objectives as Foundation for EvaluationJulianne O’Neil, M.Ed.

Siemens Healthcare

Goals

• Recognize criterion-referenced test development (CRTD) integration within overall evaluation framework

• Describe components of criterion-referenced objectives

• Write criterion-referenced objectives with those components and consider use of mind mapping

• Identify possible solutions to challenges implementing performance-based tests

4 Levels of Evaluation

Level 1(Reaction)

• Learner satisfaction/reaction following training• Customer Satisfaction

Level 2(Learning)

• Knowledge acquisition, skill development, attitude change• Measurement of learning acquisition

Level 4(Results)

• Organizational results attributed to training• Reduced costs, reduced support time, increased profits,

improved productivity

Level 3(Behavior)

• Behavior change demonstrated on the job after training• “Training Transfer”

CRTD in Evaluation Framework

Level 1(Reaction)

• Learner satisfaction/reaction following training• Customer Satisfaction

Level 2(Learning)

• HOW DO YOU KNOW WHAT THEY HAVE LEARNED?• Criterion-referenced objectives/tests

Level 4(Results)

• Organizational results attributed to training• Reduced costs, reduced support time, increased profits,

improved productivity

Level 3(Behavior)

• Behavior change demonstrated on the job after training• “Training Transfer”

Objectives’ Focus on Performance• Traditional objectives and tests focus on declarative knowledge• Knowledge not guarantee of performance improvement • What are learners going to DO on job after training?• Want to improve performance to drive organizational success

CRTD in Evaluation Framework

• Goal of Siemens Healthcare Evaluation Framework:• To ultimately evaluate Levels 1 thru 4

• Level 2 Evaluation Focus:• Identify desired performance• Establish/maintain standards for objectives/tests• Restructure objectives format criterion-referenced• Declarative knowledge performance improvement

• Level 3 Evaluation Focus:• Data collection by written/phone survey• Survey based on CRTD objectives

Components of Criterion-Referenced Objectives

A – AudienceLearner/role

B – BehaviorDesired performance/skill

C – ConditionCircumstances under which behavior is performed

D – DegreeExtent to which the new behavior is to be demonstrated

Template for a CRTD Objective

Given <Under what CONDITION the learner will perform>,

The <WHO? AUDIENCE performing the behavior>

Will <Do WHAT? BEHAVIOR>

Correctly <DEGREE - HOW OFTEN the learner must perform correctly>.

Examples of CRTD Objectives

Given an out-of-balance Patient Census in which the new Month-to-Date amount is not equal to the Actual Month-to-Date Total, the financial analyst will write the procedures required to balance the patient census with 100% accuracy.

Audience: Financial Analyst

Behavior: Write procedures to balance patient census

Condition: Out-of-balance patient census

Degree: 100% Accuracy

Examples of CRTD Objectives

The reporting analyst will track the progress of report output when presented with the Output Job Status utility to effectively reprint reports 100% of the time.

Audience: Reporting Analyst

Behavior: Track progress of report output and print reports

Condition: Output Job Status Utility

Degree: Effectively reprint 100% of time

Examples of CRTD Objectives

Using 3 sample pairs of sterile examination gloves, the nurse will demonstrate the proper technique for donning sterile gloves without tearing or touching the outside of the gloves at least 1 out of 3 tries.

Audience: Nurse

Behavior: Demonstrate proper technique for donning sterile gloves

Condition: Using sample sterile examination gloves

Degree: Without tearing or touching the outside of the gloves 1out of 3 tries

Defining 2 Types of Objectives• Topic complexity determines:

• Depth/breadth of objectives• What to test

• Standard for what is to be tested (Siemens standard: Test terminal objectives)

Terminal Objectives:Determine performance desired upon training completion• Include at beginning of course material• Describe what learners are to demonstrate• Include all content to be tested• Describe learning “above memory level”

Enabling Objectives:Enable achievement of terminal objectives.• Define “teachable” content at chapter or modular level• Describe prerequisite knowledge/skills needed to satisfy terminal objectives

Siemens-Defined Standards for CRTD

CRTD OBJECTIVES:• Terminal objectives

• Must be tested• Should be performance-based whenever possible• Must be defined using CRTD format

• Enabling objectives• May be written if needed• May optionally be tested• May optionally be defined using CRTD format

• Audience, Condition and Degree may be defined once for all objectives in group (vs. stated for each objective)

YOUR STANDARDS MAY BE DIFFERENT!

CRTD Objectives Activity

1. Instructional Goal: Select one from list in handout2. Write 1 terminal objectives in CRTD format to address chosen goal

CRTD Template:

Given <CONDITION for learner performance>,

The <learner AUDIENCE performing the behavior>

Will <desired BEHAVIOR>

Correctly <DEGREE of success>.

CRTD Objectives Activity

1. Write 2-3 enabling objectives to support the terminal objective just written

2. NOTE: These enabling objectives do not have to be written in CRTD format.

Siemens-Defined Standards for CRTDCRTD END-OF-COURSE TESTS:• Test items

• Must test terminal objectives• Optionally test enabling objectives (depending on complexity)• For performance-based objectives, practice opportunities must be provided during

class• Test item types allowed

• Performance-based• When “traditional” performance-based can’t be used:

• Multiple-choice• Matching• Sequencing

• NO:• Fill-in• Essay• True/False

YOUR STANDARDS MAY BE DIFFERENT!

Corresponding Test ItemsPerformance-based Test Item:Can test one or more objectives just by producing an end product/service.

SAMPLE TEST ITEM:Download a model report, modify the report, and add it as a custom report:

• Download report from the Report Catalog.• Save report as “LA<n>_ORE_Report_Catalog.rpt,” where <n> is your Student Number.

• Open report in Crystal and change report header name to “Operational Reports Report Catalog List for <your name>.”

• Add new report to the Report Catalog using the following values:• Report Name: “LA-<n>,” where <n> is your Student Number• Template: Your LA<n>_ORE_Report_Catalog.rpt Crystal Report• Description: Learner Assessment• Filter: Create a new filter file using the available report parameters

• Create a sub-folder under the SCR Class Reports folder with name “Folder_ <n>,” where <n> is your student number and add your new report to “Folder_ <n>”

• Locate your report output.• Create an Export Package called “LA<n>_Assessment_ <n>,” where <n> is your student

number. Be sure the Export Package contains your report.

Corresponding Test Items

Multiple-Choice Item:NOTE: This is also performance-based…

1) The Stored Procedure used by the model Dietary Report is• ORE_0100• DIETARY_REPORT_SP• ORE_0105• ORE_0030• The Dietary Report does not use a Stored Procedure

MindMap Template

Sample Instructional Objectives MindMap

Terminal and Enabling Objectives

Considerations for Performance-Based CRTD Implementation at Siemens

Considerations• Test writer resistance to change (standards restricting test item types)• Manual test scoring for “traditional” performance-based (non multiple choice)

• Automated scoring for multiple-choice (Questionmark® Perception®)

• Possible costs to maintain product software/server environments

Benefits of Performance-Based CRTD Implementation to Siemens….and Possibly to YOU!

Benefits• Provides consistency of objectives/tests across modules

• Facilitates cleaner test item construction (based on CRTD objectives)

• Allows higher fidelity testing (closer to actual job task performance)

• Permits clearer focus for instructional design

• Improves measure of learning success at end of training

• Sets performance measures foundation for post-training assessment