SNDT Women's University Introduction to Evaluation and Assessments Presented by Kathleen (Kat)...

SNDT Women's University

Introduction to Evaluation and Assessments

Presented by Kathleen (Kat) Miller, Senior Consultant with Booz Allen Hamilton

4 August, 2007

2


Evaluation Model – Donald Kirkpatrick

In 1959, Donald Kirkpatrick developed what has become one of the most popular models for evaluating education programs.

Kirkpatrick's system has four levels of evaluation.

3


Evaluation

Purpose of Evaluation

“The reason for evaluating it to determine the effectiveness of a training program.”

© Donald L. Kirkpatrick, “Evaluating Training Programs: The Four Levels,”

4


Main Reasons for Evaluation

Determine how to improve future programs

Determine whether a program should be continued or dropped.

Justify the existence of the training department

5


Kirkpatrick’s Level 1 - Reaction

Measure of “Customer Satisfaction”

Measures the emotional reaction of participants after a learning experience. – It attempts to answer questions regarding the

participants' perceptions:

– Did they like it?

– Was the material relevant to their work?

– Did it meet their expectations?

– How should the learning experience be improved?

6


Why Measure Reaction?

It provides valuable feedback and suggestions

It makes the trainees feel that their input is important

It is easy to do effectively

A positive reaction promotes motivation!

7


Guidelines for Evaluating “Reaction”

1. Determine exactly what you want to find out

2. Design a form the will quantify reaction

3. Encourage written comments and suggestions

4. Get a 100 percent immediate response

5. Get honest responses

6. Develop acceptable standards

7. Measure reactions against standards and that action

8. Communicate (as appropriate)

8


Kirkpatrick’s Level 2 - Learning

Measures whether or not a student achieved the learning objectives during the learning event or during a series of such events.

For example: we might test for written communication skills after instruction to determine if the learner is now qualified for the job.

Before Training

After Training

9


Why Measure “Learning?”

Kirkpatrick defines learning as, “the extent to which participants change attitudes, increase knowledge and/or skill as a result of attending a program.”

Was knowledge increased?

Were skills developed?

Were attitudes changed?

10


Guidelines for Evaluating “Learning”

1. Use a control group if possible

2. Use pre/post tests– Written for knowledge measurement

– Performance for skills

3. Get 100 percent response

4. Use results to take appropriate action

11


Kirkpatrick’s Level 3 - Behavior

Measures whether learners are able to apply their new knowledge and skills to their job. – Are they using those skills on the job?

– Are there other issues that are stopping them from being successful on the job?

– Has their behavior changed?

Information for Level 3 evaluations is generally gathered via surveys and personal interviews.

12


Four Conditions for Behavior to Change

The person must have the desire to change

The person must know what to do and how to do it

The person must work in the right climate

The person must be rewarded for changing

13


Guidelines for Evaluating “Behavior”

Use a control group if possible

Allow time for the change in behavior to take place

Evaluate before and after the program

Survey or interview the trainee, as well as someone who observes their behavior

Get 100 percent response

Repeat the evaluation at appropriate times

Consider cost versus benefits

14


Kirkpatrick’s Level 4 - Results

Increased Production

Improved Quality

Decreased Costs

Reduction in Accidents

Increased Sales

Reduced Turnovers

Higher Profits

15


Challenges to Evaluating “Results”

Some are intangible

Results may not occur for a long time

Many other factors impact results

The trainer cannot control all factors

Such is hard to control all those

wild cats!

16


Guidelines for Evaluating Results

Use a control group if possible

Allow time for results to be achieved

Measure both before and after

Repeat the measurement at appropriate times

Consider cost versus behavior

Be satisfied with evidence, if proof is not possible

17


Assessments

What is an assessment ?

“Any systematic method of obtaining evidence from posing questions to draw inferences about the knowledge, skills, attitudes, and other characteristics of people for a specific purpose”

© Copyright Questionmark Corporation 2004

18


Types of Assessments

Exam

Test

Quiz

Survey

19


Uses of Assessments

Diagnostic

Formative

Needs

Reaction

Summative

20


Diagnostic Assessments

Typically used in pre-learning

Identity skill gaps and needs

Identity student preferences

21


Formative Assessment

Provide feedback for the learners

Search and retrieval

May contribute to overall grades

22


Needs Assessments

Assess knowledge, skills, abilities, and attitudes

Serve as a gap analysis

Determine what should be taught based upon needs

23


Reaction Assessments

Students’ opinions are collected

Kirkpatrick's Level 1

Instructor can alter the learning experience for the future

24


Summative Assessments

Summary of knowledge/skill of the learner

Provides a quantitative grade

May result in certification

25


Stakes of an Assessment

Assessment “stakes” are the consequences to the candidate

Low

Medium

High

26


Comparison

Assessment Type Assessment Use Assessment Stakes

Exam Summative Medium, High

Test Diagnostic Low, Medium

Quiz Formative Low

Survey Needs, Reaction, Diagnostic

Low

27


Summary

Kirkpatrick’s Evaluation Model

Types of Assessments

Uses for Assessments

Assessment Stakes

28


Thank You for Your Time!

Contact me at:

[email protected]

Goodbye from your new “virtual” friend, Kat Miller

SNDT Women's University Introduction to Evaluation and Assessments Presented by Kathleen (Kat)...

Documents

Transcript of SNDT Women's University Introduction to Evaluation and Assessments Presented by Kathleen (Kat)...