Evaluation of Training and Education Activities

2

ObjectivesBy the end of this presentation, participants will be able to

List reasons why evaluation is needed Describe different types of evaluation Discuss how to use different types of

evaluation for training and education activities

Evaluation (1)

As trainers, what do we want to evaluate?

3

Evaluation (2) What have we accomplished?

What was the training? Where? How many people were trained? Cost?

How well have we done? Did the trainings go as planned? Did the participants like the training?

Was the training relevant? What were the strengths and

weaknesses?4

Evaluation (3) How effective is the training?

To what extent did the training achieve the objectives?

Did the participants increase their knowledge and skills?

Did they change their job performance? Did health status improve?

How can we improve on it? Are the training objectives appropriate,

given the available resources?5

Systematic Health Education Process

6

Development

Pilot-testing(Formative Evaluation)

Implementation(Process Evaluation)

Assess Effectiveness(Outcome and

Impact Evaluation) Evaluation

(Formative Research)

NeedsAssessment

7

When to Begin Evaluation?Consider evaluation up front (in planning phase of training project)

Allocate time Allocate resources Evaluation actually occurs at every

stage of development Determine the type of evaluation you

need

8

Levels of Evaluation Formative (before)

Identifies strengths and weaknesses of training/product before implementation

Process (during) Describes the implementation of the

training or distribution/use of the product

Summative/Outcome/Impact (after) Assesses the short-term and long-term

effects of the training/product

Four Types of Evaluation

9

Impact

Outcome

Process

Formative

Programimplementation

ends here.

Programimplementation

occurs here.

SummativeEvaluation

10

Goals and Objectives Goal

Describes the overall mission/purpose of the program Broad general statement that clearly states the

problem that will be addressed

Objectives State the expected result Specific, measurable, attainable, realistic, time-

framed Partial accomplishment of a goal

Activities Specific actions or procedures expected to occur to

achieve objectives

Formative Evaluation

11

Impact

Outcome

Process

Formative

Programimplementation

ends here.

Programimplementation

occurs here.

SummativeEvaluation

12

Formative Evaluation(Pilot Testing)

Conducted with the target audience Answers questions about whether

trainings/products are Understandable, relevant, attractive,

credible, acceptable, effective Identifies strong and weak points

Identifies confusing terms or concepts Allows for necessary revision prior to

implementation Prevents production of ineffective

products

Formative Evaluation Methods Focus groups

With various types of participants (e.g., experienced, non-experienced)

Interviews (key informant/audience) Informal discussion Open-ended questions

Understanding Appeal Utility

Test the curriculum in the “real” environment

Readability testing (literacy and comprehension) 13

14

Formative Evaluation Measures

Ease of use or intuitiveness Readability

Reading level Tone Fonts, colors, spacing

Comprehension Terminology and illustrations Concepts

TB:The Hollywood

Movie

Formative Evaluation Screen tests Movie screenings Focus groups

TB

Process Evaluation

17

Impact

Outcome

Process

Formative

Programimplementation

ends here.

Programimplementation

occurs here.

SummativeEvaluation

18

Process Evaluation(Monitoring of Training/Curriculum)

Examines the procedures and tasks involved in implementing a training/product

Assesses if the training/product is working as planned

Answers questions such as: Are the activities being implemented as

planned? Is the material being used as intended? How many participants attended the program? Were the participants satisfied with the

educational materials?

19

Process Evaluation Measures Product inventory

Hard copies distributed Web pages viewed or downloaded

Attendance Checklist and timetable review Budget, $$$$ spent Audience feedback on quality of

program/material during implementation (participant reactions)

Participant Reactions Do audience/participants think that:

Objectives are relevant to participants' work

Course content is realistic Training methods are suitable Facilitators are effective Training environment (classrooms, meals,

accommodation) contributes to learning

20

Participant Reactions Methods

Written feedback – participants write down their reactions Ensures that all participants provide comments,

not just the outspoken ones Both open- and closed-ended questions are asked

Oral feedback and observations Observations by facilitators Facilitators meet at the end of each day to

discuss their opinions and participant comments

21

Participant Reaction Form

22

Process Evaluation Ticket sales Critical reviews Rotten tomatoes

TB

Outcome Evaluation

24

Impact

Outcome

Process

Formative

Programimplementation

ends here.

Programimplementation

occurs here.

SummativeEvaluation

25

Outcome and Impact Evaluation

Answers the questions… What have you accomplished? How well have you done it? How much have you done? How effective have you been?

26

Outcome Evaluation Based on learning objectives Evaluation of learning & performance of

skills Assesses immediate or short-term effects

End of course or immediately after distribution Helps draw conclusions about effectiveness Answers questions such as:

Were learning objectives achieved? What knowledge and skills were gained as

a result of the training/product?

Outcome EvaluationEffectiveness of Self-Study Modules

on Tuberculosis Knowledge Mean Scores

Pre-test Post-testPrint 74% 91%Satellite 81% 92%Web 76% 92%

(p<.01)

27

Impact Evaluation

28

Impact

Outcome

Process

Formative

Programimplementation

ends here.

Programimplementation

occurs here.

SummativeEvaluation

29

Impact Evaluation Gathers information about the long-range

results of the training/product and improvements in health status 3 months, 6 months, one year following

implementation Answers questions such as:

Did the user’s job performance improve? What are the changes in job practice of user or

participant? What changes were implemented in the program?

Costly, time consuming, but important

30

Should You Measure Impact?

Ask yourself the following questions: Do you have the time and resources? Do you have support from management

and partners? Are you able to reach the audience at this

later date? What will motivate the audience to

participate?

Impact Evaluation Methods

31

Methods Surveys Key-informant interviews Direct observation/performance checklists Document/record reviews/treatment registers Reviewing of program indicators

Timeline Depends on the goal of the training;

however, generally 1 to 6 months after training

Summative Evaluation Domestic and international

box office revenues DVD sales Franchise opportunities

TB

Evaluation Reports

Evaluation Reports Justify program, course, or material to

stakeholders Allow for identification of areas for

improvement Provide evidence for additional needs Share lessons learned

Most Importantly…

Plan for evaluation at the beginning and conduct

throughout development process