Evaluating HRD Programs

Chapter 7

Human Resource Development

Purpose of HRD Evaluation Determine if accomplished

objectives Identify strengths and weaknesses Cost-benefit analysis Who should participate and who

benefited most Determine if program appropriate Database for decision making

Potential Questions to Be Addressed in a Process Analysis (During Training)

Was there a match between trainer, training techniques, and

training/learning objectives?

  • Were lecture portions of the training effective? Was involvement encouraged/solicited?

  Were questions used effectively?  • Did the trainer appropriately conduct the various training methodologies

(case study, role play, etc.)?

Were they explained well?       Did the trainer use the allotted time for activities?       

Was enough time allotted?       Did trainees follow instructions

Was there effective debriefing following exercises?

  • Did the trainer follow the training design and lesson plans? Was enough time given for each of the requirements? Was time allowed for questions?

Kirkpatrick’s Levels of Criteria Reaction – did trainees like the

program Learning – demonstration of

learning at end of program Behavior – actual transfer to job Results – impact on bottom line

including efficiency, productivity, cost, etc.

Reaction to Training – Part 1 of 2

Answer the following questions about the training in Active Listening skills using the scale below:1= Strongly disagree 2= Disagree 3= Neither agree nor disagree 4= Agree 5=Strongly agree  1. The training met the stated objectives 1 2 3 4 5 2. The information provided was enough so Iunderstood the concepts being taught 1 2 3 4 5 3. The practice sessions provided were sufficient 1 2 3 4 5 to give me an idea of how to perform the skill 4. The feedback provided was useful in helping 1 2 3 4 5me understand how to improve. 

Reaction to Training – Part 2 of 2

5. The training session kept my interest throughout 1 2 3 4 5

6. The pace of the Active Listening session was 1. Way too fast2.   A bit fast3.   Just right4.   A bit slow5. Way too slow

 7. What did you like best about this part of the training  8. What would you have changed Additional Comments:  Note: A similar scale would be used for each of the other components of training that were taught.

Paper & Pencil Test for Evaluation of Learning

Evaluation of learning There is no specific time limit on this test, but most should be able to finish in

about one hour. Answers to the questions should be written in the booklet provided.

 Please read each question carefully as some of the questions have more

than one part to them.

1. List four types of active listening and provide an example for each.

2. List the steps in the Conflict resolution model. After each step, provide a relevant example of a phrase that would represent that step.

3. Multiple choice or fill in the blank questions  

Potential Questions to Be Addressed in a Process Analysis (During Training)

Was there a match between trainer, training techniques, and

training/learning objectives?

  • Were lecture portions of the training effective? Was involvement encouraged/solicited?

  Were questions used effectively?  • Did the trainer appropriately conduct the various training methodologies

(case study, role play, etc.)?

Were they explained well?       Did the trainer use the allotted time for activities?       

Was enough time allotted?       Did trainees follow instructions

Was there effective debriefing following exercises?

  • Did the trainer follow the training design and lesson plans? Was enough time given for each of the requirements? Was time allowed for questions?

Possible Additions to Kirpatrick’s Model Expanding reaction measures to

include reactions to training methods Splitting reactions to assess

perceptions of enjoyment, usefulness, difficulty

Adding 5th level to include societal contributions

Adding 5th level to include return on investment

Data Collection Methods Interview Questionnaire Direct observation Tests and simulations Archival performance data Types of data – individual, group,

system wide

Research Design Internal Validity

Did a change occur? Was the change a result of the

training?

External Validity Will the change occur in other

situations with different trainees?

Threats to Internal Validity History – events occurring during

training Maturation – natural improvements

with development Testing – effects on pre-tests on

changes Instrumentation – different

measures at different point in time

Threats to Internal Validity (continued) Statistical regression – select

trainees measured at extremes in abilities/KSAs and regress to mean

Reactive effects of research situation – motivation (Hawthorne effect)

Multiple treatment effects – previous training

Threats to External Validity Representativeness of sample and

setting Differential selection – basis for

choosing trainees Experimental mortality – turnover

Experimental Designs Control group – random assignment

to treatment and control groups so trainees have similar characteristics

Two-group posttest only Two-group pretest/posttest Four group design – control for effects

of pretest and prior knowledge

Non-experimental Designs Case study – intensive, descriptive

study with after only measures One group pretest/posttest design

Quasi-experimental Designs Nonequivalent control group

design – analyze for equivalence or use multiple regression to control for demographic factors

Time series – establish base line, then training, then series of measures to determine if change has occurred

Ethical Issues Concerning Evaluation Research Confidentiality Informed consent Withholding training – can provide

later Use of deception Pressure to produce positive

results

Assessing the Impact of HRD Programs Cost-benefit – monetary costs in

relation to nonmonetary benefits Cost-effectiveness – monetary

costs in relation to monetary benefits

Return on investment = results/costs