6 Evaluating HRD3e Chapter09 Slides

7/29/2019 6 Evaluating HRD3e Chapter09 Slides

1/55

3/15/2013 HRD3e Contributed by WellsDoty, Ed.D. Clemson Univ. 1

Evaluating HRD Programs

Chapter 9


2/55

3/15/2013HRD3e Contributed by Wells

Doty, Ed.D. Clemson Univ. 2

Deliver theHRD

Programme

or intervention

Assess Needs

PrioritizeNeeds

Define

Objectives

Develop

Lesson Plans

Develop/acquire materials

select

trainer/leader

Select

Methods & Techniques

Schedule the

programme/intervention

Assessment Design Implementation Evaluation

Select

evaluation Criteria

Determine

evaluation design

Conductevaluation

of programme or

intervention

Interpret results


3/55



Effectiveness The degree to which a training (or

other HRD program) achieves its

intended purpose. Measures are relative to some starting

point.

Measures how well the desired goal isachieved.


4/55



HRD EvaluationTextbook definition:

The systematic collection ofdescriptive and judgmental informationnecessary to make effective trainingdecisions related to the selection,

adoption, value, and modification ofvarious instructional activities.


5/55



In Other WordsAre we training:

the right people

the right stuff

the right way

with the right materials at the right time?


6/55



Evaluation Needs Descriptive and judgmental

information needed.

Objective and subjective data

Information gathered according to a

plan and in a desired format.

Gathered to provide decision making

information.


7/553/15/2013HRD3e Contributed by Wells


Purposes of Evaluation Determine whether the program is

meeting the intended objectives.

Identify strengths and weaknesses. Determine cost-benefit ratio.

Identify who benefited most or least.

Determine future participants. Provide information for improving HRD

programs.


8/553/15/2013

HRD3e Contributed by WellsDoty, Ed.D. Clemson Univ. 8

Purposes of Evaluation-2 Reinforce major points to be made.

Gather marketing information.

Determine if training program isappropriate.

Establish management database.


9/553/15/2013


Evaluation Bottom Line Is HRD a revenue contributor or a

revenue user?

Is HRD credible to line and upper-levelmanagers?

Are benefits of HRD readily evident to

all?


10/553/15/2013


How Often are HRD

Evaluations Conducted? Not often enough!!!

Frequently, only end-of-courseparticipant reactions are collected.

Transfer to the workplace is evaluatedless frequently.


11/553/15/2013


Why HRD Evaluations are

Rare Reluctance to having HRD programs

evaluated.

Evaluation needs expertise and resources. Factors other than HRD cause performance

improvements, e.g.,

Economy Equipment

Policies, etc.


12/553/15/2013


Need for HRD Evaluation Shows the value of HRD.

Provides metrics for HRD efficiency.

Demonstrates value-added approachfor HRD.

Demonstrates accountability for HRDactivities.

Everyone else has it why not HRD?


13/553/15/2013


Make or Buy EvaluationI bought it, therefore it is good.

Since its good, I dont need to post-

test. Who says its:

Appropriate?

Effective? Timely?

Transferable to the workplace?


14/55



Evolution of Evaluation Efforts1. Anecdotal(short interesting story) approach: Talk to

other users.

2. Try before buy: Borrow and usesamples.

3. Analyticalapproach: Match research

data to training needs.4. Holisticapproach: Look at overall HRD

process, as well as individual training.


15/55



Models and Frameworks of

Evaluation Table 7-1 lists nine frameworks for

evaluation.

The most popular is that of D.Kirkpatrick:

Reaction

Learning Job Behavior

Results


16/55



Kirkpatricks Four Levels Reaction

Focus on trainees reactions

Learning Did they learn what they were supposed to?

Job Behavior

Was it used on job?

Results

Did it improve the organizations effectiveness?


17/55



Issues Concerning

Kirkpatricks Framework Most organizations dont evaluate at

all four levels.

Focuses only on post-training.

Doesnt treat inter-stageimprovements.

WHAT ARE YOUR THOUGHTS?


18/55



Other Frameworks/Models 1 CIPP: Context, Input, Process, Product

CIRO: Context, Input, Reaction, Outcome

Brinkerhoff: Goal setting

Program design

Program implementation

Immediate outcomes Usage outcomes

Impacts and worth


19/55



Other Frameworks/Models 2 Kraiger, Ford, & Salas:

Cognitive outcomes

Skill-based outcomes Affective outcomes

Phillips: Reaction

Learning Applied learning on the job

Business results

ROI


20/55



A Suggested Framework 1 Reaction

Did trainees like the training?

Did the training seem useful?

Learning

How much did they learn?

Behavior

What behavior change occurred?


21/55



Suggested Framework 2 Results

What were the tangible outcomes?

What was the return on investment(ROI)?

What was the contribution to the

organization?


22/55



Data Collection for HRD

EvaluationPossible methods:

Interviews

Questionnaires

Direct observation

Written tests Simulation/Performance tests

Archival performance information


23/55



InterviewsAdvantages:

Flexible

Opportunity forclarification

Depth possible

Personal contact

Limitations:

High reactive effects

High cost Face-to-face threat

potential

Labor intensive Trained observers

needed


24/55



QuestionnairesAdvantages:

Low cost to

administer Honesty increased

Anonymity possible

Respondent sets thepace

Variety of options

Limitations:

Possible inaccurate

data Response conditions

not controlled

Respondents set

varying paces Uncontrolled return

rate


25/55



Direct ObservationAdvantages:

Non-threatening

Excellent way tomeasure behaviorchange

Limitations:

Possibly disruptive

Reactive effects arepossible

May be unreliable

Need trainedobservers


26/55



Written TestsAdvantages:

Low purchase cost

Readily scored Quickly processed

Easily administered

Wide samplingpossible

Limitations:

May be threatening

Possibly no relationto job performance

Measures onlycognitive learning

Relies on norms Concern for racial/

ethnic bias


27/55



Simulation/Performance TestsAdvantages:

Reliable

Objective Close relation to job

performance

Includes cognitive,psychomotor andaffective domains

Limitations:

Time consuming

Simulations oftendifficult to create

High costs to

development anduse


28/55



Archival Performance DataAdvantages:

Reliable

Objective Job-based

Easy to review

Minimal reactiveeffects

Limitations:

Criteria for keeping/

discarding records Information system

discrepancies

Indirect

Not always usable Records prepared

for other purposes


29/55



Choosing Data Collection Methods Reliability

Consistency of results, and freedom from

collection method bias and error. Validity

Does the device measure what we want tomeasure?

Practicality

Does it make sense in terms of the resourcesused to get the data?


30/55


Type of Data Used/Needed Individual performance

System-wide performance

Economic


31/55


Individual Performance Data Individual knowledge

Individual behaviors

Examples: Test scores

Performance quantity, quality, and

timelinessAttendance records

Attitudes


32/55


System-Wide Performance Data Productivity

Scrap/rework rates

Customer satisfaction levels

On-time performance levels

Quality rates and improvement rates


33/55


Economic Data Profits

Product liability claims

Avoidance of penalties Market share

Competitive position

Return on Investment (ROI) Financial utility calculations


34/55


Use of Self-Report Data Most common method

Pre-training and post-training data

Problems: Mono-method bias

Desire to be consistent between tests

Socially desirable responses

Response Shift Bias:

Trainees adjust expectations to training


35/55


Research DesignSpecifies in advance:

the expected results of the study.

the methods of data collection to be

used.

how the data will be analyzed.


36/55


Research Design Issues Pretest and Posttest

Shows trainee what training has

accomplished. Helps eliminate pretest knowledge bias.

Control Group

Compares performance of group withtraining against the performance of asimilar group without training.


37/55


Recommended Research

Design Pretest and posttest with control group.

Whenever possible:

randomly assign individuals to the testgroup and the control group to minimizebias.

Use time-series approach to datacollection to verify performanceimprovement is due to training.


38/55


Ethical Issues Concerning

Evaluation Research Confidentiality

Informed consent

Withholding training from controlgroups

Use of deception

Pressure to produce positive results


39/55


Assessing the Impact of HRD Money is the language of business.

You MUST talk dollars, not HRD

jargon. No one (except maybe you) cares

about the effectiveness of training

interventions as measured by andanalysis of formal pretest, posttestcontrol group data.


40/55


HRD Program Assessment HRD programs and training are

investments.

Line manager often see HR and HRD ascosts, i.e., revenue users, not revenueproducers.

You must prove your worth to the

organization Or youll have to find another

organization.


41/55


Two Basic Methods for

Assessing Financial Impact Evaluation of training costs

Utility analysis


42/55


Evaluation of Training Costs Cost-benefit analysis

Compares cost of training to benefits

gained such as attitudes, reduction inaccidents, reduction in employee sick-days, etc.

Cost-effectiveness analysis Focuses on increases in quality, reduction

in scrap/rework, productivity, etc.


43/55


Return on Investment Return on investment = Results/Costs


44/55


Types of Training Costs Direct costs

Indirect costs

Development costs

Overhead costs

Compensation for participants


45/55


Direct Costs Instructor

Base pay

Fringe benefits Travel and per diem

Materials

Classroom and audiovisual equipment Travel

Food and refreshments


46/55


Indirect Costs Training management

Clerical/Administrative

Postal/shipping, telephone,computers, etc.

Pre- and post-learning materials

Other overhead costs


47/55


Development Costs Fee to purchase program

Costs to tailor program to organization

Instructor training costs


48/55


Overhead Costs General organization support

Top management participation

Utilities, facilities

General and administrative costs,such as HRM


49/55


Compensation for Participants Participants salary and benefits for

time away from job

Travel, lodging and per-diem costs


50/55


Measuring Benefits Change in quality per unit measured in

dollars

Reduction in scrap/rework measured indollar cost of labor and materials

Reduction in preventable accidentsmeasured in dollars

ROI = Benefits/Training costs


51/55


Utility Analysis Uses a statistical approach to support

claims of training effectiveness: N = Number of trainees

T = Length of time benefits are expected to last

dt = True performance difference resulting fromtraining

SDy = Dollar value of untrained job performance (instandard deviation units)

C = Cost of training

U = (N)(T)(dt)(Sdy) C


52/55


Critical Information for Utility

Analysis dt = difference in units between

trained/untrained, divided by

standard deviation in units producedby trained.

SDy = Standard deviation in dollars,

or overall productivity oforganization.


53/55



Ways to Improve HRD

Assessment Walk the walk, talk the talk: MONEY.

Involve HRD in strategic planning.

Involve management in HRD planning andestimation efforts.

Gain mutual ownership

Use credible and conservative estimates. Share credit for successes and blame for

failures.


54/55



HRD Evaluation StepsAnalyze needs.

Determine explicit evaluation strategy.

Insist on specific and measurabletraining objectives.

Obtain participant reactions.

Develop criterion measures/instrumentsto measure results.

Plan and execute evaluation strategy.


55/55

HRD3 C t ib t d b W ll

Summary Training results must be measured

against costs.

Training must contribute to thebottom line.

HRD must justify itself repeatedly as

a revenue enhancer, not a revenuewaster.

6 Evaluating HRD3e Chapter09 Slides

Documents

Transcript of 6 Evaluating HRD3e Chapter09 Slides