Training on Trial

Mid New Jersey Chapter, ASTD

May 21, 2009

Presented by:

Jim Kirkpatrick , PhD

The future of training

“Training directors might be well advised to take the initiative and evaluate their programs before the day of reckoning arrives”

- From Techniques for Evaluating Training Programs, by Don Kirkpatrick - ASTD Journal, November, - 1959

What are the charges?

“You, learning professionals, are charged with incurring training and development costs that exceed the value you bring to the business”

Who exactly is the “we” who are being charged?

The training industry

Organizational Development

Learning functions

HRD

Training departments

Us

Who is charging us?

Internal business partners

External clients

The business world

How do we know we are being charged?

“We would like you to show us your ROI”

“We know you run a lot of programs and seem quite busy. Why then are so many good people leaving us?”

“We are thinking we need to put our efforts in revenue-producing activities.”

“We are going to put that on hold for awhile.”

“We have decided to make some cuts.”

What evidence is being used brought against us?

Brinkerhoff Study

- Josh Bersin and Associates, 2008

Training Application

Did not try new skills: 15%

Tried new skills and failed: 70%

Achieved sustained new behaviors: 15%

Percentage of learning transfer

1975: Percentage of formal learning that is actually applied to the job:

15%

2005: Percentage of formal learning that is actually applied to the job

15%

From Dana Robinson, ASTD ICE, 2008

Causes of “training failure”

Preparation and Readiness: 20%Learning Intervention: 10%

Application Environment: 70%

2006 ASTD Study

Typical Learning Investment

Dr. Brent Peterson, Columbia University, 2004

P re-Work 10%

L earning E vent 85%

F ollow-Up 5%

Are we guilty?

What do we have to do to be found not guilty?

We need to extend our role beyond the traditional definitions of training and learning

The increase in “informal learning”

Where Learning Takes Place

Prior to being "trained": 20%During "training": 10%On the job: 70%

- Josh Bersin and Associates, 2008

Lets get very practical here

We need to redefine and expand our role through the entire “learning and performance” management spectrum:– Our expertise– Our involvement– Our influence– Our impact– Our value

How does this relate to you

so far?

Who will take our case?

Measurement and Evaluation

Gathering data and information to answer specific questions regarding the value of learning and performance solutions; focusing on the impact of individual programs and creating overall measures of system effectiveness, leveraging findings to provide recommendations for change and to increase organizational effectiveness.

- taken from Mapping the Future, ASTD, 2004

Why Evaluate?

1. Improve the program (or know when to retire it)

2. Maximize training effectiveness by reinforcing knowledge, skills, and attitudes

3. Demonstrate the value of training

p. 3

Level 1: REACTION To what degree participants react

favorably to the training

Kirkpatrick Four Levels

p. 4

Level 2: LEARNINGTo what degree participants acquire

intended knowledge, skills, and attitudes based on participation in

learning event

Kirkpatrick Four Levels

p. 4

Level 3: BEHAVIOR To what degree participants apply what

they learned during training on the job

Kirkpatrick Four Levels

p. 4

Level 4: RESULTS To what degree targeted outcomes occur,

as a result of the learning event(s) and subsequent reinforcement

Kirkpatrick Four Levels

p. 4

Four Level Correlations

Enjoyment (L1) and learning (L2): no correlationsRelevance (L1) and immediate learning (L2): r = .26Relevance (L1) and transfer to job (L3): r = .18

Immediate knowledge (L2) and retention (L2): r = .34Immediate knowledge (L2) and skill demo (L2): r = .18Immediate knowledge (L2) and transfer to job (L3): r = .11Retained knowledge (L2) and transfer to job (L3): r = .08Skill demo (L2) and transfer to job (L3): r = .18

Kirkpatrick Four Level Evaluation major principles

• Smile sheets, pre and post-tests, and hope for the best

• Strong connection between Levels 1 and 2

• Strong connection between Levels 3 and 4

• “The Great Divide”

• “The Missing Link”

Let’s take some initial depositions . . .

“What is your job here at the hotel?”

“I am a window washer.”

“What is your job here at the resort?”

“I am part of a team that creates great experiences for our guests!”

Group Activity

What is significant about these two window washers in relation to:

Group 1: individual employees?

Group 2: training and development professionals?

Group 3: the training and development industry?

Crossing over is not easy

What does the word “charged” mean?

• Accused

• Challenged

Common Myth

“Evaluation” only occurs at the end of the instruction design process, or at the end

of a program.

Reality

An effective evaluation plan is considered at every step in the program and training development process, not JUST at the

end.

Kirkpatrick Four Level Evaluation major principles

• The end is the beginning

• ROE is the ultimate indicator of value

• Business partnership is key

• Value must be created first

• Demonstrate value through a compelling chain of evidence

Kirkpatrick Four Level Evaluation Model

Reaction

Behavior

Results

Learning

ReactionLearningBehaviorResults

ROE

Business need

identified

Measure L1 ReactionMeasure L2 Learning

Measure L3 Behavior

Present L1 Reaction findings

Present L2 Learning findings

Present L3 Behavior findings

Present L4 Results findings

Gather final four level data / information, prepare for presentation

The Kirkpatrick Model

Identify critical behaviors and key

organizational drivers

Determine required KSAs,

Learning Objectives

Negotiate stakeholder success indicators,Business outcomes

Ana

lyze

find

ings

, adj

ust,

repe

at s

teps

as

nece

ssar

yConsider necessary learning environment

and conditions

Design and build learning program and evaluation tools

Initiate ongoing reinforcement and

monitoring

Measure L4 Results

Deliver learning program

p. 22

DOT Case Example

Challenge – to increase highway work zone efficiency, effectiveness, and safety.

The Kirkpatrick Business Partnership Model

1. Identify who is on your corporate jury2. Negotiate jury (stakeholder) expectations3. Convert expectations to Level 4 outcomes and

determine metrics4a. Identify the key new participant behaviors that will be

required to bring about the desired outcomes. If L3 behaviors are not relevant, choose L2 ongoing readiness factors

4b. Identify the few key organizational drivers that will be necessary to bring about the changes in behavior

5. (Determine and implement prerequisites)

DOT Example

1. Jury – safety managers, district engineers, traffic control, risk managers, maintenance supervisors.

2. Expectations – smooth traffic flow; increased safety for workers and motorists in highway work zones; better planning.

3. Success Outcomes – less than 15 minute delays; reduction in injuries and fatalities; increased compliment to complaint ratio.

4a. Key New Behaviors – written work zone plans; plan communicated to public; proper work zone set-up; flaggers following procedures.

4b. Success Drivers – observation, feedback and coaching by supervisors; ongoing compliance tracking; ongoing execution of formal and informal recognition programs.

Airborne Warning and Control System

Activity #21. Break up into small groups as instructed.

2. Discuss and select one high impact training program from your group members to work with.

3. Develop your answers to steps 1-4. Use flip chart pages.

4. Be prepared to share your results.

Demonstrated SKA

Training-related Activities

Business Results

Did the critical behaviors contribute to a positive impact on business results?

Did the activities build sufficient capabilities to drive the critical behaviors?

Did the target audience react favorably to the activity?

Critical Behaviors

Did the target audience exhibit a sustained behavior change in the work environment?

Level 1 Level 2 Level 3 Level 4

Measurement Strategy

ActivitiesBusiness Results

CriticalBehaviors

Business Processes

Strategy

Culture Structure

Talent

HR Processes

Business Partnership

Preparing for and presenting our case

Developing a Plan to Evaluate: Critical Criteria

Evaluation methodsEvaluation Levels

Methods

1 Reaction

2 Learning

3 Behavior

4Results

Survey ● ● ● ●

Questionnaire/Interview ● ● ● ●

Focus Group ● ● ● ●

Knowledge Test/Check ●

Work Review ● ●

Skills Observation ● ●

Presentations / Teach Bk ● ●

Action Planning ● ● ●

Action Learning ●

Key Business HR Metrics ●

1

Post Session L1

Level 1

Level 2

Level 3

Level 4

Was session delivered effectively?

Post Session Reinfcmt, Coaching & L3

Expected results will occur.Build your chain of evidence & showcase

Post Session L2

Did targeted learning occur?

Was there successful transfer of learning to behavior?

Improve Reinforcement

and/or Coaching

NO

YES

NO

NONOYES

YES

YES

Improve Session 2

3

4

5

6

7

8

9

10

11

Delivery of Learning EventL1 & L2 Formative Evaluation

Was failure due to session?

Kirkpatrick Four Level EvaluationFeedback Loop to Eliminate ‘Snags’

Example: Valdez University strategic goal scorecard – goal #1

Up from last month

Same as last month

Down from last month

Help to move our

organization from

transaction-oriented to

‘trusted advisor’

Metric Actual Target Status

1. Level 1 satisfaction scores - aggregate 93% 90%

2. % courses learning objectives matched to new directive

85% 70%

3. Level 2 skills scores - aggregate 92% 90%

4. % leaders certified as trusted advisor coaches 66% 65%

5. Level 3 scores - aggregate 48% 75%

6. Gallup scores - aggregate 77% 90%

Preparing for our closing arguments

Chain of Evidence

Gather data at all four levels and show that your training delivers true value to your organization.

Level 1Reaction

Level 2Level 2LearningLearning

Level 4Results

Level 3Behavior

Data is a critical part of our Chain of Evidence

But do not underestimate the power of testimonials

What will you do as a result of what you have learned today . . .

that will lead you to the following tomorrow?

that will hopefully lead to . . .

References• Kirkpatrick, D.L. and J.D. Evaluating Training

Programs, 3rd Ed., Berrett-Koehler Publ., Inc. San Francisco, CA, 2006

• Kirkpatrick, D.L. and J.D. Transferring Learning to Behavior: Berrett-Koehler Publ., San Francisco, CA, 2005

• Kirkpatrick, D.L. and J.D. Implementing the Four Levels, Berrett-Koehler Publ., San Francisco, CA, 2007

• Kirkpatrick, J.K., and W.K. Training on Trial, AMACOM, New York, 2009

• Contact Jim at jim.kirkpatrick@smr-usa.com

• www.smr-usa.com