Business Impact

The Long-Term Value of Using Metrics That

Matter®

KnowledgeAdvisors

17th March 2011

Objectives

Apr 10, 2023 © 2009, KnowledgeAdvisors 2

• Discuss the value proposition for measuring learning.

• Describe the long-term value of using Metrics That Matter®

Audience Poll

What is your role within your organization? A) L&D professional B) L&D Leader C) HR professional D) Business leader E) Other

Apr 10, 2023 © 2009, KnowledgeAdvisors 3

Audience Poll

Are you currently a user of Metrics That Matter® A) Yes B) No

Apr 10, 2023 © 2009, KnowledgeAdvisors 4

Agenda

Archetypal L&D Organization Why measure learning effectiveness? How do KnowledgeAdvisors’ clients

measure learning impact? What are the long-term benefits?

Apr 10, 2023 © 2009, KnowledgeAdvisors 5

Archetypal L&D Situation

Apr 10, 2023 © 2009, KnowledgeAdvisors 6

“I can’t improve it if I can’t

measure it.”

“Don’t tell me it will be a cold day in

Hell before we get a measurement

strategy.”

CLO

L&D Director

“I don’t want to leave you out in the cold on this

one, so leverage your whole team.”

L&D Manager

“It’s a very cold winter day outside. Unless we develop a measurement strategy, we’re going to

be out in it looking for new jobs.”

We Know What Leaders Want…

Apr 10, 2023 7

N = 101

© 2010, KnowledgeAdvisors

Is training effective?

Does it lead to business impact?

Why Measure Learning?

L&D is an integral part of the business designed to close gaps and help the workforce achieve business goals.

Apr 10, 2023 © 2009, KnowledgeAdvisors 8

Is L&D achieving business goals?

That’s why we measure.

How Do Clients Measure Learning?

Learning should be measured in a way that aligns with business objectives.

Usually this means demonstrating: Satisfaction with learning Knowledge and skill acquisition Application of learning on the job Demonstration of learning’s impact on the

business (e.g., sales, quality, customer sat.)

Apr 10, 2023 © 2009, KnowledgeAdvisors 9

Most Common Methodologies

Kirkpatrick’s 4 Levels of Evaluation Phillip’s ROI Methodology Bersin’s Learning Impact Analysis Brinkerhoff’s Success Case Method All of these individuals are or have been

on our Board of Directors and have influenced the development of Metrics That Matter®

Apr 10, 2023 © 2009, KnowledgeAdvisors 10

The Goal of Training Evaluation

11April 10, 2023 © 2009, KnowledgeAdvisors

Demonstrate cause and effect

Determine gaps or deficiencies in training that require continuous improvement

Performance TrainingImproved

Performance

KnowledgeAdvisors: What We Do

12

Experts in metrics that

matter KnowledgeAdvisors is a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions.

April 10, 2023 © 2009, KnowledgeAdvisors

Integrate Measurement with Management

13April 10, 2023 © 2009, KnowledgeAdvisors

Showing Business Impact

Smart Sheets

Human Capital Approach

Advanced Business

Results Evaluation

Business

Impact Templates

Actual Results Correlations

Typically employs surveys

Requires actual results data and advanced analysis techniques

Causal Modeling

ComplexSimple

Low Investment

High Investment

Determine the appropriate analysis, then select method

Business Case for L&D Measurement: A Financial Link

Shareholder Returns of companies investing in L&D measurement have 17% higher returns than the S&P 500

15

Source: McBassi Investments, 2009

April 10, 2023 © 2009, KnowledgeAdvisors

The is the cause and effect relationship we hope to demonstrate!

Long-term Use of Metrics That Matter®

Investigated long-term clients that have used MTM consistently for 5+ years

Gathered vast data sets and focused on key metrics: Knowledge and skill acquisition (L2) I will be able to apply training to the job (L3) Training will improve my job performance (L4) Training was a worthwhile investment (L5) An average of all 4 metrics

We expected that scores would rise with time because organizations monitor performance, cull under-

performing courses, and continuously improve their curricula.

Apr 10, 2023 © 2009, KnowledgeAdvisors 16

Initial Sample: Telecom Company

We examined the metrics for one company first to see if our hypothesis seemed tenable. Here are our results.

Initial Sample: Telecom Company

Each year large numbers of evaluations were collected.

Initial Sample: Telecom CompanyWe computed the performance increase as a percentage:

(2010 value – 2006 value) / (2006 value)

Initial Sample: Telecom Company

Some data were not collected every year, but the trend upward was evident even with only three years of data.

Initial Sample: Telecom CompanyThe combination of all metrics provides a good summary measure.

Despite the variability across the four metrics, the average performance still improved 14.9%.

Initial Sample: SummaryWhen all 5 metrics are graphed, it is clear that performance rises over time.

The number of evaluations is also overwhelmingly large lending undeniable reliability to the results. The average increase is also substantial.

Case Study vs. Trend

The Telecom company is only one example. Does the hypothesis hold true for other

companies or is the telecom example an anomaly?

Companies with 5 Years of DataFour organizations contributed 5 years of data:

Two insurance companies, a hospital and a telecom company

Companies with 6 Years of DataSix organizations contributed 6 years of data:

An office solutions company, an Army Base, a government defense agency, a pharmaceutical company, a US-based accounting firm, and a UK-based accounting firm

Other companies?

The analysis will continue soon with other companies: Small and medium businesses Learning Providers

We expect the results will be similar.

Apr 10, 2023 © 2009, KnowledgeAdvisors 26

Conclusion Measurement leads to performance improvement

for L&D departments The size of the improvement ranges from 3 – 5%

on average but can be as large as 14.9% Revisit Lauri Bassi’s work—investments in HC

lead to business improvement—outpacing the S&P 500 by 17%

MTM helps make learning organizations more effective through measurement.