5 keys to improve roi on e learning

Collection[BESTPRACTICES]

keys to improve the ROI of your distance

learning solutions

5

2 5 keys to improve the ROI of your distance learning solutions

35 keys to improve the ROI of your distance learning solutions

Technology can make professional learning affordable, easy, and convenient. As distance learning programs

grow ever more popular, companies have discovered that technology-driven learning can be a critical tool

to communicate new information and regulations with far-flung workers, and to help businesses that are

experiencing significant growth get employees quickly on the same page.

Many executives assume that e-learning will result in tangible business gains, such as productivity

improvements, more consistent practices, cost reductions, and time savings. The problem is that these benefits

sometimes fail to materialize. To make matters worse, many companies lack a framework to truly measure the

effects of the learning they invest in.

The return on investment for distance learning efforts needn’t be a mystery. Properly measuring return on

investment (ROI) provides an answer to the question: For every dollar invested in distance learning, how many

dollars are returned after the investment is recovered?

The basic calculation of ROI is:

This well-known equation, however, does not capture all the potential benefits and nuances of e-learning

programs. To be meaningful, ROI needs to take into account other performance measures. Jack Phillips,

Chairman of the ROI Institute, has developed a 5-level methodology for measuring ROI.

Foreword

Program Benefits – Program CostsROI (%) = x 100

Program Costs


The Evaluation Framework

• Reaction and Planned Action: Measures participants’ reactions to and satisfaction with the learning program

and participants’ plans for action

• Learning: Measures how much participants have learned, and what skills they have gained

• Application and Implementation: Measures the extent to which participants have applied their new

knowledge and skills on the job

• Business Impact: Measures the extent to which business impact measures have changed since employees

completed learning

• Return on Investment (ROI): Compares the monetary value of the business outcomes with the costs of the

learning program

Each level of evaluation provides important data that can be used on its own to measure the efficacy of distance

learning. But when taken together, a more complete and accurate measurement of ROI emerges. This improved

picture allows executives to justify spending on technology-based learning, improve programs, eliminate

unsuccessful ones, and set smarter company priorities.

This practical guide, written by CrossKnowledge in collaboration with Jack Phillips from the ROI Institute, will

help you to have a more holistic measurement of the effectiveness of your e-learning programs, by adopting a

number of key solutions to improving ROI.

1. Create a learning management system that captures critical data and adapts to your evolving needs

2. Identify your business aims and allow them to drive learning programs

3. Include responsive and dynamic content to hold learners’ attention

4. Get the right balance for qualitative learning

5. Extend your impact with innovative learning efforts


Key N°01

No efficient ROI without a smart LMS

Create a learning management system that captures critical data

and adapts to your evolving needs


01

Adopting an effective Learning Management System (LMS) is essential for getting the

best ROI on distance learning. Without a versatile LMS, companies simply cannot get

the relevant information they need in order to measure the effectiveness of learning

programs.

A good LMS allows companies to track participant completion rates and user

engagement, and to include built-in surveys and feedback mechanisms to continuously

measure the program’s effectiveness.

To optimize your LMS, you should focus on the following:

Concentrate on data

The LMS is where companies collect and analyze the data that will provide them with the

building blocks for ROI. But it can be tough to know up front what data would be useful to

collect. Step back and evaluate what you would like to learn about the distance learning

programs, and use that list of needs to inform the design of your LMS.


A large telecommunication company uses its LMS to capture a variety of data:

• course completion rates

• frequency of log-ins

• most popular courses

• time spent on content

• most popular forms of content

(video, documents, etc)

• use of social elements (message

boards, comment functions)

• reaction summaries

• planned actions

• forecasts of impact

• learning scores

• application follow-ups

• barriers to success

• enablers to success


Aim for a dedicated system

Many companies feel ill-equipped to build an LMS on their own, whether out of concern

about the cost, manpower, or technical know-how. There are plenty of off-the-shelf

learning management systems available to jumpstart e-learning efforts if your company

isn’t interested in starting from scratch. But not every size fits all. When considering

ready-made options, lean toward those that allow you to tailor systems to your specific

corporate and industry needs. Systems should also be capable of adapting over time to

match evolutions in your learning.

Design for evaluation

It’s not enough to simply capture raw data about your employees’ e-learning usage.

It’s also critical to include surveys and assessments that directly measure employees’

progress and opinions about the programs. With those evaluation measurements in

place, companies can continually assess the effectiveness of training programs, identify

weak spots, and scale up strengths. Having smart evaluation tools also allows companies

to perform small-scale tests on new training programs, gauge their usefulness, and then

expand to larger populations within the company.


01


Key N°02

Match your e-learning environment with

your strategic needs

Identify your business aims and allow them to drive learning

programs


To improve ROI, organizations must first define their business objectives. By isolating

the business need, and allowing those needs to drive the design of the learning program,

companies are more likely to achieve a positive ROI.

Clarify the business needs

Ask basic questions about your business aims and the need for the program such as “Is

this program required and why?” The business issue and subsequent need might be: a

decline in sales (which might suggest the need for updated lessons on new products); an

increase in compliance violations (suggesting the need for more consistent training across

the organization); or excessive turnover of talent (suggesting a lack of career development

offerings).

Clarify performance needs

When business needs are defined, performance needs can be identified that would

improve their desired business outcomes. For example, compliance discrepancies (business

need) may be increasing because employees are not following the correct procedure

(performance needs). Everything driving the e-learning environment should attempt to

bridge the gap between those business needs and the performance targets.

Clarify learning needs

After the performance needs are determined, the learning needs are identified. The

learning needs, when met, will close the performance gaps. For example, if employees are

not following a procedure, maybe it is because they do not know how. That is the e-learning

solution.

Determine program objectives

It’s critical to set clear objectives for the program in order to properly judge ROI. Some

objectives will be straightforward and explicit: Employees might need to demonstrate

proficiency with new software at the end of learning, or complete a specific compliance

certification. In those cases, specific mastery goals should be set.

Other objectives may have timeframes beyond the end of the learning program, such

as improvements in sales of new products. And others may be more focused on an

improvement in engagement, productivity, and motivation. Ideally the e-learning program

Match your e-learning environment with your strategic needs

02


02 Match your e-learning environment with your strategic needs

should have reaction objectives, learning objectives, application (performance) objectives,

and (business) impact objectives.

The more specific and targeted the objectives, the easier it will be to measure progress

toward their goals.

As executives attempt to match learning programs with strategic needs, they will

undoubtedly be constrained by costs. There will, after all, be a budget for an e-learning

program, and a desire to obtain the most favorable outcomes for the least investment.

Costs to consider when matching learning programs with business needs:

• Development of solutions: The cost of developing and designing the learning program,

including experts’ costs, supplies, technology, and labor.

• Acquisition of solutions: Licensed materials and technology purchased from other

sources

• Implementation and application: Salaries for coordinators and program owners, as

well as the time spent by participants in learning sessions. Also includes hardware and

software necessary for participants to engage in learning courses

• Maintenance and monitoring: Maintaining and operating learning programs in good

working order

• Evaluation and reporting: Developing an evaluation plan, collecting and analyzing data,

preparing and communicating the results


Match your e-learning environment with your strategic needs

02

A financial services company wants to update its sales training e-program, both to update the key competencies that are being taught to sales associates and to reduce the amount of time workers spend learning.

Key business need: Increase sales with updated learning while reducing time

spent in learning

Specific program objectives: The company would strive to demonstrate

an increase in sales of 15 percent within six months. To achieve this, sales

associates complete 85 percent of action items with customers using the

competencies, and 70 percent mastery in post-learning assessments.

Matching needs with objectives: In order to increase sales and instill new

sales competencies, the program was designed to improve general sales skills,

knowledge of products, customer focus skills and negotiating. The program

was also designed to allow associates with demonstrated competency in a

certain skillset to bypass training in that sector, allowing them to focus on

other areas that need improvement and reduce the overall time dedicated to

e-learning.

Costs assessed: Because sales associates already had company laptops,

technology costs were limited to the development of the software and

associated server costs. Other costs included program development expenses

(developers’ labor and testing); and participants’ time spent in learning

(salaries).

Results considered in ROI: The company was able to increase sales by 20

percent which represents $9 million in sales annually. Also, the company

reduced the number of hours new sales associates spent in training sessions,

which means they are able to start selling products earlier in their tenure.

Sales from existing clients increased as a result of the new competencies. And


02 Match your e-learning environment with your strategic needs

y

as new accounts opened and customers were retained (based in part on the

improved competencies) new business revenues increased. Participants, with

the help of managers, allocated 10 percent of the increase (one half) to the

e-learning program.

This 50 percent represented a total sales of $4,500,000 connected to the

program. When this is converted to profit (the value added) of 20%, the

monetary benefits are $4,500,000 X 20% = $900,000. The total cost of the

program is $550,000.

So for every $1 invested in the e-learning program, there is a return of .64¢

after the costs are recovered.

This example compares the cost of e-learning to the monetary benefits driven

by the e-learning program. Sometimes it is feasible to develop the ROI by

comparing the cost of e-learning to the cost of a facilitator led version. This

comparison is only appropriate when both versions produce the same impact.

$900,000 (benefits) – $550,000 (costs)ROI (%) = x 100 = 64%

$550,000 (costs)


06

Key N°03

Make your content appealing

Include responsive and dynamic content to hold

learners’ attention


03

Big investments in e-learning programs are useless if learners don’t actually use or engage

with the content. Moderator-led, in-person training is expensive, but it is often effective

because expectations and bonds are forged in the classroom. Technology-based learning, by

contrast, may not always be taken seriously, given its convenience, the fact that it is typically

done individually, and is often optional.

To maximize ROI, e-learning programs must be designed with the learner in mind.

Great care must be taken to attract and hold users’ attention. It’s a delicate line to walk:

Designers must not only make big-picture results relevant and apparent to learners

(“Why do I need to take this course?”) but also consider the minutiae that might drive

them away (such as too many clicks and screens to get to critical information and

actions). The move toward increasing gamification should also be considered carefully.

While making learning more game-like can make programs more fun, it can also create a

disconnect between the learning and a user’s job if the relevance of the game-like training

is not made explicitly clear.

In this context, the designer and developer of the e-learning programs are key to

improving ROI. In facilitator-led learning, it’s the facilitator who can make the difference.

But in distance learning, the role of designer and developer are critical if you want your

program to provide a positive pay off.

Design an interface adapted to learners’ needs and expectations

Few people want to stare at a computer screen full of nothing but text for hours at a time.

Your average e-learner is no different. Assume that the majority of your participants will

favor a graphic e-learning environment, and consider carefully the design of your content and

program flow as a result. Reduce text reading to a minimum. Make responsive design -- easy

reading and navigation with a minimum of panning and scrolling -- the first building block of

your program. And consider at the outset whether you will need the program to be available

in multiple formats, including in other languages and on alternative platforms.

Keep it logical

Having content arranged and presented logically has a huge effect on whether

information is retained by learners. If information is presented logically, participants have

a better chance of incorporating it; if it is scattershot or poorly organized, you will lose


y


03

participants’ attention in no time. You might design an introduction, main section, and

evaluation structure, or create a series of abstract information.

Increase visual interest

Humans are visual creatures, and many of us learn concepts best when we visualize them.

In order to increase the chances that content is learned and retained, make sure the

content is appealing to the eye. Incorporate pictures and videos to attract attention and

illustrated ideas. Graphics and animations can also quickly and succinctly convey complex

information in easily digested ways. Of course, don’t include graphic elements simply for

the sake of having them. The most important thing to make sure the visuals support and

enhance the message of the content.

Foster interactions between participants

If possible, create networks, whether on social media or through the LMS, so that participants

can share information, questions, success stories, and concerns. Classroom-based learning

can create tremendous bonding, commitment, and engagement, which is often lost when

learning moves online. Facilitating interactions throughout the learning program can

improve users’ dedication and replace the role of the facilitator, creating a virtuous cycle of

engagement.


An international engineering and construction firm is using and e-learning

program to teach time savings tips and the use of productivity tools. The

impact is actual time savings of the learners, when the time saved is used on

other productive tasks. The program was voluntary and designed to meet

learners’ needs. The program was logical, attractive, and interactive. Learners

could share their success with tips and tools and learn from each other.


Key N°04

Instructional design: stick to learners’ needs and

constraints

Get the right balance for qualitative learning


Improving the ROI for distance learning often means reducing logistic costs, such as those

spent on experts and facilitators. But without those experts in the classroom responding

to participants’ questions and concerns, program designers must anticipate users’ needs

and build responses to them into the learning process. Adult learners want to know why

they need to learn something. They also approach learning as problem-solving, and learn

best when they believe that the topic is of immediate value.

As a result, designers must create a system that guides learners, rationalizes their time,

and uses storytelling to make users truly feel engaged.

Make training and learning relevant to participants’ jobs

The design needs to make it clear from the outset what completely learning programs

means to users and why it is relevant to their specific roles. Make sure it is communicated

to them or built into the system why they are taking programs in the first place, what

the business need is, and how their skills and performance will be enhanced through

completion. Connecting completion of the course to the user’s success is critical for

optimal ROI.

Show that managers care

With all learning programs, no matter the format, the manager role is critical. Managers

often don’t realize they are influential in driving home the importance of learning

programs. But even small gestures and statements from managers demonstrating

support for learning can make a big difference. Similarly, saying nothing at all sends the

message that the learning isn’t of much importance. Showing support from management

can be both formal and informal. Managers might make a point of briefly mentioning

the importance of learning or asking direct reports about their progress on learning

programs.

More formally, designers might build in direct messages and statements of support from

executives and managers into the e-learning programs themselves. For instance, a video

from the CEO at a key juncture of the training pops up and voices support for the partici-

pant’s effort and reiterates how critical the program’s objectives are to the mission of the

organization. This sends a powerful message that the program is important and that the

employee’s participation and effort are valued.

Instructional design: stick to learners’ needs

and constraints

04


04

Become storytellers

Storytelling is critical for information to be conveyed in a compelling and memorable way.

Facts and figures often don’t stick in our minds at all, but stories create “sticky” memories

by attaching emotions to things that happen, which can focus learners’ attention, provoke

curiosity, sustain engagement, and ultimately, change behavior. What does storytelling

look like in a business context, and in the context of e-e-learning? Designers should

endeavor to include case studies, videos, demos, exercises, games and examples. Of

course, technology may be constrained by available budgets, but the basic arc of stories

– question, tension, closure –can be embedded in a variety of ways throughout the

program. Story-led training creates “stickier” learning, which naturally improves ROI.

Instructional design: stick to learners’ needs and constraints

y

In a steel manufacturing company, a message from the CEO introduces an

important e-learning safety program for employees. The executive stresses

the importance of this program, identifies which measures that should

improve, creates expectations, provides support, and offers an open door to

feedback about safety issues.


Credit unions that are experiencing rapid growth and changes in regulations

often have a difficult time keeping employees up to date on critical training. A

cooperative based in Utah, recently launched a Learning Through Storytelling

Program to help employees better understand important issues relevant to

the business.

Members of this company created several story-driven training videos to

help employees grasp new and developing business concepts. A recurring skit

helped trainees understand the difference between wills and trusts. Another

animated story explained the difference between various kinds of financial

partnerships.

The company said the training had a number of important benefits, including

appealing to Generation Y employees. The videos produced by the program

not only helped turn concepts that might be perceived as boring into fun

and interesting ideas, but also helped improve employees’ engagement and

learning retention.

In order to test the effectiveness of the storytelling videos, employees were

tested on concepts before and after watching the videos. The results were

impressive, showing a 266 percent increase in knowledge concerning the

difference between wills and trusts and a 489 percent increase in knowledge

concerning the difference between revocable and irrevocable trusts after

employees watching the videos.

04Instructional design:

stick to learners’ needs and constraints


Key N°05

Explore new opportunities

to maximize ROI

Extend your impact with innovative learning programs


In our rapidly changing business environment, skills must often be continuously updated.

For companies looking for additional learning opportunities, just-in-time training and

adaptive learning are natural complements to more ‘traditional’ e-learning systems. By

building on established learning systems, the additional programs can provide more ‘bang

for the buck’ by responding to individual training needs.

Just-in-time training

Just-in-time training allows workers to access learning when and where they need it.

They can tap self-guided tutorials and other tools at any time to complete tasks and

quickly update their skills. The learning is focused on “nuggets” of information that solve

problems as they arise, rather than on updating or replacing whole suites of training at

once.

The Benefit: The programs allow employees to receive the learning they need when they

need it, rather than sit through learning that might not be relevant or convenient at the

time.

Adaptive learning:

This data-driven, personalized learning adapts the presentation of materials according to

each learner’s needs, based on that individual’s position, existing skills, prior performance,

and goals. The programs are able to tailor to the participant’s specific needs based on

accumulated data, which may include information from past courses and responses to

surveys, and then anticipate what types of content and resources learners need at a

specific juncture in order to make progress.

The Benefit: The programs can adapt automatically to different skill levels, which often

results in faster and more substantive learning as a result of the personalization.

Both tracks can result in more engaged and satisfied employees, boosting companies’

retention capabilities and ROI. Continuing education and professional development have

come to be seen as a key workplace benefit in today’s new economy. By complementing

traditional e-learning options with just-in-time training and adaptive learning programs,

companies create powerful incentives for employees to stay and work for success.

Explore new opportunities to maximize ROI

05


05

Case Study:Measuring ROI in an Upgrade Selling Program: A Mobile Learning Solution.

Transoft Inc.1

Brief Summary

This project involves a mobile learning application for sales associates of a large software

firm specializing in software solutions for the trucking industry. Sales associates were

provided a mobile learning solution for their iPads, designed to describe and sell an

upgrade to its most popular software product, ProfitPro. Although the release occurred

at the same time for all sales associates, not all of them logged onto the learning portal

through their iPads, which allowed an opportunity for a comparison group. This case

study highlights the key issues in calculating the impact and ROI of a mobile learning

solution on business results.

Results

The first 25 people signed up within three days of the program’s announcement. Reaction

data were collected at the end of the fifth module and are presented in Table 1. Reactions

were as expected, with the exception of intent to use, which was very high.

Table 1. Reaction Results

Rate the Following Rating

Relevant to my work 4.3

Important to my success 4.1

Intent to use 4.7

Recommend to others 4.2

Target: 4.0 (Scale: 1 = not at all; 2 = some; 3 = average; 4 = above average; 5 = very much)

Average: 4.3

1. Elkeles, Tamar, Patricia P. Phillips, and Jack J. Phillips. Measuring the Success of Learning Through Technology: A Step-by-Step Guide for Measuring Impact and ROI on E-Learning, Blended Learning, and Mobile Learning. (Alexandra, VA: ASTD Press: 2014) p.177.

Learning measurement seemed appropriate, and quiz scores were above targets, as

shown in Table 2. Scores were slightly lower than desired for the quiz for implementation

and support. This was not a huge concern, as this information is clearly in the documents

and the product brochures.


y


05

Table 2. Learning Results

Module Quiz Categories Avg. Number of Cor-rect Responses

1 Rationale for the upgrade 4.3

2 Key features of the upgrade 4.2

3 How upgrade will increase client profit 4.3

4 Pricing options 4.1

5 Implementation and support 3.9

Possible score: 25 Target: 20

Average: 20.8

Application data seemed to be on track as shown in Table 3. Identifying pricing options

and explaining implementation and support were off a little, but overall the objectives

were met. The time for the first call was below the objective. As expected, there were

some barriers and enablers to success, as shown in Table 4. The barriers were minimal.

However, there was a concern that 9 percent of sales associates were not encouraged

by their managers to use the program. As expected, the number one enabler was

management encouragement.

Table 3. Application Results

Extent of Use in One Month Rating

Make the first call (target: 5 days) 3.5 days

Explaining the rationale for upgrade 4.2

Identifying key features of upgrade 4.3

Describing how the upgrade increases client 4.1

Identifying pricing options 3.9

Explaining implementation and support 3.8

Target: 4.0 (Scale: 1 = not at all; 2 = some use; 3 = moderate amount of use; 4 = significant amount of use; 5 = very significant amount of use)

Average: 4.06



05

Table 4. Barriers and Enablers

Barriers % Reporting

Lack of time to be involved 12%

Lack of management encouragement 8%

Took too long to complete 8%

Technology issues 4%

Other 4%

Explaining implementation and support 3.8%

Enablers % Reporting

Management encouragement 60%

Easy to use 52%

Timely 48%

Convenient 40%

Relevant 36%

Other 16%

Table 5 shows the impact data comparing the experimental group of 25 sales associates with the

control group of 22. As expected, almost every one of the control group (19 of 22) was actually

selling the upgrade though not participating in the program, but the difference of the two groups

was very impressive. The difference for that second month is then annualized, producing an

annual improvement of $1,140,000. The time of the first sale was impressive for the group

involved in the program—11 days compared to 21 days for the control group.

Table 5. Sales Data in Three Months After Launch

Group Avg. Sales (Month 3) Avg. Time to First Sale

Trial (Experimental) Group: 25 sales associates

$7,500 11 days

Comparison (Control) Group: 19 sales associates (out of 22)

$3,700 21 days

Difference = $3,800

Annualized = $3,800 x 12 x 25 = $1,140,00020% profit = $228,000

As shown in Table 6, the fully loaded costs were included to make the ROI calculation

credible.


y


05

Table 6. Costs

Needs assessment (est.) $3,000

Design and development ($56,000/110 x 25) $12,725

Mobile device (prorated) $1,700

Sales associate time $2,524

Administration time (est.) $6,000

Project management (est.) $14,500

Evaluation $15,000

Total $55,449

Table 7 shows the benefit/cost ratio and the ROI calculation. As anticipated, results

exceed the ROI objective.

Table 7. BCR and ROI Calculations

In addition to the tangible sales increase converted to money, several intangibles were

connected to the program, as presented in Table 8.

Table 8. Intangible Benefits

The complete case study is published in the book Measuring the Success of Learning Through

Technology: A Step-by-Step Guide for Measuring Impact and ROI on E-Learning, Blended Learning,

and Mobile Learning.


$228,000BCR = = 4.11

$55,449

$ 228,000 – $55,449ROI (%) = x100 = 311%

$55,449

• Made the first sale in 11 days average

• Customer satisfaction

• Brand awareness for ProfitPro

• Job satisfaction of sales associates

• Stress reduction for sales associates

• Reputation of company


www.crossknowledge.com Th

is d

ocu

men

t is

a p

ub

licat

ion

of t

he

Cro

ssK

no

wle

dge

co

mp

any.

All

bra

nd

an

d lo

gos,

reg

iste

red

or

oth

erw

ise,

are

th

e p

rop

erty

of t

hei

r re

spec

tive

ho

lder

s.

Th

is is

a n

on

-co

ntr

actu

al d

ocu

men

t. D

esig

n: I

nd

exel

- w

ww

.ind

exel

.co

m

Collection[BestPractices]

ways to link distance training

to corporate needs

5Collection[BestPractices]

ready-to-use training

innovations

5


keys for the successful gamification of your

training project

10


ways to make a success of informal

learning

7


keys for boosting learner

engagement

10tips for Rapid

Learning rollout


10


7keys to success

in social learning

ways to introduce e-learning successfully


5


7golden rules for

a successful blended learning program

5 keys to improve roi on e learning

Education

Transcript of 5 keys to improve roi on e learning