Patti Phillips, Ph.D.patti@roiinstitute.net

Show Me the Money:Moving From Impact to ROI

Objectives

• Isolate the effects of a program• Convert benefits to monetary value• Identify the intangible benefits• Tabulate the fully-loaded costs• Calculate ROI

The ROI calculation is simple.

BCR =

ROI =

Program Benefits

Program Costs

Net Program Benefits

Program CostsX 100

Try it!

$750,000

$425,000BCR =

$750,000 - $425,000

$425,000ROI = X 100

Try it!

$750,000

$425,000BCR =

$750,000 - $425,000

$425,000ROI = X 100

= 1.76:1

= 76%

What Makes a Good ROI?

1. Set the value at the same level as other investments – 15%

2. Set slightly above other investments – 25%

3. Set at break even – 0%

4. Set at client expectations

While we don’t strive for a negative ROI,

negative is not always bad!

Levels of Evaluation

Measurement Focus

0. Inputs and Indicators

The input into the project in terms of scope, volume, efficiencies, costs

Participants, Hours, Costs, Timing

1. Reaction & Perceived Value

Reaction to the project or program, including the perceived value

Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action

2. Learning

Learning to use the content and materials, including the confidence to use what was learned

Skills, Knowledge, Capacity, Competencies, Confidence, Contacts

3. Application & Implementation

Use of content and materials in the work environment, including progress with actual items and implementation

Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use

4. Business Impact

The consequences of the use of the content and materials expressed as business impact measures

Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement

Typical Measures

5. ROIComparison of monetary benefits from program to program costs

Benefit-Cost Ratio (BCR), ROI%, Payback Period

ROI Methodology

Develop EvaluationPlans and

Baseline Data

Develop EvaluationPlans and

Baseline Data

DevelopObjectivesOf Solution

DevelopObjectivesOf Solution

Stage 1Evaluation Planning

Stage 2 Data Collection

Collect DataDuring SolutionImplementation

Collect DataDuring SolutionImplementation

Level 1

Level 2

Collect DataAfter Solution

Implementation

Collect DataAfter Solution

Implementation

Level 3

Level 4

CaptureCosts

Of Solution

CaptureCosts

Of Solution

Isolate theEffects ofSolution

Isolate theEffects ofSolution

Convert Data to Monetary

Value

Convert Data to Monetary

Value

Generate Impact Study

Report

Generate Impact Study

Report

Stage 4 Communicate

Results

Stage 3 Data Analysis

Calculatethe Return On

Investment

Calculatethe Return On

Investment

Level 5

Identify IntangiblesMeasures

Identify IntangiblesMeasures

Intangible Benefits

1. Report the complete story

2. Conserve resources

3. Use the most credible sources

4. Choose the most conservative alternatives

5. Isolate the effects of the program

6. No data no improvement

7. Adjust estimates for error

8. Throw out the extreme and unsupported claims

9. Use first year benefits for short-term programs

10. Include fully-loaded costs

11. Report intangible benefits

12.Communicate results to all stakeholders

Operating Standards

Reaction

Learning

Application

Impact

ROI

Isolate the Effects of the Program

Intangible Benefits

Inputs

• Use of a control group arrangement• Trend line analysis of performance data• Use of forecasting methods of performance data• Participant’s estimate of program impact (percent)• Supervisor’s estimate of program impact (percent)• Manager’s estimate of program impact• Use of expert/previous studies• Calculate/estimate the impact of other factors• Customer input

Methods to Isolate Program Effects

* Survey of Users, N = 235

Control G

roup

Trend Line Analys

is

Forecasti

ng Meth

ods

Particip

ant Esti

mates

Manager E

stimates

Sr. Manager E

stimates

Expert

Input

Customer In

put0%

10%

20%

30%

40%

50%

60%

32%29%

5%

55%

37%

11%13%

11%

Isolating the Effects of the ProgramFrequency of Methods Used

Classic Control Group DesignWhat is the difference in improvement?

Experimental Group

Control Group

Pre-Measure

Pre-Measure Program

Post-Measure

Post-Measure

Post-Program Only DesignWhat is the difference?

Experimental Group

Control Group

Post-Measure

Program Post-Measure

2%

1%

J F M A M J J A S O N D J

ERRORRATE

1.85% Pre Program Average

1.45% Projected Average

MONTHS

Example of Trend Line Analysis

CPI Program Conducted

.7% Post Program Average

Example of Estimation

Monthly increase in credit card accounts: 175 (fact)

Contributing Factors

Consensus Impact (%)

Average Confidence (%)

Sales Training 32% 83%

Incentives 41% 87%

Management Reinforcement

14% 62%

Market Fluctuations 11% 91%

Other _________ 2% 91%

100%

Influence Fact % Contrib. Est. Impact Confidence Adjusted Impact

Sales Training

175 32% 56 83% 46.48

Fact: 175 New Credit Card Accounts

% Contribution: 32%

Est. Impact: 56

Uncertainty 17%

Margin of Error: +/- 9.52

65.52

56

46.48

Example of Estimations

Data are converted by:

• Converting output to contribution – standard value• Converting the cost of quality – standard value• Converting employee’s time – standard value• Using historical costs• Using internal and external experts• Using data from external databases• Linking with other measures• Using participants’ estimates• Using supervisors’ and managers’ estimates• Using staff estimates

Cost of A Sexual Harassment Complaint

35 Complaints35 Complaints

Actual Costs from RecordsActual Costs from Records

Additional Estimated Costs from Staff

Additional Estimated Costs from Staff

Legal Fees, Settlements, Losses, Material, Direct

Expenses

Legal Fees, Settlements, Losses, Material, Direct

Expenses

EEO/AA Staff Time, Management Time

EEO/AA Staff Time, Management Time

$852,000 Annually$852,000 Annually

Cost per complaint = $24,343$852,00035

5 Steps to Convert Impact to Money

Step 1: Focus on a unit of measure

Step 2: Determine the value (V) of each unit

Step 3: Calculate the change in performance (∆P)

Step 4: Determine the annual amount of the change (A∆P)

Step 5: Calculate the total annual value of the improvement (A∆P x V)

Example Using Internal Experts

Step 1: One grievance

Step 2: V = $6,500 (from Director of nursing and HR Experts)

Step 3: ∆P = average of 7 out of 10 grievances prevented per month

Step 4: Annual ∆P =

Step 5: A∆P x V =

Example Using Internal Experts

Step 1: One grievance

Step 2: V = $6,500 (from Director of nursing and HR Experts)

Step 3: ∆P = average of 7 out of 10 grievances prevented per month

Step 4: Annual ∆P = 84

Step 5: A∆P x V = $546,000

Example Using Standard Values

Step 1: One sale

Step 2: V = 30% profit margin

Step 3: ∆P = $20,000 revenue per month

Step 4: Annual ∆P =

Step 5: A∆P x V =

Example Using Standard Values

Step 1: One sale

Step 2: V = 30% profit margin

Step 3: ∆P = $20,000 revenue per month

Step 4: Annual ∆P = $240,000

Step 5: A∆P x V = $240,000 x .30 = $72,000

• Assessment Costs (Prorated)• Development Costs (Prorated)• Program Materials• Instructor/Facilitator Costs• Facilities Costs• Travel/Lodging/Meals• Participant Salaries and Benefits• Administrative/Overhead Costs • Evaluation Costs

Fully-Loaded Cost Profile

Financial

• Program• Skills Based Pay Systems

• Business Impact • Employee Turnover• Staffing Level• Customer and Job Satisfaction• Product Sales• Cross Selling

ROI = 258%

Telecommunications

• Program• All-Inclusive Workforce Program (AIW)

• Business Impact• Attrition Rate • Employee Satisfaction• Communication• Cooperation & Teamwork• Diversity Mix

ROI = 163%

Health Systems

• Program• Sexual Harassment Prevention Workshop

• Business Impact• Turnover Reduction• Complaint Reduction• Job Satisfaction• Absenteeism• Stress Reduction• Recruiting

ROI = 1,051%

Technology

• Program• Customer Service Training

• Business Impact (Target)• Customer Complaint Escalation

ROI = -85%

Why do people like ROI?

The people who like ROI do so because the ROI Methodology:

• Generates a balanced set of measures• Employs a step-by-step process• Bridges evaluation disciplines• Balances research and reality• Offers a flexible evaluation solution• Provides a credible approach to measurement and

evaluation

Next Steps

Assess your readiness for ROI

Identify stakeholders and their data needs

Determine the purpose of your evaluation practice

Identify programs suitable for ROI

Develop capability in the ROI Methodology

Success with measurement begins with your worldview

If you are measuring,

you are estimating.

There are no absolutes.

Sometimes the crowd

knows best.

Be able to explain what you did, how

you did it, and why you did it

that way.

A statistic is an estimate of

what probably is… maybe.

Process without

standards is no process.

The rest is merely a balancing act.

34

Benefits

Costs

35