Interactive Training Skills: Evaluation Study Executive Summary Presentation

Service Merchandise Company

Date: Wednesday, April 6, 2011

CONFIDENTIAL

Consultants:

Nancy Donovan

David Soltis

Rick Clarke

2CONFIDENTIAL

Meeting Objectives

During this session we hope to achieve the following:

a. Present the background and rationale for the ROI Study of the Customer Interactive Selling Skills Program

b. Identify key issues encountered during the study

c. Explain our ROI Methodology

d. Present the findings on 5 levels of evaluation

e. Present final conclusions and recommendations for the program

f. Question and answer period

3CONFIDENTIAL

Business Problem and Background

Business Problem:

Service Merchandise Incorporated is currently experiencing low sales growth.

Diagnosis:

Based on business analysis the problem can be attributed to a number of key factors including behaviour of sales associates and customer interaction.

Training Solution:

Pilot program developed by third party supplier which focused on enhancing the customer interactive skills of sales associates. The program was implemented across three stores (16 Learners Per Store) and included:

a. Two days of in class training

b. Three weeks of on the job application

c. One Day Discussion Session

4CONFIDENTIAL

Rationale for ROI Analysis

The ROI Study was implemented to address the following issues:

a. Reaction and Learning - Validate the level of reaction and learning that took place during the training

b. Behaviour Change - Determine if the program augmented the behaviour of sales agents (Engage in Up Selling/Positive Interaction)

c. Validate Linkage of Program to KPI - Isolate the impacts of the Program to validate the linkage of the program to specific KPI’s

d. Return on Investment - Determine if the program benefits exceeded the costs and generated a positive ROI

e. Continuous Quality Improvement – Identify if there are changes that can be made to enhance the learning experience and transfer of knowledge on to the job

f. Program Expansion - Determine if the program should be rolled out to all stores

5CONFIDENTIAL

Key Issues in the Original Study

The following is a list of key issues that our team encountered during the evaluation process:

a. Small Sample Size – The sample size consisted only of 3 stores. A stronger size of about 150-200 would have been more statistically significant.

b. Employee Turnover – Within the target learner group there is a high level of turnover.

6CONFIDENTIAL

The Alignment Process

Project

Start Here End Here

Payoff Needs

Business Needs

Performance Needs

Learning Needs

Preference Needs

Initial Analysis

Measurement and EvaluationReaction

Learning

Application

Impact

ROI

V Model

Business Alignment and Forecasting The ROI Process Model

1 1Reaction

Objectives

2 2Learning

Objectives

3Application Objectives

3

4 Impact Objectives 4

5 Impact Objectives 5

80%

5 Simple Skills

50% Use Skills

Increase in Sales

50%

7CONFIDENTIAL

Our Methodology: ROI Process Model

8CONFIDENTIAL

Our Methodology: Isolation of the Effects of the Program

Key Question: When sales data is collected three months after the program, how much of the increase in sales is directly attributed to the program

Method: Control Group Analysis

Process:

a. Group Selection - Three groups selected to be trained, three groups to be compared against (Control Group)

b. Sales Influence – Identified criteria that could influence sales including store size, store location, customer traffic and store performance

9CONFIDENTIAL

Results at Level 1

Actual

Target

Status

84.8% 80%

10CONFIDENTIAL

Results at Level 2

Learning and Confidence:

Observation: Skill Demonstration

Instrument: Questionnaire

Process: The facilitator would observe the learner and indicate if they have successfully demonstrated the skill. This provided evidence that the learner had actually learned the 5 key skills/behaviours.

Actual Target Status

5 Simple Skills Learned (Passed)

5 Simple Skills Learned

11CONFIDENTIAL

Results at Level 3

I utilize the skills taught in the program.

Strongly Agree

Agree Neither Agree Nor Disagree

Disagree Strongly Disagree

78% 22% 0% 0% 0%

Actual

Target

Status

100% 50%

12CONFIDENTIAL

Results at Level 3 (Continued)

Frequency of use of skills

With Each

Customer

Every Third

Customer

Several Times a

Day

At Least Once Daily

At Least Once

Weekly

52% 26% 19% 4% 0%

Actual

Target

Status

52% 50%

13CONFIDENTIAL

Results at Level 4

Trained Group: Avg. Weekly Sales = $12,075

Control Group: Avg. Weekly Sales = $10,449

Actual Target Status

$1,626.00 +15.56% Increase in Sales

14CONFIDENTIAL

Annualization of the Benefit

Output

AVG Weekly Sales per Employee Trained Groups $12,075.00

AVG Weekly Sales per Employee Untrained Groups $10,449.00

Increase $1,626.00

Profit Contribution (2% of store sales) $32.50

Total Weekly Improvement (x 46) $1,495.00

Total Annual Benefits (x 48 Weeks) $71,760.00

15CONFIDENTIAL

Program Costs

Item Cost

Facilitation Fees, Three Courses at $3,750 $11,250.00

Program Materials, 48@$35 per participant $1,680.00

Meals and refreshments, three days @$28 per Participant $4,032.00

Facilities, Nine Days @ $120 $1,080.00

Participants' salaries plus benefits (35% Factor) $12,442.00

Coordination and Evaluation $2,500.00

Total Costs $32,984.00

16CONFIDENTIAL

Results at Level 5

BCR =$71,769

$32,984= 2.18

ROI (%) =$71,769 - $32,984

$32,984

= 118%

X 100

Actual ROI (%)

Target Status

118% 50%

17CONFIDENTIAL

Conclusions and Recommendations

Conclusions:

a. The program had a positive impact on Average Weekly Sales (15.56% Increase between Trained and Non-Trained Groups)

b. For every dollar spent on the training program, there is a a dollar returned and an additional $1.18

c. There was an increase in a number of intangibles including job satisfaction, teamwork, confidence, customer satisfaction, corporate image, etc.

Recommendations:

a. Roll out the program throughout the store chain

b. Collect sales data for the three target stores for the remainder of the year

c. Consider online tools/job aids to support and sustain behaviour

d. Share Results with stores

e. Provide a presentation of the data to training staff, along with a workshop to educate on the ROI Process

18CONFIDENTIAL

Part B: Study Critique

19CONFIDENTIAL

Strengths and Weaknesses

Weakness:

a. Do not gather specific data on intangible impact of program (Only 1 level 4 metric)

b. Level 2 (Learning and Confidence) – Pass or fail utilized, does not indicatehow many pass/fails resulted, were participants given multiple attempts, if so,how many? This data could have been used to modify delivery of future trainingevents if required.

c. Level 3 – Was collected after three weeks after the first two days of trainingThis is a temperature check, but a second follow up would further validate thatthe behaviour has been internalized. The level 3 target has not been met for level 3 (52%). A more credible source of data would have been using observation by aStore Supervisor, or a customer survey. Given the small sample size, this could lend strength to the recommendation to implement.

e. Employee Retention is not addressed – this is not considered as a cost.Weak link to intangibles. This business impact could have been analysed in L4 by measuring % turn over in the control group and trained stores, using businessperformance monitoring 3 months after the program, and throughout the year ofdata collection.