1

DISPELLING MISPERCEPTIONS ABOUT YOUR PROGRAM’S ROI

DON’T LEAVE IT TO THE PERCEIVER

Steve Flaming, VP Analytics & ReportingHugo St. John III, Manager, Financial Analysis

YOUR ROI STORY SHOULD NOT BE AN AFTERTHOUGHT.

3

How many of you

thought about

measurement when

you implemented

your program or did

a redesign?

80%

50% 50%

30%

THE GOLD STANDARD.

Primary Parts:

1. Pre-period of time prior to launch

2. Post or measurement period

3. Representative test group

4. Matched control group

5

UP TO SNOW GOOD REWARDS – ROLLING OUT A PROGRAM.

OTHER GOOD MARKETSPortland vs SeattleCinci vs ColumbusCleveland vs DetroitNashville vs Louisville

AREAS TO AVOIDHighly volatile economies & weather patterns (e.g., Las Vegas, SF/LA, most of Florida, Chicago after October, etc.)

TEST (PILOT) MARKET – Houston

CONTROL MARKET – Dallas

FULL ADVERTISED ROLLOT FOR 12 MONTHS (MIN. OF 6)

ANALYTICS TO CONTROL FOR ANY REMAINING BIASTotal populations: 2.27M vs 1.30MPercent of families with children: 53.9% vs 55.2%Median household income: $49,399 vs $47,285Percent with a bachelor’s degree: 31.7% vs. 31.6%Unemployment rate: 6.6% vs 5.9%

Aggregate the data over the next 12 months and calculate program uplift feeling confident that we have a true test / control experiment here

CALCULATE THE IMPACT OF UP TO SNOW GOOD REWARDS.

7

EVALUATING LOYALTY PROGRAMS.

THE DIRTY UNDERBELLY OF MEASUREMENT.

Self-selection bias…and why it is all your fault!

Auto-enrollment tracks dirt in.

Waiting to measure…a dangerous game to play.

SELF-SELECTION BIAS IS NOT THE END OF THE WORLD.

“Self-selection bias points to errors due to systematic differences in relevant characteristics between those who are included in study and those who are not; specifically the personal choices of individuals.”

MACHINE LEARNING HELPS TO CONTROL FOR IT.

Non Member Member Non Member Member

Spen

d ($)

POST-MODEL COMPARISONPRE-MODEL COMPARISON

Tier 1 Tier 2 Tier 3 Tier 4

Non Member Member

Tier 1 Tier 2 Tier 3 Tier 4

Non Member Member

House

holds

(%)

Non Member Member Non Member Member

Tenu

re (Y

rs)

11

THE DIRTY SECRET ABOUT AUTO-ENROLLING MEMBERS.

Introduces bias over time via the channel of purchase:

1. Reduces size and similarity of your non-member or control group

2. Pollutes the test group

12

MEASURE EARLY; MEASURE OFTEN.

Time is not on your side when it comes to measuring the effect of your loyalty program.

The longer you wait, the harder it gets to measure loyalty—it’s not impossible, it’s simply case-dependent.

So, be sure to plan for the future!

LET’S CONSIDER OTHER ALTERNATIVES.

RETROFITTING THE GOLD STANDARD.MAKING IT RAIN

$50

$60

$70

$80

$90

$100

$110

$120

-5 -4 -3 -2 -1 0 1 2 3 4 5

Non Rewards Rewards

Requirements• Member data

• Start date/1st transaction/Conversion• Non Member data

• Start date/1st transaction• Pre data

• What did everyone do before the program?

• Post data• What did everyone do after the

program?

Pre-period? We don’t need no stinking pre-period.

Requirement and Considerations:

Key: Take advantage of the buildup to build a pseudo pre-period

Data available• Member data (post program only)

• Start date/1st transaction/Conversion• Non-member data (post program only)

• Start date/1st transaction

$50

$60

$70

$80

$90

$100

$110

$120

0 1 2 3 4 5

Non Rewards Rewards

OPTION 1 –NO DATA PRIOR TO PROGRAM START.

OPTION 2 – NOT TRACKING NON-LOYALTY DATA.Ugh… I hate time series models. They are the worst.

Requirements and considerations.

Key: Utilizing a widely-accepted approach called time series analysis which projects out an expected pre-period.

Data available• Only need total sales

$50

$60

$70

$80

$90

$100

$110

$120

-5 -4 -3 -2 -1 0 1 2 3 4 5

Option 2 Pre/Post Only

Rewards

Uplift

Expected sales if no loyalty program

OPTION 3 -POST-PERIOD DATA ONLY FOR LOYALTY MEMBERS.Hold on, I have a Ouija Board in my closet somewhere.

Requirements and considerations:

Key: Very little you can do to analytically assign a value to incremental sales at this point.

Alternate KPIs become the measure of success:• NPS• OSAT• Emotional Loyalty• Active Customer• Churn Reduction• Lifetime Value

50

60

70

80

90

100

110

120

-5 -4 -3 -2 -1 0 1 2 3 4 5

Rewards

UNIVERSAL CONTROL

GROUPS

YOU CAN’T MEASURE YOUR PROGRAM… SO, NOW WHAT?

CAMPAIGN CONTROL

GROUPS

NPS/OSAT

EMOTIONAL LOYALTY

SCORING

GETTING FROM GOOD TO GREAT.

PROGRAM REDESIGN.

During program redesign, teams should use what they already know to forecast the better future they want to step into.

WHAT YOU NEED BEFORE YOU START.

Complete data from the current program, which should include:- Unique member transaction history for earn and burn- Campaign-specific data as they relate to changes in member behavior- All costs incurred that would not exist, save for having the program (e.g.,

rewards costs, vendor management fees, etc.) in order to focus on profitability

These data and results plug directly into the financial loyalty model!

INCORPORATE UNCERTAINTY.

When you redesign your program, you need to model and simulate:- Changes to expected enrollment- Shifts in tier migration- Increases in transaction behavior- New redemption patterns (e.g., due to higher earning potential

across members, new redemption options, etc.)

WHAT THIS MIGHT LOOK LIKE.

INCREMENTAL SALES

PROGRAM YEAR

$7.2 million increase in cumulative incremental sales by year 3 of program

$5.7 million decrease in cumulative incremental sales by year 3 of program

COST OF BENEFITS

PROGRAM YEAR

$6.0 million increase in total cost of benefits by year 3 of program

$5.2 million decrease in total cost of benefits by year 3 of program

PROGRAM YEAR

THE OFT NEGLECTED PART OF ROI.

We often focus on the numerator of ROI – incremental sales.

The denominator is important and can be used as evidence of program value.

Cost savings can include things like:- Increased app usage- Reduction in call center hours- Paperless communications- Auto-billing

These all represent changes to the program value proposition that will drive higher ROI—you should model and simulate these changes, too!

AIM FOR AN ELEGANT SOLUTION.

MEASUREMENTREVISITED.

Any time a loyalty program’s value proposition changes significantly, teams have an opportunity to use the Gold Standard for measuring the new program’s performance compared to the old program.

Yes, you can go home again.

27

CLOSING THOUGHTS.

28

PROGRAM MEASUREMENT CHECKLIST.

• Can you build a pre-period?• Representative non-member group?• Machine learning to lessen self-selection bias?• Long-term UCG?• Campaign-level control groups?• Non-market valuation (e.g., OSAT/NPS/ELS)?

29

PROACTIVELY USE YOUR DATA.

• Meticulously “test and learn” potential program changes • Focus on how key member segments react • Remember program changes include cost-savings• Model out how a full program redesign, based on these learnings, affects the

program’s ROI• Design the new rollout with the Gold Standard in mind

30