Holiday catalog testing with Christopher Werler

Holiday Catalog Testing

Catalog University Pub7/20/2016

• Why testing this year matters next year

• Best practices for test set up and results

assessment that empower you with dependable

data for predicting future performance.

• Test variables that provide actionable learnings

most likely to most significantly improve

performance

$1.0M $1.2M$1.5M

$1.8M $1.8M 2.0M 2.0M

2.5M3.0M 3.0M

$6.0M $6.3M

$7.5M $7.2M

Circulation

Revenue

Year 1 Actual Year 2 Actual Year 3 Actual Year 4 Fcst Year 4 Actual

Test Winner More Costly Device Offer Lists Lists

Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ

Spend $1M $1.2M $1.5M $1.8M $1.8M

CPP $0.50 $0.60 $0.60 $0.60 $0.60

Circ 2M 2M 2.5M 3M 3M

Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40

Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M

Incr ROAS 4.0 5.0 4.2 4.2 4.0

$ Chg YOY 50% 4% 20% 15%

20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY

Assumes a margin and ROI target that is acceptable for this fictitious business.

Conservative long-term vision…

what any manager would like to see

$1.0M $1.2M$1.5M

$1.8M $1.8M 2.0M 2.0M

2.5M3.0M 3.0M

$6.0M $6.3M

$7.5M $7.2M

Circulation

Revenue

Spend $1M $1.2M $1.5M $1.8M $1.8M

CPP $0.50 $0.60 $0.60 $0.60 $0.60

Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40

Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M

Incr ROAS 4.0 5.0 4.2 4.2 4.0

$ Chg YOY 50% 4% 20% 15%

20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY

$1.0M $1.2M$1.5M

$1.8M $1.8M 2.0M 2.0M

2.5M3.0M 3.0M

$6.0M $6.3M

$7.5M $7.2M

Circulation

Revenue

Spend $1M $1.2M $1.5M $1.8M $1.8M

CPP $0.50 $0.60 $0.60 $0.60 $0.60

Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40

Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M

Incr ROAS 4.0 5.0 4.2 4.2 4.0

$ Chg YOY 50% 4% 20% 15%

20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY

$1.0M $1.2M$1.5M

$1.8M $1.8M 2.0M 2.0M

2.5M3.0M 3.0M

$6.0M $6.3M

$7.5M $7.2M

Circulation

Revenue

Test Winner More Costly Device Offer Lists Lists Lists

Winner Rollout Status Quo More Costly Device Better Offer Expanded Circ Expanded Circ

Spend $1M $1.2M $1.5M $1.8M $1.8M

CPP $0.50 $0.60 $0.60 $0.60 $0.60

Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40

Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M

Incr ROAS 4.0 5.0 4.2 4.2 4.0

$ Chg YOY 50% 4% 20% 15%

20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY

$1.0M $1.2M$1.5M

$1.8M $1.8M 2.0M 2.0M

2.5M3.0M 3.0M

$6.0M $6.3M

$7.5M $7.2M

Circulation

Revenue

Spend $1M $1.2M $1.5M $1.8M $1.8M

CPP $0.50 $0.60 $0.60 $0.60 $0.60

Incr $/Circ $2.00 $3.00 $2.50 $2.50 $2.40

Incr $ $4.0M $6.0M $6.3M $7.5M $7.2M

Incr ROAS 4.0 5.0 4.2 4.2 4.0

$ Chg YOY 50% 4% 20% 15%

20% 0% 50% 25% 25% 4% 20% 20% 20% 20% 20% 15%Change YOY

Execute roll-outs and tests each year with a

view across at least 3 holiday seasons…

• THIS year’s ROLL-OUTS are informed by LAST year’s test WINNERS.

– That’s just hard work – execution. Has to be done perfectly.

• THIS year’s TESTS are selected informed by LASTyear’s test results (unless you’re testing for the first time) – plus totally new ideas – all directed towards maximizing performance NEXT year.

– Testing is the vital – and fun – part if we are to achieve long-term growth, and it can easily be overlooked since it distracts from the execution and performance of THIS year’s campaign.

– Don’t let testing get lost! Make testing a priority!

– Plus – it’s FUN! It’s THE fun!

This Year Next YearLast Year

Simple Campaign Structure

Mailed Circulation Audience Device Offer IHDExpiration

Mail Control 1 25,000 Catalog Save $10 14-Nov 31-Dec

Test A - Costlier Device 25,000 Catalog + Bells/Whistles Save $10 14-Nov 31-Dec

Test B – Better Offer 25,000 Catalog Save $30 14-Nov 31-Dec

Test C – Later IHD 25,000 Catalog Save $10 1-Dec 31-Dec

Test D - Cheaper Device 25,000 Plastic Postcard Save $10 14-Nov 31-Dec

Test E – Earlier Exp. Date 25,000 Catalog Save $10 14-Nov 15-Dec

Rollout Control 1,825,000 Catalog Save $10 14-Nov 31-Dec

Total Circulation Mailed 2,000,000 - - - - - - - - - - - -

No Mail Control Audience - - - - - - Start End

No Mail Control 1 25,000 - - - - - - 14-Nov 31-Dec

Total No Mail Control 50,000 - - - - - - - - - - - -

Audience Audience - - - - - - - - - - - -

Total Required 2,050,000 - - - - - - - - - - - -

Building Testpools

• For our example:– Take the 2.05M Households

– Split into 10 cohorts, n-thing off for equal balance, so that if all cohorts were treated equally in the mail they should all behave the same

• We need 9 cohorts of 25k and 1 rollout cohort of 1.825M

– For house cohorts, check that they’re balanced by checking each cohort’s transaction history – they should each have the same 0-6 month revenue per household

– For prospect cohorts, test that they’re balanced by checking if zips are equally represented

Building Testpools

Example of n-thing

We have an audience of 100k

We want 5 testpools of 10k each• No Mail Control• Mail Control• Test 1 of 3• Test 2 of 3• Test 3 of 3

We want the remainder, 50k, for the rollout

50% 10.0% 10.0% 10.0% 10.0% 10.0%

HouseholdRollout

NoMail

Control

Etc….

Test Pool Size: What’s Optimal?

• Best Practice– Work with an analyst who is fully informed on your campaign

structure and your testing objectives to determine optimal test pool size

• Second-Best Practice– 100 orders minimum “rule of thumb”

• Have your test pool size large enough that you are confident that you will definitely get 100 orders or more

• Assume prospect response will be just 20-30% of house response

• If possible, shoot for test pools sizes that will deliver much more than 100 orders – particularly if the results of your tests will inform decisions on a grand scale

– Example: we don’t want make a change to a 20M rollout campaign that would significantly impact costs or revenue based on a test conducted with a testpool size of just 5,000

No Mail Control vs. Mail Control Results

Audience

FullyLoaded

Matchback

FullyLoaded

Matchback/Household

Incremental

$/Household

Incremental

Revenue

NoMailControl1 25,000 298,750$ 11.95$

NoMailControl2 25,000 296,250$ 11.85$

TotalNoMailControl 50,000 595,000$ 11.90$

MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$

MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$

TotalMailControl 50,000 750,000$ 15.00$ 3.10$ 155,000$

Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$

TotalMailControls/Rollout 1,875,000 28,125,000$ 15.00$ 3.10$ 5,812,500$

$15.00-$11.90=$3.10

– Matchback Incremental Lift Assessment Model (above)

• Mail Control Matchback $ or Orders minus No Mail Control

Matchback $ or Orders (you can also add in Pass-Along Promo Code

Redemption $ or Orders) = Incremental Result

Audience

FullyLoaded

Matchback

FullyLoaded

Matchback/Household

Incremental

$/Household

Incremental

Revenue

NoMailControl1 25,000 298,750$ 11.95$

NoMailControl2 25,000 296,250$ 11.85$

MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$

MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$

Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$

$15.00-$11.90=$3.10

Audience

FullyLoaded

Matchback

FullyLoaded

Matchback/Household

Incremental

$/Household

Incremental

Revenue

NoMailControl1 25,000 298,750$ 11.95$

NoMailControl2 25,000 296,250$ 11.85$

MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$

MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$

Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$

$15.00-$11.90=$3.10

Audience

FullyLoaded

Matchback

FullyLoaded

Matchback/Household

Incremental

$/Household

Incremental

Revenue

NoMailControl1 25,000 298,750$ 11.95$

NoMailControl2 25,000 296,250$ 11.85$

MailControl1 25,000 373,750$ 14.95$ 3.00$ 75,000$

MailControl2 25,000 376,250$ 15.05$ 3.20$ 80,000$

Rollout 1,825,000 27,375,000$ 15.00$ 3.10$ 5,657,500$

$15.00-$11.90=$3.10

Tests vs. Mail Control

(Rollout) Results

• Test A delivered $2.50 more in incremental revenue vs. Mail Control (Rollout). • Next year, when Test A is rolled out to the 2M circulation, it will deliver $5M

more in incremental revenue vs. this year (but cut that back 15%).

• Test B Offer improved results by $0.80• Test C Later In-Home Date was a LOSER ($2.50)• Test D Cheaper Device was a LOSER ($1.10)• Test E Earlier Expiration Date had NO IMPACT

AudienceFullyLoadedMatchback

FullyLoaded

Matchback/Household

Incremental$/Household

Incremental

vs.IncrMailControl

TotalMailControl 50,000 750,000$ 15.00$ 3.10$

TestA-CostlierDevice 25,000 437,500$ 17.50$ 5.60$ 2.50$TestB-Offer 25,000 395,000$ 15.80$ 3.90$ 0.80$

TestC-In-HomeDate 25,000 312,500$ 12.50$ 0.60$ (2.50)$TestD-CheaperDevice 25,000 347,500$ 13.90$ 2.00$ (1.10)$

TestE-ExpirationDate 25,000 375,000$ 15.00$ 3.10$ -$

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

more in incremental revenue vs. this year (but cut that back 5-15%).

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

(Rollout) Results

FullyLoaded

Matchback/Household

Incremental

vs.IncrMailControl

$5.60 - $3.10 = $2.50

Results Assessment Models

• 2 primary ways to measure results

– Matchback Incremental Lift Model• Match back your Mail and No Mail Controls

• Mail minus No Mail Control + Pass-Alongs = Incremental Revenue

– Promotion Code Redemptions• Match Back Redemptions

• Measure Pass-Alongs– promo code usage by customers not mailed – somehow they

received the coupon

• Total Redemptions

• Redemption Entries (entered into account but not used) vs. Usage

– Matchback Incremental Lift Model• Match back your Mail and No Mail Controls

received the coupon

– Matchback Incremental Lift Model (Best)• Match back your Mail and No Mail Controls

received the coupon

What results can show and not show

• Matchbacks Incremental Lift Model only tells you

incremental number of responders and revenue –

we don’t know WHO was the incremental

responder

– And your catalog drives not just incremental responders,

but incremental revenue from those responders who were

going to order anyway whether they received the catalog

or not – they spent more than they might have if they

hadn’t received it

– You can look at incremental revenue by product category

• Promo Code Redemptions tell you who engaged

– But NOT who would have engaged anyway if you hadn’t

mailed at all

responder

mailed at all

responder

mailed at all

responder

mailed at all

How many tests in holiday season?

• Conduct as many tests as practical within bounds of circ size, budget, keeping it simple, and prioritizing tests likely to improve performance the most

• Some tests will be winners, most will be losers vs your roll-out control– You may lose some short-term revenue due to testing but

that’s the cost of increasing revenue next year.

• One chance per year – don’t miss it – so certainly don’t under-test.

• More tests increase chance you find something that will move the meter and have a meaningful impact on next year’s results

What to test?

Here are come

categories:

– Lists

– Devices

– Promotional

Offers

– Merchandise

– Timing

– Expiration

– Covers

– Messaging

– Design

– Color

– Frequency

– Digital Integration (Catalog + Email,

Catalog + Display)

What to test?

Here are come

categories:

– Lists

– Devices

– Promotional

Offers

– Merchandise

– Timing

– Expiration

– Covers

– Messaging

– Design

– Color

– Frequency

Catalog + Display)

What to test?

Here are come

categories:

– Lists

– Devices

– Promotional

Offers

– Merchandise

– Timing

– Expiration

– Covers

– Messaging

– Design

– Color

– Frequency

Catalog + Display)

What matters most?Factors most likely to increase performance

• In my experience, what will move the meter the most is the interaction between:(Assuming timing & merch mix are adequate.)

– Lists• Relevance

• Breadth

• Depth

– Device “in-the-mail”• You may call it a “Form Factor”?

• Catalog / Pages / Merch / Bells & Whistles / Wraps / Wallet Cards / Blow-Ins / Inserts / Envelopes / Things to Peel / Greeting Card / Postcard

– Promotional Offer (Incentive)• Save! Free! Survey! Exclusive Access!

• Be uncomfortably generous

• Breadth

• Depth

• Breadth

• Depth

• Breadth

• Depth

Test: House Lists

• Segmentation RFMP– Recency

– Frequency

– Monetary

– Product

– Seasonal

– Inquiries / Non-Buyers

• Reactivation Scoring (lapsed “look-alike” modeling)

• Match your e-mail only customers to postal addresses and mail them

• Match your anonymous site visitors to postal addresses and mail them

Test: House Lists

– Frequency

– Monetary

– Product

– Seasonal

Test: House Lists

– Frequency

– Monetary

– Product

– Seasonal

Test: House Lists

– Frequency

– Monetary

– Product

– Seasonal

Test: Prospect Lists

• Test from many list categories:– Vertical Lists (ex: Pottery Barn Buyers)

– Compiled Lists (ex: Kindergarten Teachers)

– Look-Alike Modeled Lists (co-ops)

– Subscriber Lists (magazines)

– Association Lists (conferences)

– Business Partner Lists• Do you sell to same target, but not competitive products?

• Use each other’s lists

– Competitor List Swaps• Trust, good relations, and a common belief that both parties

benefit (that both sell more) by mailing each other’s lists are required. This is a rare arrangement, but it works for those who partner to make it possible.

Use Promo Codes AND Matchbacks for

list results analysis

• Promo Codes are very helpful for LIST analysis – NMCs (No Mail Controls) may be too small at list segment

level to meaningfully measure the incremental revenue

– You may not want to pay for the prospect NMC names for every list – particularly at large cohort sizes that provide statistically significant data

– You will at least get a read on the promo code performance of list segments relative to each other

– If your match back reports end up unreliable for some reason, promo code redemption usage is a back-up path to results analysis (though not preferrable vs. match back and incremental lift model).

– In my experience, the incremental performance of prospect lists lands at 90-110% of the promo code redemptions – not too far off (house lists not so)

• Therefore, assign a separate promo code group for each list test, each segment depth

Test: Device

• Test devices that make you feel uncomfortably radical – don’t just be a “catalog”– They stand out in the mailbox, violate, and engage,

triggering a pause for further consideration, an interception on the way to the trash can

– What sparkles? Catches the eye? Feeling of touch?

• Thick plastic postcards

– What conveys value?

• Wallet Cards (they look like credit or gift cards)

– What conveys a trigger to take action?

• Sticky Notes (anything that peels)

• It’s only a test! • Remind everyone at your company that only a small portion of

the mailing will receive this radial device – it won’t “destroy the brand” if mailed once to a small cohort

OBJECTIVE: Intercept the trash can and get recipient from the direct mailer / catalog to your web site. Everything about your device should be serving this objective!

Costlier Devices can PAY out BIG time!

• Costlier devices can have significant impact in the mailbox.– You’re not just competing with your compeitors in your

sector, you’re competing with whatever else arrives in the recipient’s mailbox that day – are you standing out?

• If $0.10 extra spend per piece in the mail for some violating bells & whistles engagement brings in $2.50 more revenue, you have a ROAS of 25 on that incremental $0.10 spend – an insanely high ROAS.

• A blow-in card, with an extra ink-jetted promo code promoting a featured product, costs about $0.01. If that lifts your result $0.05 you have a ROAS of 5.

Test: Promotional Offers

• Here’s what I am seeing from my experience:

– Save 10-30% isn’t meaningfully increasing response

– Save $ offers seem to perform better than Save % offers

– BOGO where margins make it possible tends to be powerful

– Free Shipping isn’t meaningfully increasing response

– Free Gifts aren’t meaningfully increasing response

– Free Products – if you can give products away, do!

– Multiple offers (promo codes sprinkled throughout book, product targeted, on top of an overall order offer) tends to increase response

– Surveys – do you have a credible reason to ask your customers or prospects to give you their opinion? Do you have an incentive for them to do it without any obligation to purchase anything at all for that incentive? These can be very powerful, depending on the sector and target market.

– Invitations – premium offers for special events or access to special information. Sneak peeks. These are effective with folks who care about access, information and membership than about price and promotions.

Test: Merchandise

• More or less merch?

• Breadth? Depth? – Pack more merch into your existing catalog size

– Cut pages – can you spend less?

– Add in a centerfold stitched-in insert of extra pages, a “gift guide” booklet

– Try a postcard instead

• Plastic postcards perform 2-3x better than card-stock postcards

– Try a greeting card instead – very little merch

• Invitation

• Wallet Card

• Works great with existing customers

Test: Promotional Offer Expiration Dates

• I’ve seen both short-term expiration dates and

extended expiration dates perform well

• Test to see what works best for your business

• Don’t test small differences – test big differences:

– Nov 30

• Careful! Don’t run the risk that your offer expires before it lands

in mailboxes!

– Dec15

– Dec20

– Dec31

– Jan15

Test: Timing – when you’re landing in

mailboxes

• Holiday catalogs are hitting homes in October

these days…what works best for your catalog?

• Don’t guess! Test!

• Test a variety of dates

– Pre-Halloween

– Post-Halloween

– Pre-Thanksgiving

– Land in mailbox Thanksgiving weekend

– Post-Thanksgiving

Test: Covers

• Covers are difficult to test because a control won’t help you; you can’t repeat the same cover twice (or the same cover next year)

• Cover wraps, on the other hand, can be repeated and serve as a roll-out control to be beat each year

• If you’re going to test cover imagery, try:– “product” vs. “aspirational”

• Bottles of Wine VS.

• Beautiful people laughing and drinking wine at a Vineyard

– If “aspirational” works best, then always use “aspirational” covers of people partying at a Vineyard until you find some other approach that works better

– Variations on the same theme: every cover is of friends on a dramatic outdoor excursion

Test: Covers

Test: Messaging

• In my experience, messaging relevant to a

targeted audience can increase results a bit –

– Never ordered?

– Ordered just once?

– Previous products purchased indicates they qualify

for relevant upsell?

Test: Colors

• Color in a vacuum has no impact except to the

degree that it:

– Helps or impairs the conveyance to the recipient of

your intended message

– Helps recipients recognize your company, such as

Shutterfly, Netflix

Test: Colors

degree that it:

Shutterfly, Netflix

Test: Colors

degree that it:

Shutterfly, Netflix

Test: Colors

degree that it:

Shutterfly, Netflix

Test: Digital Integration

• Email Test– Cohort A: Send Catalog Only

– Cohort B: Send Email Only

– Cohort C: Send Catalog + Email

• Display– Cohort A: Send Catalog Only

– Cohort B: Serve Display Only

– Cohort C: Send Catalog + Serve Display

• In my experience this integration isn’t a major lever, is a lot of work and cost with little payout– Difficult to read results, so much noise in those channels already

– Most catalog recipient households are probably getting hit with emails and display ads anyway.

– Not clear this integration has any impact, but it does cost!

• Don’t conclude from my experience – try it yourself…

Test: Frequency

– Test A • Single-Touch Mail Control Touch 1 of 1 (NOV15)

– Test B – The total combined revenue of…• 2-Touch Mail Touch 1 of 2 (NOV15)

• 2-Touch Mail Touch 2 of 2 (DEC1)

• Trigger Direct Mail – 24-hour follow-up to account activity (log-in, placed order, new customer)

Test: Frequency

– Test A • Single-Touch Mail Control Touch 1 of 1 (NOV15)

– Test B – The total combined revenue of…• 2-Touch Mail Touch 1 of 2 (NOV15)

• 2-Touch Mail Touch 2 of 2 (DEC1)

• Trigger Direct Mail – 24-hour follow-up to account activity (log-in, placed order, new customer)

Take-aways

• If you’re not testing, take some steps now to test something meaningful this year – it can be simple

• If you can test:– be sure to set up tests using best practices so that the

results are actionable

– Prioritize• Lists

• Promotional Offers

• Devices

• Don’t believe what anyone else says about what is working – test it yourself before taking action. – Believe in the best practice methods for testing.

– Believe in what you have learned yourself by testing using best practices!

Take-aways

• Devices

Take-aways

• Devices

Thanks!

Contact Info:

Christopher Werler

[email protected]

781-962-7902

Holiday catalog testing with Christopher Werler

Marketing

Transcript of Holiday catalog testing with Christopher Werler