Testing for SuccessHow to Effectively Utilize A/B Split Testing

Fishbowl Best Practices Webinar

Hosted by:

Drew HebbleSenior Marketing Specialist

AGENDA

• The Basics

• Getting Started

• Improving Open Rates

• Improving ClickThrough Rates

• After the A/B Split Test

• Q & A

THE BASICS: WHAT IS AN A/B SPLIT TEST & HOW DOES IT WORK

What is an A/B Split Test?

A simple method for comparing the performance of two different versions of an e-mail campaign

How Does an A/B Test Split Work?

•A small subset of the total member list is selected as a test group and divided into two equal parts.

•Each half of the test group receives a different version of the same message and a winner is identified

The higher-performing message is blasted to the remainder of the list.

How Will A/B Split Testing Help my eClub?

Understanding the behavior of your e-club members will help you construct a messaging strategy that better engages them

THE BETTER YOUR PROGRAM PERFORMS AGAINST THESE KEY METRICS, THE GREATER THE SALES IMPACT!

Engaged members drive:

– Higher open rates

– Higher click-through rates

– Lower unsubscribe rates

GETTING STARTED: CHOOSING THE APPROPRIATE A/B SPLIT TEST

Getting Started: What Should You Test?

Common test variables include:

Subject lines

Creative design

Call to action language

Call to action placement

Call to action style

Send times (time of day or week)

From lines

Actions based on findings

Each A/B split test should isolate a specific message component that has the potential to provide actionable results

Getting Started: What Should I Measure?

Open Rate

• Key Test Variables– Subject line

– Time of day & day of week

Click-through Rate (CTR)

• Key Test Variables

– Call to action

• Language,

• Style

• Placement

Open CTR

What’s Most Common

Getting Started: Guidelines for a Clean & Effective Test

1. Restrict each test to 1 variable

2. Make sure that your results can be measured

3. Construct a fair comparison

A comparison of two subject lines sent at different times of day makes it difficult to directly correlate results with to one of the variables

Comparing two non-clickable calls to action would not provide an action to measure

“Free appetizer today!” vs. “Your October Restaurant Newsletter” are too different to provide a fair comparison.

IMPROVING OPEN RATES: A/B SPLIT TESTING OF SUBJECT LINES

Subject Line Testing: Good Examples

• Offer “teaser” vs. Specific offer

Enjoy this Special Birthday Gift!

Enjoy your Free Entrée Birthday Gift!

• Personalization

Drew, enjoy a free dessert on us!

Enjoy a free dessert on us!

• Subject line length

We’ve got it all!

We’ve got it all! Good food, music, giveaways and more!

AB

AB

AB

Subject Line Testing: Good Examples

• Symbols

☂ Rainy day special! ☂

Rainy day special!

• Placement of offer copy

Free Appetizer for you and a Friend!

You and a Friend can enjoy a Free Appetizer!

Adjust testing types, based on your branding, assumptions and historical data

AB

AB

Subject Line Testing: Is It a “TRUE” Test?

Have a Free Appetizer on us! vs. Drew, we’d like to offer you a Free Appetizer – today only!

♥ You’re going to love this great deal! ♥ vs. A Free Entrée for you and a friend!

CAUTION… UNFAIR

COMPARISONAHEAD

Subject Line Testing: “I’ve sent my test, now what?”

1. Review key performance metrics for each version of the message (opens, click-throughs, and unsubscribes)

2. Send the top performing subject line to remaining members

3. Monitor winning subject line performance?

4. Analyze the results and use them to improve on next campaign?

The results of each test should add one more piece to the puzzle

What Does Winning Performance Look Like…..

Open Rate ClickThrough Rate Unsubscribe Rate

Subject Line A: Enjoy this Special Birthday Gift!

0.19 0.012 0.0012

Subject Line B: Enjoy your Free Entrée Birthday Gift!

0.23 0.014 0.0013

3%

8%

13%

18%

23%

IMPORTANT! Consider ALL messaging

metrics, NOT just open rate

IMPROVING CLICKTHROUGH RATES: A/B TESTING OF CALL TO ACTION

• Question vs. statement

• Click Here vs. (an implied clickthrough icon)

Call to Action Testing: Language

Want a Free Appetizer?

Click Here for your Free Appetizer

• Left vs. right

• Top vs. bottom

• Inside an image vs. stand alone

Happy Birthday on Us! 

Click Here for a free

brownie during the week of

your birthday.

Happy Birthday on Us! 

Click Here for a free brownie

during the week of your birthday.

Call to Action Testing: Placement

Call to Action Testing: Visual Style

To help you through this rainy weather, Click here to enjoy a free appetizer with any entrée purchase.

To help you through this rainy weather

Click Here for your free appetizer with any entrée purchase.

• Clickable body copy vs. button

• Animation vs. static image

• Different color copy/button

• Different images on each version

• One has a clickable call to action, the other has no measurable variable

Call to Action Testing: Is It A “TRUE” Test

Click Here for your free appetizer with any entrée purchase.

AVOID UNFAIR COMPARISONS

Improving CTR: “I’ve sent my test, now what?”

Want a Free Appetizer?

Click Here for your Free Appetizer

Version A (2.8% CTR) Version B (1.7% CTR)

1. Review click-through performance metrics for each version of the message

2. Send top performing call to action to remaining members3. Monitor winning call to action performance across the remaining audience?4. Analyze the results and use them to improve on next campaign?

AFTER THE A/B TEST:ANALYZING RESULTS & APPLYING LEARNING’S

There Was No Clear Winner…What Should I Do?

• Both versions performed below your average?

– Negative results can be as useful as positive ones. Learn from what your guests don’t respond to, and try a different strategy on your next campaign.

• Click-through results don’t lean one way or the other?

– Review your testing method to ensure that there the difference between the two message versions is significant.

A “Click here to RSVP” vs. “Click below to RSVP” call to action may

not yield a different result

There Was No Clear Winner…What Should I Do?

One version had a much higher open rate, while the other version had a far superior clickthrough rate?

Subject line test results

A) 23% open rate & 1.1% CTR

B) 21% open rate & 3.3% CTR

It’s important to consider the amount of variation between metrics.

Depending on the focus of your message, it’s sometimes better to sacrifice one slightly lower metric for another that is much greater.

Acting on Your Results

Make Assumptions based on learning and test on future campaigns

• Shorter subject line outperformed a longer subject line?

• Red call to action yielded a higher click-through than a blue one?

If YES – You’re on the right track to better understanding your audience. If NO –

Was there a difference on this test that may have swayed results?

Refine your methodology and continue to test until results are consistent.

Are you able to replicate the results

Three Key A/B Testing Takeaways

1. ALWAYS test variable at a time

2. ALWAYS review the performance of your entire mailing, NOT just the A/B test

3. ALWAYS apply your learning's to future campaigns

4. NEVER stop testing and refining

Questions?

Thanks for your time!

Upcoming Fishbowl Webinars

October 18th Optimizing your website to attract new e-club members

November 15th Messaging Best Practices

Resources

For more information on Fishbowl’s Creative and strategic Professional Services offerings:

Enterprise Clients: Please contact your Account Director

Local Clients: Please contact your Sales Representative

Technical questions on how to do A/B subject line tests?Call our Support line 800.883.1984 or

email clientsupport@fishbowl.comSign up for a webinar: http://

www.fishbowl.com/feature/webinar_enterprise_series