Many thanks to:

Growth Hacking Meetup

- Episode 4 -

9 A/B Testing Use Cases

But first… some A/B Testing mistakes startups make ALL the time

11

A/B tests are ended too early Statistical significance (large sample size, etc) is what

tells you version A is actually better than version B

As an optimizer, your job is to figure out the truth. You have to put your ego aside.

12

Tests are not run for full weeksConversion rates can vary greatly depending on the day of the week. Always test a full week at a time.

13

Always pay attention to external factorsIs it Christmas? External factors definitely affect your

test results. When in doubt, run a follow-up test.

14

Testing random ideas without a hypothesis You’re wasting precious time and traffic.

Never do that.

By the way, if you want to test button colors “green vs orange” is not the essence of A/B testing. It’s about understanding the target audience.

Use your traffic on high-impact stuff. Test data-driven hypotheses.

15

They give up after the first test failsRun a follow-up test, learn from it, and

improve your hypotheses.

“I have not failed 10,000 times. I have successfully found 10,000 ways that will not work.” - Thomas Alva Edison, inventor

16

They’re ignoring small gains Only 1 out of 8 A/B tests drive

significant change.

17

They’re not running tests at all timesEvery single day without a test is a wasted day.

Testing is learning.

Now, for the second part… 12 Surprising A/B Test Results to Stop You Making Assumptions

Test 1: Which Copy Increased Trial Sign-Ups?

Result

In this test, Version B increased sign-ups by 38% – a big rise.

However, you would think version A was the better design.

Why does Version B work? Simply because the copy is better. Lesson: landing pages don’t have to be pretty to be effective.

Test 2: Which Landing Page Got 24% More Lead Generation Form Submissions?

Picture vs No picture

Result

Surprisingly, Version A was the page that got the 24% increase in submissions, simply by removing

the image from the page.

This is a great example of why you should confirm your assumptions with quantitative testing.

Test 3: Which Radically Redesigned Form Increased B2B Leads By 368.5%?

Result

Version A is an obvious winner, but it’s not just the big red button that makes the difference.

Version A keeps things really tight and uses grouping to visually shrink the impact of the form

When designing your landing page, don’t overestimate your user’s tolerance, goodwill, and patience.

Test 4: Does Matching Headline & Body Copy to Your Initial Ad Copy Really Matter?

Result

Version B looks as if it should be better: the headline copy is snappier, the sub-head clearer, but in tests version A increased leads by 115%.

Why? Version A was designed to tie in with the ads that drive users to the page. Lesson: making the elements of the sales funnel work together increases efficacy.

Test 5: Which Landing Page Increased Mortgage Application Leads by 64%?

Result

Video can be a very effective tool in communicating lots of information in a compact form. But the presence of video couldn’t save Version B; Version A increased leads by 64%.

Test 6: Does adding testimonials to your Homepage increase Sales?

Result

It would seem that this would have very little effect, but in practice this small change increased sales by 34% – a big margin.

Why is this? Having ‘social proof’, even in this basic form, humanizes the conversion experience

Test 7: Does Social Proof Increase Conversions? ‘Big Brand Prices’ vs. Consumer Ratings

Result

‘Social proof’ is a powerful tool that can have a demonstrable effect on conversion outcomes.

But in this case, customers were just looking for the cheapest offer. Asking them to consider an additional piece of information – customer satisfaction – made

them back away from the conversion.

Test 8: Does an Email Security Seal Help or Hinder Lead Generation Form Completions?

Result

Actually, users were put off by seeing it in this context, assuming that they were

about to pay for something

Test 9: Which Page Got an 89.8% Lift in Free Account Sign Ups?

Result

Version B wins: It has three bullet points, as opposed to words in speech bubbles.

Removing a distraction from the page and reducing the risk of users navigating away

from the page worked better

Whether you are right or wrong: First, make sure you get the basics right before you start testing, and second, always be testing, because unless you test, you can never be absolutely sure.

You are Welcome to Join the Adventure

hey@thefamily.co