ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

November 3–6, 2014

#CZLCHI | @ClickZLive

The Global Conference Series Designed by Digital Marketers, for Digital Marketers

3 Common Mistakes in Online Testing(And How to Avoid Them)Taylor KennedyMECLABSSenior Manager, Optimization & Strategy

@OptimizeDude

November 3–6, 2014 | #CZLCHI | @ClickZLive

30%

16%

Do you understand your results?60

%

12%

43%

8%

62%

6%

@OptimizeDude


30%

16%

60%

12%

43%

8%

62%

6%


What does it mean?

@OptimizeDude


You’re (probably) testing wrong if you:

• Are stuck optimizing a part of your funnel that lacks significant impact on the bottom line

• Have been blindly following “best practices”

• Are not further establishing customer theory with each test

@OptimizeDude


Our testing process


Case StudyOptimizing the wrong part of the funnel

@OptimizeDude


Cart Test 1: Control

@OptimizeDude


Cart Test 1: Treatment

@OptimizeDude


Cart Test 1: Results

Results were flat.

Shopping Cart Conversion Rate

Rel. Difference

Level of Confidence

Control 14.3% NA ---

Treatment 13.5% - 4.99% 44%

?

@OptimizeDude


Cart Test 2: Control

@OptimizeDude


Cart Test 2: Treatment

@OptimizeDude


Cart Test 2: Results

Results were flat, even trending downward.


Rel. Difference

Level of Confidence


Treatment 11.1% -19.26% 88%

?

@OptimizeDude


Why aren’t tests getting results?

5.7% 41%All visitors Add to cart Payment info Convert40%

@OptimizeDude


Through proper planning you:Identify the highest impact area of your site to test.

Find potential validity threats linked to normal traffic that may impact results.

Discover visitor behavior that helps you create long-term test plans.

@OptimizeDude


Product Page 1: Control

@OptimizeDude


Product Page 1: Treatment

@OptimizeDude


Product Page Test 1: Results

• “Add to Cart” increased by over 15%• Conversion increased by 40%


Rel. Difference

Level of Confidence


Treatment 1.50% 40.05% 98%

!

@OptimizeDude


What motivated the cart test?A. Needed improvements were obvious

B. Our client or boss told us to

C. Testing close to the conversion event has the biggest ability to increase revenue

D. The data told us to


You’ve been following “best practices”Case study examples

@OptimizeDude


How are successful and unsuccessful ecommerce companies approaching testing?

MarketingSherpa asked marketers:

@OptimizeDude


@OptimizeDude


Your peers are following best practices too.

@OptimizeDude


How often do you hear the following?

“Keep your call-to-action above the fold”

“Video converts”“Use a compelling image”

“Mobile-friendly is critical”

“Include privacy and security statements to establish trust”

@OptimizeDude


Security Seal Test: Control

@OptimizeDude


Security Seal Test: Treatment 1

@OptimizeDude



@OptimizeDude


Security Seal Test: Results

Shopping Cart ClickthroughRate

Rel. Difference

Level of Confidence


Treatment 1 66.95% 1.2% 50%

Treatment 2 65.00% -1.7% 57%

Treatment 3 64.60% -2.3% 54%

Results were flat.?

@OptimizeDude


Mobile Test: Control

@OptimizeDude


Mobile Test: Treatment

@OptimizeDude


Mobile Redesign Test: Results

Results were significantly negative.


Rel. Difference

Level of Confidence

Control 0.5% NA ---

Treatment 0.3% -40.75% 96%

!


Redefine best practice:• There are no shortcuts to customer wisdom.

• Just because a test worked for someone else, doesn’t mean it will work for you.

• Rigorous, methodology-driven, hypothesis testing ensures you are generating real customer insights.

@OptimizeDude


Framework

©C = 4m + 3v + 2(i-f) - 2a

@OptimizeDude


Framework

©C = 4m + 3v + 2(i-f) - 2a

Probability of conversion

@OptimizeDude


Framework

©C = 4m + 3v + 2(i-f) - 2a

Motivation of user

@OptimizeDude


Framework

©C = 4m + 3v + 2(i-f) - 2a

Clarity of value proposition

@OptimizeDude


A bad hypothesis:“By reducing friction on the page associated

with product selection, we will see an increase in clickthrough.”

• Unclear link to a particular problem• Not related to specific elements on a

particular page or process• Not linked to information or data about the

page

@OptimizeDude


A bad hypothesis:“By reducing friction on the page associated

with product selection, we will see an increasein clickthrough.”

• Unclear link to a particular problem• Not related to specific elements on a

particular page or process• Not linked to information or data about the

page

@OptimizeDude


A good hypothesis:

CHANGING __________________ INTO__________________

WILL_______________________

What your analysis indicates is the

problem

What change you think will solve the

problem

The effect it will have on your KPI

@OptimizeDude


A good hypothesis:“Changing the presentation of products from a single column format into a dual column format will increase the number of product

options visible and boost clickthrough to the product details page”

@OptimizeDude


A good hypothesis:“Changing the presentation of products from a single column format into a dual column format will increase the number of product

options visible and boost clickthrough to the product details page.”

@OptimizeDude


A good hypothesis is:• A proposed statement

• Made on the basis of limited evidence related to a specific problem

• Can be proved or disproved

• Used as a starting point for further investigation

@OptimizeDude


Motivation Test: Control

@OptimizeDude


Motivation Test: ControlHypothesis:

Changing the right-column form into a call-to-action requiring no personal information will reduce friction and boost clickthrough to the subsequent page.

@OptimizeDude


Removed form completelyTreatment 1:

@OptimizeDude


Removed form, added starting priceTreatment 2:

@OptimizeDude


Kept form, added starting priceTreatment 3:

@OptimizeDude


Homepage Motivation Test: Treatments

Shopping Cart Rel. CTR Level of Confidence

Control -- ---

Treatment 1 202% 99%

Treatment 2 149% 99%

Treatment 3 - 20% 99%

@OptimizeDude


Remember:• Proper planning allows you to run tests

with clear outcomes predetermined in advance

• Formulate test sequences that generate big-picture customer analysis

!


Key takeawaysFollow a “testing process.” Don’t just “test.”

1. Always perform at least a basic site data analysis (funnel analysis) before beginning testing.

2. Analyze each page through the mind of your prospect to identify areas of opportunity for testing.

3. Formulate hypotheses that enable you to learn from each and every test. That will lead you to a more complete picture of your ideal prospect.


http://bit.ly/OptimizeTKPresentation Link

http://bit.ly/OptimizeTK

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

Marketing

Transcript of ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)