Class 2: How to create solid hypotheses and set up powerful A/B Tests

Presentation Title

#OptimizelySchoolwww.summerschool2015.eu

What is Summer School?

Optimizely Summer School – Class 1


Annemarie de JongOptimization Consultant

[email protected] @annemariedejong

Class 2How to create solid hypotheses and set up powerful A/B tests


Company Goal

Business Unit

Optimization Goals

Experiment Goals

#OptimizelySchool

Improve annual profit by being a reliable go to store for outdoor

Company Goal

Reduce shipping & handling costs from returns

Business Unit

Increase UX of the order process

Optimization Goals

Relation size calculator and UX

Experiment Goals

#OptimizelySchool

Success metrics differ per industry

Ecommerce/ Travel Media Lead Gen/ B2B

Product page views Pages per visit Key content page views

Add to cart rate Ad impressions Content engagement

Checkout completion rate Ad viewability AD unit CTR Funnel entrance rate

Average order value CPM CPC Entry field completion

Conversion rate Subscription rate Conversion rate


There is not enough motivating imagery

Page is overly focused on price

Call to Action is too far down the page

Problem Solution Result

How to find testing ideas

Add product visuals

Balance price with discount

Duplicate Call to Action at top of page

Increased conversion rate

Increased conversion rate

Increased Pageviews on Checkout Page

The goal of the Hypothesis

Gathering insights on your users

A good hypothesis enables you to:1. Focus on the right goals & KPIs

2. Focus on the right elements3. Collect data to make data-driven decisions

#OptimizelySchool

If______ Then______ Due to______change result

(quantifiable)rationale



IF we add a size calculator to every product page



IF we add a size calculator to every product page THEN the conversion rate will increase



IF we add a size calculator to every product page THEN the conversion rate will increase DUE TO the increased trust of buying the right size product.

Hypothesis: removing elements will allow for more product copy and less clutter on the page.

Good hypothesis: if we divide the homepage in a 80-20 ratio between product related and non product related items

then we will see an increase in article page views,

due to the reduction of distracting elements on the page customers can better focus on article content.

Make sure you learn from every test

A bad hypothesis = No actionable insights

Hypothesis:

If we show text suggestive of the range of products we offer

then we can increase the product page views and conversions.

Due to users arriving at their preferred products easier.

Hypothesis:

If we integrate weather information from Skymosity into the predictive text in the search bar

then we can increase the product page views and conversions.

Due to users arriving at their preferred products easier.

Hypothesis:

If we hide the coupon code field behind a link

then we can increase the conversions and reduce the drop off rate

due to users not leaving site to go find a coupon code

Hypothesis:

If we add a tab to search for vacation packages

then we can increase the average order value

due to the fact that users will engage with a feature they could not previously considered.

Hypothesis:

If we mention all shows on the homepage

then we will increase the overall engagement.

Due to the better visibility of all shows.

+600% in Engagement

• is easy to find if you follow the Problem, Solution, Result pattern

• Consists of 3 fundamental building block: IF… THEN…DUE TO

• Shows a change in a quantifiable metric

• Gives actionable insights (no matter the outcome)

So a good hypothesis…

Annemarie de JongOptimization Consultant

[email protected] @annemariedejong

Sign Up for the next Class!

Until Class 3!

Class 2: How to create solid hypotheses and set up powerful A/B Tests

Marketing

Transcript of Class 2: How to create solid hypotheses and set up powerful A/B Tests