A/B Testingwhat’s that and how it works

Konrad Synoradzki Allegro Group 2013

What is A/B testing?

A/B testing is comparing two versions of a web

page to see which one performs better.

Konrad Synoradzki Allegro Group 2013

web-page visitors

Konrad Synoradzki Allegro Group 2013

slice visitors

Konrad Synoradzki Allegro Group 2013

Then compare two web pages by showing the two variants (let's call them A and B) to similar visitors at the same time.

Konrad Synoradzki Allegro Group 2013

example

Konrad Synoradzki Allegro Group 2013

example

Konrad Synoradzki Allegro Group 2013

example

Konrad Synoradzki Allegro Group 2013

What can you test?• Media mentions

• Awards and badges

• Banners efficiency

• Rotation speed

• New version of page

• Removing ads

Konrad Synoradzki Allegro Group 2013

• Headlines

• Sub headlines

• Paragraph Text

• Testimonials

• Call to Action text

• Call to Action Button

• Links

• Images

• Content above or below the fold

• Social proof

Types of tests

Konrad Synoradzki Allegro Group 2013

A/B testingBest used to measure the

impact of a two to four

variables on interactions with

the page.

Examples:

• new layout

• Labels on button

Konrad Synoradzki Allegro Group 2013

Multivariate Testing• compares a higher

number of variables,

• reveals more information about how these variables interact with one another.

• traffic to a page is split between different versions of the design.

Examples

• Test headline, buttoncolor, button text and image on landing page

Konrad Synoradzki Allegro Group 2013

Multi-Page Testing• A/B test on multi-page

Example:

• Colors on primary

action in process

• Badges on promo items

Konrad Synoradzki Allegro Group 2013

Getting StartedWhat is and how to use optimizely?

Konrad Synoradzki Allegro Group 2013

Getting Started

Konrad Synoradzki Allegro Group 2013

Step 0: Implement the snippet

Getting Started

Konrad Synoradzki Allegro Group 2013

Step 1: Enter your URL on Optimizely.com

Getting Started

Konrad Synoradzki Allegro Group 2013

Step 2: Add page variations

Getting Started

Konrad Synoradzki Allegro Group 2013

Step 3: Set up goals for tracking

Getting Started

Konrad Synoradzki Allegro Group 2013

Step 4: Target and allocate

Report page

Konrad Synoradzki Allegro Group 2013

Short moviehttps://fast.wistia.net/embed/iframe/gxttknv8aa?auto

Play=true&controlsVisibleOnLoad=true&version=v1&vo

lumeControl=true

Konrad Synoradzki Allegro Group 2013

Technical detailsbut how??

Konrad Synoradzki Allegro Group 2013

Organizing Experiments

Konrad Synoradzki Allegro Group 2013

User Roles

Konrad Synoradzki Allegro Group 2013

Each Project Has Its Own Project Code (Snippet)

Konrad Synoradzki Allegro Group 2013

or asynchronous version

Optimizely Snippet Contents

Konrad Synoradzki Allegro Group 2013

• Active Experiments• Draft Experiments• Project Goals• Optimizely execution and tracking logic• Optimizely jQuery Settings (optional – version 1.6.4 included by

default)

Two modes

Konrad Synoradzki Allegro Group 2013

Optimizely translates every change made in the visual editor

into a line of jQuery code, which we call Variation Code:

Konrad Synoradzki Allegro Group 2013

Optimizely Testing Process

Konrad Synoradzki Allegro Group 2013

Order of Execution

Konrad Synoradzki Allegro Group 2013

Snippet Timing Logic

Konrad Synoradzki Allegro Group 2013

Optimizely Architecture Overview

Konrad Synoradzki Allegro Group 2013

Optimizely Architecture Overview

Konrad Synoradzki Allegro Group 2013

• Google App Engine is used to store and manage your account and experiment configuration settings, including your account login and password information. Passwords are hashed and stored securely using 160 bit cryptographic hashes.

• Amazon Elastic Compute Cloud and Simple Storage Service is used to store your conversion/traffic logging data, which Optimizely uses to compute experiment results.

• Akamai is used to store and serve your Optimizely snippet JavaScript file(s). This file is publicly accessible by necessity.

Collecting data

Konrad Synoradzki Allegro Group 2013

The visitor’s browser makes an asynchronous request to https://log.optimizely.com, which logs the following information:

• The Optimizely­­assigned unique user ID • IDs of the experiments and variations this visitor has seen • Your Optimizely account number • Visitor’s browser/OS version • Visitor’s IP address, with last octet removed • A string identifying this conversion goal, such as the page URL • Date and time

that’s all

Konrad Synoradzki Allegro Group 2013

question?

Konrad Synoradzki Allegro Group 2013

Konrad Synoradzki Allegro Group 2013