A/B Testingwhat’s that and how it works
Konrad Synoradzki Allegro Group 2013
What is A/B testing?
A/B testing is comparing two versions of a web
page to see which one performs better.
Konrad Synoradzki Allegro Group 2013
web-page visitors
Konrad Synoradzki Allegro Group 2013
slice visitors
Konrad Synoradzki Allegro Group 2013
Then compare two web pages by showing the two variants (let's call them A and B) to similar visitors at the same time.
Konrad Synoradzki Allegro Group 2013
example
Konrad Synoradzki Allegro Group 2013
example
Konrad Synoradzki Allegro Group 2013
example
Konrad Synoradzki Allegro Group 2013
What can you test?• Media mentions
• Awards and badges
• Banners efficiency
• Rotation speed
• New version of page
• Removing ads
Konrad Synoradzki Allegro Group 2013
• Headlines
• Sub headlines
• Paragraph Text
• Testimonials
• Call to Action text
• Call to Action Button
• Links
• Images
• Content above or below the fold
• Social proof
Types of tests
Konrad Synoradzki Allegro Group 2013
A/B testingBest used to measure the
impact of a two to four
variables on interactions with
the page.
Examples:
• new layout
• Labels on button
Konrad Synoradzki Allegro Group 2013
Multivariate Testing• compares a higher
number of variables,
• reveals more information about how these variables interact with one another.
• traffic to a page is split between different versions of the design.
Examples
• Test headline, buttoncolor, button text and image on landing page
Konrad Synoradzki Allegro Group 2013
Multi-Page Testing• A/B test on multi-page
Example:
• Colors on primary
action in process
• Badges on promo items
Konrad Synoradzki Allegro Group 2013
Getting StartedWhat is and how to use optimizely?
Konrad Synoradzki Allegro Group 2013
Getting Started
Konrad Synoradzki Allegro Group 2013
Step 0: Implement the snippet
Getting Started
Konrad Synoradzki Allegro Group 2013
Step 1: Enter your URL on Optimizely.com
Getting Started
Konrad Synoradzki Allegro Group 2013
Step 2: Add page variations
Getting Started
Konrad Synoradzki Allegro Group 2013
Step 3: Set up goals for tracking
Getting Started
Konrad Synoradzki Allegro Group 2013
Step 4: Target and allocate
Report page
Konrad Synoradzki Allegro Group 2013
Short moviehttps://fast.wistia.net/embed/iframe/gxttknv8aa?auto
Play=true&controlsVisibleOnLoad=true&version=v1&vo
lumeControl=true
Konrad Synoradzki Allegro Group 2013
Technical detailsbut how??
Konrad Synoradzki Allegro Group 2013
Organizing Experiments
Konrad Synoradzki Allegro Group 2013
User Roles
Konrad Synoradzki Allegro Group 2013
Each Project Has Its Own Project Code (Snippet)
Konrad Synoradzki Allegro Group 2013
or asynchronous version
Optimizely Snippet Contents
Konrad Synoradzki Allegro Group 2013
• Active Experiments• Draft Experiments• Project Goals• Optimizely execution and tracking logic• Optimizely jQuery Settings (optional – version 1.6.4 included by
default)
Two modes
Konrad Synoradzki Allegro Group 2013
Optimizely translates every change made in the visual editor
into a line of jQuery code, which we call Variation Code:
Konrad Synoradzki Allegro Group 2013
Optimizely Testing Process
Konrad Synoradzki Allegro Group 2013
Order of Execution
Konrad Synoradzki Allegro Group 2013
Snippet Timing Logic
Konrad Synoradzki Allegro Group 2013
Optimizely Architecture Overview
Konrad Synoradzki Allegro Group 2013
Optimizely Architecture Overview
Konrad Synoradzki Allegro Group 2013
• Google App Engine is used to store and manage your account and experiment configuration settings, including your account login and password information. Passwords are hashed and stored securely using 160 bit cryptographic hashes.
• Amazon Elastic Compute Cloud and Simple Storage Service is used to store your conversion/traffic logging data, which Optimizely uses to compute experiment results.
• Akamai is used to store and serve your Optimizely snippet JavaScript file(s). This file is publicly accessible by necessity.
Collecting data
Konrad Synoradzki Allegro Group 2013
The visitor’s browser makes an asynchronous request to https://log.optimizely.com, which logs the following information:
• The Optimizelyassigned unique user ID • IDs of the experiments and variations this visitor has seen • Your Optimizely account number • Visitor’s browser/OS version • Visitor’s IP address, with last octet removed • A string identifying this conversion goal, such as the page URL • Date and time
that’s all
Konrad Synoradzki Allegro Group 2013
question?
Konrad Synoradzki Allegro Group 2013
Konrad Synoradzki Allegro Group 2013
Top Related