Would you bet your job on your A/B test results?
@QubitGroup #QubitABtest
Friday, 28 February 14
Agenda
@QubitGroup
• 5 digital experience technology trends that will guide your technology investment — Anjali Yakkundi, Forrester Research, Inc.
• 5 fundamental A/B testing mistakes and how to avoid them—Will Browne, Qubit
• How to run a successful A/B testing program—Melanie Kyrklund, Staples
• Q&A
#QubitABtest
Friday, 28 February 14
5 Digital Experience Technology Trends To Guide Your Investment Anjali Yakkundi, AnalystFebruary 27, 2014
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited4
Burberry brings digital into the store
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited5
Room Number
Address & directions
Starwood’s Preferred Guest Streamlines The Check-In Process
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited6
Agenda
›Why digital experience technology matters ›Key digital customer experience
technologies ›Five trends that will drive your digital
experience technology investment›Recommendations
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited7
Agenda
›Why digital experience technology matters ›Key digital customer experience
technologies ›Five trends that will drive your digital
experience technology investment›Recommendations
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited8
Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report
Age of the customer
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited9
Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report
Your customers are always connected…
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited10
…often on multiple devices
Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, August 2012
Friday, 28 February 14
You need to sustain your customer’s attention with engaging experiences.
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited12
Engaging experiences requires strong technology management
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited13
Agenda
›Why digital experience technology matters ›Key digital customer experience
technologies ›Five trends that will drive your digital
experience technology investment›Recommendations
Friday, 28 February 14
Digital experience reference architecture
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited15
Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report
Many solutions make up the digital experience ecosystem
TestingOptimization Analytics
(mobile, social, Web)
PortalsSite search
Commerce platform Web content management
Online video platform
Digital asset management
Product content management
Recommendation engine
Campaign management
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited16
Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report
Testing and optimization solutions are key to engagement and measurement
TestingOptimization Analytics
(mobile, social, Web)
PortalsSite search
Commerce platform Web content management
Online video platform
Digital asset management
Product content management
Recommendation engine
Campaign management
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited17
Agenda
›Why digital experience technology matters ›Key digital customer experience
technologies ›Five trends that will drive your digital
experience technology investment›Recommendations
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited18
#1- This is the beginning of the end of the all-in-one-suite
Only 9% prefer a suite approach
Base: 233 Digital Experience Professionals Source: March 2013, “Digital Experience Delivery Online Survey”
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited19
#2- Tools must create business value out of data
Solutions must effectively harness customer data, product data, social media data, and other
information to create and optimize experiences.
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited20
#3- Tools must support personalized experiences
Situation Real-time optimization Prediction
• Device / Channel • Location• Time of day• Weather
• Historical past• Immediate past • Current session
• Likely next steps
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited21
#4- Look for more than software: look for solutions ›Look for solutions that help marketers and business
users quickly adapt to changing customer needs. This can be achieved by:
•Keeping IT out of critical publishing paths •Prioritizing user interface needs
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited22
#5- Solutions must prioritize multichannel
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited23
Agenda
›Why digital experience technology matters ›Key digital customer experience
technologies ›Five trends that will drive your digital
experience technology investment›Recommendations
Friday, 28 February 14
© 2013 Forrester Research, Inc. Reproduction Prohibited24
Recommendations
›You can’t do it all at once — prioritize key customer touchpoints ›Assess current technology gaps– most
organizations tend to have gaps around:•Optimization•Personalization•Multichannel delivery
›When sourcing new solutions, look to vendors that can help you connect the dots.
Friday, 28 February 14
Thank youAnjali [email protected]@Ayakkundi
Friday, 28 February 14
5 fundamental A/B testing mistakes
and how to avoid them
Will Browne, [email protected]
@QubitGroup #QubitABtest
Friday, 28 February 14
Why do I have to test?
Friday, 28 February 14
A/B testing is an essential tool
@TheGrahamCooke
Friday, 28 February 14
A/B testing is an essential tool
@TheGrahamCooke
• It is the best, simplest way to prove the business value of your digital technology
@QubitGroup #QubitABtest
Friday, 28 February 14
A/B testing is an essential tool
@TheGrahamCooke
• It is the best, simplest way to prove the business value of your digital technology
• It is the only way to get close to proving cause and effect in what you do
@QubitGroup #QubitABtest
Friday, 28 February 14
But there are costs associated with A/B testing
Friday, 28 February 14
But there are costs associated with A/B testing
• The monthly cost of your testing platform
@QubitGroup #QubitABtest
Friday, 28 February 14
But there are costs associated with A/B testing
• The monthly cost of your testing platform
• The salaries and time of your developers and team
@QubitGroup #QubitABtest
Friday, 28 February 14
• The monthly cost of your testing platform
• The salaries and time of your developers and team
• The lost revenue from implementing changes that do nothing for your site (or actively harm it)
But there are costs associated with A/B testing
@QubitGroup #QubitABtest
Friday, 28 February 14
We’re here to show you how to reduce these costs, and increase ROI
@QubitGroup #QubitABtest
Friday, 28 February 14
We’re here to show you how to reduce these costs, and increase ROI
• Find the ideas that are likely to have a real positive effect on user behavior (true winners)
@QubitGroup #QubitABtest
Friday, 28 February 14
We’re here to show you how to reduce these costs, and increase ROI
• Find the ideas that are likely to have a real positive effect on user behavior (true winners)
• Detect as many true winners as possible
@QubitGroup #QubitABtest
Friday, 28 February 14
We’re here to show you how to reduce these costs, and increase ROI
• Find the ideas that are likely to have a real positive effect on user behavior (true winners)
• Detect as many true winners as possible
• Minimize the chance of finding tests that look like winners, but have no true effect
@QubitGroup #QubitABtest
Friday, 28 February 14
39
Find the ideas that are likely to have a real positive effect on user behavior
@QubitGroup #QubitABtest
Friday, 28 February 14
Size does matter
Friday, 28 February 14
Detect as many true winners as possible
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
Friday, 28 February 14
Imagine testing for weighted coins
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
• How many coin flips would be enough to prove that one coin was weighted?
@QubitGroup #QubitABtest
Friday, 28 February 14
Imagine testing for weighted coins
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
• How many coin flips would be enough to prove that one coin was weighted?
• It is impossible to be 100% certain, but the more data you have, the closer you can get
@QubitGroup #QubitABtest
Friday, 28 February 14
Imagine testing for weighted coins
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
• How many coin flips would be enough to prove that one coin was weighted?
• It is impossible to be 100% certain, but the more data you have, the closer you can get
• We follow scientific standards and wait until there is an 80% chance of detecting a difference
@QubitGroup #QubitABtest
Friday, 28 February 14
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
Minimize the chance of finding tests that look like winners,but have no true effect
Friday, 28 February 14
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
• We generally assume that with good testing there is a 5% false positive rate
Minimize the chance of finding tests that look like winners,but have no true effect
@QubitGroup #QubitABtest
Friday, 28 February 14
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
• We generally assume that with good testing there is a 5% false positive rate
• We can test this using A/A tests (showing both groups the exact thing)
Minimize the chance of finding tests that look like winners,but have no true effect
@QubitGroup #QubitABtest
Friday, 28 February 14
We’ve found that tests based on data are
7.7x more effectivethan tests based on intuition alone
(Qubit, 2013)
• We generally assume that with good testing there is a 5% false positive rate
• We can test this using A/A tests (showing both groups the exact thing)
• So out of 100 A/A tests, 5 would be winners, despite doing absolutely nothing
Minimize the chance of finding tests that look like winners,but have no true effect
@QubitGroup #QubitABtest
Friday, 28 February 14
Splitting tests too much can lead to false positives
Friday, 28 February 14
Get a second opinion
Friday, 28 February 14
Validate your results
We recommend a
95/5 splitto confirm your winning variation is still winning
“Most winning A/B test results are illusory”, Qubit 2014
@QubitGroup #QubitABtest
Friday, 28 February 14
A worked example: Imagine Mr Bean’s jelly beans shop
@QubitGroup“Most winning A/B test results are illusory”, Qubit 2014
Friday, 28 February 14
A worked example: Imagine Mr Bean’s jelly beans shop
@QubitGroup“Most winning A/B test results are illusory”, Qubit 2014
• Mr Bean has a 1% conversion rate
• He runs 20 different tests in one year
• There are 2 good tests with uplifts of 5%
• There are also 2 bad tests with uplifts of -5%
@QubitGroup #QubitABtest
Friday, 28 February 14
TruthCorrect method
Naive method
@QubitGroup #QubitABtest
Friday, 28 February 14
TruthCorrect method
Naive method
No. of tests 20 20 20
Reported winners 2 3 9
@QubitGroup #QubitABtest
Friday, 28 February 14
TruthCorrect method
Naive method
No. of tests 20 20 20
Reported winners 2 3 9
After validation 2 2 9
@QubitGroup #QubitABtest
Friday, 28 February 14
TruthCorrect method
Naive method
No. of tests 20 20 20
Reported winners 2 3 9
After validation 2 2 9
Reported uplift 2 x 5% 2 x ~5%
9 x (5-40)%
@QubitGroup #QubitABtest
Friday, 28 February 14
Correct method Naive method
Reported uplift 2 x 5% 9 x (5-40)%
Delivered uplift 2 x 5% 2 x 5% + (? x -5%)
@QubitGroup #QubitABtest
Friday, 28 February 14
Correct method Naive method
Reported uplift 2 x 5% 9 x (5-40)%
Delivered uplift 2 x 5% 2 x 5% + (? x -5%)
Wasted resource None7 unnecessary
tests
@QubitGroup #QubitABtest
Friday, 28 February 14
Correct method Naive method
Reported uplift 2 x 5% 9 x (5-40)%
Delivered uplift 2 x 5% 2 x 5% + (? x -5%)
Wasted resource None 7 unnecessary tests
Customer experience
Improved Degraded
Your understanding
Improved Confused
@QubitGroup #QubitABtest
Friday, 28 February 14
Correct method Naive method
Reported uplift 2 x 5% 9 x (5-40)%
Delivered uplift 2 x 5% 2 x 5% + (? x -5%)
Wasted resource None 7 unnecessary tests
Customer experience
Improved Degraded
Your understanding
Improved Confused
Your manager Happy Angry
@QubitGroup #QubitABtest
Friday, 28 February 14
So, to conclude, those 5 mistakes are:
• Mistake 1: Not A/B testing
• Mistake 2: Not prioritizing tests properly
• Mistake 3: Not waiting for significance and sample size
• Mistake 4: Testing too many things at once
• Mistake 5: Not validating the tests after implementation
@QubitGroup #QubitABtest
Friday, 28 February 14
Questions you should ask your testing provider
• How do you prioritize tests?
• How do you calculate the sample size required?
• How do you account for the multiple testing problem?
• How do you validate winning tests?
@QubitGroup #QubitABtest
Friday, 28 February 14
Thank you
If you have any detailed questions please contact
Will Browne, [email protected]
@QubitGroup #QubitABtest
Friday, 28 February 14
How to run a successful A/B testing program
Melanie KyrklundSenior Ecommerce Analyst
Friday, 28 February 14
Testing programs have the potential to drive the online customer experience
66
Understand your customers and respond to their needs
Optimize the user journey
Create personalized experiences
Quickly respond to market pressures
Rapidly align marketing & onsite experience
Validate development items
Every visit represents an opportunity to establish a relationship with a customer
Customer experience
Personalization Persuasion
Targeted offers
User experience,
functionality & content
Friday, 28 February 14
The reality of running a testing program
67
For many organizations, testing represents a cultural change with many political and operational barriers to be overcome:
Resource & operational factors
Buy-in / Multiple stakeholders
Alignment with other business priorities and development plans
Instilling a data-driven culture
Friday, 28 February 14
A lot of testing is required to deliver results
68
1/5 tests or 1/10 tests will deliver statistically significant wins
A rapid pace of testing is required to build a program that is delivering consistent results month over month
Friday, 28 February 14
Two factors are critical in building a successful testing program
69
Approach Speed
MethodologyIdea Generation
Operational ProcessPrioritization
Friday, 28 February 14
The right approach to optimization
70
Be inspired – every visit is an opportunity to establish a relationship with a customer
Use data to understand how visitors are using your website, what is working and what isn’t
Customer feedback to understand why visitors are not purchasing on your website
If working over multiple websites & countries, look for insight at local level first and then identify commonalities at a later stage
Persuasion research, understand the psychology of your customers
Supplement with quick UX testing
Thoroughly research & quantify each opportunity (visitors reached, expected uplift and incremental revenue)
UX optimization Personalization Persuasion
Friday, 28 February 14
Idea Generation
71Friday, 28 February 14
Idea Generation
71Friday, 28 February 14
Idea Generation
71Friday, 28 February 14
Idea Generation
71Friday, 28 February 14
Speed – Operational Processes
72
Testing Tool selection : Map out the most efficient implementation process
Operational efficiency Pick managed services (analysis, implementation) as required
Prioritization
Build a test pipeline logging:
Idea & hypothesisExpected uplift & incremental revenue Implementation estimatesTest duration Prioritize tests weekly, to keep up momentum of testing and
focus on biggest wins
Friday, 28 February 14
Conclusion: A rapid, data-driven approach is required
73
Testing programs have the potential to drive the online customer experience, however a rapid data-driven approach is required to deliver results
Plan the required resources upfront
Consider ease of use, test time to market and managed services in tool selection
Map out the most efficient end to end process to get tests live
Process supported by data-driven idea generation & prioritization
Friday, 28 February 14
Thank you
Any questions?
Friday, 28 February 14
75
bit.ly/qABresearch
More A/B testing material from Qubit
bit.ly/qABinfographic
@QubitGroup#QubitABtests
Friday, 28 February 14
Top Related