Would you bet your job on your A/B test results?

@QubitGroup #QubitABtest

Friday, 28 February 14

Agenda

@QubitGroup

• 5 digital experience technology trends that will guide your technology investment — Anjali Yakkundi, Forrester Research, Inc.

• 5 fundamental A/B testing mistakes and how to avoid them—Will Browne, Qubit

• How to run a successful A/B testing program—Melanie Kyrklund, Staples

• Q&A

#QubitABtest

5 Digital Experience Technology Trends To Guide Your Investment Anjali Yakkundi, AnalystFebruary 27, 2014

Burberry brings digital into the store

Room Number

Address & directions

Starwood’s Preferred Guest Streamlines The Check-In Process

Agenda

›Why digital experience technology matters ›Key digital customer experience

technologies ›Five trends that will drive your digital

experience technology investment›Recommendations

Agenda

Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report

Age of the customer

Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report

Your customers are always connected…

…often on multiple devices

Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, August 2012

You need to sustain your customer’s attention with engaging experiences.

Engaging experiences requires strong technology management

Agenda

Digital experience reference architecture

Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report

Many solutions make up the digital experience ecosystem

TestingOptimization Analytics

(mobile, social, Web)

PortalsSite search

Commerce platform Web content management

Online video platform

Digital asset management

Product content management

Recommendation engine

Campaign management

Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report

Testing and optimization solutions are key to engagement and measurement

TestingOptimization Analytics

(mobile, social, Web)

PortalsSite search

Commerce platform Web content management

Online video platform

Digital asset management

Product content management

Recommendation engine

Campaign management

Agenda

#1- This is the beginning of the end of the all-in-one-suite

Only 9% prefer a suite approach

Base: 233 Digital Experience Professionals Source: March 2013, “Digital Experience Delivery Online Survey”

#2- Tools must create business value out of data

Solutions must effectively harness customer data, product data, social media data, and other

information to create and optimize experiences.

#3- Tools must support personalized experiences

Situation Real-time optimization Prediction

• Device / Channel • Location• Time of day• Weather

• Historical past• Immediate past • Current session

• Likely next steps

#4- Look for more than software: look for solutions ›Look for solutions that help marketers and business

users quickly adapt to changing customer needs. This can be achieved by:

•Keeping IT out of critical publishing paths •Prioritizing user interface needs

#5- Solutions must prioritize multichannel

Agenda

Recommendations

›You can’t do it all at once — prioritize key customer touchpoints ›Assess current technology gaps– most

organizations tend to have gaps around:•Optimization•Personalization•Multichannel delivery

›When sourcing new solutions, look to vendors that can help you connect the dots.

Thank youAnjali Yakkundiayakkundi@forrester.com@Ayakkundi

5 fundamental A/B testing mistakes

and how to avoid them

Will Browne, Qubitwill.browne@qubitdigital.com

Why do I have to test?

A/B testing is an essential tool

@TheGrahamCooke

• It is the best, simplest way to prove the business value of your digital technology

@TheGrahamCooke

• It is the best, simplest way to prove the business value of your digital technology

• It is the only way to get close to proving cause and effect in what you do

But there are costs associated with A/B testing

• The monthly cost of your testing platform

• The salaries and time of your developers and team

• The lost revenue from implementing changes that do nothing for your site (or actively harm it)

We’re here to show you how to reduce these costs, and increase ROI

• Find the ideas that are likely to have a real positive effect on user behavior (true winners)

• Detect as many true winners as possible

• Minimize the chance of finding tests that look like winners, but have no true effect

Find the ideas that are likely to have a real positive effect on user behavior

Size does matter

Detect as many true winners as possible

We’ve found that tests based on data are

7.7x more effectivethan tests based on intuition alone

(Qubit, 2013)

Imagine testing for weighted coins

(Qubit, 2013)

• How many coin flips would be enough to prove that one coin was weighted?

(Qubit, 2013)

• It is impossible to be 100% certain, but the more data you have, the closer you can get

(Qubit, 2013)

• It is impossible to be 100% certain, but the more data you have, the closer you can get

• We follow scientific standards and wait until there is an 80% chance of detecting a difference

(Qubit, 2013)

Minimize the chance of finding tests that look like winners,but have no true effect

(Qubit, 2013)

• We generally assume that with good testing there is a 5% false positive rate

(Qubit, 2013)

• We can test this using A/A tests (showing both groups the exact thing)

(Qubit, 2013)

• We can test this using A/A tests (showing both groups the exact thing)

• So out of 100 A/A tests, 5 would be winners, despite doing absolutely nothing

Splitting tests too much can lead to false positives

Get a second opinion

Validate your results

We recommend a

95/5 splitto confirm your winning variation is still winning

“Most winning A/B test results are illusory”, Qubit 2014

A worked example: Imagine Mr Bean’s jelly beans shop

@QubitGroup“Most winning A/B test results are illusory”, Qubit 2014

A worked example: Imagine Mr Bean’s jelly beans shop

@QubitGroup“Most winning A/B test results are illusory”, Qubit 2014

• Mr Bean has a 1% conversion rate

• He runs 20 different tests in one year

• There are 2 good tests with uplifts of 5%

• There are also 2 bad tests with uplifts of -5%

TruthCorrect method

Naive method

TruthCorrect method

Naive method

No. of tests 20 20 20

Reported winners 2 3 9

TruthCorrect method

Naive method

After validation 2 2 9

TruthCorrect method

Naive method

After validation 2 2 9

Reported uplift 2 x 5% 2 x ~5%

9 x (5-40)%

Correct method Naive method

Reported uplift 2 x 5% 9 x (5-40)%

Delivered uplift 2 x 5% 2 x 5% + (? x -5%)

Wasted resource None7 unnecessary

Wasted resource None 7 unnecessary tests

Customer experience

Improved Degraded

Your understanding

Improved Confused

Wasted resource None 7 unnecessary tests

Customer experience

Improved Degraded

Your understanding

Improved Confused

Your manager Happy Angry

So, to conclude, those 5 mistakes are:

• Mistake 1: Not A/B testing

• Mistake 2: Not prioritizing tests properly

• Mistake 3: Not waiting for significance and sample size

• Mistake 4: Testing too many things at once

• Mistake 5: Not validating the tests after implementation

Questions you should ask your testing provider

• How do you prioritize tests?

• How do you calculate the sample size required?

• How do you account for the multiple testing problem?

• How do you validate winning tests?

Thank you

If you have any detailed questions please contact

research@qubitdigital.com

Will Browne, Qubitwill.browne@qubitdigital.com

How to run a successful A/B testing program

Melanie KyrklundSenior Ecommerce Analyst

Testing programs have the potential to drive the online customer experience

Understand your customers and respond to their needs

Optimize the user journey

Create personalized experiences

Quickly respond to market pressures

Rapidly align marketing & onsite experience

Validate development items

Every visit represents an opportunity to establish a relationship with a customer

Customer experience

Personalization Persuasion

Targeted offers

User experience,

functionality & content

The reality of running a testing program

For many organizations, testing represents a cultural change with many political and operational barriers to be overcome:

Resource & operational factors

Buy-in / Multiple stakeholders

Alignment with other business priorities and development plans

Instilling a data-driven culture

A lot of testing is required to deliver results

1/5 tests or 1/10 tests will deliver statistically significant wins

A rapid pace of testing is required to build a program that is delivering consistent results month over month

Two factors are critical in building a successful testing program

Approach Speed

MethodologyIdea Generation

Operational ProcessPrioritization

The right approach to optimization

Be inspired – every visit is an opportunity to establish a relationship with a customer

Use data to understand how visitors are using your website, what is working and what isn’t

Customer feedback to understand why visitors are not purchasing on your website

If working over multiple websites & countries, look for insight at local level first and then identify commonalities at a later stage

Persuasion research, understand the psychology of your customers

Supplement with quick UX testing

Thoroughly research & quantify each opportunity (visitors reached, expected uplift and incremental revenue)

UX optimization Personalization Persuasion

Idea Generation

71Friday, 28 February 14

Idea Generation

Speed – Operational Processes

Testing Tool selection : Map out the most efficient implementation process

Operational efficiency Pick managed services (analysis, implementation) as required

Prioritization

Build a test pipeline logging:

Idea & hypothesisExpected uplift & incremental revenue Implementation estimatesTest duration Prioritize tests weekly, to keep up momentum of testing and

focus on biggest wins

Conclusion: A rapid, data-driven approach is required

Testing programs have the potential to drive the online customer experience, however a rapid data-driven approach is required to deliver results

Plan the required resources upfront

Consider ease of use, test time to market and managed services in tool selection

Map out the most efficient end to end process to get tests live

Process supported by data-driven idea generation & prioritization

Thank you

Any questions?

bit.ly/qABresearch

More A/B testing material from Qubit

bit.ly/qABinfographic

@QubitGroup#QubitABtests

Would you bet your job on your A/B test results?

Business

Transcript of Would you bet your job on your A/B test results?

Go from Hating Your Job to Loving Your Job

Your Job and Your Rights

You Can Bet Your Life on God

AlphA bet - ASTRINIastrini-design.pl/wp-content/uploads/2017/09/folder... · 2017. 9. 8. · AlphA bet AlphA bet AlphA bet AlphA bet AlphA bet AlphA bet AlphA bet AlphA bet AlphA

Your Best Bet for IIT-JEEtopperstestprep.com/ebrouchure.pdf · Your Best Bet for IIT-JEE TOPPERS - the Brainchild of Mr. Praveen Kumar, a Civil Engineering graduate from IIT Kharagpur

Why First Contact is Your Best Bet for Driving Technology Adoption

Do You Own Your Job, or Does Your Job Own You?

Keranique Your Best Bet for Getting Back Thicker, Fuller Hair

You Bet Your Life: Why We Need a National Health Program

YOUR BEST BET

Volunteer Development: Your Best Bet for Success Carolyn Dale Linda Steve.

Your first job. Your next job. What you need to know

Neurons I bet your neurons are all fired up now!.

Find and Keep Your Dream Job, the · Your dream job, unfortunately, won't fall into your lap out of the blue one day. That happens, but it's so rare that nobody should ever bet on

Which crowdfunding platform is your best bet

Bet, Call, Check your SEO activity - Lukasz Zelezny #SiGMA2014 Malta

WHEN YOU GAMBLE WITH SAFETY YOU BET YOUR LIFE!€¦ · WHEN YOU GAMBLE WITH SAFETY... YOU BET YOUR LIFE! Virginia Ship repair aSSociation 21St annual Safety & health Seminar Wednesday,

BET ON IT: Design Your University Web Site To Enhance Your Admissions Goals

MARCH 2014 Managing the information that drives the ...docs.media.bitpipe.com/io_11x/io_114834/item... · lutely, indisputably, you-bet-your-job, 100% certain that your disaster recovery

YourBest Bet for Success Volunteer Development: Your Best Bet for Success Originally presented by: Carolyn Ashton, Oregon State University Dale Leidheiser,