A/B Testing - What your mother didn't tell you
-
Upload
curtis-poe -
Category
Marketing
-
view
5.737 -
download
3
description
Transcript of A/B Testing - What your mother didn't tell you
P.O.P. Copyright 2013, All Around The World January 14, 2014
A/B TesAng
What your mother didn’t tell you hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
What is A/B TesAng?
• TesAng customers, not code • Show different versions • Pick the best one
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Who Relies On A/B TesAng?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Transform Your Company
• Test your customers, not your code • Stop guessing what works
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Should your paycheck depend on my hunch?
Conversion Funnel
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Conversion Funnel
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
→
A Basic Test
• Your current behavior is the base • A variant is a new version (just one change) • You can have one or more variants
• But we assume only one for this talk
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
10,017 Visits 9,983 Visits
125 Conversions 132 Conversions
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
9,950 Visits 10,550 Visits
125 Conversions 155 Conversions
The Numbers (95% Confidence)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
Version Visits Conversion Rate Not Random
Red 10,017 125 1.25% —
Blue 9,983 132 1.32% 51%
The Numbers (95% Confidence)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
Version Visits Conversion Rate Not Random
Red 9,950 125 1.26% —
Blue 10,550 155 1.47% 94%
Which Sells More Product?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Which Creates More Profit?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
What Confidence Level?
Is 99% beDer than 90%?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
The Cycle
Create Experiment
Only one difference per version!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Roll the Dice
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Track Performance
Sta:s:cs
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Choose the Best
Pa:ence
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Create Experiment
Wash, rinse, repeat
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Choose the Best
• Ignore the first day • Full-‐business cycle • What if you’re wrong?
• False posiAves
hDp://www.einarsen.no/is-‐your-‐ab-‐tesAng-‐effort-‐just-‐chasing-‐staAsAcal-‐ghosts/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
LimitaAons of A/B TesAng
• Doesn’t tell you why • Side effects • Doesn’t tell you loyalty
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Benefits
• Hyundai: 62% increase in test drive requests • Artsy Editor: 47% increase in click-‐throughs • Visual Website OpAmizer: 50% signup increase • AssessmentDay: 62% increase in sales • SmileyCookie: 41% increase in sales • BaseKit: 25% increase in conversions • Mobal: 27% increase in sales • You Need A Budget: 85% increase in downloads
hDp://visualwebsiteopAmizer.com/case-‐studies.php
What Are They TesAng?
• Add-‐to-‐cart buDon placement • Call to acAon buDons • Using popup forms • Removing content • Link color • Adding tesAmonials • NavigaAon menus • Removing signup forms
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Gemng Started
• Vendor or in-‐house? • Go for quick wins • Run an AA test • Be pa:ent!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Problems
Ego January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Problems
• Forgemng caching • Non-‐random selecAon
• Forgemng selecAon
• TesAng only the view • Performance
• Dead code • Profit versus conversion
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Tips
• Most experiments fail • Don’t roll the dice prematurely
• A/B test bugs • Test “stupid” ideas • Don’t require rollouts!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
SegmentaAon
• New versus returning customers • Different browsers • Accept-‐Language • Time of day
• IP country • Gender? Age? Something else?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
See Also
• The MulA-‐Armed Bandit hDp://stevehanov.ca/blog/index.php?id=132
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Conclusion
• Lots of money lying on the table • Egos are bad • Data is good • Customer behavior trumps expert opinion
• Customer behavior trumps code behavior
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
QuesAons?
hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/