The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk
-
Upload
manuel-da-costa -
Category
Data & Analytics
-
view
629 -
download
0
description
Transcript of The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk
![Page 1: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/1.jpg)
The Sensationalisation of A/B TestingAnd why it doesn’t work for you
#MEASUREFEST
8th October 2014
![Page 2: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/2.jpg)
Manuel da Costa
@digital_tonic
DIGITAL TONIC
![Page 3: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/3.jpg)
We have a problem with A/B testing
#MEASUREFEST
![Page 4: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/4.jpg)
Let’s see how good you are at A/B testing
#MEASUREFEST
![Page 5: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/5.jpg)
#MEASUREFEST
CONTROL
![Page 6: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/6.jpg)
#MEASUREFEST
VARIATION
![Page 7: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/7.jpg)
#MEASUREFEST
VARIATION
25% increase in conversions
![Page 8: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/8.jpg)
#MEASUREFEST
![Page 9: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/9.jpg)
#MEASUREFEST
![Page 10: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/10.jpg)
#MEASUREFEST
Clickthroughs 606% and Purchases 11%
![Page 11: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/11.jpg)
#MEASUREFEST
![Page 12: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/12.jpg)
![Page 13: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/13.jpg)
![Page 14: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/14.jpg)
#MEASUREFEST
![Page 15: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/15.jpg)
What we don’t see
What was the sample size?
How long did they run the test for?
Significance of results?
What was the target segment for the test?
Did they re-test ?
![Page 16: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/16.jpg)
THROWINGcrap against the wall hoping it sticks
#MEASUREFEST
![Page 17: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/17.jpg)
#MEASUREFEST
AB/ TESTING
![Page 18: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/18.jpg)
You need aFramework
#MEASUREFEST
![Page 19: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/19.jpg)
#MEASUREFEST
![Page 20: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/20.jpg)
Observe
#MEASUREFEST
Start here
Your customers
![Page 21: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/21.jpg)
Identify the leaks in your funnel
![Page 22: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/22.jpg)
![Page 23: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/23.jpg)
![Page 24: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/24.jpg)
Get customer insights
![Page 25: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/25.jpg)
Brainstorm
#MEASUREFEST
We believe that changing X for Y customers will result in ZX = variables you want to test
Y= target segment you want to run the test on Z = the outcome i.e. change conversions or user behaviour
![Page 26: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/26.jpg)
Prioritise your test ideas
Ease of testingImpact on bottom line
![Page 27: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/27.jpg)
#MEASUREFEST
AB/ TESTING
Select your sample size
Set your test duration
QA your test
Integrate with analytics
![Page 28: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/28.jpg)
Don’t stop your test too early
![Page 29: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/29.jpg)
Review & Iterate
#MEASUREFEST
Segment your data!!!
Statistical significance http://abtestguide.com/calc/
Re-test
What did you learn?
![Page 30: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/30.jpg)
A B/ Testing
ObserveYour customers
BrainstormReview & Iterate
#MEASUREFEST
![Page 31: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/31.jpg)
Need inspiration for your A/B Testing?
LOOK AT YOUR DATA
#MEASUREFEST
![Page 32: The sensationalisation of A/B Testing & Why it doesn't work for you - Measurefest October 2014 talk](https://reader033.fdocuments.in/reader033/viewer/2022060118/558a379fd8b42aaa728b46ec/html5/thumbnails/32.jpg)
http://bit.ly/digitaltonicmeasurefest
Interested in more Conversion Optimisation stuff?Get the slides, resources and more
#MEASUREFEST
Follow me on twitter : @digital_tonic Drop me a line : [email protected]