15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL...
-
Upload
marlee-jopling -
Category
Documents
-
view
216 -
download
0
Transcript of 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL...
![Page 1: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/1.jpg)
![Page 2: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/2.jpg)
15 A/B testing mistakes I see all the damn time (and what to do about it)
by Peep Laja, ConversionXL Call to Action Conference 2014
![Page 3: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/3.jpg)
“You should test that!”
![Page 4: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/4.jpg)
![Page 5: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/5.jpg)
“Dude wtf?”
![Page 6: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/6.jpg)
@peeplaja
Typical scenario: A company runs 100 tests /
year. After 12 months their
conversion rate is still the same.
![Page 7: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/7.jpg)
@peeplaja
Testing mistake #1:Precious time
wasted on stupid tests
![Page 8: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/8.jpg)
@peeplaja
Testing mistake #2:
You don’t have a data-driven,
learning-oriented hypothesis
![Page 9: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/9.jpg)
@peeplaja
Testing mistake #3:You think you
know what will work
![Page 10: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/10.jpg)
@peeplaja
Testing mistake #4:You run tests
when you have no traffic
![Page 11: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/11.jpg)
@peeplaja
Testing mistake #5:
Your absolute conversions are
way too low
![Page 12: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/12.jpg)
![Page 13: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/13.jpg)
@peeplaja
You need *at least* 250-400 conversions PER
variation.
Way more if you want to segment the data
![Page 14: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/14.jpg)
@peeplaja
Our "thumb rule" is: 3000-4000 conversions per variation and 3-4 week test duration. That is enough traffic so we can even talk about valid data if we drill down into segments.
Testing "sin" nr.1: search in segments for uplifts although you have no statistical validity, e.g. 85 versus 97 conversions. That’s bullshit.
![Page 15: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/15.jpg)
@peeplaja
Testing mistake #6:
Your tests don’t run long enough
![Page 16: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/16.jpg)
![Page 17: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/17.jpg)
@peeplaja
Testing mistake #7:
You don’t test full weeks at a time
![Page 18: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/18.jpg)
![Page 19: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/19.jpg)
@peeplaja
Testing mistake #8:
Test data is not sent to third-party
analytics
![Page 20: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/20.jpg)
![Page 21: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/21.jpg)
@peeplaja
Pro tip: If you have over 500,000
pageviews / mo, you don’t (necessarily) need to spend
$100,000 on GA Premium. Get unsampled data with Analytics
Canvas
![Page 22: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/22.jpg)
@peeplaja
Testing mistake #9:
You give up after your first test for
a hypothesis fails.
![Page 23: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/23.jpg)
![Page 24: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/24.jpg)
Full case study: http://conversionxl.com/case-study-how-we-improved-landing-page-conversion/
79.3% uplift in conversions
![Page 25: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/25.jpg)
@peeplaja
Testing mistake #11:
You’re not aware of validity threats
![Page 26: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/26.jpg)
@peeplaja
History effectInstrumentation effect
Selection effect
![Page 27: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/27.jpg)
@peeplaja
Testing mistake #12:
You’re ignoring small gains
![Page 28: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/28.jpg)
@peeplaja
5% monthly increase in your conversion rate will result in
80% uplift for the year. That’s how math works.
![Page 29: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/29.jpg)
@peeplaja
Testing mistake #13:
You don’t know html/css/jquery -
and thus only test the simple stuff
![Page 30: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/30.jpg)
@peeplaja
Testing mistake #14:
You’re not running tests at all times
![Page 31: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/31.jpg)
@peeplaja
Testing mistake #15:
You’re late for your lunch break!
![Page 32: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/32.jpg)
Follow me @peeplaja
@peeplaja
Keep reading ConversionXL!
Thank you
![Page 33: 15 A/B testing mistakes I see all the damn time (and what to do about it) by Peep Laja, ConversionXL Call to Action Conference 2014.](https://reader036.fdocuments.in/reader036/viewer/2022070415/56649cad5503460f9496fc4b/html5/thumbnails/33.jpg)