A/B testing #eMetrics London
50
-
Upload
alexandra-paun -
Category
Technology
-
view
61 -
download
0
description
product development, a/b testing, case studies
Transcript of A/B testing #eMetrics London
A/B Testing
Alexandra Paun@ToastingIdeas
Pre Test SignificancesA/A Testing
User Base
Control
75%
Test
25%
7/16/1
3
7/17/1
3
7/18/1
3
7/19/1
3
7/20/1
3
7/21/1
3
7/22/1
3
7/23/1
3
7/24/1
3
7/25/1
3
7/26/1
3
7/27/1
3
7/28/1
3
7/29/1
3
7/30/1
3
7/31/1
3
8/1/1
3
8/2/1
3
8/3/1
3
8/4/1
3
8/5/1
3
8/6/1
3
Revenue
TestControl
7/16/1
3
7/17/1
3
7/18/1
3
7/19/1
3
7/20/1
3
7/21/1
3
7/22/1
3
7/23/1
3
7/24/1
3
7/25/1
3
7/26/1
3
7/27/1
3
7/28/1
3
7/29/1
3
7/30/1
3
7/31/1
3
8/1/1
3
8/2/1
3
8/3/1
3
8/4/1
3
8/5/1
3
8/6/1
3
Revenue
TestControl
Pre test significances biased results
The Viral Gate Test
7/22/13 7/23/13 7/24/13 7/25/13 7/26/13 7/27/13 7/28/13 7/29/13 7/30/13 7/31/13 8/1/13 8/2/13 8/3/13 8/4/13 8/5/13
Revenue
ControlTest
7/16/1
3
7/17/1
3
7/18/1
3
7/19/1
3
7/20/1
3
7/21/1
3
7/22/1
3
7/23/1
3
7/24/1
3
7/25/1
3
7/26/1
3
7/27/1
3
7/28/1
3
7/29/1
3
7/30/1
3
7/31/1
3
8/1/1
3
8/2/1
3
8/3/1
3
8/4/1
3
8/5/1
31,100,000
1,150,000
1,200,000
1,250,000
1,300,000
1,350,000
1,400,000
1,450,000
1,500,000
Daily Active Users
ControlTest
7/22/1
3
7/23/1
3
7/24/1
3
7/25/1
3
7/26/1
3
7/27/1
3
7/28/1
3
7/29/1
3
7/30/1
3
7/31/1
3
8/1/1
3
8/2/1
3
8/3/1
3
8/4/1
3
8/5/1
31,260,000
1,280,000
1,300,000
1,320,000
1,340,000
1,360,000
1,380,000
1,400,000
1,420,000
1,440,000
Daily Active Users7 Day Moving Average
ControlTest
7/22/13 7/23/13 7/24/13 7/25/13 7/26/13 7/27/13 7/28/13 7/29/13 7/30/13 7/31/13 8/1/13 8/2/13 8/3/13 8/4/13 8/5/1316.00
16.50
17.00
17.50
18.00
18.50
3 Day Retention
ControlTest
Viral gate
Too good to be true?
The Opportunity Cost Test
Could This Be Making More Money?
User Base
Cheap Plasma
10%
Expensive Plasma
90%
3/6/1
3
3/8/1
3
3/10/1
3
3/12/1
3
3/14/1
3
3/16/1
3
3/18/1
3
3/20/1
3
3/22/1
3
3/24/1
3
3/26/1
3
3/28/1
3
3/30/1
3
4/1/1
3
4/3/1
3
4/5/1
3
4/7/1
3
4/9/1
3
4/11/1
3
4/13/1
3
4/15/1
3
4/17/1
3
4/19/1
3
4/21/1
3
4/23/1
3
4/25/1
3
4/27/1
3
4/29/1
3
5/1/1
3
5/3/1
3
5/5/1
3
5/7/1
3
5/9/1
3
5/11/1
3
5/13/1
3
Revenue
Cheap PlasmaExpensive Plasma
2/28/1
3
3/3/1
3
3/6/1
3
3/9/1
3
3/12/1
3
3/15/1
3
3/18/1
3
3/21/1
3
3/24/1
3
3/27/1
3
3/30/1
3
4/2/1
3
4/5/1
3
4/8/1
3
4/11/1
3
4/14/1
3
4/17/1
3
4/20/1
3
4/23/1
3
4/26/1
3
4/29/1
3
5/2/1
3
5/5/1
3
5/8/1
3
5/11/1
3
5/14/1
30
100,000
200,000
300,000
400,000
500,000
600,000
700,000
Daily Active Users
Cheap PlasmaExpensive Plasma
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
200,000 Plasma bubbles sold
Cheap PlasmaExpensive Plasma
3/6/1
3
3/9/1
3
3/12/1
3
3/15/1
3
3/18/1
3
3/21/1
3
3/24/1
3
3/27/1
3
3/30/1
3
4/2/1
3
4/5/1
3
4/8/1
3
4/11/1
3
4/14/1
3
4/17/1
3
4/20/1
3
4/23/1
3
4/26/1
3
4/29/1
3
5/2/1
3
5/5/1
3
5/8/1
3
5/11/1
3
5/14/1
30
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
100,000
Fire bubbles sold
Cheap PlasmaExpensive Plasma
Question everything
Long Run Testing
How long should I keep this test running for?
Long enough
2/28/1
3
3/2/1
3
3/4/1
3
3/6/1
3
3/8/1
3
3/10/1
3
3/12/1
3
3/14/1
3
3/16/1
3
3/18/1
3
3/20/1
3
3/22/1
3
3/24/1
3
3/26/1
3
3/28/1
3
3/30/1
3
4/1/1
3
4/3/1
3
4/5/1
3
4/7/1
3
4/9/1
3
4/11/1
3
4/13/1
3
4/15/1
3
4/17/1
3
4/19/1
3
4/21/1
3
4/23/1
3
4/25/1
3
4/27/1
3
4/29/1
30
200,000
400,000
600,000
800,000
1,000,000
1,200,000
1,400,000
Daily Active Users
TestControl
3/6/1
3
3/8/1
3
3/10/1
3
3/12/1
3
3/14/1
3
3/16/1
3
3/18/1
3
3/20/1
3
3/22/1
3
3/24/1
3
3/26/1
3
3/28/1
3
3/30/1
3
4/1/1
3
4/3/1
3
4/5/1
3
4/7/1
3
4/9/1
3
4/11/1
3
4/13/1
3
4/15/1
3
4/17/1
3
4/19/1
3
4/21/1
3
4/23/1
3
4/25/1
3
4/27/1
3
4/29/1
318
18.5
19
19.5
20
20.5
21
21.5
22
22.5
1 Day Retention
TestControl
3/6/1
3
3/8/1
3
3/10/1
3
3/12/1
3
3/14/1
3
3/16/1
3
3/18/1
3
3/20/1
3
3/22/1
3
3/24/1
3
3/26/1
3
3/28/1
3
3/30/1
3
4/1/1
3
4/3/1
3
4/5/1
3
4/7/1
3
4/9/1
3
4/11/1
3
4/13/1
3
4/15/1
3
4/17/1
3
4/19/1
3
4/21/1
3
4/23/1
3
4/25/1
3
4/27/1
3
4/29/1
3
Revenue
TestControl
3/6/1
3
3/8/1
3
3/10/1
3
3/12/1
3
3/14/1
3
3/16/1
3
3/18/1
3
3/20/1
3
3/22/1
3
3/24/1
3
3/26/1
3
3/28/1
3
3/30/1
3
4/1/1
3
4/3/1
3
4/5/1
3
4/7/1
3
4/9/1
3
4/11/1
3
4/13/1
3
4/15/1
3
4/17/1
3
4/19/1
3
4/21/1
3
4/23/1
3
4/25/1
3
4/27/1
3
4/29/1
3
Revenue
TestControl
Could this be better?
Why did I not test this?
5/20/1
3
5/21/1
3
5/22/1
3
5/23/1
3
5/24/1
3
5/25/1
3
5/26/1
3
5/27/1
3
5/28/1
3
5/29/1
3
5/30/1
3
5/31/1
3
6/1/1
3
6/2/1
3
6/3/1
3
6/4/1
3
6/5/1
3
6/6/1
3
6/7/1
3
6/8/1
3
6/9/1
30
200,000
400,000
600,000
800,000
1,000,000
1,200,000
Daily Active Users
5/20/1
3
5/21/1
3
5/22/1
3
5/23/1
3
5/24/1
3
5/25/1
3
5/26/1
3
5/27/1
3
5/28/1
3
5/29/1
3
5/30/1
3
5/31/1
3
6/1/1
3
6/2/1
3
6/3/1
3
6/4/1
3
6/5/1
3
6/6/1
3
6/7/1
3
6/8/1
3
6/9/1
30.00%
5.00%
10.00%
15.00%
20.00%
25.00%
1 Day Retention
5/20/1
3
5/21/1
3
5/22/1
3
5/23/1
3
5/24/1
3
5/25/1
3
5/26/1
3
5/27/1
3
5/28/1
3
5/29/1
3
5/30/1
3
5/31/1
3
6/1/1
3
6/2/1
3
6/3/1
3
6/4/1
3
6/5/1
3
6/6/1
3
6/7/1
3
6/8/1
3
6/9/1
30
10,000
20,000
30,000
40,000
50,000
60,000
70,000
New Users
5/20/1
3
5/21/1
3
5/22/1
3
5/23/1
3
5/24/1
3
5/25/1
3
5/26/1
3
5/27/1
3
5/28/1
3
5/29/1
3
5/30/1
3
5/31/1
3
6/1/1
3
6/2/1
3
6/3/1
3
6/4/1
3
6/5/1
3
6/6/1
3
6/7/1
3
6/8/1
3
6/9/1
30
5,000
10,000
15,000
20,000
25,000
New Users Segementation
FeaturingRequests
Make your life easy. Run an A/B test.
• A/A Testing• Too good to be true (?)• Could this be better?• Long run tests• Test everything
Questions?
@ToastingIdeas
