JSConf Budapest - Impostor Syndrome, am I suffering enough to talk about it?
The anatomy of an A/B Test - JSConf Colombia Workshop
-
Upload
eduardo-shiota-yasuda -
Category
Technology
-
view
2.253 -
download
0
Transcript of The anatomy of an A/B Test - JSConf Colombia Workshop
![Page 1: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/1.jpg)
A/B testing workshop “In God we trust, all others must bring data”
JSConf Colombia Workshop 2015
![Page 2: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/2.jpg)
@shiota github.com/eshiota
slideshare.net/eshiota eshiota.com
![Page 3: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/3.jpg)
![Page 4: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/4.jpg)
![Page 5: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/5.jpg)
A/B
![Page 6: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/6.jpg)
A/B tests measure how a new idea (version B/variant/test) performs agains an existing implementation (version A/base/control).
![Page 7: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/7.jpg)
![Page 8: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/8.jpg)
![Page 9: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/9.jpg)
Buy now Buy nowversus
![Page 10: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/10.jpg)
coin flip
Buy now
Buy now
50%
50%
![Page 11: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/11.jpg)
When the user sees or is affected by the idea, they are tracked and become part of the test.
![Page 12: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/12.jpg)
Buy now
Buy now
track(my_experiment)
![Page 13: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/13.jpg)
Data about the website is generated as users browse through pages and do their tasks.
![Page 14: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/14.jpg)
product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…
![Page 15: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/15.jpg)
When there’s enough information to make a decision, you can either stop the test (keeping version A) or choose version B, directing all traffic to it.
![Page 16: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/16.jpg)
Buy now Buy now
Duration: 14 days Visitors: 45.140 (22.570 per variant)
339 (1.5%) 407 (1.8%)
20% up
144.500 COP 147.390 COP
2% up
Number of purchases:
Average price:
![Page 17: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/17.jpg)
coin flip
Buy now
Buy now
50%
50%
![Page 18: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/18.jpg)
B
Buy now
100%
![Page 19: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/19.jpg)
"But my design is obviously more beautiful and intuitive than what we have now! Why should I run an A/B test?” — the majority of designers
![Page 20: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/20.jpg)
Quiz time!(prizes included)
![Page 21: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/21.jpg)
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?
![Page 22: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/22.jpg)
Reduced bounce rate in 1.7%
![Page 23: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/23.jpg)
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?
![Page 24: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/24.jpg)
Which performed better?
Increased CTR in 203%
![Page 25: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/25.jpg)
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?
![Page 26: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/26.jpg)
Which performed better?
43.4% more purchases
![Page 27: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/27.jpg)
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?
![Page 28: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/28.jpg)
Both were statistically equivalent
Which performed better?
![Page 29: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/29.jpg)
Intuition vs
Historical Analysis vs.
Experimentation
![Page 30: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/30.jpg)
We have a 2/3 chance of being wrong when trusting our intuition.
![Page 31: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/31.jpg)
People behave differently each season/month/day of the week.
![Page 32: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/32.jpg)
Different cultures lead to different patterns of usage.
![Page 33: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/33.jpg)
Data analysis alone provides correlation but not causation.
![Page 34: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/34.jpg)
![Page 35: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/35.jpg)
Running your A/B test(in 5 simple steps)
![Page 36: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/36.jpg)
Step 1: Hypothesis
![Page 37: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/37.jpg)
Analyse all possible inputs to come up with an hypothesis to work on.
![Page 38: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/38.jpg)
• Usability research • Benchmarking • Surveys • Data mining • Previous experiments
![Page 39: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/39.jpg)
Hypothesis:
“If users from South America countries relate more to the website, they will book more.”
![Page 40: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/40.jpg)
Step 2: Idea
![Page 41: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/41.jpg)
Idea:
“If we add the country’s flag next to the website’s logo, users will relate more to the brand.”
![Page 42: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/42.jpg)
![Page 43: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/43.jpg)
![Page 44: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/44.jpg)
Step 3: Setup
![Page 45: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/45.jpg)
• Who will participate? • What is the primary metric? • Any secondary impacts? • How will it be implemented?
![Page 46: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/46.jpg)
• Users from Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Guyana, Paraguay, Peru, Suriname, Uruguay and Venezuela, on all platforms
• Conversion (net bookings) uplift is expected • We expect more returning customers
![Page 47: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/47.jpg)
<h1 class="main-header__logo logo"> <% if user.is_from_south_america && track_experiment(:header_flag_for_south_america) == "b" %> <span class="main-header__logo__country-flag"> <%= user.country %> </span> <% end %> <%= image_tag "logo.png" %> </h1>
![Page 48: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/48.jpg)
Step 4: Monitoring
![Page 49: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/49.jpg)
Keep checking the metrics to see if anything’s terribly wrong.
![Page 50: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/50.jpg)
Avoid checking too often, let your test get enough users and enough runtime.
![Page 51: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/51.jpg)
![Page 52: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/52.jpg)
Step 5: Data, decisions, and next steps
![Page 53: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/53.jpg)
When you reach the expected runtime, number of visitors or effect, look at the data and take a decision.
![Page 54: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/54.jpg)
product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…
![Page 55: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/55.jpg)
Optimizely dashboard
![Page 56: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/56.jpg)
• How were the primary and secondary metrics impacted?
• What were the results isolated by each country?
• What were the results isolated by each language?
• Did any particular platform (desktop, mobile devices, tablets) perform better?
• Was the impact on returning customers any higher than first time visitors?
![Page 57: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/57.jpg)
Based on the gathered data, plan for next steps.
![Page 58: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/58.jpg)
• Should we add a copy to the flag? • Should we add a tooltip to the flag? • Should we increase/decrease the flag size? • Should we restrict it just for desktop users? • Should we try this for a single country, or
other countries?
![Page 59: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/59.jpg)
What can you test?
![Page 60: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/60.jpg)
(almost) Everything.
![Page 61: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/61.jpg)
You can test a small design change.
![Page 62: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/62.jpg)
versus
![Page 63: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/63.jpg)
You can test large design changes.
![Page 64: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/64.jpg)
versus
![Page 65: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/65.jpg)
You can test different copies.
![Page 66: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/66.jpg)
versus
Submit
Book now
![Page 67: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/67.jpg)
You can test technical improvements and measure page load time, repaints/reflows, and conversion impact.
![Page 68: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/68.jpg)
versus
jQuery 1.11.3
jQuery 2.1.3
![Page 69: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/69.jpg)
You can even test back-end optimisations and measure page load time, rendering time, CPU and memory usage etc.
![Page 70: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/70.jpg)
if track_experiment(:my_optimized_query) @users = my_optimized_query else @users = do_the_normal_thing end
![Page 71: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/71.jpg)
Live coding(I hope that works.)
![Page 72: The anatomy of an A/B Test - JSConf Colombia Workshop](https://reader034.fdocuments.in/reader034/viewer/2022051520/58e4a2271a28aba3458b62a1/html5/thumbnails/72.jpg)
Find the code at:
https://github.com/eshiota/ab_workshop
Additional links:
https://www.optimizely.com/ https://github.com/splitrb/split/
http://whichtestwon.com http://unbounce.com/
http://blog.booking.com/hamburger-menu.html http://blog.booking.com/concept-dne-execution.html
Gracias!