Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments...
Transcript of Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments...
![Page 1: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/1.jpg)
A/B Tests in Google Analytics To Make UI
Decisions
![Page 2: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/2.jpg)
David SchlotfeldtCo-owner of Plaudit Design
Who do you think you are!?
![Page 3: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/3.jpg)
What’s this about?
“A/B Tests in Google Analytics to Make UI Decisions”First, a couple comments on the title...
● Applicable beyond “UI” - making any decisions you can test on your website (e.g. price of a product)
● “A/B” but beyond “A” and “B” to allow for many variations
![Page 4: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/4.jpg)
Covered in This Presentation
● What “AB Tests” Are
● Running an Experiment
○ Howto Setup an Experiment in GA
○ Reading Results of an Experiment During and After
![Page 5: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/5.jpg)
What “AB Tests” Are
![Page 6: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/6.jpg)
Explaining, by explaining why
Modified questions on contact form
![Page 7: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/7.jpg)
Why an increase?
Because my form change frickin rocked!!
..or…
● We are in a seasonal industry● Sales department picked up involvement in expos● Started a large social marketing campaign● Many reasons!
If we could see what would have happened if we didn’t make the change...
![Page 8: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/8.jpg)
Same blue line...
Modified questions on contact form
![Page 9: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/9.jpg)
AB Tests
● Allow you to compare with to without
● Split users into the control group (they see the original) and those who will see changes (variations)
● Isolate the effects of the change○ Everything beyond the change we care to measure
modifies the results of each segment equally.
![Page 10: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/10.jpg)
Types of TestsA/B - Testing two variations of a page.
Multivariate - Testing various combinations of components on a single pagee.g. 5 different images x 4 different headlines = 20 variations
A/B/N - Testing two or more variations of a page.
== A/B/N model == Experiments
![Page 11: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/11.jpg)
Running an Experiment
![Page 12: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/12.jpg)
Example
Will be using a real example throughout this presentation
AB test selected because it is:● Recent● Small
Perfect for after presentation group discussion
![Page 13: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/13.jpg)
Example
Goal: Increase sales leads
Metric: Online contact form submissions
Identified: A low number of users who view the first step of the contact form move on to the second step
![Page 14: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/14.jpg)
![Page 15: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/15.jpg)
![Page 16: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/16.jpg)
... ContinuedWe believe the first step is overwhelming users. It contains a large number of options and each has a paragraph explanation.
Hypothesis:If we simplify and decrease the number of options a user chooses between at any given time then form submissions will increase.
![Page 17: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/17.jpg)
Steps to running an experiment:
1. Plan2. Create Variations3. Configure Experiment4. Let it Run5. Analyze Results
![Page 18: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/18.jpg)
1) Plan
![Page 19: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/19.jpg)
What variations will be tested?Original always one variation
When testing multiple changes at once attempt to have a variation per combination
How many variations?
● Consider how long you want the experiment to run for and the traffic available when selecting number of variations.
● Up to 10 variations per experiment in Google Analytics (According to docs but you can configure 35 + the original.)
![Page 20: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/20.jpg)
How to evaluate?We’ll expand on the following as we walk through configuring an experiment.
Need to decide:● Metric that will be used to evaluate the results
○ Page viewed, session duration, bounce, etc.● Percentage of traffic to include in the experiment● Distribute traffic evenly across variants (yes or no)
Control length of experiment:● Specify minimum time the experiment will run● Minimum confidence threshold to be achieved
before GA declares a winner
![Page 21: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/21.jpg)
Our ExampleEvaluate: Increase contact form submissions
VariationsVariation 1 - Present options in multiple steps and remove descriptions. Doing so we believe users won’t become overwhelmed and their level of commitment will increase with each selection they make.
NOTE: Ideally we would have ran multiple variations to test multiple changes, but as noted earlier multiple variations increases timeline and best to minimize changes per variation.
![Page 22: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/22.jpg)
![Page 23: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/23.jpg)
![Page 24: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/24.jpg)
2) Create Variations
![Page 25: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/25.jpg)
Develop the Variations● HTML, CSS, blah, tech stuff, blah
● Each variation will either be○ Their own page○ Differences contained on one page
● Which approach you use depends on how you integrate (More on this later)
![Page 26: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/26.jpg)
3) Configure Experiment
![Page 27: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/27.jpg)
Create Experiment1. Login2. “Reports” (Top)3. “Behavior” > “Experiments” (Left side)4. Click “Create experiment” (Right side)
![Page 28: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/28.jpg)
![Page 29: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/29.jpg)
![Page 30: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/30.jpg)
Percentage of users to include in experiment.
Those not included will see the original.
When deciding consider:● Volume of traffic to page● Number of variation● Riskiness of the changes
![Page 31: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/31.jpg)
![Page 32: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/32.jpg)
Multi-armed bandit experiments● “Distribute traffic evenly across all variants”
○ “off” by default○ “on” GA does not adjust traffic dynamically based on variation
performance
● GA uses multi-armed bandit for experiments which has the goal of finding best performing and adjust distribution of users as experiment progresses
● Twice per day adjusts the fraction of traffic that each variation will receive going forward.
● Adjustments made using a statistical formula that considers sample size and performance metrics together.
● You benefit by traffic being moved towards winning variations gradually, instead of having to wait for a "final answer"
![Page 33: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/33.jpg)
Multi-armed bandit experiments
![Page 34: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/34.jpg)
![Page 35: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/35.jpg)
Two Options
Options presented are:1. “Manually insert the code”2. “Send the code to webmaster”
These options are:● Easy to implement● Simply copy and paste Javascript● Code will select a variation for each user and
redirect them to the URL of the variation.
![Page 36: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/36.jpg)
Oh wait! There are two more options!
Additional options not presented in wizard
● Both require programming
● Both do not redirect to a separate page
● URL fields for variations can be fake. E.g. “1.com”, “2.com”. Validation errors can be safely ignored
Both are more involved so I’m only introducing them here.
![Page 37: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/37.jpg)
Option 3: Javascript API Client SideAdvanced Javascript Implementation
More Info: https://developers.google.com/analytics/solutions/experiments-client-side
<!-- Load the Content Experiment JavaScript API client for the experiment -->
<script src="//www.google-analytics.com/cx/api.js?experiment=YOUR_EXPERIMENT_ID"></script>
<script>
// Ask Google Analytics which variation to show the user.
var chosenVariation = cxApi.chooseVariation();
</script>
![Page 38: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/38.jpg)
● Requires backend coding - PHP, Java, .net, etc. (Google provides great documentation on exactly how to code this.)
● No delay/flash in page load/initialization waiting for JS
● Able to test more - e.g. change price of a product
Option 4 : Server Side
![Page 39: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/39.jpg)
Option 4 : Server Side1. Store and Refresh - Periodically update local data about
experiments
2. Choose Variationa. For the current user interaction (e.g. pageview), is there an
experiment running?b. Has the user been previously exposed to this experiment?c. Should the user be included in the experiment?d. Choose a variation for the user
3. Send experiment and select variation to Google Analytics (Normally injecting JavaScript into the page)
More Info: https://developers.google.com/analytics/solutions/experiments-server-side
![Page 40: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/40.jpg)
4) Let it Run
![Page 41: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/41.jpg)
Let it run!● Last step of wizard: Click “Start Experiment”
● Automatically stops when clear winner or hits 3 months max
List of Running Experiments:
View of an Experiment:...
![Page 42: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/42.jpg)
View of an experiment
![Page 43: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/43.jpg)
Once start you can modify
● Name of experiment and variation pages
● Percentage of users in the experiment pages
● If email notifications are sent
● Variations can be disabled.○ e.g. If you feel one of your variations is clearly under
performing.○ Will shorten experiment since more users split between less
variations.
● Stop an experiment. Keep in mind it cannot be restarted
![Page 44: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/44.jpg)
5) Analyze Results
![Page 45: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/45.jpg)
It finished! (Or you got tired of waiting…)
Now what?
● Original page only - Users will no longer be presented variations
● Best variations - If a variation won, modify the site
● If variations existed at different URLs, set up permanent redirects to new page for those who bookmarked a variation
● Run another test! With more insight from that experiment consider running a new one for additional improvement
![Page 46: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/46.jpg)
Analyzing Beyond Who Won● Create segment to analyze behavior of the users who were
served a specific variation
● Can use segment to analyze and compare any report in Google Analytics○ How they used the site: Behavior reports (e.g. Users Flow,
Behavior Flow, etc)○ How they found the site: Acquisition reports (e.g.
Channels, AdWords, etc)
More Information: https://support.google.com/analytics/answer/4380401?hl=en&ref_topic=1745208
![Page 47: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/47.jpg)
Wrapping UpThe Preso
![Page 48: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/48.jpg)
Summary● Experiments allow you to isolate the effect of changes● Google Analytics uses A/B/N model
● The process to running an experiment:1. Plan
a. What variations will be tested?b. How to evaluate?
2. Create Variationsa. Development
3. Configure Experimenta. Setup in Google Analyticsb. Configure on site with “four” potential approaches
i. “Manually insert the code”ii. “Send the code to webmaster”iii. Javascript API Client Sideiv. Server Side <-- My preference
4. Let it Runa. Watch reports.b. Modify if needed including disabling variations.
5. Analyze Resultsa. Put winner in place.b. Use segments to further analyze
![Page 49: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/49.jpg)
Critique - Room for improvement!What changes could be tested in the next experiment to further improve conversions?
Process1. In small groups brainstorm ideas2. Share ideas with the larger group3. As a group select top ideas4. Define variations
![Page 50: Decisions Analytics To Make UI A/B Tests in Google · Make UI Decisions” First, a couple comments on the title... Applicable beyond “UI” - making any decisions you can test](https://reader034.fdocuments.in/reader034/viewer/2022050204/5f57eec6abb4af7c2229ec17/html5/thumbnails/50.jpg)
The end!Until next time