How to Create Winning A/B Tests through Stronger Research
-
Upload
visual-website-optimizer -
Category
Marketing
-
view
816 -
download
0
description
Transcript of How to Create Winning A/B Tests through Stronger Research
![Page 1: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/1.jpg)
How to Create Winning A/B Tests through Stronger
Research
![Page 2: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/2.jpg)
View the video of this webinar here
![Page 3: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/3.jpg)
About Siddharth Deswal
● Senior Marketer at VWO
● Passionate about analytics, optimization and storytelling
● Connect on Twitter @SiddharthDeswal
![Page 4: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/4.jpg)
Research - The Scientific Method
Problems are usually well defined: Increase sales, signups, leads generated, etc.
Neglected part of conversion rate optimization (or just lip-service paid)
HiPPOs rule over here (Highest Paid Person’s Opinion)
Most users focus on these steps
![Page 5: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/5.jpg)
1. Only one in seven tests results in a winner
2. Many people don’t know what to test after the initial experiments
3. Tests with results that don’t move the needle
Why research
![Page 6: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/6.jpg)
Analytics
![Page 7: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/7.jpg)
Analytics - Top sources
![Page 8: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/8.jpg)
Analytics - Understand entry points
![Page 9: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/9.jpg)
Analytics - Mapping exit pages
![Page 10: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/10.jpg)
Analytics - Key pages
![Page 11: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/11.jpg)
Segment for better insights
1) New visitors who don’t bounce
![Page 12: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/12.jpg)
Segment for better insights
1) New visitors who don’t bounce (continued)
![Page 13: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/13.jpg)
Segment for better insights
2) Visitors who behave similar to those who convert
![Page 14: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/14.jpg)
Segment for better insights
2) Visitors who behave similar to those who convert (continued)
![Page 15: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/15.jpg)
Segment for better insights
3) Finally, analyze the visitors who behaved like converters but did not convert by looking at their top sources, entry points and exit pages
![Page 16: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/16.jpg)
Usability Testing
![Page 17: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/17.jpg)
Usability Testing
Putting 5 to 10 users in front of your website or landing pages and observing them can teach you far more than most other forms of research.
Caveat: these users have to be representative of your target customers.
![Page 18: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/18.jpg)
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
Usability Testing - Asking questions
1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed
1) User opens ‘Live Chat’ and asks rep what to do, or2) Takes to Twitter to complain to the brand, or3) Looks for the ‘Returns’ page, or4) Something that none of us were able to anticipate
![Page 19: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/19.jpg)
Usability Testing - Asking questions
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
Useful when trying to understand roadblocks in a specific process but the question has biased the user towards a particular action
Better when you want to understand how users react to problems without any bias
![Page 20: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/20.jpg)
Usability Testing - Measurement
Quantitative measurement1. Number of clicks to solve the problem
2. Time taken to solve the problem
3. Number of page loads to solve the problem
4. Errors while attempting to solve the problem
5. Number of goals completed while solving the problem
![Page 21: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/21.jpg)
Usability Testing - Measurement
Qualitative insights1. What catches the participant’s attention in the first 5/10/15 seconds after the landing page has
loaded
2. How quickly is the headline able to communicate the business’s primary offering
3. How does the participant search for information
4. How convinced is the user with and without trust signs/badges
5. Questions that the user generates while attempting to solve the problem
6. How pleasant the user finds the website
7. Distractions that make participants leave the conversion funnel
8. Points where participants fumble, are confused or have to turn to you for further direction
![Page 22: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/22.jpg)
User Testing - Attensee.com
![Page 23: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/23.jpg)
On-site surveys
![Page 24: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/24.jpg)
On-site surveys
![Page 25: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/25.jpg)
On-site surveys - SaaS
Page Question Insights drawn
Pricing / Features / Trial-Signup What’s stopping you from signing up with us?
Common concerns, objections, anxieties
Top exit pages Is there something you were looking for that you couldn’t find on the website?
Reasons that users leave, Information that they needed to make a decision but couldn’t find
Top landing pages / Homepage How familiar are you with [business offering]?
Understand how much convincing/explaining do your visitors need
Top landing pages / Homepage Where did you come to know of us?
Top marketing channels
![Page 26: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/26.jpg)
On-site surveys - SaaS
![Page 27: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/27.jpg)
On-site surveys - eCommerce
Page Question Insights drawn
Product page What’s stopping you from adding this product to cart?
Common concerns, objections, anxieties or ‘just looking’
Top exit pages (product or category)
Is there something you were looking for that you couldn’t find on the website?
Reasons that users leave, Information that they needed to make a decision but couldn’t find
Top landing pages / Homepage Where did you come to know of us?
Top marketing channels
Checkout / Cart What’s stopping you from buying this product?
Why are visitors not completing purchases in spite of adding to cart
![Page 28: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/28.jpg)
On-site surveys - Exit intent
![Page 29: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/29.jpg)
On-site surveys - Exit intent
WAITTell Us Why You’re Leaving And Get A 10% Discount Coupon On Your Next
Purchase
I’m leaving because● I don’t want to create an account before buying● I found a better price elsewhere● I can’t find a coupon code● I don’t see free-shipping● I was just browsing● Other
![Page 30: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/30.jpg)
Visual Analysis
![Page 31: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/31.jpg)
Visual Analysis - Heatmaps
![Page 32: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/32.jpg)
Heatmaps case study - original
![Page 33: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/33.jpg)
Heatmaps case study - Heatmap
![Page 34: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/34.jpg)
Heatmaps case study
![Page 35: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/35.jpg)
Heatmaps case study - Result
25% increase in clicks on app store buttons!
![Page 36: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/36.jpg)
Visual Analysis - Scrollmaps
![Page 37: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/37.jpg)
Hypothesis
![Page 38: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/38.jpg)
Hypothesis
Research
Hypothesis
Image credithttp://www.pasosadelantepv.com/evaluation-treatment.html
![Page 39: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/39.jpg)
Hypothesis Framework
The Change:Effect model
Image credit http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
![Page 40: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/40.jpg)
Hypothesis Framework
The Change: Effect model depends on
1. Presumed problem
2. Proposed solution
3. Anticipated result
![Page 41: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/41.jpg)
Hypothesis Framework
The Change: Effect model
Image credit http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
![Page 42: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/42.jpg)
Putting it all together1. Start by creating a segment that shows only those visitors who came from your largest or most
important traffic source
2. Get a list of the top landing and exit pages for this traffic source
3. Run user tests where representative participants are asked to solve a problem that takes them
through these pages
4. Make note of the key concerns, objections and difficulties expressed by test participants
5. Run on-site surveys on the key pages
![Page 43: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/43.jpg)
Putting it all together6. Again, make a note of the concerns, objections and reasons given by respondents
7. Run heatmaps and scrollmaps on your key pages to understand visitor behavior in their natural
environment
8. In our experience, once you’ve completed the analysis of all the previous steps, the problems on
your landing pages and websites automatically jump out, i.e. they become so obvious that
everyone is easily able to identify them
9. Create hypotheses that attempt to fix these problems
10. Run your A/B tests!
![Page 44: How to Create Winning A/B Tests through Stronger Research](https://reader033.fdocuments.in/reader033/viewer/2022051323/547e68bcb379596a2b8b5485/html5/thumbnails/44.jpg)
Thank You!