How to Run AB Tests that Get REAL Results
-
Upload
kissmetrics-on-slideshare -
Category
Technology
-
view
7.416 -
download
0
Transcript of How to Run AB Tests that Get REAL Results
![Page 1: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/1.jpg)
How to Run A/B Tests that Get REAL Results
Lars LofgrenMarketing Analyst - July 2012 [email protected] - Confidential - Do not distribute
![Page 2: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/2.jpg)
@larslofgren #KISSwebinar
![Page 3: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/3.jpg)
We’ll cover...
The wrong way to A/B test1
5 places to get ideas for testing2
Get the most from your A/B testing tools3
![Page 4: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/4.jpg)
HOW MOST PEOPLEA/B Test
![Page 5: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/5.jpg)
A/B testing is totally rad! We should do it!
![Page 6: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/6.jpg)
Everyone loves button colors
VS
![Page 7: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/7.jpg)
Next up? Layouts!
![Page 8: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/8.jpg)
What about photos?
VS
![Page 9: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/9.jpg)
And conversions went up by 4%! Party time!
![Page 10: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/10.jpg)
There’s a better way...
Testing random elements is very inefficient. You will not find the big
wins this way.
![Page 11: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/11.jpg)
You’re also not learning
Red vs green buttons doesn’t help you understand your customer.
![Page 12: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/12.jpg)
GETQualitative
![Page 13: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/13.jpg)
Qualitative data definition
Data that isn’t a number.
![Page 14: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/14.jpg)
Start with qualitative insights
Use other sources of data to get a basic understanding of what your
customers want.
![Page 15: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/15.jpg)
Understand first, then do the A/B test
You’ll focus on the right tests and increase your conversions a LOT
faster.
![Page 16: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/16.jpg)
The A/B Testing Process
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
![Page 17: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/17.jpg)
Why is this process better?
Helps you find the 40% increase in conversions instead of only the 4%.
![Page 18: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/18.jpg)
WHERE IS THE QUALITATIVEData?
![Page 19: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/19.jpg)
Surveys
Location #1
![Page 20: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/20.jpg)
Use one-question surveys on your site
![Page 21: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/21.jpg)
Also try long surveys
![Page 22: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/22.jpg)
Survey best practices
Keep it short1
Only ask for answers that you’ll use2
Start with open-ended questions3
![Page 23: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/23.jpg)
Feedback
Location #2
![Page 24: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/24.jpg)
Do you have a feedback email?
![Page 25: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/25.jpg)
They know exactly what problems keep coming up.
Talk to your support team
![Page 26: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/26.jpg)
They talk to customers every day.
Also talk to your sales team
![Page 27: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/27.jpg)
Reach Out
Location #3
![Page 28: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/28.jpg)
Get them on the phone and deeply understand how they use your
product.
Nothing beats talking to customers personally
![Page 29: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/29.jpg)
If local, take them out to lunch or coffee.
![Page 30: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/30.jpg)
Focus on asking good questions and listening.
This is not a pitch
![Page 31: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/31.jpg)
User Activity
Location #4
![Page 32: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/32.jpg)
You’ll need customer analytics for this.
Look at activity of individual people
![Page 33: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/33.jpg)
KISSmetrics People report
![Page 34: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/34.jpg)
Confirm by looking at other people.
When you have an insight...
![Page 35: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/35.jpg)
Usability Tests
Location #5
![Page 36: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/36.jpg)
Have someone use your site or product while you watch.
What’s a usability test?
![Page 37: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/37.jpg)
Grab someone off the street
![Page 38: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/38.jpg)
Amazon’s Mechanical Turk - mturk.com
![Page 39: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/39.jpg)
UserTesting.com
![Page 40: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/40.jpg)
Places to Find Qualitative Insights
Surveys1
Feedback2
Reach Out3
User Activity4
Usability Tests5
![Page 41: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/41.jpg)
The A/B Testing Process
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
![Page 42: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/42.jpg)
After you’ve found the big wins and you have a full-time growth team.
When should you test the small stuff?
![Page 43: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/43.jpg)
A/B TESTINGTools
![Page 44: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/44.jpg)
Great for testing copy, photos, and layouts. Terrible for bigger tests like
pricing.
A/B testing tools only measure the next step
![Page 45: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/45.jpg)
Critical for measuring how your changes impact customers over the
long term.
Tie A/B tests back to the customer life-cycle
![Page 46: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/46.jpg)
How do we actually do this?
![Page 47: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/47.jpg)
First, make sure you’re using customer analytics
![Page 48: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/48.jpg)
Then use Visual Website Optimizer or Optimizely
OR
![Page 49: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/49.jpg)
Now all your A/B testing data is in KISSmetrics
![Page 50: How to Run AB Tests that Get REAL Results](https://reader035.fdocuments.in/reader035/viewer/2022062219/55d52fe1bb61eb8c0b8b46f1/html5/thumbnails/50.jpg)
For a demo of KISSmetrics or to learn how KISSmetircs can help grow your
business, contact Ben Sardella at: