Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)
-
Upload
andrew-capland -
Category
Marketing
-
view
379 -
download
0
Transcript of Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)
![Page 1: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/1.jpg)
Video A/B Testing Deconstructed: What? How? Where?
(with REAL Data & Examples)
#WistiaFest / @acapland
![Page 2: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/2.jpg)
#WistiaFest / @acapland
Hey, I’m Andrew
• Lead the Growth team at Wistia
• Huge marketing & data nerd
• Surfer & snowboarder
• 90’s hip-hop fan
![Page 3: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/3.jpg)
#WistiaFest / @acapland
I like testing things
![Page 4: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/4.jpg)
#WistiaFest / @acapland
I testing things
![Page 5: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/5.jpg)
#WistiaFest / @acapland
![Page 6: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/6.jpg)
#WistiaFest / @acapland
We test things, like CTAs
![Page 7: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/7.jpg)
#WistiaFest / @acapland
And page layouts
![Page 8: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/8.jpg)
#WistiaFest / @acapland
And value props
![Page 9: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/9.jpg)
#WistiaFest / @acapland
And self-guided tours
![Page 10: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/10.jpg)
#WistiaFest / @acapland
Then we redesign and do it all again
![Page 11: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/11.jpg)
#WistiaFest / @acapland
testing worksBecause marketers are crazy
![Page 12: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/12.jpg)
#WistiaFest / @acapland
WHY aren’t marketers testing
with video?
![Page 13: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/13.jpg)
#WistiaFest / @acapland
Video is more compelling than static content
![Page 14: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/14.jpg)
#WistiaFest / @acapland
![Page 15: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/15.jpg)
SCREEN SHOT OF MY TALK
#WistiaFest / @acapland
That’s my post!
ads
![Page 16: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/16.jpg)
WHY?
#WistiaFest / @acapland
![Page 17: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/17.jpg)
#WistiaFest / @acapland
Video is a powerful tool that humanizes your brand, and compels
your viewers to take action
Marketers don’t understand the conversion power of video
![Page 18: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/18.jpg)
#WistiaFest / @acapland
Action Increase signups from website
Drive product activation Accelerate sales process
Decrease churn
Emotional Storytelling
Humanize brand Establish connections
Let people get to know you
![Page 19: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/19.jpg)
#WistiaFest / @acapland
Action Increase signups from website
Drive product activation Accelerate sales process
Decrease churn
You can easily test and measure these!
![Page 20: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/20.jpg)
#WistiaFest / @acapland
You need to be testing your videos
![Page 21: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/21.jpg)
Two main ways to a/b test with video
#WistiaFest / @acapland
Video content Video placement
![Page 22: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/22.jpg)
#WistiaFest / @twitter-handle
1. Why should I test my videos?
2. Where should I test my video?
3. Video testing lessons from Wistia
Let’s dig in more:
![Page 23: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/23.jpg)
#WistiaFest / @acapland
1. Why should I test my videos?
![Page 24: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/24.jpg)
#WistiaFest / @acapland
This is the Wistia creative team
Danielle
Dan
Liat
Trevor
Chris
![Page 25: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/25.jpg)
#WistiaFest / @acapland
They make awesome things
Danielle
Dan
Liat
Trevor
Chris
![Page 26: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/26.jpg)
#WistiaFest / @acapland
Hey we’re redesigning our About Us page,
should it have a video?
Of course, let’s make it weird. Everyone
loves weird!
Oh yeah! That would be sick.
Let’s put it on the top of the page.
Let’s film it with our new lens. It’ll
look awesome. Oh, I‘m jacked up!
![Page 27: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/27.jpg)
#WistiaFest / @acapland
About Us page
![Page 28: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/28.jpg)
#WistiaFest / @acapland
80’s themed pizza party to celebrate!
![Page 29: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/29.jpg)
#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
Ruh roh
Conversion Rate
![Page 30: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/30.jpg)
#WistiaFest / @acapland
![Page 31: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/31.jpg)
#WistiaFest / @acapland
Was it this?
![Page 32: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/32.jpg)
#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
Too late
Test to avoid unexpected results
![Page 33: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/33.jpg)
#WistiaFest / @acapland
JAN FEB MAR APR MAY JUNE JULY
Conversion)Rate
More please
Test to understand what works
![Page 34: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/34.jpg)
#WistiaFest / @acapland
LEARN:
WHAT WORKSWHAT DOESN’T WORK
![Page 35: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/35.jpg)
#WistiaFest / @acapland
Don’t embed it and forget itSee Margot de Cunah’s talk
Don’t publish videos and just
hope for the best
![Page 36: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/36.jpg)
#WistiaFest / @acapland
We get 30k monthly visits to our page, but few signups. I wonder
how we could get more?
And what video might get us the most signups?
Hmmn. I wonder if a video would help?
We could film a few different versions? Oh, I am jacked up!
![Page 37: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/37.jpg)
#WistiaFest / @acapland
1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate
![Page 38: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/38.jpg)
#WistiaFest / @acapland
Sometimes your most boring ideas are the most effective
![Page 39: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/39.jpg)
#WistiaFest / @acapland
You need to be testing your videos
Learn wicked fast
![Page 40: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/40.jpg)
#WistiaFest / @acapland
2. Where should I test my videos?
![Page 41: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/41.jpg)
#WistiaFest / @acapland
Hint: not everywhere
![Page 42: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/42.jpg)
#WistiaFest / @acapland
Test in actionable places where you can learn
![Page 43: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/43.jpg)
#WistiaFest / @acapland
Also, somewhere that gets a decent amount of traffic
![Page 44: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/44.jpg)
#WistiaFest / @acapland
Product page(s) Email
About us page Pricing page
Onboarding flow Homepage
![Page 45: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/45.jpg)
#WistiaFest / @acapland
Optimizely Sample size calculator - wi.st/sizecalc
A/B confidence calculator - wi.st/statsig
How to run an a/b test
![Page 46: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/46.jpg)
#WistiaFest / @acapland
3. Lessons we learned from testing video at Wistia
![Page 47: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/47.jpg)
ReminderThere are two main ways you can a/b test with video
#WistiaFest / @acapland
Video content Video placement
![Page 48: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/48.jpg)
#WistiaFest / @acapland
1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate
Remember the system
![Page 49: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/49.jpg)
#WistiaFest / @acapland
Test 1: Welcome to Wistia page
![Page 50: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/50.jpg)
#WistiaFest / @acapland
Test 1: Is this the best video for our goal?
![Page 51: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/51.jpg)
#WistiaFest / @acapland
Test 1: Version A
![Page 52: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/52.jpg)
#WistiaFest / @acapland
Test 1: Welcome to Wistia page
Hypothesis: Teaching people about
the tool will increase account activations.
![Page 53: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/53.jpg)
#WistiaFest / @acapland
![Page 54: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/54.jpg)
#WistiaFest / @acapland
Test 1: What do you think?
8% increase in activations
![Page 55: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/55.jpg)
#WistiaFest / @acapland
Well, like duh!
![Page 56: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/56.jpg)
#WistiaFest / @acapland
Learnings
Straightforward educational content is more helpful
(for new users)
![Page 57: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/57.jpg)
#WistiaFest / @acapland
Test 1: Educational follow-up
Hypothesis: Highlighting product
differentiation will increase activations
![Page 58: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/58.jpg)
#WistiaFest / @acapland
Test 1: Follow-up
![Page 59: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/59.jpg)
#WistiaFest / @acapland
Test 1: What do you think?
3% decrease in activations
![Page 60: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/60.jpg)
#WistiaFest / @acapland
Learnings
Teaching how to use tool more helpful than highlighting features
(for new users).
![Page 61: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/61.jpg)
#WistiaFest / @acapland
Test 2: Wistia product page
![Page 62: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/62.jpg)
#WistiaFest / @acapland
Test 2: Should this page have a video?
![Page 63: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/63.jpg)
#WistiaFest / @acapland
Test 2: Automated Emails
![Page 64: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/64.jpg)
#WistiaFest / @acapland
Test 2: Product Page
Hypothesis: Static content above the fold may be more
effective.
![Page 65: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/65.jpg)
#WistiaFest / @acapland
Test 2: What do you think?
9% increase in signups!
![Page 66: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/66.jpg)
#WistiaFest / @acapland
Test 2: Learnings
New visitors may want to browse quickly on the
product page
![Page 67: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/67.jpg)
#WistiaFest / @acapland
Test 3: Onboarding flow
![Page 68: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/68.jpg)
#WistiaFest / @acapland
Test 3: Is this the best video?
![Page 69: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/69.jpg)
#WistiaFest / @acapland
Test 3: Onboarding flow
Hypothesis: Sharing how to use all
the features will increase action with
new users.
![Page 70: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/70.jpg)
#WistiaFest / @acapland
![Page 71: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/71.jpg)
#WistiaFest / @acapland
Test 3: What do you think?
30% increase in usage!
![Page 72: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/72.jpg)
#WistiaFest / @acapland
Test 3: Learnings
Reinforced our “how to” learning. Showing better than telling.
Context is king.
![Page 73: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/73.jpg)
#WistiaFest / @acapland
You should be testing your videos
Everyone wins!
![Page 74: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/74.jpg)
#WistiaFest / @acapland
RECAP
1. Set clear goals 2. Test to learn faster & understand impact 3. Test where it’s actionable (not everywhere) 4. Test your way to success
![Page 75: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/75.jpg)
#WistiaFest / @acapland
Use technology to find hidden gems
Collect data (Google Analytics to ID high traffic pages) Watch users (FullStory, Inspectlet)
Survey visitors (Qualaroo, Intercom)
![Page 76: Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)](https://reader035.fdocuments.in/reader035/viewer/2022081604/58729e081a28ab07208b520d/html5/thumbnails/76.jpg)
#WistiaFest / @acapland
Optimizely Sample size calculator - wi.st/sizecalc
A/B confidence calculator - wi.st/statsig
A/B Testing Resources