Prove it! A/B Testing for Nonprofits
-
Upload
charity-express -
Category
Documents
-
view
2.213 -
download
0
description
Transcript of Prove it! A/B Testing for Nonprofits
![Page 1: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/1.jpg)
Prove It! A Practical Guide to A/B Testing for Nonprofits
@NetTuesday // @bradyjosephson // #ntvan
![Page 2: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/2.jpg)
4 More Years.
![Page 3: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/3.jpg)
Culture of Learning.
@NetTuesday // @bradyjosephson // #ntvan
![Page 4: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/4.jpg)
Tonight.
• 3 Forms of Testing• 4 Steps to Testing
– 6 Sources for Insights– 5 Things to Avoid
• 1 Case Studies & 3 Other Examples
• 7 Resources for Testing
• 1 Live Test• ? Questions
• Tweet– #ntvan– @NetTuesday– @bradyjosephson
• Poll– @poll– Text to: 37607
• Me– [email protected]– recharity.ca– recharity.ca/category/
testing/
@NetTuesday // @bradyjosephson // #ntvan
![Page 5: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/5.jpg)
3 FORMS OF TESTINGHacking. Tracking. Cracking.
@NetTuesday // @bradyjosephson // #ntvan
![Page 6: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/6.jpg)
Hack.
@NetTuesday // @bradyjosephson // #ntvan
![Page 7: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/7.jpg)
Track.
@NetTuesday // @bradyjosephson // #ntvan
![Page 8: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/8.jpg)
Crack.
@NetTuesday // @bradyjosephson // #ntvan
![Page 9: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/9.jpg)
4 STEPS TO TESTINGInsights. Hypothesis. Test. Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 10: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/10.jpg)
Insights.
Things that provoke curiosity and make you want to test.
6 Sources for Insights• Survey. • Feedback. • Analytics. • Focus Group. • Social Media. • Your Brain.
@NetTuesday // @bradyjosephson // #ntvan
![Page 11: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/11.jpg)
Insights. // Dissonance.
Text REDCROSS to 30333
@NetTuesday // @bradyjosephson // #ntvan
![Page 12: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/12.jpg)
Hypothesis.
Tries to answer an important question with an expected result.
• If True? – Would we do differently?
• If False? – What would we do differently?
• Needed?– Is this question already answered?
@NetTuesday // @bradyjosephson // #ntvan
![Page 13: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/13.jpg)
Test.
The execution to get the data/results you need to make a decision.
Choose the (right) metric.State what is significant.Prepare your data.Control your variables.
@NetTuesday // @bradyjosephson // #ntvan
![Page 14: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/14.jpg)
Analyze.
Looking at the unfiltered data for a winner, what you learned and what it means moving forward.
Which one won?What else did you learn?How confident are you in the result?What are the next steps?– Run another test?– Change something in your strategy?
@NetTuesday // @bradyjosephson // #ntvan
![Page 15: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/15.jpg)
5 THINGS TO AVOIDConfounding Variables. Small Sample Size. Lack of Purpose. Wrong Metric. Skewed Data.
@NetTuesday // @bradyjosephson // #ntvan
![Page 16: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/16.jpg)
Confounding Variables.
Too many variables making it hard to know what caused what.
• Try to change as few things as possible and iterate more
• The more moving pieces you have the less you can trust the outcome
@NetTuesday // @bradyjosephson // #ntvan
![Page 17: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/17.jpg)
Sample Size.
Smaller the size the less confident you can be in the result.
• If you only have small sample sizes, run more tests
• Smaller sample sizes allow for more custom/manual work
@NetTuesday // @bradyjosephson // #ntvan
![Page 18: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/18.jpg)
No Purpose.
What changes with the outcome of this test or finding?
• Think of a decision you recently made. Now think of what information you wish you had to make that decision. That’s purpose.
@NetTuesday // @bradyjosephson // #ntvan
![Page 19: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/19.jpg)
Wrong Metric.
Choosing to measure the wrong thing to “prove” the winner.
• Email open rates measure subject lines (firstly) and timing (secondly) so unless you are testing those things it is a useless metric for a test.
@NetTuesday // @bradyjosephson // #ntvan
![Page 20: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/20.jpg)
Skewed Data.
Bad data gets compounded. Start with good data and you have a chance.
• Start capturing good/clean data right now• Don’t try to capture everything all at once• Just because you can track it doesn’t mean
that you should
@NetTuesday // @bradyjosephson // #ntvan
![Page 21: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/21.jpg)
1 CASE STUDY & 3 OTHER EXAMPLES
Email Appeals. Email Opens. Email Signups. Facebook Ads.
@NetTuesday // @bradyjosephson // #ntvan
![Page 22: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/22.jpg)
Email Appeals.
• Insight. – Current site revolves around projects, we are suggesting
it should revolve around stories.
• Question. – If we can prove more connection to stories, not projects,
it will help with site and communications revision.
• Hypothesis. – Appeals centered around stories, not projects, will
produce a greater response rate measured by gifts made and greater than 20%.
@NetTuesday // @bradyjosephson // #ntvan
![Page 23: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/23.jpg)
Email Appeals.
• Test. – Send two similar emails to randomized group
with slightly different funding focus.
• Analyze. – See which email has the greater response rate
as determined by gifts made.
@NetTuesday // @bradyjosephson // #ntvan
![Page 24: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/24.jpg)
Email Appeals. // Test.
• Randomized Group.– Excel and random function
• Set Up Goals.– Connect MailChimp and Google analytics
• Complications.– Small sample size– Additional emails
@NetTuesday // @bradyjosephson // #ntvan
![Page 25: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/25.jpg)
Email Appeals. // Test.
Choose the (right) metric.– Number of gifts.
State what is significant.– 20% difference.
Prepare your data.– Randomized in Excel.
Control your variables.– Length, design, landing page, timing all the same.
@NetTuesday // @bradyjosephson // #ntvan
![Page 26: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/26.jpg)
Email Appeals. // Test.
@NetTuesday // @bradyjosephson // #ntvan
![Page 27: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/27.jpg)
Email Appeals. // Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 28: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/28.jpg)
Email Appeals. // Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 29: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/29.jpg)
Email Opens. // Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 30: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/30.jpg)
Email Signups. // Test.
@NetTuesday // @bradyjosephson // #ntvan
![Page 31: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/31.jpg)
Email Signups. // Test.
@NetTuesday // @bradyjosephson // #ntvan
![Page 32: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/32.jpg)
Email Signups. // Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 33: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/33.jpg)
Facebook Ads. // Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 34: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/34.jpg)
7 RESOURCES FOR TESTINGUnBounce. MailChimp. Google. KISS Metrics. HubSpot. Wufoo. Excel.
@NetTuesday // @bradyjosephson // #ntvan
![Page 35: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/35.jpg)
Resources.
Formswufoo.com
Analytics (and everything)Search “Google” on bing.com Marketing
hubspot.com
DataYour Computer
LTV & Awesome Blogwww.wufoo.com
Landing Pagesunbounce.com
Emailsmailchimp.com
@NetTuesday // @bradyjosephson // #ntvan
![Page 36: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/36.jpg)
1 LIVE TESTOn You.
@NetTuesday // @bradyjosephson // #ntvan
![Page 37: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/37.jpg)
Live Test. // A.
If you sign up to get emails from me today, I’ll send you a free copy of our eBook “Tool Kit for Charities”.
To sign up:• Text 599154 to 37607
OR• Tweet @poll with 599154
@NetTuesday // @bradyjosephson // #ntvan
![Page 38: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/38.jpg)
Live Test. // B.
If you sign up to get emails from me today, you’ll be entered for a chance to win a $25 gift card to Starbucks.
To sign up:• Text 599156 to 37607
OR• Tweet @poll with 599156
@NetTuesday // @bradyjosephson // #ntvan
![Page 39: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/39.jpg)
Live Test. // Analyze.
@NetTuesday // @bradyjosephson // #ntvan
![Page 40: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/40.jpg)
? QUESTIONS & THANKSFrom You.
@NetTuesday // @bradyjosephson // #ntvan
![Page 41: Prove it! A/B Testing for Nonprofits](https://reader033.fdocuments.in/reader033/viewer/2022061223/54c46dab4a7959a77a8b45d2/html5/thumbnails/41.jpg)
THANKS!From me. @bradyjosephson // recharity.ca
@NetTuesday // @bradyjosephson // #ntvan