Ntc 2016 step right up testing presentation 16x9

53
Step Right Up! Everyone’s a Winner! Steve Kerhli, PETA & Steve Daigneault, M+R

Transcript of Ntc 2016 step right up testing presentation 16x9

Page 1: Ntc 2016 step right up testing presentation 16x9

Step Right Up! Everyone’s a Winner!

Steve Kerhli, PETA & Steve Daigneault, M+R

Page 2: Ntc 2016 step right up testing presentation 16x9

Ask String on Main Donation Form

Test Control

Vote: Control Test Tie

Winner: Test

Page 3: Ntc 2016 step right up testing presentation 16x9

Renewal Language to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Page 4: Ntc 2016 step right up testing presentation 16x9

Higher Ask String for Prospects

Test Control

Vote: Control Test Tie

Winner: Tie

Page 5: Ntc 2016 step right up testing presentation 16x9

SMS Deadline Reminder

Test Control

Vote: Control Test Tie

Winner: Test

Page 6: Ntc 2016 step right up testing presentation 16x9

Nudge to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Page 7: Ntc 2016 step right up testing presentation 16x9

Font Type and Size

Test Control

Vote: Control Test Tie

Winner: Test

Page 8: Ntc 2016 step right up testing presentation 16x9

$5 Ask to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Make your tax-deductible gift

of $50 or whatever you can

afford now. Help us meet our

urgent Year-End Challenge goal

of $125,000.

Make your tax-deductible gift

now. Pitch in $5 or more to help

us meet our urgent Year-End

Challenge goal of $125,000.

Page 9: Ntc 2016 step right up testing presentation 16x9

Adding PayPal as Payment Option

Test Control

Vote: Control Test Tie

Winner: Test

Page 10: Ntc 2016 step right up testing presentation 16x9

“Renew” Language to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Page 11: Ntc 2016 step right up testing presentation 16x9

Another Nudge

Test Control

Vote: Control Test Tie

Winner: Tie

Page 12: Ntc 2016 step right up testing presentation 16x9

Cents Per Day

Test Control

Vote: Control Test Tie

Winner: Control

Page 13: Ntc 2016 step right up testing presentation 16x9

Yes, Another Nudge Test

Test Control

Vote: Control Test Tie

Winner: Test

Page 14: Ntc 2016 step right up testing presentation 16x9

Multi-Button vs. Single Button

Test Control

Vote: Control Test Tie

Winner: Tie

Page 15: Ntc 2016 step right up testing presentation 16x9

Quick Donate Based on HPC

Test Control

Vote: Control Test Tie

Winner: Tie

Page 16: Ntc 2016 step right up testing presentation 16x9

EOY Lightbox

Test Control

Vote: Control Test Tie

Winner: Tie

Page 17: Ntc 2016 step right up testing presentation 16x9

$10 Ask to Lapsed and Deeply Lapsed Donors

Test Control

Vote: Control Test Tie

Winner: Test

Page 18: Ntc 2016 step right up testing presentation 16x9

“Read More” Button

Test Control

Vote: Control Test Tie

Winner: Tie

Page 19: Ntc 2016 step right up testing presentation 16x9

Nudge to Prospects + HPC<$125 Donors

Test

Control

Vote: Control Test Tie

Winner: Test

Page 20: Ntc 2016 step right up testing presentation 16x9

Video on Donation Form

Test Control

Vote: Control Test Tie

Winner: Tie

Page 21: Ntc 2016 step right up testing presentation 16x9

Reverse Ask String

Test Control

Vote: Control Test Tie

Winner: Test

Page 22: Ntc 2016 step right up testing presentation 16x9

Button Colour

Test Control

Vote: Control Test Tie

Winner: Test

Page 23: Ntc 2016 step right up testing presentation 16x9

Email Recruitment Lightbox

Version 2 Version 1

Vote: Version 1 Version 2 Version 3

Winner: Version 2

Version 3

Page 24: Ntc 2016 step right up testing presentation 16x9

Donate vs. Membership Lightbox at EOY

Test Control

Vote: Control Test Tie

Winner: Test

Page 25: Ntc 2016 step right up testing presentation 16x9

Donate vs. Membership Lightbox at EOY

Test Control

Vote: Control Test Tie

Winner: Tie

Page 26: Ntc 2016 step right up testing presentation 16x9

Streamlining Donation Form

Test Control

Vote: Control Test Tie

Winner: Control

Page 27: Ntc 2016 step right up testing presentation 16x9

OMG, Another Nudge Test

Test Control

Vote: Control Test Tie

Winner: Test

Page 28: Ntc 2016 step right up testing presentation 16x9

Recommendations for Nudge Tests

• Tested variations of nudge tests to 10 organizations.

• 50% saw improvement in at least one key metric. No organization saw nudges depress response or revenue.

• Recommended language: • “Most people are donating at least $X right

now.”

Page 29: Ntc 2016 step right up testing presentation 16x9

Upper Case vs. Sentence Case Lightbox

Test Control

Vote: Control Test Tie

Winner: Test

Page 30: Ntc 2016 step right up testing presentation 16x9

Multistep vs. Single Column Form

Test Control

Vote: Control Test Tie

Winner: Tie

Page 31: Ntc 2016 step right up testing presentation 16x9

Another PayPal Payment Option

Test Control

Vote: Control Test Tie

Winner: Test

Page 32: Ntc 2016 step right up testing presentation 16x9

Ask String vs. Open Field

Test

Control

Vote: Control Test Tie

Winner: Test

Page 33: Ntc 2016 step right up testing presentation 16x9

Right Call-Out vs. Centered

Test Control

Vote: Control Test Tie

Winner: Test

Page 34: Ntc 2016 step right up testing presentation 16x9

Review vs. Donate Button

Test Control

Vote: Control Test Tie

Winner: Tie

Page 35: Ntc 2016 step right up testing presentation 16x9

Paragraph Spacing

Test Control

Vote: Control Test Tie

Winner: Control

Page 36: Ntc 2016 step right up testing presentation 16x9

One Column vs. Two Column

Test Control

Vote: Control Test Tie

Winner: Tie

Page 37: Ntc 2016 step right up testing presentation 16x9

PETA Tests

Page 38: Ntc 2016 step right up testing presentation 16x9

$5 Ask to Non-Donors

Test

Control

Vote: Control Test Tie

Winner: Test

Page 39: Ntc 2016 step right up testing presentation 16x9

$5 Ask vs $10 Ask to Non-Donors

Test Control

Vote: Control Test Tie

Winner: Control

Page 40: Ntc 2016 step right up testing presentation 16x9

Copy Length on Donation Form

Test Control

Vote: Control Test Tie

Winner: Test

Page 41: Ntc 2016 step right up testing presentation 16x9

Single Image vs. Multi-Image

Test Control

Vote: Control Test Tie

Winner: Test

Page 42: Ntc 2016 step right up testing presentation 16x9

Larger Link Size

Test Control

Vote: Control Test Tie

Winner: Test

Page 43: Ntc 2016 step right up testing presentation 16x9

Animal Specific Donation Forms

Test

Control

Vote: Control Test Tie

Winner: Test

Page 44: Ntc 2016 step right up testing presentation 16x9

Renewal Language to Current + Lapsed Donors

Test Control

Vote: Control Test Tie

Winner: Test

Page 45: Ntc 2016 step right up testing presentation 16x9

Membership Buttons on Homepage

Test Control

Vote: Control Test Tie

Winner: Test

Page 46: Ntc 2016 step right up testing presentation 16x9

Tools for Running a Strong Testing Program

Page 47: Ntc 2016 step right up testing presentation 16x9

✳ The purpose of hypotheses is to make sure we're thinking critically about what we want to test and why and to make sure we learn from the test.

✳ To help create a hypothesis, fill in the blanks:

Changing (control) to (variation) will improve (test objective) because (reason you think the change will help) .

✳ If you craft a good hypothesis, you will learn something from every test, regardless of the outcome.

Crafting the hypothesis

Page 48: Ntc 2016 step right up testing presentation 16x9

✳ Before running a test, figure out the sample size needed for the test.

✳ To do that, you’ll need to understand how much of a change you’ll be able to detect.

✳ If you’re evaluating specific audiences or pages (e.g. mobile traffic), do this for each page or segment that you will evaluate

Determine Sample Size needed

Page 49: Ntc 2016 step right up testing presentation 16x9

Baseline Conversion Rate - the current average for the metric you are trying to improve (e.g. response rate, action rate)

Minimum Detectable Effect (MDE) - the amount of change in the conversion rate that you want to be able to detect. We recommend 25% or lower.

Elements of the sample size calculator

Minimum Audience Size - the total number of people you need to include in the experiment, per variation.

Page 50: Ntc 2016 step right up testing presentation 16x9

Sample Testing Roadmap

Page 51: Ntc 2016 step right up testing presentation 16x9

Sample Test Plan

Page 52: Ntc 2016 step right up testing presentation 16x9

Sample Results Tracker

Page 53: Ntc 2016 step right up testing presentation 16x9

All In One Place