The Freemium Model and A/B Testing
description
Transcript of The Freemium Model and A/B Testing
Confidential & Proprietary
The Freemium Model & A/B Testing
HostingCon 2011
By Haroon MokhtarzadaCEO & Co-Founder, Webs.com
Confidential & Proprietary
Introduction
2
Haroon MokhtarzadaCo-Founder and CEO
Entrepreneur from the age of 10. Co-founded Webs in 2001 with brothers Zeki and Idris while studying Economics at the University of Maryland. Obtained J.D. from Harvard Law School in 2005 and thereafter joined Webs full-time.
Founding Vision“Build a free website creation tool so easy that Mom can use it”
FundingRaised $12.1 million from Novak Biddle, Columbia Capital and prominent angels in August 2006.
Confidential & Proprietary
About Webs
3
Webs is a leading provider of online “front door” solutions for micro businesses across web, mobile and social platforms.
40 million websites created since inception
Started as a tool for anyone to easily create a website for free and evolved to focus on online solutions for micro businesses
Recently acquired and relaunched Pagemodo.com (Freemium Facebook page creation tool)
We run an average of 3 different A/B tests at any given time
Confidential & Proprietary
Freemium – A Success Story
4
As a disruptive business model, freemium has been the driving force behind many successful startups and $billions in value creation
Confidential & Proprietary
Simple Fundamentals
5
Freemium is a business model that works well in scenarios where the marginal cost of users can be driven towards 0 over time or through economies of scale
$ Per Active User
Time (or # users)
Variable Cost
ARPU
Confidential & Proprietary
What Is a Freemium Product/Service?
Freemium is NOT…
Free trial or sample – user forced to pay after period of time
Shareware – user forced to pay to realize actual benefit of product
6
Confidential & Proprietary
What Is a Freemium Product/Service?
Freemium IS…
A free, usable product/service
That attracts users through its price tag and free user base
And monetizes with optional paid premium features(typically subscription based)
7
Confidential & Proprietary
Freemium is merely one option:
Popular web business models:
Sell a product to customers (premium) Upsell a product to your users (freemium)
Sell other people’s products to your users (ads)
Sell your users’ data (ex: addthis)
8
Than
ks!
Confidential & Proprietary
Why we “chose” Freemium 10 yrs ago
Broke - No marketing budget
Clueless - Uncertainty about subscription/pricing model
Frugal - Cost per user was low
9
Confidential & Proprietary
Challenges for Freemium Players
Low Conversion - People who may have been willing to pay normally do not pay because the free product is sufficient
Low ARPU – makes it difficult to accelerate growth via traditional marketing methods
Infrastructure – millions of free users can create non-trivial and costly infrastructure problems
Customer Support – free users often still demand support
10
Confidential & Proprietary
Benefits of the Freemium Model
Viral – Rapid adoption, opens new, affordable, channels for getting users
Disruptive – Compete with even the most entrenched premium competitors by giving it away for free
Superior UI/UX – Offer a better product through rapid testing / customer development 11
Confidential & Proprietary
The Freemium Offering on Webs
12
Free Starter Enhanced Pro
Sitebuilder & Apps
Included Included w/ Prem. Themes
Included w/ Prem. Themes
Included w/ Prem. Themes
Pages 10 Unlimited Unlimited Unlimited
Domain Name Included Included
Email Addresses
5 15
Webstore Items
5 5 15 Unlimited
Storage 40Mb 400Mb 1Gb 5Gb
Customer Support
Community 24/7 Live Chat 24/7 Live Chat 24/7 Live Chat
Confidential & Proprietary
Leveraging our Free Users
13
Direct Marketing from Free UsersLinks on bottom of sitesBranding - Logo on bottom of sitesSEO value from websites back to Webs.comURL has webs.com in it
Indirect Marketing from Free UsersWord of mouthSize / Credibility
Confidential & Proprietary
How Premium Businesses Can Leverage Freemium
14
Offer a limited version for free (but be careful NOT to cannibalize)
Create new, related free or freemium product, and use as marketing funnel for your paid product
Partner with a freemium product, offer their premium product to your audience
Confidential & Proprietary 15
Part II: Testing… Testing… 1 2 3
HostingCon 2011
Confidential & Proprietary
The Freemium Conversion Funnel
16
Visitors
Registrations
Engagements
Referrers
$
“What gets measured gets improved”
Confidential & Proprietary
Conversion Timeline
17
Understanding When Your Users Convert Is Key
Our likelihood of conversion decreases over timeWhy? Users are publishing a new website and want all features at time of launchImplications: Focus on upselling EARLY in customer lifecycle
Other models:User more likely to convert as time goes on:
- Dropbox (space fills up)- Mailchimp (email list grows)
Implications: Deemphasize upsell up front, and focus on getting as much usage as possible
Confidential & Proprietary 18
Simple A/B Testing
Define some goal event
Create test treatments
Test - randomly split visitors into test & control groups; measure success to goal
Good for conversion events with immediacyEx: Homepage, signup page, shopping cart
Confidential & Proprietary 19
Simple A/B TestingWhat to test?
Calls-to-action (buttons)Copy, Color, Size, Placement, Hover states
Forms/FieldsDescriptors, Size, Length/number, etc.
TextHeadlines, Value Props, Links, Amount (long/short)
Imagery
Segment results by source, use landing pages* Bonus Points
Confidential & Proprietary 20
AbTests.com
Great site for testing inspiration
Confidential & Proprietary
Case Study
21
Shortened Signup Flow
Tested shorter 1-page flow against 2-page flow with more fields
Shortened version created 10,000 more signups over 3 weeks
Going 100% translates to over 25,000 more signups/month!
Control Test
Confidential & Proprietary 22
A/B Test Reporting (GWO)
Confidence increases over timeLow variation is goodGoogle Website Optimizer is good because:
Provides visitors a consistent experienceMeasures & reports confidence / significanceOffers multivariate (combination) testing
Confidential & Proprietary 23
Interpreting ResultsWhen A/B Testing is Not Enough
Freemium Problem: Signup != $$$
A/B signup test only told us which produced more USERS, but does NOT tell us which produced more $$
Example: Asking more questions in signup will reduce signups but may increase user success rate and make more $$
Confidential & Proprietary 24
Database Driven Cohort Analysis
Solution:
A system that permanently fixes a user to a test group, allowing one to run results analysis on any variable that is also attached to that user (i.e. $ spent)
Offers the ability to compare users who received different treatments through your entire funnel over time
Confidential & Proprietary 25
Cohort Funnel Analysis Example
Control Test 1 Test2
Homepage 10000 10000 10000
Signup 3000 (30%) 2900 (29%) 3300 (33%)
Control Test 1 Test2
Homepage 10000 10000 10000
Signup 3000 (30%) 2900 (29%) 3300 (33%)
Publish 1800 (60%) 1850 (64%) 1950 (59%)
Purchase 180 (10%) 203(11%) 195 (10%)
A/B testing would have shown Test2 as the winnerTest 1 makes more money. Tradeoff?
* Dummy data
Confidential & Proprietary 26
How to Set Up Cohort Testing
Quick & Dirty Shortcut: Mod on userid(ex: show even users one treatment and odd users another)
Thanks Eric Ries!
In Cohort?
Coin Toss
DisplayTreatment
Assign to Cohort
User Enters
Yes
No
Confidential & Proprietary
Visualizing Results
27
6/15/2
011
6/16/2
011
6/17/2
011
6/18/2
011
6/19/2
011
6/20/2
011
6/21/2
011
6/22/2
011
6/23/2
011
Cumulative $
Control TEST
Day to day reporting on metrics like Premium $ can often be lumpy.
Looking at a cumulative view helps visualize more smoothly
Diverging lines means one is superior
Parallel lines apart means one got lucky;)
Confidential & Proprietary 28
Segmentation & Cohort AnalysisAdvantages
Integrated front- and back-end testing
Complete ownership of user data
Ability to track impact to user long after user-facing experience has gone away
Provide users with consistent test experience that integrates your back-end - Great for pricing & promotional tests
Ability to “slice” data by internal user variables
Confidential & Proprietary
Advanced Segmentation
29
Total Personal Group Business
Conversion Rates by Segment
Control Test
Total/Aggregate results can hide the real story
Not all users are the same
Segmenting data and “slicing” can produce surprising gains
Treatments can be applied to different segments accordingly Total Personal Group Business
Conversion Rates by Segment
Control Test
Confidential & Proprietary
Visualizing Advanced Segmentation
30
Cumulative Charts
6/15/2
011
6/16/2
011
6/17/2
011
6/18/2
011
6/19/2
011
6/20/2
011
6/21/2
011
6/22/2
011
6/23/2
011
Segment 1
Control TEST
6/15/2
011
6/16/2
011
6/17/2
011
6/18/2
011
6/19/2
011
6/20/2
011
6/21/2
011
6/22/2
011
6/23/2
011
Segment 3
Control TEST
6/15/2
011
6/16/2
011
6/17/2
011
6/18/2
011
6/19/2
011
6/20/2
011
6/21/2
011
6/22/2
011
6/23/2
011
Segment 2
Control TEST
Different reports for each segmentCan use end results to find interesting variable combinations, then check for significance
Confidential & Proprietary
Why A/B Testing is Crucial
31
Lots of Leverage – since your free customers are often your largest referring source, improving your funnel can have compounding effects on both virality and conversion.
Users constantly surprise – its impossible to guess what users will do. That’s why all great companies use testing.
Validate ideas and time/energy spent on product efforts
Better Serve Your Users – build a better product and improve your intuition about your customers
Confidential & Proprietary
Summary
32
The freemium model is here to stay
When implemented properly, freemium is a disruptive business model allowing rapid growth in new or existing markets
Improving conversion is crucial to success, and requires rapid iteration and A/B testing
Rapid iteration and testing cannot be done without large influx of free users, favoring those who have achieved scale
Successful freemium products are always-improving, and tend to have the superior metrics & UX/UI (out of sheer necessity!)
Premium players who do not want to disrupt legacy business can can partner with freemium players to leverage their assets
Confidential & Proprietary 33
Q&AHaroon Mokhtarzada, CEO & Co-Founder of Webs@haroon, [email protected]