Practitioner Web Analytics Presentation May 2010
-
Upload
jim-williams -
Category
Business
-
view
466 -
download
3
description
Transcript of Practitioner Web Analytics Presentation May 2010
I’m Jim Williams,
Head of Customer
Analytics
Split Testing to Optimize Digital Sales
www.weeworld.comwww.ju2.comwww.twitter.com/ju2ltd
Weeworld
• Weeworld is a social networking site/
virtual world
• WeeWorld is the home of the
WeeMee
• The world's favourite avatar
• 33M weemees and growing
•Users Express themselves visually by
dressing up the weemee with latest
fashions and accessories
Hi… I’m a WeeMee
Building out community
I’m so popular, I have 10,000 friends!
Rooms: extension of your WeeMee
A new room is created about like every 10 seconds.
******
World: Synchronous interaction
• Doesn’t have to be 3D, that’s not important as far as building a Virtual World is concerned
Hey you got a boyfriend?
Maybe yes, maybe no.
Getting Gold Points to Spend on Site
•Subscriptions•PayPal•Prepaid Cards•PSMS Mobile Payments•Surveys•Offers•Pay by Cash•Launch Credit Cards
•Original Get Points Page too Complicated•Get Points Page Redesign
New Tabbed Design
Credit Card Tab Surveys Tab
Which Page Design gave the highest Revenue per User?
Which Design is Best?
Wanted to answer 2 questions:
• Is the tabbed design better than the original design
• Which tab should be default
Average Revenue per User
Tab or not to Tab that is the question?
Objective: To Identify whether new tabbed design more effective than original.
Challenges:
•All transactions completed on third party sites•Could not guarantee to capture conversion•Unable to use Google Website Optimizer•Multiple micro transactions•User experience must be consistent
Whys life so complicated?
Original Get Points Page
Allocate page versions according to userID
MOD5- 0 Original 20%- 1 or 2 Credit Card 40%- 3 or 4 Surveys 40%
Track average revenue per user ARPU
Statistics 95% confidence IntervalsNo significant differences between groups prior to tests
New Tabbed Design
Credit Card Tab Surveys Tab
Which Page Design gave the highest Revenue per User?
Improvement in Revenue per Login
Version Response(improvement in
revenue per login)
p<0.05
Original - -
Credit Card 7.7% yes
Surveys 7.1% yes
Tabbed design 7% more effective than original one page design no difference between tabs
Difficulties – Thing about your control
• Don’t be impatient• Change only one thing at a
time• Original Page had no credit
card option• Was result due to new design
or just credit card?• Another follow up experiment
showed that the design was better than Original -Credit Card
Change only one thing?
Difficulties – Don’t breakout the champagne too early!
• Cyclical Effects - don’t breakout the champagne too early.
• Improvement in ARPU much higher early in experiment
• Maintaining Interest 28 days to see result.
Difficulties – Avoid Bias
• Site constantly changing• Marketing initiatives• Store Promotions• Avoiding bias• Live in real world
That’s biased it?
Difficulties -
• PayPal Revenue dropped• Cannibalization by Credit Card• Average Transaction Value fell
28%• Changed price points• Reversing option order removed
fall in average transaction value
Lessons Learned
• Using userID for split testing valid – continue to use• Tabbed Design increased revenue per user compared to
original design even with credit card. • Give careful consideration to controls• Change only one thing at a time.• Try to avoid bias with other changes on the site• Average transaction values very sensitive to price points• Maintain stakeholder interest throughout experiment
Questions?