Post on 11-Aug-2014
description
Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014
Hiten Shah hnshah@gmail.com
Download the slides: h!p://kiss.ly/leanprod2014
Continuous Learning & Improvement
Don’t know where to start?
Wondering if you’re working on the right things?
Or, you just want to hear me babble!
Lean Analytics helps you Optimize for Learning
Metrics → Hypothesis → Experiment → Act
What is Hello Bar?
Metrics Figure out what to improve
Hypothesis Make an educated guess Experiment Test your guess
Act Decide what’s next
Our funnel showed a low installation rate
Hello Bar Funnel MetricsSteps Visitors Homepage
URL EntriesCreated First Bar
Completed Registration
Installed
Totals 15068 1868 716 688 398% prev step 12.40% 38.33% 96.09% 57.85%% drop off -87.60% -61.67% -3.91% -42.15%% of visitors 100% 12.40% 4.75% 4.57% 2.64%
We achieved a 40% increase for our installa2on rate with just this one first experiment.
The Lean Analytics Cycle At Work
Original Hello Bar installation screen
Metrics Figure out what to improve
Hypothesis Make an educated guess Experiment Test your guess
Act Decide what’s next
Find out why people don’t install Hello Bar
Didn’t Meet My Expectations
I Had Trouble Installing It
I Don’t Have A Website
Not Ready To Install
Other
Hello Bar Experiment Hypothesis
If more installa2on op2ons are given, then more people will add Hello Bar to their website due to the need for a WordPress plugin and ability to email the code to someone else.
Metrics Figure out what to improve
Hypothesis Make an educated guess Experiment Test your guess
Act Decide what’s next
Created a new flow to test
I can install code myself.
41.67% responded with “WordPress Plugin”
25% responded with “email my developer / web designer”
I need help.
Metrics Figure out what to improve
Hypothesis Make an educated guess Experiment Test your guess
Act Decide what’s next
Full funnel metrics for the experiment
VariationSteps Visitors Homepage
URL EntriesCreated First Bar
Completed Registration
Installed
Totals 7456 906 342 329 276% prev step 12.15% 37.75% 96.20% 83.89%% drop off -87.85% -62.25% -3.80% -16.11%% of visitors 100% 12.15% 4.59% 4.41% 3.70%
ControlSteps Visitors Homepage
URL EntriesCreated First Bar
Completed Registration
Installed
Totals 7469 925 353 341 197% prev step 12.38% 38.16% 96.60% 57.77%% drop off -87.62% -61.84% -3.40% -42.23%% of visitors 100% 12.38% 4.73% 4.57% 2.64%
Resulted in a 30% installation rate increase
HTTP://ISVALID.ORG
Experiment resulted in a 40% improvement
ACer a dozen more experiments, our installa2on rate is now hovering between 5% and 7%
That’s an addi;onal 89% increase!!!
The Lean Analytics Cycle At Work
Metrics → Hypothesis → Experiment → Act
How You Can Put It Into Practice
Metrics Figure out what to improve
“What gets measured, gets managed.” Peter Drucker
Spreadsheets and Calculations
HELLOBAR HTTP://HELLOBAR.COM
Steps Visitors Homepage URL Entries
Created First Bar
Completed Registration
Installed Created Second Bar
Totals 15068 1868 716 688 398 193% prev step 12.40% 38.33% 96.09% 57.85% 48.49%% drop off -87.60% -61.67% -3.91% -42.15% -51.51%% of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%
Hypothesis Make an educated guess
“A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be!er it will be.”
— Michael Aagaard, h!p://kiss.ly/abthypo
Research Tools & Tac;cs
Get feedback right when they cancel
Experiment Test your guess
Only 1 out of 5 tests win.
4 out of 5 tests cost you money.
$
1) Which top metric are we trying to improve?
2) What can you anchor your metric against?
3) What is your hypothesis about why this metric is at x%?
4) What do we expect from this experiment?
5) What was the result and what did we learn from it?
HTTP://KISS.LY/6QSAB
Answer key questions for each experiment
Don’t copy “best practices” blindly
Learn what works best for you
Act Decide what’s next
Explore the results and make data informed decisions.
THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM
KISSmetrics A/B Testing Report
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
We know that people love Google Analytics
KISSmetrics Experiment Hypothesis
If people authen2cate with Google, then more people will sign up due to their exis2ng familiarity with Google Analy2cs and alignment with our winning headline.
KISSmetrics Homepage Test Control
KISSmetrics Homepage Test Variation
KISSmetrics Homepage Test Signup Conversion Rate
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
KISSmetrics Homepage Test Received Data
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
KISSmetrics Homepage New Control
Inspiration
KISSmetrics Homepage New Test Variation
KISSmetrics Homepage Test Signup Conversion Rate
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
KISSmetrics Homepage Test Received Data
KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST
Metrics Figure out what to improve
Hypothesis Make an educated guess Experiment Test your guess
Act Decide what’s next
Hiten Shah hnshah@gmail.com
@hnshah
You will get all you want in life if you help enough other people get what they want.
Zig Ziglar
Download the slides: h!p://kiss.ly/leanprod2014