Lean Analytics Cycle

96
Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014 Hiten Shah [email protected] Download the slides: hp://kiss.ly/leanprod2014

description

 

Transcript of Lean Analytics Cycle

Page 1: Lean Analytics Cycle

Lean Analytics Lean Product & Lean UX Silicon Valley - July 2014

Hiten Shah [email protected]

Download the slides: h!p://kiss.ly/leanprod2014

Page 2: Lean Analytics Cycle
Page 3: Lean Analytics Cycle

Continuous Learning & Improvement

Page 4: Lean Analytics Cycle

Don’t know where to start?

Page 5: Lean Analytics Cycle

Wondering if you’re working on the right things?

Page 6: Lean Analytics Cycle

Or, you just want to hear me babble!

Page 7: Lean Analytics Cycle

HTTP://KISS.LY/LEANLYTICSBOOK

Page 8: Lean Analytics Cycle

Lean Analytics helps you Optimize for Learning

Page 9: Lean Analytics Cycle

Lean Analytics Cycle

AVINASH KAUSHIK HTTP://KISS.LY/LEANAC

Page 10: Lean Analytics Cycle

Metrics  →  Hypothesis  →  Experiment  →  Act

Page 11: Lean Analytics Cycle

HTTP://HELLOBAR.COM

Page 12: Lean Analytics Cycle

What is Hello Bar?

Page 13: Lean Analytics Cycle
Page 14: Lean Analytics Cycle
Page 15: Lean Analytics Cycle
Page 16: Lean Analytics Cycle
Page 17: Lean Analytics Cycle
Page 18: Lean Analytics Cycle
Page 19: Lean Analytics Cycle
Page 20: Lean Analytics Cycle
Page 21: Lean Analytics Cycle

 Metrics    Figure  out  what  to  improve  

 Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess  

 Act    Decide  what’s  next

Page 22: Lean Analytics Cycle

Our funnel showed a low installation rate

Hello Bar Funnel MetricsSteps Visitors Homepage

URL EntriesCreated First Bar

Completed Registration

Installed

Totals 15068 1868 716 688 398% prev step 12.40% 38.33% 96.09% 57.85%% drop off -87.60% -61.67% -3.91% -42.15%% of visitors 100% 12.40% 4.75% 4.57% 2.64%

Page 23: Lean Analytics Cycle

We  achieved  a  40%  increase  for  our  installa2on  rate  with  just  this  one  first  experiment.

The Lean Analytics Cycle At Work

Page 24: Lean Analytics Cycle

Original Hello Bar installation screen

Page 25: Lean Analytics Cycle

 Metrics    Figure  out  what  to  improve  

 Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess  

 Act    Decide  what’s  next

Page 26: Lean Analytics Cycle

Find out why people don’t install Hello Bar

Page 27: Lean Analytics Cycle

Didn’t Meet My Expectations

Page 28: Lean Analytics Cycle

I Had Trouble Installing It

Page 29: Lean Analytics Cycle

I Don’t Have A Website

Page 30: Lean Analytics Cycle

Not Ready To Install

Page 31: Lean Analytics Cycle

Other

Page 32: Lean Analytics Cycle

Hello Bar Experiment Hypothesis

If  more  installa2on  op2ons  are  given,  then  more  people  will  add  Hello  Bar  to  their  website  due  to  the  need  for  a  WordPress  plugin  and  ability  to  email  the  code  to  someone  else.

Page 33: Lean Analytics Cycle

 Metrics    Figure  out  what  to  improve  

 Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess  

 Act    Decide  what’s  next

Page 34: Lean Analytics Cycle

Created a new flow to test

Page 35: Lean Analytics Cycle

I can install code myself.

Page 36: Lean Analytics Cycle

41.67% responded with “WordPress Plugin”

Page 37: Lean Analytics Cycle

25% responded with “email my developer / web designer”

Page 38: Lean Analytics Cycle

I need help.

Page 39: Lean Analytics Cycle

 Metrics    Figure  out  what  to  improve  

 Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess  

 Act    Decide  what’s  next

Page 40: Lean Analytics Cycle

Full funnel metrics for the experiment

VariationSteps Visitors Homepage

URL EntriesCreated First Bar

Completed Registration

Installed

Totals 7456 906 342 329 276% prev step 12.15% 37.75% 96.20% 83.89%% drop off -87.85% -62.25% -3.80% -16.11%% of visitors 100% 12.15% 4.59% 4.41% 3.70%

ControlSteps Visitors Homepage

URL EntriesCreated First Bar

Completed Registration

Installed

Totals 7469 925 353 341 197% prev step 12.38% 38.16% 96.60% 57.77%% drop off -87.62% -61.84% -3.40% -42.23%% of visitors 100% 12.38% 4.73% 4.57% 2.64%

Page 41: Lean Analytics Cycle

Resulted in a 30% installation rate increase

HTTP://ISVALID.ORG

Experiment resulted in a 40% improvement

Page 42: Lean Analytics Cycle

ACer  a  dozen  more  experiments,  our  installa2on  rate  is  now  hovering  between  5%  and  7%  

That’s  an  addi;onal  89%  increase!!!

The Lean Analytics Cycle At Work

Page 43: Lean Analytics Cycle

Metrics  →  Hypothesis  →  Experiment  →  Act

How You Can Put It Into Practice

Page 44: Lean Analytics Cycle

Metrics  Figure  out  what  to  improve

Page 45: Lean Analytics Cycle

“What gets measured, gets managed.” Peter Drucker

Page 46: Lean Analytics Cycle

Baselines

DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

Page 47: Lean Analytics Cycle

Funnels

KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

Page 48: Lean Analytics Cycle

SaaS Funnel

KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

Page 49: Lean Analytics Cycle

eCommerce Funnel

KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

Page 50: Lean Analytics Cycle

Revenue

BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE

Page 51: Lean Analytics Cycle

Spreadsheets and Calculations

HELLOBAR HTTP://HELLOBAR.COM

Steps Visitors Homepage URL Entries

Created First Bar

Completed Registration

Installed Created Second Bar

Totals 15068 1868 716 688 398 193% prev step 12.40% 38.33% 96.09% 57.85% 48.49%% drop off -87.60% -61.67% -3.91% -42.15% -51.51%% of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%

Page 52: Lean Analytics Cycle

Hypothesis  Make  an  educated  guess

Page 53: Lean Analytics Cycle

“A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the be!er it will be.”

— Michael Aagaard, h!p://kiss.ly/abthypo

Page 54: Lean Analytics Cycle

Research  Tools  &  Tac;cs

Page 55: Lean Analytics Cycle

Customer Development

HTTP://WWW.CUSTDEV.COM

Page 56: Lean Analytics Cycle

Jobs To Be Done #JTBD

HTTP://JOBSTOBEDONE.ORG

Page 57: Lean Analytics Cycle

Usability Testing

HTTP://USERTESTING.COM

Page 58: Lean Analytics Cycle

Live Chat

HTTP://OLARK.COM

Page 59: Lean Analytics Cycle

Mouse Tracking

HTTP://CRAZYEGG.COM

Page 60: Lean Analytics Cycle

Long Form Surveying

HTTP://SURVEYMONKEY.COM

Page 61: Lean Analytics Cycle

On-Page Surveying

HTTP://QUALAROO.COM

Page 62: Lean Analytics Cycle

Personalized Messaging

HTTP://INTERCOM.IO

Page 63: Lean Analytics Cycle

Email a survey to people

HTTP://SURVEYMONKEY.COM

Page 64: Lean Analytics Cycle

Ask people why they canceled

HTTP://SURVEYMONKEY.COM

Page 65: Lean Analytics Cycle

Get feedback right when they cancel

Page 66: Lean Analytics Cycle

Exit Surveys

HTTP://QUALAROO.COM

Page 67: Lean Analytics Cycle

Discover what’s missing

HTTP://QUALAROO.COM

Page 68: Lean Analytics Cycle

Find out what words people use to describe your product

HTTP://QUALAROO.COM

Page 69: Lean Analytics Cycle

Learn about people’s expectations

HTTP://INTERCOM.IO

Page 70: Lean Analytics Cycle

Learn what frustrates people

HTTP://INTERCOM.IO

Page 71: Lean Analytics Cycle

More actionable reading…

QUALITATIVE DATA HTTP://KISS.LY/QUALABT

Page 72: Lean Analytics Cycle

Experiment Hypothesis Formula

DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

Page 73: Lean Analytics Cycle

Experiment  Test  your  guess

Page 74: Lean Analytics Cycle

Only  1  out  of  5  tests  win.

4  out  of  5  tests  cost  you  money.

$

Page 75: Lean Analytics Cycle

1)  Which  top  metric  are  we  trying  to  improve?  

2)  What  can  you  anchor  your  metric  against?  

3)  What  is  your  hypothesis  about  why  this  metric  is  at  x%?  

4)  What  do  we  expect  from  this  experiment?  

5)  What  was  the  result  and  what  did  we  learn  from  it?

HTTP://KISS.LY/6QSAB

Answer key questions for each experiment

Page 76: Lean Analytics Cycle

Prioritize your experiments

HTTP://KISS.LY/ABPIE

Page 77: Lean Analytics Cycle

Don’t copy “best practices” blindly

Page 78: Lean Analytics Cycle

Learn what works best for you

Page 79: Lean Analytics Cycle

Act  Decide  what’s  next

Page 80: Lean Analytics Cycle

Explore  the  results  and  make  data  informed  decisions.

THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM

Page 81: Lean Analytics Cycle

KISSmetrics A/B Testing Report

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Page 82: Lean Analytics Cycle
Page 83: Lean Analytics Cycle

We know that people love Google Analytics

Page 84: Lean Analytics Cycle

KISSmetrics Experiment Hypothesis

If  people  authen2cate  with  Google,  then  more  people  will  sign  up  due  to  their  exis2ng  familiarity  with  Google  Analy2cs  and  alignment  with  our  winning  headline.

Page 85: Lean Analytics Cycle

KISSmetrics Homepage Test Control

Page 86: Lean Analytics Cycle

KISSmetrics Homepage Test Variation

Page 87: Lean Analytics Cycle

KISSmetrics Homepage Test Signup Conversion Rate

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Page 88: Lean Analytics Cycle

KISSmetrics Homepage Test Received Data

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Page 89: Lean Analytics Cycle

KISSmetrics Homepage New Control

Page 90: Lean Analytics Cycle

Inspiration

Page 91: Lean Analytics Cycle

KISSmetrics Homepage New Test Variation

Page 92: Lean Analytics Cycle

KISSmetrics Homepage Test Signup Conversion Rate

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Page 93: Lean Analytics Cycle

KISSmetrics Homepage Test Received Data

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Page 94: Lean Analytics Cycle

Growth!

HTTP://KISSMETRICS.COM

Page 95: Lean Analytics Cycle

 Metrics    Figure  out  what  to  improve  

 Hypothesis    Make  an  educated  guess        Experiment    Test  your  guess  

 Act    Decide  what’s  next

Page 96: Lean Analytics Cycle

Hiten Shah [email protected]

@hnshah

You will get all you want in life if you help enough other people get what they want.

Zig Ziglar

Download the slides: h!p://kiss.ly/leanprod2014