Post on 25-Mar-2018
Team Intelligence For Data-Driven Product Development usenotion.com
Team Intelligence For Data-Driven Product Development usenotion.com
Table of ContentsAcquisition
Activation
Referral
Retention
Revenue
Team
p.7
p.12
p.17
p.21
p.26
p.31
Conversion From Lead Source 8
Conversion to Signup 9
Conversion from Trial 10
Customer Acquisition Cost 11
Average Sessions per User 13
Average Session Time 14
MAU/WAU/DAU 15
Feature Utilization 16
Net Promoter Score 18
Product Invite Conversion 19
Social Media Referrals 20
Customer Churn 22
Complaints per Customer 23
Feature Requests 24
Revenue Churn 25
Average Revenue per User 27
Average Revenue per Employee 28
Customer Lifetime Value 29
Monthly Recurring Revenue 30
Backlog Burn Rate 32
Cost of Development per Unit 33
Cycle Time 34
Data Removal Efficiency 35
Lead Time 36
Team Health 37
Throughput 38
Total Defects 39
Work in Progress 40
Velocity 41
Team Intelligence For Data-Driven Product Development usenotion.com
The AARRRT! of Great Productimportant to think about the best fit for your users’ goals. There are a number of KPIs that are commonly tracked in many of the most successful technology companies. You may have come across business KPIs like ARPU or LTV, but sometimes it’s hard to see how those metrics are relevant to the process of creating and building the product.
This guide will take you through many of the most popular KPIs for product develop-ment, business, and customer success. Our goal is to teach you how measuring and track-ing these metrics will help you to make better, more profitable products.
For product people, finishing features or dealing with bugs can take up so much time that it’s hard to see the bigger picture. But to create really loveable, effective product for your customers, you need to understand how the work your team is doing fits in with the needs of your customers and the broader vision of your company.
Data is a great way to measure and test the effectiveness of your work, but many of us feel like we don’t have time or even know where to start when collecting and comparing data.
KPIs FTW!Key Performance Indicators vary acrossdifferent kinds of companies, and it’s
Team Intelligence For Data-Driven Product Development usenotion.com
Great KPIs are:• Actionable: Create a goal for the KPI. If the
indicator’s value changes, you’ll know what actions to take next.
• Measurable: Choose KPIs that are specific and can easily be tracked and recorded.
• Comparable: Knowing how your KPIs inter-relate can help you discover how moving one lever can affect other outcomes.
• Valid: Indicators aren’t really useful unless they are based on good data and correct assumptions. Testing can help to ensure that the basis of your KPIs connects with the goals you’re trying to achieve.
Performance indicators need to give youactionable information about how to improve.
To create good indicators, first assess yourobjectives and your strategy to meet goals that further the objective.
Every KPI should have a direct correlationto performance towards the goal. The KPImay help the team detect problems orimpediments to the goal and also help driveawareness of actions to further the goal.So think about how a data point based on measuring something will tell you whetheryou are reaching a goal.
The Recipe for Great KPIs
Leading indicators tend to be predictive and can change quickly. They help you under-stand the direction things are going. Leading indicators are hard to measure and easy to influence.
Ideally, every decision-maker and stakeholder in your business will be able to see KPIs in a dashboard that shows where you are meeting goals, where you are off-target, and what factors correlate.
When a team is able to meet or achieve a KPI, then the business goals have been met for that particular objective. The next step is to set the goal higher for that same KPI—or focus on a new KPI with a new objective.
• User-focused: KPIs work best when you understand that improving them directly impacts the success of your customer.
• Owned: Make sure there is someone who is accountable for each KPI so it doesn’t get lost.
• Shared: KPIs are best used to help the entire organization understand how things are going and what issues need to be addressed.
Lagging indicators track output. They are easy to measure but hard to improve or in-fluence, since they deal with past work. Often these are your financial numbers, and they give you a concrete picture of where you are today.
Team Intelligence For Data-Driven Product Development usenotion.com
typically you will be reviewing those numbers at least weekly.
We built Notion to give you an easy, beauti-ful way to look at KPIs with your team. In the following guide, you’ll see the recipes we’ve created to make tracking your indicators straightforward and simple.
How often should you measure?Most KPIs need to be reviewed at least monthly—though we’ve found that data-focused PMs are looking at these numbers every day—to get a picture of where they are going and what might be moving the needle.
With campaign-focused metrics, considerthe goals you’re trying to hit and track often enough to see if you need to iterate. Some metrics won’t change frequently—such as Revenue per Employee—and some are tied to data you may only be collecting periodically—such as Net Promoter Score. Your Team metrics will be tied to your development cycle, and
Team Intelligence For Data-Driven Product Development usenotion.com
The Metrics
Discovering where customers hang out andwhat kinds of messages motivate them can be useful for product personas. And knowing how much it costs to acquire customers is important to compare with how much value the productdelivers to users.
AcquisitionAcquisition metrics help you understand which sources or channels are the most useful fordriving customer growth. Typically, thesenumbers are marketing and sales-focused, but they are valuable for product as well.
Marketing folks will tell you about TOFU (the Top Of the FUnnel, not a tasty soy product). Basically,you need to identify real indicators that suggest your efforts to bring new customers to theproduct are successful.
8
Acquisition Metrics
Conversion from Lead SourceLead Sources are purely marketing metrics, but you
may want to keep them in mind, simply because
understanding what’s driving customers to fill out
a landing page form or reply to an email can be
informative to what users expect from the product.
Unfortunately, product teams tend to ignore
marketing metrics and often vice-versa. More
communication between groups can lead to better
customer onboarding, better retention and ultimately,
better product.How to calculate
9
Acquisition Metrics
Conversion to Signup RateConversions can be helpful to keep an eye on
since they can give you a good picture of what users
are looking for. For example, if you have great
conversions at signup but a low conversion rate
from trial, it could be that you are promising some-
thing you can’t deliver on to customers, that you
are initially selling to the wrong person, or product
usability is a problem.
Based on the action you want your user to
accomplish, you can measure conversions for
any behaviour, but signing up and paying for your
service are key metrics.
How to calculate
10
If you have lots of customers signing up for a 14 or
30-day trial and your Conversion from Trial rate is
low, you may want to conduct surveys or user
interviews to find out where you’re going wrong.
Growth Hacker Lincoln Murphy suggests tracking
the “Common Conversion Activities” your success-
ful customers perform, such as Feature Utilization,
Lead Source and reading your blog or helpcenter
materials. Compare these metrics in a dashboard
like Notion’s to assess what actions to take.
Acquisition Metrics
Conversion From Trial Rate
How to calculate
11
Acquisition Metrics
Customer Acquisition Cost (CAC)
This metric measures how much it costs, on
average, for your company to acquire a user.
Sometimes companies, especially in ecommerce,
also measure Cost Per Action (CPA) which applies
to the cost of signup rather than the total costs
associated with acquisition, which include sales
salaries, promotions and marketing campaigns.
In general, early-stage companies will spend as little
as possible on customer acquisition until they are
sure they have a working product with product-
market fit. Once you are prepared to spend money
to acquire customers, you’ll need to be very clear on
who to spend it on, so some form of user personas
are recommended. You’ll also need to understand
the LTV (customer lifetime value) so you know if
spending more makes sense.
How to calculate
The Metrics
Every company will define “active” and “utilization” differently. A user could be “active” if they login on a regular basis, or perform an action likeinstalling code or making a profile.
Some products need hours of use every day to make it useful, whereas some will work just fine if users visit weekly or a couple times a month. We’re curious what works best for you, so drop us a line and we’ll incorporate your thoughts into a future lesson in the School of Little Data.
So, your marketing and sales teams are amazing and you have tons of users signing up. All done, right? If you’re working on product, this is where the journey actually gets interesting. By under-standing what your users are actually doing, you can see if they are understanding the vision you have to make them successful, or just getting stuck. Data can give you insight into where your product is most useful and where it’s superfluous or even annoying. With this knowledge, you can reduce Churn and increase LTV.
Activation
13
Activation Metrics
Average Sessions Per UserA user’s session can be defined as the interval of
time in which you are receiving events from that
user if the time between events is less than 30
minutes.
Average Sessions per User is a simple metric that
will help you understand how your user in engaging
with your product. In many cases, it’s great when a
customer is using your product frequently. Based
on the job your product does, you can speculate
how many times your customer would optimally use
your product, and set goals.
How to calculate
14
Activation Metrics
Average Session Time can give you insight into the
engagement of your customers. After all, the more
time they are on the site, the better, right?
However, AST typically includes time that your
customer is “in the product” but may not be using
it all. You can analyze this metric better in conjunc-
tion with Average Sessions per User and specific
product utilization metrics.
Average Session Time
How to calculate
How to calculate
15
Activation Metrics
A number of measurements can help you understand
how users are exploiting your product, and which
features they don’t care—or possibly know—about.
These will tend to be somewhat specific to your
product, but generally, you will want to collect data
on how many customers used a feature, whether
they used it repetitively (x2-∞), or whether they “con-
verted” within the feature (performed the action you
designed for, such as creating a report). Typically this
metric will be most relevant for the onboarding peri-
od, the first 30–90 days.
Feature Utilization
16
Activation Metrics
This metric seems basic until you have to define
“Active.” You might imagine that the user is “Active”
if they visited a dashboard or installed a piece of
code. But different types of users may be measured
by different definitions of “Active”. Ideally, you will
define it by an action or set of actions that suggest
the customer is meaningfully using the product.
DAU, WAU, and MAU are often considered vanity
metrics. You may find it useful, though, to compare
the MAU/DAU ratio to see how “sticky” your product
is. It’s harder to retain customers who only use your
product once in a while.
Daily/Weekly/Monthly Active Users (DAU/WAU/MAU)
How to calculate
The Metrics
in your product or analytics from your marketing site or from social media.
If you have sharing or referral features in-app, you will want to understand who is activatingreferrals and to whom you are being referred.
ReferralMaking your product shareable makes it much cheaper and easier to acquire new customers.So you’ll likely want to understand the ways your customers are referring you.
Some of the most successful companies rely on referrals within the product. Dropbox famously grew by offering a credit for referrals, and Uber let users give free rides to friends. Make sure your referrals can be tracked and added to your tracking system. This may involve tracking events
18
0-6 Detractors: customers who are likely to conveynegative impressions to their network
7-8 Passives: customers who are mildly positive toindifferent about the product
9-10 Promoters: customers with high satisfactionwho will actively evangelize about the product
Referral Metrics
NPS was developed to give a metrically significant,
simple way of understanding customer satisfaction.
In traditional NPS surveys, you ask your customers
“How likely are you to recommend our product?”
on a scale from 0-10, with 10 as the most likely.
Typically, this is the only question in the survey, with
the option of providing qualitative feedback by
asking an open-ended question like “Why did you
pick that number?”
A principle of both Lean methodology and NPS is
“closing the loop,” meaning that customers who
offer feedback should be contacted and heard.
This process gives you an opportunity to make a
detractor into a promoter, and also gives you
insights into the customer’s experience.
Asking for an assessment of “satisfaction with”
versus “likely to recommend” has been shown to
be as or more effective to NPS.
Net Promoter Score (NPS)
How to calculate
How to calculate
19
Referral Metrics
Many software products include a way to invite new
users, since ideally, your existing customers are the
most effective source of generating qualified new
customers.
You want to make sure that the methods you’re
using to generate these referrals are effective.
Tracking how well your inproduct referral engines
are functioning can help you understand what
methods work best, what customers are promoting
you to others and how to better encourage new
customers to use the product.
Product Invite Conversion
20
Referral Metrics
Typically social media stays in the “marketing
domain” and is certainly a “vanity metric,” but
keeping track of engagement numbers like shares
via Twitter or Facebook relative to product release
cycles can help you understand your customers
as well as help in thinking about ways to generate
shares within the product.
You can also think about sentiment when tracking
social media. It’s great when users are engaged,
but a little less awesome when they are all unhappy.
On the other hand, negative sentiment can give you
visceral feedback on your product pain points.
Referrals via Social Media
How to calculate
The Metrics
Another key KPI related to Retention is Churn, which can tell you if your rate of losing customers is a cause for concern.
Retention matters for SaaS companies far more than for many other businesses. For example, if you sell diamond engagement rings, you likely don’t expect or need your customers to come back often. On the other hand, a company that sells subscription-based marketing software will likely lose a customer if the user doesn’t return to the product on a regular basis.
Retention can be the most important product metric for SaaS and is often overlooked. Build-ing retention into your product should start with your first touch with a user, ensuring that they have a complete understanding of the ways your product can help them and how to achieve those goals.
To measure retention, you will typically look at activation metrics (i.e. how you defined an “Active User”) and calculate the average instances ofactivation per user or per account.
Retention
22
Retention Metrics
Customer ChurnA typical solution is to calculate churn as (number
of customers who churned) divided by the sum
of (customers at the beginning of the period) +
(customers at the end of the period) divided by 2.
Unfortunately, this won’t give much of a picture of
the trend of churn: if there is a significant difference
between January and March, it will be hidden. One
way to attack this problem is to calculate churn with
a daily average, then multiply over the timeframe
you want to measure.
Churn is a key metric that touches everyone at the
company, so experiment with different formulas
until you find the one that works best for you and
make sure everyone understands your goals.
How to calculate
23
Retention Metrics
Complaints per CustomerAlso known as “Active Issues,” this metric lets you
know how many issues customers are having at
a given time. Knowing what issues customers are
having can help you make better product decisions,
while understanding Resolution Rate (how fast
issues are resolved) can be instructive about the
kinds of solutions needed.
For example, you may simply need to improve UI
so that customers better understand how to use
a feature. On the other hand, there may be more
complicated issues related to product-market fit
or persistent technical issues. Using user issues in
the development process is essential to creating
products your customers love.
How to calculate
24
Retention Metrics
Feature RequestsFeature Requests might be considered more of a
qualitative metric than an actionable measure. That
said, tracking requests by category can give you a
sense of how well your product is loved or even
understood.
For example, if you notice that there are persistent
feature requests for a feature you already have, you
need to do a better job of uncovering the feature for
new and existing customers. If you notice growing
feature requests in a certain category, you may want
to investigate further to find out if there is market
potential for growth in that area.
25
Retention Metrics
Revenue ChurnCompare Customer Churn and Revenue Churn to
help you understand your user trends. For example,
you could have a net increase in revenue with an
increase in customer churn if you gained a number
of people on a $99/month plan but lost many more
customers at your $5/month level. Low-value
customers are typically more likely to churn because
they’ve made less of an investment overall in your
product.
New sales don’t count. You can also calculate
Revenue Churn in regards to certain segments,
like RC based on downgrades or RC based on
cancellations.
How to calculate
The Metrics
Revenue metrics also help you plan product de-velopment, since you don’t want to build a lot of features if no one is there to pay for them. At an earlier-stage company, you may not be able totell much from a snapshot of revenue, butcomparing these metrics from month to month can be a good way to see if you are heading in the right direction.
You love your product and making and improving it can be fun and challenging. But at the end of the day, you need to make money so you can eat and pay for things like cool quadcopter drones (or, you know, whatever YOU need). Revenuemetrics help you see how well your product is resonating with customers based on whether they find it valuable enough to pay for it.
Revenue
27
Measure the average amount of revenue each customer pro-
duces in connection with data on upgrades and downgrades
as well as Churn to understand how you can increase ARPU.
ARPU is sometimes considered a vanity metric. If you’re just
dividing the sum of your MRR by your number of customers
(the simplest way to calculate ARPU), you might not learn
enough that’s actionable.
To improve the way you use ARPU, consider a cohort analy-
sis, or just compare signups from the last six months versus
older customers (or whatever sample makes sense for you).
If ARPU for new customers is higher than for ones who have
been around a while, then you might be on a better track
than if the reverse is true. ARPU can reflect whether you are
selling to the right people at a company or if there is prod-
uct-market fit. Your ARPU needs to be high enough that you
can afford to acquire the type of customers you serve.
You will use ARPU to calculate another KPI, LTV (customer
lifetime value).
Average Revenue Per User (ARPU)
How to calculate
28
Revenue Metrics
Customer Lifetime Value (LTV)LTV gives your marketing team a baseline on CAC
(Customer Acquisition Cost). You don’t want to
spend more on getting a customer than they are
worth in their lifetime. Typically, the taget ratio of
LTV:CAC is 3:1 or better.
For more accurate results, you can compare LTV of
more recent customers to your earlier customers to
see if you are trending higher.
Another interesting aspect of LTV is churn. Statisti-
cally, lower-value plans tend to churn more, since
the customer tends to be less invested. So let’s
all charge $10,000 a month! Unfortunately, the
corollary generally tends to be fewer customers
on pricier plans. You’ll need to look at the total
revenue per plan level to get a more balanced
picture.
How to calculate
29
Revenue Metrics
Monthly Recurring Revenue (MRR)Though the more punk rock among you may
associate this acronym with “Maximum Rock & Roll,”
Monthly Recurring Revenue is a bedrock of business
KPIs. You know that you need to know how much
money is coming in each month, but then what?
Comparing MRR with factors like team performance,
customer satisfaction, and feature utilization can
offer useful insights and areas to test.
How to calculate
30
Average Revenue Per EmployeeThis metric can help you in two ways. First of all, you
can compare it to industry standards to see how
you compare to your competitors. For example, the
average RPE for publicly traded SaaS companies is
about $200k.
Even more importantly, it can help you to under-
stand when you have the opportunity to hire, and
what effect adding or subtracting staff has on the
overall growth of your company. As a lagging
indicator, RPE is more helpful in understanding
how the company is performing than as a metric to
optimize for.
Revenue Metrics
How to calculate
The Metrics
KPIs such as DRE, Total Defects/Bugs, lines of code, and other metrics are typical ProductQuality measurements. These metrics can be helpful to understand the technical quality of your product, but it’s a good idea not to become too obsessive about metrics like these relative to others that focus more on the larger goals you have in delivering success to your customers. Development teams can also look at Churn and Customer Complaints to get a better overallpicture of product quality.
For Product Managers, team statistics can help to drive quality, efficiency and accountability during product development. In the past, it was hard to visualize the success of the development teamin relation to other metrics. After struggling with this issue in our previous companies, we createdNotion to help you track and compare your team’s success, their morale, and effectiveness against business metrics like revenue or custom-er behavior.
Team
32
Team Metrics
Your Backlog Burn Rate tells you how fast your team
is accomplishing the tasks and stories in the back-
log. This metric gives context to your Velocity and
Sprint Burndown charts so you can determine if
your team is prepared to tackle upcoming projects.
Use this metric to help plan new features and up-
coming sprints.
Read more about Backlog Burn Rate in the School of
Little Data.
Backlog Burn Rate
How to calculate
33
Team Metrics
This KPI measures how much it costs you to actually
make your product, primarily in terms of staff and
software.
Use this KPI to compare with MRR and ARPU to see
if you are pre- or post-breakeven. You don’t have
to know the exact salaries of your team—you can
simply take their estimated average annual salary x
1.15 (adding 15% overhead for benefits and payroll
taxes).
Cost of Development Per Unit
How to calculate
34
Team Metrics
Cycle Time measures the completion rate of your
product. It is always a subset of Lead Time, which
measures the time from a ticket being created to
the time the work is completed. Cycle TIme mea-
sures the actual time it took to build.
Cycle TIme is the inverse of Throughput. It’s related
to Lead Time by the following equation: Lead Time =
Cycle Time * WIP, according to Little’s Law.
Cycle Time
How to calculate
35
Team Metrics
Defect Removal Efficiency (DRE)DRE is a percentage of the bugs identified and
corrected internally compared to the total bugs
in the complete release life cycle. So, DRE is the
percentage of bugs caught by code reviews, unit
tests, and QA.
To use a DRE effectively, you may want to classify
bugs by common sources, like Design, Build,
Requirements, or Deployment to Production. You
can also create an overall DRE to assess general
quality in the product.
Read more about DREs in the School of Little Data.
How to calculate
How to calculate
36
Team Metrics
Lead Time is the average time it takes between the
creation of a ticket and its completion. You can cal-
culate Lead Time as Cycle Time multiplied by WIP or
as WIP divided by Throughput.
Some consider Lead Time to be the overall measure
of product development process. Being quick and
responsive to new customer stories can give a
product an edge in a competitive market.
On the other hand, good product management
requires thoughtful feature development rather
than answering every customer’s demand, so Lead
Time may be deceptive. A good practice is to
compare this metric in relation to actual customer
satisfaction or NPS.
Lead Time
How to calculate
37
Team Metrics
How your team actually feels about the work they
are doing can be instructive for the health and
success of the product itself.
By tracking how your team feels about product
quality, morale, and efficiency, you can correlate
team health with product development. Of course,
it’s a great idea to have a sense of how your team
feels for other reasons, like keeping them happy
and productive.
To facilitate this metric, Notion includes the Team
Polling feature, a lightweight survey tool that keeps
track of how team members assess accuracy,
defects, morale and other aspects of their work.
Read more about Team Health in the School of Little
Data.
Team Health
38
Team Metrics
Throughput is a simple metric that compliments
Velocity when used correctly. Velocity alone doesn’t
reflect the quality of work along with how much
effort it took to do, but by adding Velocity and
Throughput, you’ll be able to understand the real
results you are getting.
Software development doesn’t ever “finish” so you
need to have a metric that helps you understand
how much you are getting done over time.
Track throughput over long periods as well as
sprints to help you ensure you are consistent and
sustainable. Throughput should be tied to your
specific goals rather than inconsistent factors like
story points, which can vary over different teams.
Throughput
How to calculate
39
Team Metrics
Bugs are a problem for camping or cookouts and
they’re even worse when they start piling up in your
product.
Having a picture of how many bugs you have is criti-
cal to product planning and will often correlate with
Customer Complaints. This metric is a component of
DRE.
Total Defects/Bugs
How to calculate
40
Team Metrics
Your WIP is a simple key performance indicator
used to track how much work your team is doing at
any given moment.
Typically, too much in your WIP will lead to decreased
quality and performance for your team. Limiting
context shifting for your developers leads to better
and faster work, and limiting your WIP to essential
components can improve your overall efficiency.
Learn more about WIP in the School of Little Data.
Work In Progress (WIP)
How to calculate
41
Team Metrics
Velocity is a leading indicator that measures the
effort needed to deliver value. Units can consist
of features, user stories, requirements or backlog
items. Use this metric along with Throughput and
Cycle Time to evaluate your development process.
If Velocity is constant, it can be helpful in planning
sprints. For example, if you plan a sprint over a two-
week period and one of those days were a holiday,
you might estimate a 10% reduction in Velocity
(assuming you’re not working weekends!)
Don’t focus on maximizing Velocity, since your DRE
can go down significantly if you’re only worrying
about what you got done, versus the quality of your
work.
Strong product management and ownership, fre-
quent and regular release cycles, and effective UX
design prior to development will improve velocity.
Velocity
How to calculate
Team Intelligence For Data-Driven Product Development usenotion.com
get in the way of a good experience. You may need to ramp up if your customers generally need more features. And you need to under-stand clearly how everything you are building will address real user stories.
No one metric can tell you everything, solook at your KPIs in relation to each other. We built Notion to give you a clear, sharable visual interface to compare your KPIs and increase product insight for your whole team.Let us know how we can help you think about setting up or analysing your KPIs. Now that you’ve done all this learning, we’d like to offer one more recipe: The Pirate’s Perfect Dark & Stormy. Don’t forget to stir!
KPIs help you understand your progress. A good Product Manager also needs to consider the customer’s experience through the pro-cess of using a product. When you decide what KPIs to track, consider how your customers are actually interacting with you and what would show you that they are successfully using your product. Don’t focus on “vanity metrics” that don’t really help you understand the needs of the customer and if you’re offering the value that they’re looking for.
When you look at your team’s success, keep in mind that your main goal is to create value for your users. You may need to slow down if you’re generating a lot of bugs or errors that
The Map to Real Treasure
Team Intelligence For Data-Driven Product Development usenotion.com
AARRRT!
How to calculate