Is your startup going in the right direction? Introduction to metrics for startups

Why should you care about metrics?

During all my years as a mobile developer and now, as a product owner, I was always fascinated by the particular breed of people I loved working with — founders. No matter the domain — weather app, challenger bank or new femtech device — the audacious vision they usually had and the I-don’t-know-but-I’ll-figure-it-out attitude is something that made working for them extremely rewarding. The most accurate description of this “founders phenomenon” and the impact it has on workers can be found in Steve Jobs’ biography written by Walter Isaacson. In it he states that Jobs possessed extraordinary ability to convince himself, and others around him, to believe almost anything with a mix of charm, charisma, bravado, hyperbole, marketing, appeasement and persistence. It was said to distort his co-workers’ sense of proportion and scales of difficulties and to make them believe that whatever impossible task he had at hand was possible. Jobs was creating a reality distortion field around himself, and so are you dear Founder, that’s why it’s so pleasurable to work for you. But is it optimal in the long run?

A good proportion between people blindly following the vision and the people asking tough questions is a must for a business to succeed. That’s why I used to ask questions. A lot of them. Trying to keep management as close to reality as possible:

• Are you sure that we found product-market fit only because 1000 people downloaded our app last weekend?
• What is our unique value proposition? Why would users even choose us among all the competitors?
• It’s nice that we have 100 monthly active users, but are they in our target audience? How much time are they spending in the app? Can we monetize them in a way, which will exceed the cost of user acquisition?
• How do you know we should pivot now?

All these questions are meant to start meaningful conversations and help everyone in a team to align and be on the same page. But probably the most important thing about them is that they could all be answered with data — correct metrics set up in correct places visualized in a correct way. This is exactly why you should care about metrics in your startup. They aid in preserving balance between your reality distortion field and the harsh market reality, which leads to better decision making, which, in the end, can lead to your company becoming a unicorn.

In this blog post I’ll give you a solid introduction to the world of analytics and metrics for startups. I’ll start by showing you the tool that Pragmatic Coders very often uses with founders during our workshop sessions and explain why it’s important with regard to the topic we’re discussing. Later, I’ll describe what a good metric is and what are the most common types of metrics, including the dreaded vanity metrics. Next, you’ll read some of my thoughts about setting targets in your analytics. I’ll close the article with an introduction to the most popular framework used with metrics in startups — Lean Analytics framework.

The Lean Canvas

In the previous paragraph, I’ve stressed the importance of trying to find a good balance between a reality distortion field and a market reality. I have also mentioned how valuable a candid discussion is for business health. What’s more, you’ve probably already started thinking about what to track in your startup using some common metrics, but maybe you wonder if you missed anything? From my experience, there is one tool that can address all of the issues above. It’s called the Lean Canvas.

You can think of it as a one-page definition of all crucial areas in your startup. Plan a meeting with your team to fill it just before you decide what metrics to choose and you’ll be amazed with the level of clarity the Lean Canvas can provide. During workshops with our clients we always use sticky notes to complete each block in the canvas, since they can be easily discarded, changed or moved, promoting more flexibility, even long after the meeting is over. Take a look below for a short description of the most important sections from the Lean Canvas.

Problem

You should definitely try to fill this block first. Don’t start with a solution, try defining a problem you’re solving. For example, Uber solved a problem of unreliable taxi rides, which were often too costly and sometimes took forever to arrive without the client knowing how much longer he should wait.

Customer segments

Now go to this section. Do you know for whom you’re solving the problem? Is it only one group of customers or are there multiple ones? At the very beginning you need to be focused. For example, Facebook, at first, wasn’t targeted for a mass audience — you had to be a university student to get in. Founded in 2004, it opened its doors for all people over the age of 13 only in 2006.

Unique value proposition

Your app or website will do a lot of things, but is at least one of them significantly different from the ones the competitors already provide? Try to be concise, clear and honest. For example, all of my friends didn’t create a Revolut account because they liked the UI or wanted a new place to store their money. They loved the fact that Revolut provides the easiest and cheapest way to exchange currency and then use it abroad.

Solution

You’ve already defined common problems for given customer segments. How are you going to solve them? For example, the problem of unreliable taxi rides was solved by Uber with a mobile app containing an interactive map. On it, users could see where their taxi was. In addition, before they got in, they could see an estimate of the fare they would have to pay.

Channels

Connecting with customers and potential customers is crucial not only during the launch of a product, but at every step of the startup building process. Do you know how to reach them? Where can you find them? For example, if I were to create an online course about programming for beginners I might check active Facebook groups with people trying to learn a new programming language, go to student job fairs or stand outside technology university in my hometown, waiting for computer science freshmans to come out after classes. A carefully prepared LinkedIn post wouldn’t help me much, or could even attract an incorrect type of audience.

Revenue streams

How are you planning to make money? Subscription? Freemium? Ads? One-time payment like e-commerce? For example, Airbnb takes the majority of its revenue from booking service fees charged to both guests and hosts.

Cost structure

What are some of the most important costs you’ll have to deal with? For example, many of Pragmatic Coders clients, during their first stages of product building, spend the majority of money on development costs including UX services.

Unfair advantage

The things you’ll put here can’t be easily copied or bought, making your startup stand out even more. For example: patents, existing community, domain experts in the team, established personal brand with large following, years of industry experience, insider information.

Key metrics

Filling all of the previous blocks should have given you a clarity to come up with metrics that will indicate if your startup is going in the right direction. But don’t dive into this block just yet! In the next paragraph I’ll show you what are the characteristics of effective metrics for your startup along with a few common pitfalls you might come across.

What makes a good metric for a startup?

Below you’ll find a list of the most important characteristics of an effective metric. There is no need to follow each bullet point religiously, every business is different, but based on my experience, having a check mark next to all of them makes you, as a founder, less susceptible to being uninformed about the real condition of the product you worked so hard to put in the market.

Simple

This one is crucial. Imagine that a metric you’re trying to come up with is put on a large screen in your office, next to a team trying hard to make the product succeed. Would they understand it at first glance? Would they know how it’s being calculated? Would it require any explanation whatsoever for a new employee? The harder it is for team members to grasp, the harder it will be for them to act or even notice when the metric will go in the wrong direction. Complexity is your enemy here not only when it comes to quality but also quantity. That’s why Alistair Croll and Benjamin Yoskovitz in their book “Lean Analytics: Use Data to Build a Better Startup Faster” popularized the term OMTM — One Metric That Matters. The idea behind it is that at every stage of your business there is one number that you should focus on. It doesn’t mean that you ignore everything else, but, to promote focus, you should actually work on improving only this number, until you move to the next stage of your startup (you can read a bit more about these stages in the last paragraph). I highly recommend this approach.

Comparable

When defining a metric, ask yourself if it’ll be easy to compare. You might want to be able to compare it to some standard points of reference. For example, here you’ll find email marketing benchmarks prepared by Mailchimp, based on the vast amount of data they have. Find your industry and check if there is room for improvement in metrics such as average open rate, average click rate or unsubscribe rate. Maybe you’re developing an iOS app, which generates the majority of revenue from targeted ads? Here’s an interesting link to a blog post from one of the largest mobile analytics providers — Flurry. Is your opt-in rate below or above the average?

A good metric can also be compared to competitors. Say you’re developing a product that will directly compete with Brand24 — a media monitoring tool. Their CEO is quite vocal on LinkedIn about the milestones they achieve. Use it to set realistic targets for your metrics.

To make your metric comparable to other time periods, user segments and cohorts stick to a ratio, rather than a plain number. Ratios are comparable by nature, make trends more visible and help to define whether the startup is really growing. For example, the number of new users the app gained this month does not tell us much until we divide it by a one-year average of the number of new users creating an account every month.

Actionable

A good metric translates to a change of behavior in your startup and informs business decisions. It should be constructed in a way that empowers employees and management to act when the metric goes down or suddenly skyrockets. This characteristic is especially visible when performing product experiments, during which an experimenter starts with setting a clear hypothesis, for example “Our vacation rental business will generate 30% more revenue in the next 3 months if we use professional photos of the apartments in our marketplace”. What should founders do if the hypothesis is correct? Is a metric change leading to an action? Is revenue the correct number to track in this case or can founders be informed a bit earlier that they’re onto something? Whereas the answer to the last question remains debatable (maybe an increase in the number of bookings on a property compared to a previous month might be a better option), the answer to the first two can be observed on Airbnb website.

Accurate

There are three most common factors that might mess with the accuracy of metrics in a startup: technical, data related and psychological. Technical are often the simplest to fix. For example, data from the analytics tool might not be reaching the server due to a bug in implementation or misconfiguration, that’s why it’s so crucial to thoroughly test every user path before deploying the code to production. When you’re receiving product statistics, but they are really polluted e. g. they have missing values that might be interpreted as zeros, you are dealing with data related factors and probably need a small help from a data scientist. Inaccuracy connected with human psychology tends to be a bit harder to tackle, but it’s crucial for founders to be aware of this phenomenon. It can be summarized by a famous quote from a TV series “Dr House”:

For example, which of the below metrics would you consider more accurate to evaluate your new app idea?

1. Percent of your family members and friends, who answered “Yes” to the question “Do you like my new app idea?”.
2. Percent of people from a simple targeted Facebook ad, who clicked the link taking them to the landing page of your new app, which you haven’t even started developing yet.

The description of ways to alleviate this problem might require another full-scale blog post, so for now I’ll just leave you with increased awareness of its existence and move on to the most common types of metrics you’ll come across.

Vanity metrics

A few months ago I was looking for a reliable Facebook group about investing. I was still new to this and wanted to see what others think about some of the decisions I’ve made. After short research I found two:

1. 15 000 users and on average over 5 posts per day.
2. 700 users and on average 1 post per day.

I chose the first one and posted a question. After 3 days my post had 0 reactions and 3 comments from what appeared to be bots asking me to make some shady investments in crypto. Questions from other users seemed to be following the same pattern. I didn’t give up, joined the second Facebook group with 700 users and immediately got an absolutely fabulous set of responses from people, who knew a ton about investing. Now imagine owners of these groups having a small party with their friends. Who will impress more? “I have a small but very active Facebook group with 700 members” or “That’s nice, but I already crossed the 15k milestone and have 700 members join me every week”?

Vanity metrics are metrics that make founders look and feel good, but have very little to do with the actual performance and health of the startup. They make for a great headline in the press release or story during a party, but don’t provide enough context to make informed business decisions. In other words, they tick all the boxes from the “What makes a good metric for a startup?” paragraph except for the most important one — “Actionable”.

Vanity metrics can sometimes be tricky to spot, so below are a few common examples. Since every metric has a potential to become a vanity metric, be careful with the suggestions I’ve put in the brackets and embed them in your startup’s business context first.

• Page views (probably better metrics to focus on: time spent on page, number of new subscribers per 1000 page views)
• App installs (probably better metrics to focus on: time spent in app, monthly active users)
• Number of users on the email list (probably better metrics to focus on: open rate, click rate)
• Social media followers (probably better metrics to focus on: average post engagement)
• Total revenue (probably better metrics to focus on: customer lifetime value, cost of user acquisition)

Other categories of metrics for startups

I’ve already described characteristics of practical metrics: simple, comparable, actionable and accurate. They are opposite to dreaded vanity metrics, which don’t have that much to do with the real condition of a startup. There are however other contrasting categories that metrics can fall into. Use them to further refine and discuss with your team the reason for tracking a given number.

Qualitative vs quantitative metrics

I see a tendency among founders to quantify everything and, trust me, I’ve been there myself. After all, it’s easier and less chaotic to create a simple poll asking users to rate a sign up experience of the app from 0 to 10 and then plot the data using an Excel spreadsheet. Many people also think that this way of collecting information is statistically significant since they get thousands of responses in their surveys. It has to be, right? Well, not quite. The quantitative way, based on numbers, plots and order, is often not the best one. Personally, I’ve learnt more about flawed sign up experience in my app after watching only 6 users go through it, then by looking at the numbers in the analytics tool. Both qualitative (output from an in-depth user interview) and quantitative (rating of sign up experience from a user survey) metrics have their places when looking for product-market fit, but, from my experience, talking to users directly and listening to them usually gives founders the best results at the beginning of the journey with their startup.

Leading vs lagging metrics

Imagine that the churn rate in your SaaS company was 2% last month, which means that, compared to the beginning of the last month, 2% of subscribers discontinued using the software and stopped paying for it. Churn rate is a solid metric to track. It’s definitely simple to understand, can be compared to competition, provokes action (e. g. when after removing an old feature your churn rate increases, you should probably put it back) and is usually very accurate since it affects the bottom line. It is however a lagging metric because the damage is already done — the user deleted the account or canceled a subscription and, most of the time, there’s very little you can do about it. On the other hand, the leading metric of the event of losing a subscriber could have given you an indication that something was about to happen way before it actually did. For example, a number of software crashes or a number of complaints in a given month both may lead to user being very frustrated with your service and, in effect, discontinuing to use it a few weeks later. In order to have a full picture of the business reality, startups need to track both leading and lagging metrics.

Correlated vs causal metrics

Correlation does not imply causation. Some time ago, I found a nice explanation of this old statistics rule reading The Guardian: “Just because people in the UK tend to spend more in the shops when it’s cold and less when it’s hot doesn’t mean cold weather causes frenzied high-street spending. A more plausible explanation would be that cold weather tends to coincide with Christmas and the new year sales”. Finding the causation, especially for such valuable metrics like monthly recurring revenue, is the Holy Grail for many startup founders. It could possibly give the opportunity to control the factor that’s causing the metric to move and eventually swing it in the company’s favor. Looking for causation starts with finding a correlation and then performing an experiment by controlling every other factor. It is very difficult to achieve, especially in a dynamic startup environment, but can give a hint about what’s working and what’s not.

Setting targets for startup metrics

Once you carefully craft a practical set of metrics, you will eventually reach the point of asking “Am I doing ok?”. It’s nice to know that your churn rate is 2%, but should it be better? Maybe it would be more beneficial to stick to 2% and improve on other numbers in the analytics? Setting targets for startup metrics is always tricky. There are myriad ways of going about it, so let me just focus on a very subjective approach stemming from my experience building apps used by millions of users.

I believe that it’s enough to set ANY target for a metric as long as it loosely fulfills 3 requirements. First of all, it must not be too outrageous. Yes, you are a visionary, but not everybody in your team is. The goal should challenge them, not make them cry or laugh at you when you’re not watching. Setting a correct magnitude of the metric’s target can increase morale and productivity among you and your colleagues. Secondly, it must not be too low. The business has to make money from the users engaging with the product. Now is a good time to take an educated guess if e. g. gaining five new users per day for the next month is enough to eventually make your startup sustainable. Use common sense, industry knowledge, available competition reports and benchmarks to avoid under or overestimation. Lastly, the target has to be time-bound. In other words, the success criteria for a metric require a specific value and a date before which this value should be reached. The self-imposed time constraint will force you to frequently ask the most important question you can probably ask yourself as a founder in a startup: “Why did it fail and what can I do to avoid it next time?”. Maybe the metric target was too ambitious? Maybe there’s a huge leak in the conversion funnel? Or maybe it’s finally time to pivot?

I want to emphasize my point about using common sense, industry knowledge, available competition reports and benchmarks to set targets for metrics. The order here is not random — common sense is the most important thing in this process. Take, for example, a metric often referred to as time spent in an app. If its value exceeds 30 minutes per user you might be happy, right? Maybe now it’s time to even increase the target to an hour? Think again. What if users are spending all this time writing complaints via in-app support chat? Or what if your app is a productivity one, created to save user’s time? It’s always a good idea to be careful and never allow yourself to compare metrics to industry benchmarks or competition reports without further diving into the business context of what’s exactly being tracked.

The Lean Analytics framework

After spending all this time learning about motivation behind tracking metrics, the Lean Canvas, characteristics of practical metrics, different categories of metrics and setting targets you might be a little overwhelmed. That’s why I want to conclude this article with a short introduction to the Lean Analytics framework, originally proposed by Alistair Croll and Benjamin Yoskovitz. It consists of five consecutive stages. Your business can only be at one stage at the time and you can move onto the next one only if one of the metrics (often defined as OMTM — One Metric That Matters, see paragraph: “What makes a good metric for a startup?”) shows clear indication of reaching the “gate”. The Lean Analytics framework promotes focus and can help you come to order with various metrics you want to track.

The stages of Lean Analytics framework

1. Empathy (“gate” needed to move forward: I’ve found a real poorly met need a reachable market faces).
2. Stickiness (“gate” needed to move forward: I’ve figured out how to solve the problem in a way they will accept and pay for).
3. Virality (“gate” needed to move forward: I’ve built the right product/features/functionality that keeps users around).
4. Revenue (“gate” needed to move forward: The users and features fuel growth organically and artificially).
5. Scale (“gate” needed to move forward: I’ve found a sustainable, scalable business with the right margins in a healthy ecosystem)

You can read more about this and other analytics frameworks in Chapter 5 of “Lean Analytics”. And since a picture is worth a thousand words, below I’m attaching an illustration comparing most of them:

Summary

In this blog post we took a closer look at the most important aspects any founder needs to take into account when measuring success (or failure) of a startup. We talked about the motivation behind tracking metrics, the Lean Canvas, characteristics of a good metric, vanity metrics and other metrics categories, targets and finally we wrapped up with an introduction to the Lean Analytics framework proposed by Alistair Croll and Benjamin Yoskovitz. After understanding key points I made in this article, you should have a pretty good idea of what to track, but most importantly why to track it. I do hope that this knowledge will help founders remain visionary, but also bring them closer to reality, whic