The ultimate guide to analyzing data: Part 2

How to interpret metric trends and identify risks and opportunities buried in the data

This is part two of a series on how to analyze your business to understand trends and uncover risks and opportunities. Last week I covered:

1. What metrics to track: How to establish the revenue equation and driver tree for your business

2. How to track: How to set up monitoring and avoid common pitfalls.

In this post, we’ll dig into how to analyze the data to interpret trends and identify risks and opportunities:

3. Extracting insights: How to identify issues and opportunities in a structured and repeatable way! We’ll go over the most common types of trends you’ll come across, and how to make sense of them.

Let’s get into it.

Chapter 3: Extracting insights from the data

Having a ton of data is great, but how do we translate it into insights?

You won’t have time to dig into every metric on a regular basis, so prioritize your time by first looking at the biggest gaps and movers:

• Where are you missing your goal? Where do you see unexpected outperformance?

• Which metrics are tanking? What trends are inverting?

Once you pick a trend of interest, you’ll need to dig in and identify the root cause so you can come up with targeted solutions.

In order to provide structure for your deep dives, I am going to go through the key archetypes of metric trends you will come across and provide tangible examples for each one based on real-life experiences.

1. Net neutral movements

When you see a drastic movement in a metric, first go up the driver tree before going down. This way, you can see if the number actually moves the needle on what you ultimately care about; if it doesn’t, finding the root cause is less urgent.

Example scenario: In the image above, you see that the visit-to-signup conversion on your website dropped massively. Instead of panicking, you look at total signups and see that the number is steady.

It turns out that the drop in average conversion rate is caused by a spike in low-quality traffic to the site; the performance of your “core” traffic is unchanged.

2. Denominator vs. numerator

When dealing with changes to ratio metrics (impressions per active user, trips per rideshare driver etc.), first check if it’s the numerator or denominator that moved.

People tend to assume it’s the numerator that moved because that is typically the engagement or productivity metric we are trying to grow in the short-term. However, there are many cases where that’s not true.

Examples include:

• You see leads per Sales rep go down because you just onboarded a new class of hires, not because you have a demand generation problem

• Trips per Uber driver per hour drop not because you have fewer requests from riders, but because you increased incentives and more drivers are online

3. Isolated / Concentrated Trends

Many metric trends are driven by things that are happening only in a specific part of the product or the business and aggregate numbers don’t tell the whole story.

The general diagnosis flow for isolating the root cause looks like this:

Step 1: Go down the issue tree

• Keep decomposing the metrics until you isolate the trend or can’t break the metrics down further. Similar to how in mathematics every number can be broken down into a set of prime numbers, every metric can be broken down further and further until you reach the fundamental inputs.

• By doing this, you are able to isolate the issue to a specific part of your driver tree which makes it much easier to pinpoint what’s going on and how you should react.

Step 2: Segment the data

• By segmenting the data you can figure out if a specific area of the business is the culprit. By segmenting across the following dimensions, you should be able to catch > 90% of issues:

  • Geography (region / country / city)

  • Time (time of month, day of week, etc.)

  • Product (different SKUs or product surfaces (e.g. Instagram Feed vs. Reels))

  • User or customer demographics (age, gender, etc.)

  • Individual entity / actor (e.g. sales rep, merchant, user)

Let’s look at a concrete example:

Let’s say you work at DoorDash and see that the number of completed deliveries in Boston went down week-over-week.

Instead of brainstorming ideas to drive demand or increase completion rates, let’s try to isolate the issue so we can develop more targeted solutions.

The first step is to decompose the metric “Completed Deliveries”:

Based on this driver tree, we can rule out the demand side. Instead, we see that we are struggling recently to find drivers to pick up the orders (rather than issues in the restaurant <> courier handoff or the food drop-off).

Lastly, we’ll check if this is a widespread issue or not. In this case, some of the most promising cuts would be to look at geography, time and merchant. The merchant data shows that the issue is widespread and affects many restaurants, so it doesn’t help us narrow things down.

However, when we create a heatmap of time and geography for the metric “delivery requests with no couriers found”, we find that we’re mostly affected in the outskirts of Boston at night:

What do we do with this information? Being able to pinpoint the issue like this allows us to deploy targeted courier acquisition efforts and incentives in these times and places rather than peanut-buttering them across Boston.

In other words, isolating the root cause allows us to deploy our resources more efficiently.

Other examples of concentrated trends you might find:

• Most of the in-game purchases in an online game are made by a few “whales” (so you’ll want to focus your retention and engagement efforts on these)

• The majority of support ticket escalations to Engineering are caused by a handful of support reps (giving you a targeted lever to free up Eng time by training these reps)

4. Mix Shifts

One of the most common sources of confusion in diagnosing performance comes from mix shifts and Simpson’s Paradox.

Mix shifts are simply changes in the composition of a total population. Simpson’s Paradox describes the counterintuitive effect where a trend that you see in the total population disappears or reverses when looking at the subcomponents (or vice versa).

What does that look like in practice?

Let’s say you work at YouTube (or any other company running ads for that matter). You see revenue is declining and when digging into the data, you notice that CPMs have been decreasing for a while.

CPM as a metric cannot be decomposed any further, so you start segmenting the data, but you have trouble identifying the root cause. For example, CPMs across all geographies look stable:

Here is where the mix shift and Simpson’s Paradox come in: Each individual region’s CPM is unchanged, but if you look at the composition of impressions by region, you find that the mix is shifting from the US to APAC.

Since APAC has a lower CPM than the US, the weighted aggregate CPM is decreasing.

Again, knowing the exact root cause allows you to tailor your response. Based on this data, you can either try to reignite growth in high-CPM regions, think about additional monetization options for APAC, or focus on making up the lower value of individual impressions through outsized growth in impressions volume in the large APAC market.

Outlook: External Benchmarking

Looking at your internal data is not the only way to uncover risks and opportunities. You can also benchmark your business’ performance against other companies to get an idea of where you positively or negatively deviate from the norm.

I’m not going to go into detail here since this topic deserves a proper deep dive that would make this already lengthy post way too long. I’ll publish a separate follow-up post on this in the near future.