Measurement vs. Metrics in Product Analytics Instrumentation

Introducing the Series: Principles for Product Instrumentation Success

I meet lots of teams that are apprehensive about deciding what to measure. They are worried they will be unable to answer important questions. Worried they will track the wrong , and pick the wrong KPIs. Worried that they’ll end up with unusable and untrustworthy data. And worried they’ll waste their developer teammate’s time.

I can totally relate to these concerns! So in this series of posts I want to share some tips that reduce the risk of things going wrong, and increase the chance that things will go…right!

The series is broken up into seven key principles:

We’ll start with something super simple and not linger too long on the theory and definitions.

Measurement vs. Metrics

The National Institute of Science and Technology as follows:

Practically, applied to products, this means that we measure “events”—things that happened that we care about (thank you )—and then combine these measures together to derive metrics. We call important metrics key performance indicators (KPIs).

To measure things that happened that we care about, we typically add code to our existing code, and that code records the event (somewhere) along with a timestamp and additional context.

A helpful way to think about the difference between measurement and metrics is to consider a product like (a fitness tracking device). A Fitbit has a variety of sensors:

• 3-axis accelerometer
• Optical heart rate monitor
• GPS
• Vibration motor
• NFC
• Altimeter

These sensors are used to measure things like heart rate (every couple seconds), your location and altitude, when you take a step, etc. From those measurements, :

• Active Zone Minutes
• Cardio Fitness Score
• Time spent in light, deep and REM sleep
• Steps (a big deal for Fitbit users, maybe more of a KPI)
• Distance, calories burned, active minutes, hourly activity and stationary time
• Resting heart rate

Product analytics are not that different.

When we instrument products for product analytics, you can think of it as

1. measuring when events of interest happen, along with
2. additional information about the event (the where, how, how much, etc.), and
3. the information about the actor: the human or machine that triggered the event.

Why is the distinction between measurement and metrics important?

When a team asks what they should track, I ask them for examples of things that happened that they care about. They look at me funny and say “noooo, what metrics should we use, what do our competitors use?” I’m thinking about measurement related to the product experience (to enable meaningful metrics), and they’re thinking about generic metrics.

Here’s a second reason this distinction is important. Let’s go back to Fitbit. If you’ve followed Fitbit over the years, you’ve observed that they have added some measurements (and improved the way they measure), but they’ve added far more metrics and new insights. With that core set of measurements, you are able to unlock a dizzying array of metrics, insights, and guided/personalized functionality. Similarly, with products, a smart mixed-pattern approach (see next week’s post) sets you up for great insights, both in the present, and in the future when they become available.

The full series:

Part 1: 

Part 2: 

Part 3: 

Part 4: 

Part 5: 

Part 6: 

Part 7: