Measurement vs. metrics in product analytics instrumentation

Introducing the Series: Principles for Product Instrumentation Success

I meet lots of teams that are apprehensive about deciding what to measure. They are worried they will be unable to answer important questions. Worried they will track the wrong product metrics, and pick the wrong KPIs. Worried that they’ll end up with unusable and untrustworthy data. And worried they’ll waste their developer teammate’s time.

I can totally relate to these concerns! So in this series of posts I want to share some tips that reduce the risk of things going wrong, and increase the chance that things will go…right!

The series is broken up into seven key principles:

1. Measurement vs. metrics
2. Mixing approaches to measurement
3. Staying grounded in the customer domain
4. Learning how to “see” data
5. Unlocking the long tail of insights & t-shaped instrumentation
6. Learning how to ask better questions
7. Working small and working together

We’ll start with something super simple and not linger too long on the theory and definitions.

Measurement vs. Metrics

The National Institute of Science and Technology differentiates measures and metrics as follows:

Practically, applied to products, this means that we measure “events”—things that happened that we care about (thank you Alberto Brandolini)—and then combine these measures together to derive metrics. We call important metrics key performance indicators (KPIs).

To measure things that happened that we care about, we typically add code to our existing code, and that code records the event (somewhere) along with a timestamp and additional context.

A helpful way to think about the difference between measurement and metrics is to consider a product like Fitbit (a fitness tracking device). A Fitbit has a variety of sensors:

• 3-axis accelerometer
• Optical heart rate monitor
• GPS
• Vibration motor
• NFC
• Altimeter

These sensors are used to measure things like heart rate (every couple seconds), your location and altitude, when you take a step, etc. From those measurements, a Fitbit offers metrics like:

• Active Zone Minutes
• Cardio Fitness Score
• Time spent in light, deep and REM sleep
• Steps (a big deal for Fitbit users, maybe more of a KPI)
• Distance, calories burned, active minutes, hourly activity and stationary time
• Resting heart rate

Product analytics are not that different.

When we instrument products for product analytics, you can think of it as

1. measuring when events of interest happen, along with
2. additional information about the event (the where, how, how much, etc.), and
3. the information about the actor: the human or machine that triggered the event.

Why is the distinction between measurement and metrics important?

When a team asks what they should track, I ask them for examples of things that happened that they care about. They look at me funny and say “noooo, what metrics should we use, what do our competitors use?” I’m thinking about measurement related to the product experience (to enable meaningful metrics), and they’re thinking about generic metrics.

Here’s a second reason this distinction is important. Let’s go back to Fitbit. If you’ve followed Fitbit over the years, you’ve observed that they have added some measurements (and improved the way they measure), but they’ve added far more metrics and new insights. With that core set of measurements, you are able to unlock a dizzying array of metrics, insights, and guided/personalized functionality. Similarly, with products, a smart mixed-pattern approach (see next week’s post) sets you up for great insights, both in the present, and in the future when they become available.

The full series:

Part 1: Measurement vs. Metrics

Part 2: Use a Mixed Pattern Approach to Instrumenting your Product

Part 3: Keeping the Customer Domain Front and Center

Part 4: Learning How to “See” Data

Part 5: The Long Tail of Insights & T-Shaped Instrumentation

Part 6: Asking Better Questions

Part 7: Working Small and Working Together