Measuring to Control vs. Measuring to Learn and Improve

I spend a good deal of time facilitating measurement/metrics related workshops, and I’ve noticed a pattern. When teams embrace measurement as a catalyst for learning and a way to encourage aligned autonomy, their brainstorming efforts are a lot more productive. Team members feel safe to surface their assumptions and core beliefs. The ideas flow, and the teams are much more likely to converge on “good enough” metrics and models.

Unfortunately, this is not always the case. We’ve all experienced abuse of metrics, KPIs, and quantitative goal setting. Examples abound. A product manager cherry-picks data to make their case. A team is pushed to make stretch-goals, they cut corners, and those cut corners come back to haunt them. A model is criticized for not being “perfect”, so in the future people either refrain from establishing models all together, or they add layers of false certainty to get “buy in”. Or, as I experienced recently on a coaching call, a leader rips apart a metric instead of being more curious about the underlying assumptions surrounding the metric (probably because it wasn’t very flattering to the current strategy).

Teams that are worried about these anti-patterns will have a hard time establishing the safety necessary to learn, and learn about how they learn (see double loop learning).

Sometimes, there isn’t obvious dysfunction. The environment is reasonably healthy, and the resistance is more about perfectionism and naturally high expectations. We’ve all read about companies coming up with powerful insights that changed the way they operate (e.g. see Facebook’s “7 friends in 10 days”), or about magic A/B tests that cracked the free-trial to paid customer upgrade puzzle. What you don’t often hear about is just how long it took to converge on those insights, and if those insights stood the test of time.

Nine times out of ten it was an extended learning process. There were missteps, u-turns, and red herrings. The team iterated on the model. Tried something. Elicited new insights. They shared reports that proved to be “wrong”, and set out to be less wrong with each increment and iteration. It’s hard work, which makes it valuable.

So you have two (sometimes related) problem flavors: one centered around abusing metrics and low safety, and the other centered around perfectionism and fear of uncertainty. Both flavors can make it very hard for teams to get the most out of measurement, insights, designing and testing models, and experimentation. Which leaves the obvious question: how do you battle metrics abuse, improve psychological safety, and leave teams more comfortable with uncertainty?

At least based on my coaching experience, it can make a world of difference to have a senior leader in the room who openly admits that they don’t have all the answers. The impact is almost immediate. The leader’s “I really don’t know” quickly turns into other people admitting what they don’t know, and also ramps up the levels of visible curiosity. Product development seems to attract passionate puzzle/problem solvers. “I don’t know” is a powerful incentive for the team to dig deeper. Related, is being sure to mention that a lack of certainty is actually a signal of untapped value and impact.

Second, it is extremely important to stress the iterative nature of model design and testing, and whenever possible decouple certainty in the model from compensation and assessing performance. Recently I worked with a startup that could not settle on a basic growth model that represented their beliefs and bets. It was getting contentious. They’d get close, but then back away when the people in the room grappled with the reality of their teams being assessed and “graded”.

They had a perfectly useful model that represented the uncertainty inherent in any startup, but they couldn’t agree because the model was serving dual purposes: learning and control/management. Steve Blank famously defined a startup as “an organization formed to search for a repeatable and scalable business model”. The operative word is search, which implies learning and uncertainty. Most measurement efforts require lots of learning. The perfect model is an aspiration. In terms of incentives, it is extremely important to reward this cycle of gradually improving confidence in models vs. only incentivizing the big, visible, (seemingly) certain wins.

Third, sometimes you need a change of scenery. A quick hack I’ve used recently is to try to get the team out of their head—and product (and politics and fear)—and think about metrics in an unrelated context like city health and wellness, predicting relationship success, or modeling a successful college “career”. What is incredible about this exercise is how creative people can be once they step away. Topics like causation vs. correlation, confidence, etc. become much easier to discuss. A central facet of these activities is that we start with beliefs and “made up” measures (e.g. “argument resolution time”), to lessen the intimidation of picking the “right” metric.

Finally, though I am not really a fan of skunkworks and siloed innovation efforts, I do have an easier time coaxing teams to take more risks, and expose their core beliefs, when they are dealing with a product (or decision) that has a more contained blast radius. The core success metric for your “main” product is likely wrapped up in politics and egos. The same may not be true for a new product, a specific, tangible near-term decision, or a product area known for being difficult to use and in need of love. If you’re experiencing resistance and hesitancy, consider tackling a less contentious problem that you have already committed to solve. This, in turn, will make it easier to brainstorm and test different measures.

Wrapping up, rest-assured that no team—and I’ve met some of the most data savvy—magically gets this. At Amplitude we are always revising our understanding, tweaking, cleaning up data, and reassessing. That’s how it works. I highly recommend reading How to Measure Anything: Finding the Value of “Intangibles” in Business by Douglas W. Hubbard for a thoughtful approach to tackling measurement problems. And finally, realize that representing reality—what you know, know you don’t know, etc.—is the first step towards meaningful progress. Reflect the reality in the room as a starting point.