Product Metrics Trap

Vanity metrics.

Updated: 20^th July 2020

Product managers are expected to be driven by data. The results of their decisions must be supported by metrics that are examined over time to determine if the goal was reached or not. However, there is a trap in the numbers that can easily give Product Managers a false impression of success: vanity metrics.

The concept of vanity metrics was introduced by Eric Ries, author of The Lean Startup. Precisely, it defines measurements that look good on paper and to the media. Conversely, these metricst provide little insight into what’s the cause and what to do next. You can see examples of vanity metrics every day such as an increase in visits, number of downloads or user growth.

Certainly, achieving 1,000,000 downloads can be the result of a Product Manager’s job. Equally, it can also stem from the sales effort, or customer support, or a combination of all (or even none of these). Moreover, what will you do next? How can you replicate this behavior? Can it be improved? To avoid falling into measuring vanity metrics, Ries proposes three criteria to test “good” metrics:

Actionable

The metrics you decide to measure must have a clear cause-effect relationship. This will allow you to know exactly what to do to if you want to replicate or improve results. You can learn from every action you take.

Accessible

Metrics are nothing if they are not understood by the people whose decision they must influence. As a Product Manager, you need every team who you work with to understand what it means that the churn rate went from 5% to 10%. There are two options to make this happen. Either simplify the language with which you present your results, or train everyone in the technical terms that need to be used.

Auditable

Good metrics are unbiased. Anyone measuring them should be able to come to the same conclusion. This will allow you to build trust from the teams you work. Also, it will limit finger-pointing when things don’t go as expected. An auditable metric should be able to stand the test of real customer discussion: what your customers say validate what your reports do.

Conclusion

As a new Data Product guy, vanity metrics might seem attractive but provide little long-term value to you and your product. Avoid falling into that trap by using actionable, accessible and auditable metrics. The best Product Managers understand the difference between the two, so following these simple guidelines will set you off to a great start in your new career path.

Subscribe to my newsletter

To get latest Product and Data blogs.

Want to get in touch?
Drop me a line!

And coffee's on me ☕.

You can connect with me on links provided below.