The Hard Thing About Product Analytics

I spent the last four years investing in high impact startups and helping them find product-market fit. While they all professed a desire to be data-driven and iterate rapidly, only a few realized that goal. Implementing a good product analytics process and creating a transparent, data-informed culture across an entire team continue to be extremely hard problems — only getting harder as every growing company becomes a “big data” company and wonders how “AI” will affect them. These are tired buzzwords, but have unfortunately real implications.

What’s different about product analytics?

Gartner defines product analytics as a specialized application of business intelligence towards gathering feedback on products. This definition is the crux of the problem.

Historically, the role of BI has been maintaining the central repository of truth about the history of a company. What was sales 3 years ago? What was the operating budget last month? How many paying customers do we have this week? In the trade-off between speed and accuracy, the latter always won. The analysis could be batched, as long as it was comprehensive. It was never required to be self-serve because the “truthfulness” had to be audited by a centralized data team before public consumption. If you needed to run some ad-hoc analysis yourself, you export data to the all-powerful Microsoft Excel. Hello, pivot tables!

None of this rings true for an agile product team in today’s move-fast-break-things culture. The ability to explore all data on user behavior in real-time could be the competitive advantage for a team.

Sample of behavioral cohorts, conversion funnels and retention analysis from the Amplitude Demo.

Product analytics needs to be real-time, collaborative and self-serve in order to align product vision during rapid cycles of iteration. It needs to be easy to use in order to create that elusive data culture. BI tools are simply not set up to serve these two particular goals.

So how are most companies solving their “data problem” today?

The older generation of companies — tech companies launched during the first dot-com boom and every other consumer business—rely heavily on centralized BI teams now managing data lakes and warehouses using Informatica/Hadoop/BigQuery/Redshift et al.

This often creates massive bottlenecks and data breadlines in companies. As Tomasz Tunguz elaborates in his book, the data poor are waiting in queues with their resource coupons, and often asking fewer questions as a result.

Companies that were born during the second internet boom and the dawn of the mobile era typically embed analysts in each team for fast results. This creates data silos in organizations with each team relying on their own tools to answer questions. It has led to this famously terror-inspiring market landscape of big data that continues to scare CIOs everywhere and seems firmly here to stay.

Another effect of this trend is a massive surge in demand for analysts and data scientists. Everyday, consumers create more and more data that teams need to access and explore, with analysts to manage and learn the specialist tools needed to access each silo. Some companies hire engineers to build and maintain their own in-house data analytics stack. Executives who are skeptical about the massive investments being made in data infrastructure often demand to see evidence that any of it is worth it. Those who do invest in building their own custom analytics often need to change it as soon as they are done, because of evolving business models and technology frameworks.

Ok, now we have data engineers, analysts and scientists. Isn’t it fixed?

If so, it was probably very expensive and you were very lucky. Congrats! However, when you look under the hood, here’s what you might find:

• Data scientists who want to work on advanced predictive models, but instead are answering basic questions for executives and spending more time cleaning data than answering anything at all.
• Engineers who want to build fancy machine learning features into your product, but instead are fielding endless last-mile requests from analysts across marketing, product and biz ops.

That’s two expensive and less-than-fulfilled groups of people you might be massively under-utilizing in your organization.

Instead of dedicating their valuable time towards building the future, they are stuck helping everyone else catch up to the present.

So what does the future of building product look like?

Despite all the advances made in machine learning, what we see and touch today within products is limited — just better tailored recommendations for what to click on next. What else you would like see, read and buy, but not how. The product itself is the same for every user, a one-size-fits-none experience.

The first, small step towards building products that can learn and adapt is to build teams that can learn and adapt instantly. Product managers, growth teams, marketers, customer success, analysts and executives who can both ask and answer any question about users. Organizations that can collaborate on experiments and share insights to make better choices. Teams where everyone is empowered to make objective decisions.

This needs an entire generation of enterprise analytics tools native to the big data stack (Redshift, Spark, et al) that are not only future-ready but also easy for everyone to use. So far, self-serve tools in product and marketing analytics have struggled to go beyond counts/page-views to sophisticated queries, leaving teams frustrated. Product instrumentation and maintenance of tracking plans continue to be tedious chores for large teams. The future will belong to a platform that not only generates insights instantly, but also uses best in class ML to help teams discover what questions to ask!

What I’m excited to be working on…

We have established by now that it’s damn hard to build a good product analytics suite that actually gets adopted across an entire company. The good news is there are a bunch of amazing startups working on this exact problem.

I recently joined the extremely talented team at Amplitude who share my vision for the future of product development. We’re building an analytics platform that helps teams build better products through access to the behavioral layer of user data. Our solution has given us strong traction with great feedback from early adopters at Square, Instacart, and even behemoths like Disney, Intuit, and Microsoft. If you are interested in the problem we are solving at Amplitude get in touch!

This post was originally published on Medium on January 17, 2017.