Making the Leap to Embedded Product Analytics

Prior to joining Amplitude to work on the Professional Services team, I was deeply entrenched in the business intelligence and analytics world. In that world, product management and product development happened elsewhere in the organization. We made sure we had clean data, answered questions, and designed reports, but we were not a part of acting on those insights. We existed on an island with a single bridge—questions in, answers out.

Rachel Herrera

The transition to working with Amplitude customers—product and growth teams working in the trenches with access to rich behavioral data—has been a huge eye-opener. There’s so much potential for doing meaningful and impactful work when teams can experiment and take action. In this post, I’d like to share some of the things I have learned, as well as some tips for getting started on this journey.

The Key Difference Between Traditional Analytics and Product Analytics

There are two key differences I have discovered as I work more with our customers:

1. Behavioral data unlocks answers to new, important questions
2. Product teams work differently

The Data

Let’s start with the data. When I was working as an analyst, my focus would be to collect and organize the data necessary to answer questions like, How many appointments were booked, and how does that compare to bookings at the same time last year? I would typically build a new query to pull this data from various backend databases and surface metrics in Tableau for others to use, and that was that. We’d get part of the story but not the whole story. I usually wasn’t aware of why they wanted this information or what decisions they made once they had it.

I was often only looking at a small part of the user experience, and questions were limited by the specific data points we were tracking. We weren’t answering more complex questions like: Did people who booked an appointment tend to view a certain type of content? What was the impact of content sharing on social media? How did the number of articles read correlate with their decisions later in the booking process?

We needed to go deeper, but couldn’t. We just weren’t capturing those user-based actions. Truth be told, I didn’t really imagine we could (or should) capture those things. Even if we had been able to track those things through behavioral analytics, we would not have been in a position to act on our insights. Business intelligence tools like Tableau and Power BI have a wide range of capabilities, but building conversion funnels or creating cohorts of users took heavy dev work and wasn’t supported in the UI.

The Teams

This is where empowered product teams come in. The real magic is in empowering front-line product and growth teams with that complex data.

Product teams have a different dynamic. There’s a camaraderie and shared purpose on product teams that is hard to describe. I had always thought that cross-functional collaboration was just a “pie in the sky” idea because I had never really seen it work with siloed analytics teams. We did thoughtful work, but there was a hard boundary between us and the internal customer (not to mention us and the external customer!).

Now, working directly with Amplitude customers (mostly cross-functional product teams and growth teams), I understand that collaboration isn’t just a pipe dream. I first noticed that members of the team have a vested interest in their work working. They really care. Second, it became immediately apparent just how critical alignment is to the overall health of the team. Data is part of that, but there is a human component as well.

The reality is that product teams are constantly converting what they learn into action (which generates more learning, and more options…a virtuous cycle). This changes their needs when it comes to analytics; identifying the problem is helpful, but they’re also looking for recommended paths forward and want the ability to measure the impact of their changes.

The Evolving Analyst Role

What I have noticed in talking to our customers is that the analyst role changes a great deal once you give teams access to a product like Amplitude.

Here’s why: Imagine being heads down trying to pull together a “Big Report for an Important Presentation” only to get sidetracked multiple times a day by little adhoc questions and fire drills. That’s the typical day-in-the-life for someone on your average shared analytics team. It’s hectic and you’re pulled in a million different directions. However, once teams have access to Amplitude, the nature of the work changes.

The shift boils down to leverage. Expertise is still in demand. There are still perplexing questions that need attention. Some teams may even need a crash course in analytics, a run-through of the taxonomy, and help with troubleshooting. By interacting with multiple teams, you can share best practices and help uplevel the whole product development organization.

But once you get them going and teach them how to fish, so to speak, you’re able to really focus on high-value work and even dip your toes into new types of analysis and data science. A bonus is that you also have the benefit of fusing a new source of fine-grained product data with the data in your current data warehouse—opening up new possibilities and insights.

Building initial confidence. Where do you start?

The beauty of Amplitude is that it makes complex behavioral data accessible; however, you first have to trust the data. After making sure your engineers know how to check that the code is sending data correctly, get the basics down by focusing on tracking discrete events before jumping into conversion rates.

From there, you can track a handful of flows that you are interested in measuring (e.g. a sign-up flow, a checkout flow). In a short amount of time, you will begin seeing the value, trusting the data, and—in turn—maintain the rigor of your product. With our Amplitude customers, we use a data trust score to nail down whether or not the team is confident about the information they are seeing. That trust is imperative for moving forward.

Challenges and Opportunities

The above advice will work in some cases, but for enterprises, there may be larger issues. Often, the biggest challenge is being okay with something not working.

Being comfortable asking questions and having dialogue between teams is critical, especially among people of different seniority levels. Those with more experience in the enterprise may have less interest in changing their approach, as opposed to a smaller startup where they are learning and changing together.

An example win from one company:

I recently received an email from one of our Amplitude customers. Their team traditionally had a hard time defending their decisions to leadership without having the data to prove their suggestions were better than the HIPPO’s of the large company.

In a recent presentation, the product manager pulled up the Amplitude data in a well organized Notebook (see more info on Notebooks here), which allowed the team to explain why and how an alternative approach would be more beneficial than the one leadership wanted to go with. Often, simply showing the data can open up conversations that might not otherwise take place in a traditional top-down organization. This doesn’t mean you’ll always get the perfect, definitive answer from the data, but you can usually reduce uncertainty and increase confidence.

Looking Ahead

At Amplitude, we continue to find ways to better serve our clients. People love the product and they love the data; however, once they hit that first wall while taking a more impact-focused approach, it can get frustrating. We want to help with that.

Our Embedded Analytics program is a new workshop designed to help Amplitude customers map the future state of their products and services. We focus on something that they currently have in flight, a new campaign for example, and see how it is performing.

The workshop spends some time mapping how they’ve worked in the past to how they can achieve those same objectives in Amplitude. It is about mapping the current state to the future state in a hands-on, tactical way.

By focusing on a simple example, and piggybacking on something they do today, we can spark their confidence. Confidence is like the chewy nugget-y center of our embedded analytics workshops. It is essential to success.

Moving to a dynamic product analytics approach with collaborative teaming allows for a much richer and successful product experience for companies and their clients. While not without its growing pains, the key is being open to a new way of working together.