Building a Single Source of Truth for Org-wide Activation Metrics
If you want to build a single view of metrics across an entire portfolio of products, then this one is for you. In this edition of ‘How I Amplitude’, Claudia Canamas-Donnelly, Senior Product Analyst at Appfire, explains how she created a process for aggregating metrics across almost 200 apps into a single Amplitude dashboard, for her team’s Activation reporting use case.
“For higher level stakeholders interested in gauging the health of our business as a whole, we don't want them having to jump from chart to chart.”
Background
Appfire is a portfolio company - they have over one hundred different products. The brand mostly makes B2B apps that help businesses manage their workflow better.
Claudia is on the growth team, which strongly focuses on product-led growth (PLG) and activation.
As Appfire has done this foundational PLG work and started tracking activation metrics for individual apps, the brand has come to prefer Amplitude for collecting data and for ad hoc analytics, but also for official metrics reporting.
Activation Journey Maps
Activation at Appfire is defined as the process of taking customers from initial sign-up all the way to establishing a habit around the product. Appfire’s definition is inspired by the PLG content found in Reforge, an industry-approved online professional development platform.
Proper activation is key to retaining customers down the line, so Appfire’s Growth team puts a lot of care into having well-defined activation flows for each one of their products. Activation can be broken down into four main milestones:
- Signed up
- Setup moment
- Aha moment
- Habit moment
The sign-up moment is the first action a user takes to start using the product, e.g. creating an account or an initial visit to a website. It’s followed by the setup moment, which is when the user has completed the necessary actions that set them up for success in the product. For Appfire products, this typically means further configuration of app settings – whatever is needed for the customer to be able to get value from the app.
Next comes the Aha! moment—the first time the user has experienced the core value proposition of the app.
Lastly, there is the Habit moment, which is the point where the user has experienced the app’s core value proposition enough times to establish a habit around getting value from the app. At this point, we start to feel more confident that our product is becoming an ingrained part of the user’s routine.
These four moments all together are what we call the activation journey map (AJM).
From AJMs to KPIs
To implement these activation milestone moments in Amplitude, Appfire uses Amplitude native custom events.
Custom events make it easy to define moments in the activation journey by using events already in the event plan. Once a custom event has been created for each of the four AJM milestones, we can add them all into a funnel chart.
With that, the first metric is defined: activation rate. This is the percentage of sign-ups that form a habit around the product.
Bear in mind: mapping four AJM milestones to four discrete funnel steps is a simplified example. Oftentimes, an AJM milestone must be captured as two or more funnel steps. This is especially true for the Habit Moment, as it can be a longer process for our customers to go from their first realization of the product’s value to fully forming a habit around the product.
Topline activation KPIs
Once a basic funnel is built, two more activation metrics can be introduced. The first is Time to Value, which Appfire defines as the median amount of time it takes customers to reach the Aha! Moment after completing Sign-up.
Similarly, there is the Time to Habit metric, which is the median amount of time it takes to complete the full activation funnel, from sign-up to the final habit moment.
For a single product, these two metrics are tracked quite easily in Amplitude, thanks to the funnel chart’s native Time to Convert option. However, for a portfolio company like Appfire, where many stakeholders focus on multiple apps, it’s not always ideal to have this tracking done at the individual app level.
Stakeholders often want to look at data that is more high-level than that for a single app. They care to learn about the activation performance of Appfire products at an aggregate level, e.g. the activation metrics for all apps of a certain category, or all apps sold on a specific marketplace.
Appfire’s Growth team wanted to be able to easily group metrics from individual products, and then zoom out to get a single top-line view.
Portfolio-wide reporting: Technical Requirements
So, what is the best way to report on portfolio-level metrics in a way that’s concise and proves useful to higher-level stakeholders interested in the health of the business as a whole?
First, the team considered using Amplitude's portfolio add-on for this situation, as it enables analysis and user tracking across projects by duplicating source project events and putting them into a new single analytics project.
However, in addition to the fact that this add-on is only available for Enterprise plans, the team found some limitations with the add-on that disqualified it as a viable solution for this reporting use case. The Appfire team required a solution that:
- could aggregate metrics from various funnel charts that could possibly differ in number of funnel steps, length of conversion window, and other settings.
- did not skew metrics in favor of apps with the highest customer volumes.
- met specific data privacy standards. At Appfire, there were many concerns with granting access to all source projects that could feed into one single portfolio analytics project, since individual app projects often contain sensitive information, like a user's contact info or revenue data that we may not want to share company-wide.
With these requirements in mind, Appfire's Growth team went about designing their own solution.
Portfolio-wide reporting: The Solution
Now we get into the details of the solution, which takes advantage of Amplitude’s:
Step 1: Creating charts for each individual product is the starting point. Since Appfire’s reporting use case is related to Activation, they used AJM funnel charts described earlier.
Step 2: Equipped with a funnel chart per product, the team then uses a script + the Dashboard REST API to query each chart and pull out key information needed to compute the defined metrics per product (activation rate, time to value, and time to habit). The Dashboard API provides all of the needed information per chart in JSON format.
Step 3: From there, the team unpacks each JSON response, restructures the funnel chart data into a format that is better suited for delivering top-line metrics, and sends it back into Amplitude using the HTTP V2 API. The team accomplished this by making a new analytics project in Amplitude specifically dedicated to this solution, and sending it ‘dummy’ events that contain the desired metrics (computed from the original funnel chart) as event properties. One of these events was sent per app, per week, which effectively provides a historical record of each app’s activation metrics, all in one analytics project!
Step 4: Once the aggregated data is back in Amplitude, the team wants to present it in a way that makes it easy to slice and dice the metrics at different levels, e.g. by marketplace, app category, app hosting type, etc. For this final reporting layer, the Appfire team took advantage of Amplitude’s built-in dashboards and notebooks.
What are the benefits?
There are a wide range of benefits from this reporting solution.
Firstly, users get a concise view of activation for the entire portfolio of products, as well as a high-level troubleshooting view. The team achieves their initial goal of preventing stakeholders from having to jump from chart to chart to get an idea of how different products are performing.
Next, because their process takes weekly snapshots of activation metrics, values from previous weeks don’t keep shifting after they’ve been reported on. This means that the team avoids a situation where one person checks a chart on a certain day and reports the metric value seen that day, only for a different person to check the chart a few days later and find that the metric value has changed – either due to new data or shifting cohort populations behind the scenes. This solution avoids that confusion, helping to maintain a single source of truth.
Additionally, this solution allows the Appfire team to include a built-in rolling window for their metrics, which acts to smooth out noise caused by low data volumes (an especially common problem for B2B companies).
Lastly, the solution allows for cross-portfolio comparison without compromising on data privacy standards. This enables Appfire to give everyone in their team access to the one analytics project they created. It’s easier for permissions maintenance, and allows stakeholders to compare apps against one another on KPI performance without accessing sensitive information of individual projects.
Looking ahead
Despite the many benefits of this solution, there are certain limitations with a process that relies on external syncs that the Appfire team would like to address in the next phase of development.
For example, the same "weekly snapshot" approach that lets the team enjoy a stable single source of truth for historic metric values, can also prove quite rigid and inflexible in the face of backfills or updates to activation methodology. It is an ongoing goal of the team to keep dashboard maintenance as easy and quick as possible, so that end users can trust the data is clean and relevant to their reporting use cases.
Since the release of this dashboard in October 2023, the Appfire team has been focused on internal education and adoption. There is no point to optimizing the solution further if there is no stakeholder buy-in!
Accessibility: From an administrative standpoint, the Appfire team wants to make these tools accessible and easy to find. The team has a space in Amplitude dedicated to PLG work, public for all Appfire employees to see. All official charts and dashboards related to product-led growth metrics are stored in this space, including the dashboard discussed in this post. The team uses Amplitude's native blue check mark feature and pins certain artifacts to the top of the space so that users can find them more easily.
Furthermore, the team has been encouraging product managers to use the dashboard to monitor metrics and has held frequent demos of the tool to demonstrate its value.
Business value: The organization as a whole has put recent emphasis on activation improvements, which has increased demand for this type of aggregated activation view and has only emphasized the business impact of the dashboard.
The Growth team's reporting solution has allowed Appfire stakeholders to start speaking the same language around activation metrics, become more confident that we are aligned in our reporting, and further encouraged data-driven decision making.
Join the community!
Connect with Claudia and other practitioners in our community. The Cohort Community exists to help people become better analysts. We focus on actionable programs, sharing best practices, and connecting our members with peers and mentors who share in the same struggles.