5 Best Practices Marketers Often Overlook When Measuring Retention

This blog is part 3 of a 3-part Reforge blog takeover series, where Reforge experts help product and growth leaders improve their retention. Like Austin’s perspective and looking for more? Be sure to check out his Reforge course on MarTech this Fall.

Strong user retention is critical when it comes to building a successful product. However, many teams encounter difficulties in accurately measuring and extracting valuable insights from retention analyses. In this post, I will guide you through the five most commonly overlooked tips for retention analysis:

1. Data hygiene must be proactive (not reactive)
2. Be careful not to over-rely on analyzing client-side events
3. Evolve your retention metric as your product and users evolve
4. Add context to your retention curves using lifecycle and product indicators
5. Use retention data to source better top-of-funnel leads

By identifying these opportunities, you can significantly improve your ability to analyze retention and optimize it for the sustained growth of your product. Get ready to dive deep and unlock valuable strategies for success.

We’ll start at the beginning before measurement, then help define our key metrics, and finally move toward best practices in analysis.

Data hygiene must be proactive (not reactive)

If you’ve ever lived the nightmare of spending hours cleaning up data row-by-row just to get to the report you need, you can understand exactly why proactive data hygiene is important. Data hygiene is the ongoing process of validating and updating data to ensure data is up-to-date, accurate, complete, standardized, and reliable.

Data hygiene plays a crucial role in maintaining a healthy and efficient data ecosystem. Rather than being reactive and fixing issues as they arise (or the hours before a big report is due), it is essential to adopt a proactive ongoing approach.

By practicing good data hygiene from the outset, organizations can ensure the accuracy and reliability of their data as they grow. It enables teams to better leverage data for meaningful insights. Not having a strategy here will cause major problems down the road.

Establish data governance policies and standardization

At a minimum, organizations should audit their data to identify any duplicates, discrepancies and inconsistencies. They should identify and document the sources of their data and their data model so they can easily understand what each element means. This will make it easier for engineers and analysts to clean up the mess if any anomalies are found. Additionally, it’s helpful to also add simple visuals of the action or event in reference so that non-technical team members can also use and help govern the data plan.

Your data hygiene management doesn’t have to be fancy or difficult. Early start-ups function with low-fi Excel sheets outlining clearly what each instrumented event means. Platforms including Amplitude, however, offer more advanced data hygiene tools that enable you to track, monitor, and flag issues quickly.

Taking a proactive stance toward data hygiene is essential for organizations to thrive in the increasingly data-driven landscape. By embracing good data hygiene practices, organizations can foster a culture of data-driven decision making and gain a competitive edge. It is crucial to recognize the importance of integrating data management tools and processes early on, rather than waiting until issues arise or it becomes too late to rectify problems.

Be careful not to over-rely on analyzing client-side events

In Reforge’s Martech course, we not only teach you how to track client-side events, but also Server-side events, and even offline, or “Synthetic” Events. Let’s take a minute to explain what each one is and why they are important.

Client-side events

Client-side events refer to the events that are triggered and handled on the client side of a web application. These events are typically associated with user interactions in the browser, app, or device, such as clicking a button or submitting a form.

For example, when a user clicks on a button to open a modal window, a client-side event is triggered to handle the action of opening the modal. When teams think about retention, client-side events are often the default events to measure. These are the easiest to record, and most obvious events to track, because they are human behaviors.

Server-side events

On the other hand, server-side events are events that occur and are processed on the server side of a web application. These events are often related to server processes or database transactions.

For instance, when a user submits a form, to save data, a server-side event is triggered to handle the data processing and storage. Server-side events are typically implemented using server-side programming languages like PHP, Python, or Java. Server-side events may be happening with or without the human involved.

These are also not difficult to instrument, but they are often overlooked as retention metrics.

For example, let’s look at home security systems. Here the measure of retention is hopefully not human interaction, as a client side event would only happen in the case of an emergency, or something going wrong. Instead, this type of business would either want to track a server side event, like credit card payment each month which happens automatically via servers.

Offline or synthetic events

Offline or synthetic events are artificially generated or simulated in a web application. These events are not directly triggered by user interactions or server processes. Instead, they happen offline.

This could include a visit to a store, a purchase at a brick and mortar, or something like changing the batteries on a home security system. For each of these, we have to go through a few extra steps to convert the offline event into a synthetic event inside of the MarTech stack. Offline events are particularly important for companies that aren’t purely digital in nature. Teams may choose to avoid these due to their perceived complexity, but if the user is getting value offline, we’ll want to find a way to track this.

Evolve your retention metric as your product and users evolve

Reforge and Amplitude both have detailed guidance on how to choose a North Star Metric that I won’t revisit in this blog. Instead I want to highlight that all of the analyses in the world won't help you if your retention metric isn't right or doesn't reflect your latest users and product strategy. Teams need to define these metrics well and update them regularly. From my vantage point in Martech, I often see BU leaders set a North Star Metric and leave it, but from the data that I live in every day, I see macro shifts over time on two main axes: Frequency and Critical Events.

When you measure retention at your company, you've determined a critical event that constitutes whether or not someone is retained, that occurs within a certain time frame or frequency. Some examples might be playing three games within the first day of use, or how many users complete a weekly meditation session.

Once this critical event and frequency is set, BU’s then drill into their heads that the action they set is the most important, because the success of the business is measured based on it. Unfortunately, this core action will naturally change over time, or the business will evolve and measuring actions downstream start to matter more.

Critical events can change over time

For example, let’s say you release a new product or new feature that unlocks value to the customer in a unique or improved way. Or, perhaps your product has many actions that drive value, but you started measuring retention of a junior audience, and have since upleveled your target ideal customer profile (ICP). Now the action that drives value for your new audience may be different. For example, if your original ICP was college students and you’ve grown to enterprise companies, the action that drives value may have changed.

Unfortunately, after drilling into the single core action over time, teams may accidentally overlook updating the action they want to focus on. Tools like Amplitude are great because they make it easy to explore data without needing to know SQL and help automate audience segmentation and discovery. This levels the playing field in terms of who can weigh in and help the business understand which users and what events are worth measuring and using in your company’s objective function.

Frequency of critical events might also change

For each critical event, we also need to set a given frequency. Unfortunately, leaders often rely on frequencies investors expect to hear like daily active users (DAUs) and weekly active users (WAUs), rather than truly understand the natural frequency of the given action, which might be bi-weekly usage or even bi-annual usage. Additionally, when leaders do change the critical action measured, they can be prone to overlook updating the frequency.

The most serious danger of evaluating retention using the wrong frequency is that businesses can make faulty decisions about their customer loyalty. For instance, a company relying on set frequencies may think they have high customer loyalty if customers keep coming back to purchase within those set times. In reality, however, these customers might be purchasing from the company out of convenience and not true loyalty, which might be at a higher frequency or rate. Similarly, a business might underestimate their retention rate when relying solely on natural frequency because they do not take into account repeat purchases made outside of the natural cycle.

What is natural frequency and why does it matter?

Natural frequencies, or what Amplitude calls Usage Intervals, measure customer usage over time in order to better gauge user engagement. For example, a company such as Spotify might track user activity every day or week, depending on how often people use their service. This approach enables more accurate assessments of engagement than methods based on set frequencies, which assume that customers will retain their engagement with a product or service at a consistent rate without taking into account any fluctuation in usage patterns.

For instance, a company like Dropbox might use monthly or yearly retention metrics even though users might not access their service every single month or year. These set metrics offer less insight into the nuances of customer usage than natural frequency measurements and can lead to incorrect assumptions about customer loyalty and engagement if used as the primary source of information.

Amplitude provides guidance around determining critical events and usage interval frequencies in its Mastering Retention playbook.

Add context to retention curves using lifecycle and product indicators

Retention measures whether the customer remains active, defined as a critical action taken at a set frequency. Retention Indicators are events that happen before the critical event that predict or instigate the user to be active. Some examples of retention indicators might be sharing a song within a music app or activating a daily reminder to complete meditations in a meditation app.

Indicators play a crucial role in enhancing the understanding of retention curves, providing valuable insights into user behavior and the factors that drive them back to a product or service. By measuring these indicators, businesses can gain a deeper understanding of what motivates users to return, enabling them to optimize their strategies for improved customer retention. Most businesses fail to link indicators that help color their retention curves, and fail to understand the nuances between the two leading indicators: lifecycle and product indicators.

Lifecycle indicators

Lifecycle indicators are marketing activations, automated or manual, that encourage a user to return to the product. Examples of lifecycle indicators include push notifications, text messages, or other personalized communication methods that prompt users to return to the product. For example, common lifecycle indicators include giving someone a coupon or sending them a notification about a new feature.

Lifecycle indicators can be powerful boosters of retention. Measuring their relative effectiveness can help you identify new opportunities to boost retention, but these indicators are manufactured and occasionally cost the business money to activate, so they are less organic. It’s also worth noting that these indicators often require data from external sources that can be combined into a data warehouse to further analyze and measure the impact of these indicators.

Product indicators

On the other hand, product indicators focus on the specific features or elements within a product that contribute to user retention. These indicators provide insights into the effectiveness of specific product functionalities or enhancements in driving user engagement and retention. Examples include receiving a communication through usage of the product, getting access to new features of content, etc.

For example, if I receive a DM in Instagram, I may open the app and perform other behaviors, but the DM was the impetus for me to return to the platform. The benefit here is that product indicators tend to lead to far more organic retention, and yet, they are hard to spin-up out of thin air without a strong team and deep customer insights.

In short, supplementing retention curves with indicators, such as lifecycle indicators and product indicators, enables businesses to gain a more comprehensive understanding of user behavior and the drivers of user retention. Although lifecycle indicators focus on the different stages of user interaction with a product, product indicators provide insights into specific features or enhancements that encourage user engagement. That said, one of the benefits of measuring and analyzing these indicators is that businesses can make data-driven decisions to optimize their strategies and ultimately improve customer retention.

Use retention data to source better top-of-funnel leads

Retaining users is critical for any business, and understanding their retention curve is key to achieving long-term success. By analyzing and extracting insights from retention data, we can uncover valuable information that can be used to calculate a retention score—an invaluable metric for predicting Lifetime Value (LTV).

To begin, we can leverage the retention curve to determine the likelihood of user retention. By grouping users based on different attributes, such as the day they were retained, we can gain a deeper understanding of user behavior and engagement. This segmentation allows us to identify patterns and trends, enabling more targeted strategies to improve retention. Amplitude refers to this as “behavioral cohorting” and offers a user segmentation guide to help you get started. You'll have the opportunity to observe a retention curve that showcases a 30-day rolling retention metric. This valuable information depicts the proportion of active users at the end of the 30-day period compared to the initial number of users at the start of that period. For a detailed overview on how to measure a 30-day rolling metric, Reforge has a program dedicated to this topic.

Scoring retention-related activities

In order to track a 30-day rolling retention metric, we are measuring user actions over a period of time. We can take this a step further by actually assessing the value that each user brings to the business. Actions such as returning to the website, logging in, viewing content, and sharing videos can all contribute to the overall retention score. Each of these actions is assigned a point value, with some activities being worth more points than others, helping us quantify user engagement and loyalty.

Then, we can compare these retention scores to the actual retention of our existing users, and further refine our calculation over time. Once confident, we can upload these retention scores and associated customer profiles upstream to our ad-networks and tell it to find lookalike audiences to your existing customers with high retention scores.

Maximize marketing activities with better retention measurement

In summary, mining retention data provides invaluable insights into user behavior and enables us to calculate a retention score—an important predictive metric for determining the Lifetime Value of users. By leveraging this knowledge, businesses can tailor their marketing campaigns and engagement strategies to drive long-term success and maximize user retention.

If you enjoyed Austin’s perspective, be sure to check out his upcoming course on Reforge, MarTech.