Mastering Retention

As we defined in the Retention Lifecycle Framework, a new user is someone who was using your product for the first time during the current period that you’re measuring.

First, take your new user cohort that you created in Section 3.3, and plot your baseline retention for new users.
Remember, you can use either N-day, bracket, or unbounded retention, as discussed in Section 3.1.

Investigate user properties & segment your retention curve.
Once you create your new user cohort, look at user properties to get a high level understanding of who these users are. Measuring the breakdowns of key user properties can help you identify trends and groups of users you should study more closely.

You should also segment your retention curve by your major user properties (e.g. platform, location, attribution source) to identify any differences to investigate.

Example: Where new users are coming from
For example, here we can see the breakdown of where new users are coming from.

When we segment the new user retention curve by country, we see that while the overall retention curve looks similar for different countries, there are some significant differences. For example, by Day 7, users in India have 9.6% higher retention than users in Brazil.

If you see differences like this, it’s worth digging into the user experiences in these countries to find contributing factors. Are there ways that this company could get their retention in Brazil and other countries to match India?

Now it’s time to look for personas of users who are new to your product. Remember, behavioral personas describe distinct ways that people use your product. In Chapter 5, we identified personas of current users, and now we’ll look for specific ways that new users behave—which may have some overlap with your current user personas. For more background on behavioral personas and why they’re important, flip back to Chapter 4.

Studying the personas of your new users can help you understand:

• Why users are trying out your product
• Where new users are coming from
• Any patterns of early behavior that might impact retention (positively or negatively) later on

Example: New user personas of a B2B SaaS company

We work with a B2B SaaS company with a platform that enables marketers and other professionals to easily make professional animated videos. When they looked at new user personas on their website, the first thing they established was one important way to group new users: New users who had created an account vs. those who had not.

Users who had created an account were likely to be video "producers," whereas the SaaS company hypothesized that users without accounts were potentially video "consumers" who were on the platform simply to view video content. This company decided to separate users who had created an account from those who had not, before looking for distinct personas within each group. This is a great example of using qualitative knowledge of your product to group users before using quantitative data analysis methods.

Discovering new user Personas in Amplitude

To start, the team created two behavioral cohorts in Amplitude: ‘New User – Account’ and ‘New User – No Account’. They then ran each cohort through the Personas tool to find clusters of users who behave similarly (see Chapter 4 for a refresher on how Personas works).

For the New User – Account cohort, they found that most users were taking actions related to producing videos — ‘Producers.’

Using the Personas feature, the SaaS company identified two main groups within ‘Producers’:

• Producers – Heavy Asset Users: These users launched the video maker and added lots of "assets"—basically any object or component—to videos
• Producers – Low Asset Users: These users also launched the video maker, but did not use many assets

Looking at retention of ‘Heavy’ versus ‘Low’ Asset Users, they found that Heavy Asset users have significantly better retention (see chart).

These findings helped highlight that using assets in a video early on is probably important for understanding the platform’s value, increasing the chances of retention.

For the New User – No Account, they found some interesting personas as well.

• Consumers: These people are mainly watching videos that were created on the platform. Moving forward, it will be important to measure Consumers separately from Producers.
• Potential Leads: These people are visiting the company’s website and performing actions consistent with potential customers of the platform. The main differentiating action was visiting the pricing page—most users in other personas were not doing this at all.

Takeaway
Overall, this company found that the product experience and messaging should be very different for these personas, which they easily identified via Personas in Amplitude. The company decided to dig deeper into the ‘Potential Leads’ persona to learn more about their current behavior on the website and their conversion rate to signing up for a free trial, and plan to optimize the website experience for this persona.

By measuring the retention of users in each persona, you can compare the best-retained clusters to those with lower retention. For personas with high retention, you can hypothesize that their behaviors while they are new users are related to their retention. Further testing can confirm whether that’s true.

First impressions matter

One of the most important parts of understanding new user retention is to analyze your onboarding funnel. Bad onboarding leads to a bad new user retention. Onboarding is the first phase in new user retention, so increasing the number of users who successfully onboard will shift your entire retention curve up.

Define your onboarding funnel

Many apps have a well-defined sequence that users move through during their first-time experience. If that’s the case for you, defining your onboarding funnel is pretty easy: simply track an event for each step, and that’s your funnel.

If your first-time user experience is more open-ended and flexible, think about the key events that a user needs to do before they can start getting value out of your product (and see Section 4.3.5 for how Pathfinder can help you identify those event sequences).

Measure the retention impact of your onboarding flow

If you have a defined onboarding flow or tutorial like the SaaS company example, you should also measure the impact your onboarding has on retention. Do people who complete the onboarding flow have higher retention than those who don’t? How important is it to get users to complete the tutorial?

Example: Onboarding funnel for B2B SaaS

The B2B SaaS company that we just discussed above includes a tutorial in their onboarding process that introduces new users to the platform. They created a funnel to capture the major steps in their onboarding process from signing up to completing this tutorial.

At each step in the funnel, you can measure the percentage of users who continued from the previous step, and the percentage of users who dropped out of the funnel and didn’t continue.

Start with the largest drop-offs:
Anytime you’re diagnosing a funnel, it helps to start with the largest drop-off to see what you can improve. Here you can see that the largest drop-off is between the ‘Begin Tutorial’ and ‘Completed Tutorial’ steps. Only about one-third of the users who start the tutorial go on to complete it.

Takeaway
In this case, the company decided to start tracking specific steps within the tutorial to learn where users are losing interest. They plan on using that data to redesign the tutorial so that it is useful to their users while still getting across the most important tutorial information.

Example: Retention impact of an onboarding tutorial

Here’s an example from one of our customers, who has a marketplace app where users can buy and sell items. The first time a user opens the app, they get put into a tutorial flow that points out key features, but they can exit the flow if they want.

As the funnel chart below shows, the app has a 26% conversion rate from the onboarding flow start to finish. In other words, only 26% of users finish the onboarding tutorial.

This company created behavioral cohorts of new users who completed the flow, and those who did not. Remember that in Amplitude, you can use the Microscope feature to create cohorts straight from any chart.

When they compared the retention of the two cohorts over the first 30 days, they found that users who completed onboarding had about 5% higher (unbounded) retention than users who did not.

Takeaway
Now, 5% may not seem like a huge difference, but over time this 5% can add up. Incremental improvements like this can add up to many more users retained in your product over time. For this company, we recommend that they investigate the following:

• What value is the onboarding flow adding for users who complete it? Is there a specific feature that they’re using earlier?
• Is there a way to then resurface that value later in the user experience for people who skip the onboarding flow?
• Why are 74% of new users abandoning the onboarding flow? Could it be improved so that it still conveys the essential information but has a higher completion rate?

What do "dropped-off" users do instead?

Path analysis can be really useful for seeing what users are doing once they drop out of a funnel, like the onboarding flows we looked at earlier. Are these users leaving the app for good, or are they doing something else instead?

For example, let’s look at Pathfinder for the marketplace app, where about 74% of new users are abandoning the onboarding tutorial. What are those users doing instead?

This shows that of users who abandon the onboarding flow, only about 5% are actually leaving the app right after abandoning onboarding—so that’s encouraging. 84% of the time, the next action is to view product listings, followed by viewing product details, viewing other product listings, or searching for items. Some users even start the process of listing an item to sell.

Takeaway
Based on this data, it seems like the vast majority of users who abandon the onboarding flow end up using the product as expected. This may be the case because the marketplace app is intuitive enough for users to understand as they explore it, and doesn’t require a structured onboarding tutorial.

We recommend the marketplace company try the following to see how they impact retention:

• Streamline the onboarding flow so it has a higher completion rate.
• Try getting rid of the onboarding flow. Dump users straight into the app and have contextual tooltips that explain features and encourage users to take action as they encounter them.

The prevailing wisdom around new user retention has been to benchmark N-day retention on some arbitrarily appointed days: D1, D7, D30, D90, much like the graph of mobile retention that we showed at the beginning of the chapter. As we’ve discussed in earlier chapters, this approach is problematic because products have very different expected usage patterns. You expect someone to play a mobile game or use a music streaming service at different frequencies than an e-commerce site or on-demand food delivery app.

Using arbitrary ‘Day N’ benchmarks for retention is problematic because products have very different expected usage patterns.

Instead, we’ll provide you with a framework for analyzing new user retention that you can adapt to the usage patterns of any product.

After a user starts using your product, they progress through three phases: Onboarding to Value Discovery to Habit Formation. Users who don’t make the transition from one phase to the next are dormant users.

Onboarding

A first-time user of your product completes the onboarding experience and uses the product for the first time. In this phase, it’s critical that you get users to experience your product’s core value as quickly as possible.

Value Discovery

After onboarding, you have a limited window of time to continue showing your core value to a new user. During this time, make sure users are experiencing the core value as many times as possible so that they get a good understanding of how your product helps them or improves their current way of doing things.

Habit Formation

Once a user has discovered value in your product, it’s time to make sure they develop a habit so that they keep coming back over time.

We’ve already discussed Habit Formation, in which a new user becomes a current user, in Section 5.3.

In this chapter, we want to understand the transitions that occur during the Onboarding and Value Discovery phases. What are the behaviors that contribute to pushing someone over the line into the next phase?

Defining the timeframes for the Onboarding and Value Discovery phases

For any product, we’re setting the Onboarding phase to Day 0, the first day that someone opens your app. The length of the Value Discovery phase, however, is determined based on your product’s usage interval, which you calculated in Chapter 2. The table above outlines the phases for products of different usage intervals:

Once you have the timeframes for your product, create the cohorts of new users who return during those timeframes. You’ll use these cohorts to understand the transitions between the phases.

How to create cohorts for Onboarding and Value Discovery in Amplitude

You can use custom bracket retention in Amplitude to quickly create cohorts for each phase of new user retention.

In the Retention view, choose Custom Bracket retention as your method.

Then simply enter the number of days that should be in each bracket. For a product with a weekly usage interval, that’s 1 day for Onboarding, 7 days for Value Discovery, and 7 days for Habit Formation. Once you apply those brackets to the retention chart, you will get a retention curve of the Onboarding and Value Discovery Phases.

In this chart, only 6% of users are successfully onboarded and make it to the Value Discovery phase.

To create a cohort for each phase, simply click on the data point and click ‘Create Cohort’. For example, clicking the Day 1-7 data point will give you the Value Discovery Cohort: users who were new on Day 0 and then came back anytime during Days 1-7.

In Section 5.3, we walked through a process to discover the behaviors that drive completing the habit formation phase. We will repeat that process now to identify behaviors that lead to a new user becoming successfully onboarded.

Again, here are the basic steps to follow for finding these triggers:

1. Create a base cohort of new users during your usage interval.
2. Create a retained cohort of users who were in the base cohort and were retained in the Value Discovery period.
3. Create a dormant cohort of users who were in the base cohort and were not retained in the Value Discovery.
4. Compare your retained and dormant cohorts to look for behaviors that are present in the retained cohort, but not in the dormant cohort.
5. Once you’ve formed some hypotheses of actions that might qualify as triggers, measure the difference in retention between users who do that action, and users who don’t do that action.

Or, use Compass

Alternatively, you can simply enter your new user and value discovery cohorts into Amplitude Compass to get a list of potential drivers. Compass will find the user actions that are most correlated to successful onboarding. For a more complete description of how Compass works, refer back to Section 5.3.

Example: Onboarding drivers for a mindfulness app

Here’s an example Compass report from our customer who makes a mindfulness app. While some of these events are too generic, like ‘Application: Launched’ and ‘End Session’, you can see that ‘Meditation Session Completed’, an event that fires when a user completes a meditation session, is second on the list.

Let’s take a closer look at the ‘Meditation Session Completed’ event. According to Compass, completing this event at least one time within the first day of use is moderately predictive of successful onboarding.

The ‘Compare Retention Rates’ graph also shows that users who complete a meditation session their first day have better retention than new users overall.

Based on this Compass report, it seems that getting a new user to perform one meditation session in their first 24 hours is a good indicator that they will successfully pass the Onboarding phase.

Takeaway
This company decided to tailor their first-time user experience to get a user to complete a meditation within their first session in the app.

Now, we’ll repeat the same process for the transition from the Value Discovery to Habit Formation phase.

Example

Let’s continue the example from the mindfulness app. In Compass, we’ll now set the ‘Value Discovery’ cohort as the base cohort, and the ‘Habit Formation’ cohort as the target predict cohort.

The resulting Compass report shows that in the first 7 days, completing meditation sessions is still the most correlated action for retaining from the Value Discovery phase to the Habit Formation phase.

The best predictor for successful Value Discovery is completing at least 3 meditation sessions in the first week.

Takeaway
Based on these findings, the company decided to test ways to get users to complete 2 additional meditation sessions between days 1-7 in the app, for a total of 3 meditation sessions in the first week.

Apply what you’ve learned: Getting more users through the Onboarding and Value Discovery phases

Once you’ve discovered your own drivers for Onboarding and Value Discovery, you know the milestones that you need to get new users through to increase their chances of staying retained.

Sometimes you’ll find that the drivers are similar for both the Onboarding and Value Discovery phases, in which case it may make sense to consolidate your approach to focus on one time period. This was the case for one of our customers’ products, which helps users find and book fitness classes near them. Their highest correlated event for both phases was booking 1 class, so we recommended that they focus on getting users to book 1 class during the Value Discovery Phase, instead of limiting the focus to only the Onboarding time period.

To put these insights into action, think about ways you can get more users to pass these milestones during their early experience.

This will improve your overall retention rate by shifting the curve up. Here are some suggestions based on the mindfulness app example.

Onboarding

• Provide an onboarding experience that includes completing a short meditation session, so a user immediately gets the core value of the app.
• If a user doesn’t complete a meditation during their first session, ask them to set a reminder (via push notification) to complete a meditation session at a later time.

Value Discovery

• On a user’s second day, if they haven’t yet done a meditation session, send a push notification reminding them to meditate.
• Present users with a goal of completing three sessions in their first week, with some incentive if they reach the goal.
• Introduce the idea of meditation "streaks"—if a user has already meditated for two days in a row, send them a notification on the third day reminding them to meditate so that they don’t break their streak.

By experimenting with different methods, you can find the most effective ways to get users through the phases of new user retention.

Remember, the phase that comes after Value Discovery is Habit Formation, which we covered in Section 5.3.

More examples of Onboarding & Value Discovery drivers

To help get your own ideas flowing, here are some other examples that our customers have found and recommendations for getting more users to complete those actions.

PRO TIP:
Don’t forget retention detractors. We focus most of this playbook on finding positive correlations with retention so that you can get more users to do the things that improve retention. However, it’s important to also think about the factors that might have a negative retention impact. Fixing these ‘retention detractors’ can sometimes provide really big wins.

Improve product quality first. The overall quality and performance of your product can have a big impact on retention. Some obvious culprits are bugs, crashes, and slow load times— if your app performance is unreliable, it doesn’t matter how much value your product can theoretically provide. Users will quickly give up and abandon your app. If you have any major performance issues, make sure you spend engineering resources to fix those first before implementing your retention strategy.

Here are some ways to find detractors via your analytics data:

• Find bugs or crash events, if you’re logging these in your user analytics.
• Find events that have a low correlation with retention. In Amplitude, you can find these low correlation events in Compass.
• Segment your dormant users by different user properties to identify any trends. For example, you might notice that your retention on Android devices has taken a sudden dip, and pinpoint a bug in your latest release.
• Use Amplitude’s Personas feature (or a similar clustering algorithm) to identify groups of users with low retention, and see what events or properties they have in common.

To recap, the key components of new user retention analysis are:

• Identify important user properties and behavioral personas of your new users
• Diagnose your onboarding funnel
• Identify the actions that drive users to complete the Onboarding and Value Discovery phases of new user retention
• Don’t forget about retention detractors!

Now that you’ve completed new user retention analysis, summarize what you’ve found and form some hypotheses to test. You can use the New User Retention worksheet at the end of this chapter to take notes and keep organized.

Key questions to ask yourself

• Who are your new users, and what are their behavioral personas or significant user properties? Is there a specific persona that you should focus on?
• Why do you think some of your behavioral personas retain better than others? Are there particular behaviors that seem to positively (or negatively) impact retention?
• How does the first-time onboarding experience affect later retention? Are there certain steps that can be improved? Is a tutorial or structured onboarding flow beneficial for your product?
• What are the key actions that you identified as drivers for successfully passing the Onboarding and Value Discovery phases of new user retention? What are some methods you can test to get more new users to cross those milestones?
• Did you identify any retention detractors? How can you improve them?
• What experiments can you run to determine whether a certain action or sequence of actions is critical to your new users retaining?

Track improvement over time

As you start testing some of your hypotheses and trying out new ways to improve your new user retention, it’s important to keep track of your metrics to see what is and isn’t working.

Remember, the overarching goal of new user retention analysis is to set new users up to become current users. Keep this in mind as you form your metrics and KPIs for new user retention.

Here are some metrics you can track over time to measure your progress:

• The percentage of new users who become current users.
• Retention over time of your new users and of important behavioral personas.
• In particular, you can set up a bracket retention curve that follows your New to Onboarding to Value Discovery to Habit Formation phases. You might choose to focus on improving one part of this curve first, depending on where the largest opportunity for improvement is.
• Conversion rate over time through your onboarding funnel.

A new user is someone who is using your product for the first time during the current interval that you’re measuring. New user retention analysis will help you understand how your new users are onboarding and discovering value in your product.

New user diagnostic checklist

Run through the metrics below to get a baseline understanding of your new users. Refer back to Ch. 4 for a refresher on any of these methods.

• Create a cohort of your new users
• Plot the baseline retention curve of new users
• Segment the retention curve by user properties
• Measure conversion through your critical path funnel
• Identify common user flows
• Measure stickiness for your critical event
• Measure session metrics

Checklist item 1: New user behavioral personas

Identify any behavioral personas within your new users and list them here.

Checklist item 2: Define & measure your onboarding funnel

List the steps in your onboarding funnel (you may not need all the rows) and the conversion rate between steps, noting where your largest drop-offs between steps are.

Checklist item 3: Segment your funnel by user properties and personas

Make note of any interesting trends you see.

You should also:

• Measure the retention impact of completing your onboarding flow
• Look at what "dropped off" users do instead

Checklist item 4: Drivers of onboarding & value discovery

Use the process in Sections 6.4-6.6 to identify the behavioral drivers of the Onboarding and Value Discovery phases of new user retention. List those drivers here and some ideas you have for how to get more users to perform those actions.

Checklist item 5: Take action: hypotheses & next steps

Ask yourself these questions as you form hypotheses and come up with experiment ideas.

• Who are your new users, and what are their behavioral personas or significant user properties? Is there a specific persona that you should focus on?
• Why do you think some of your behavioral personas retain better than others? Are their particular behaviors that seem to positively (or negatively) impact retention?
• How does the first-time onboarding experience affect later retention? Are there certain steps that can be improved? Is a tutorial or structured onboarding flow beneficial for your product?
• What are the key actions that you identified as drivers for successfully passing the Onboarding and Value Discovery phases of new user retention? What are some methods you can test to get more new users to cross those milestones?
• Did you identify any retention detractors? How can you improve them? What experiments can you run to determine whether a certain action or sequence of actions is critical to your new users retaining?

Checklist item 6: Metrics for tracking improvement over time

As you start testing some of your hypotheses and trying out new ways to improve your new user retention, it’s important to keep track of your metrics to see what is and isn’t working.

Remember, the overarching goal of new user retention analysis is to set new users up to become current users. Keep this in mind as you form your metrics and KPIs for new user retention.

We suggest tracking these metrics over time to measure your progress:

• The percentage of new users who become current users.
• Retention over time of your new users and of important behavioral personas. In particular, you can set up a bracket retention curve that follows your New to Onboarding to Value Discovery to Habit Formation phases. You might choose to focus on improving one part of this curve first, depending on where the largest opportunity for improvement is.
• Conversion rate over time through your onboarding funnel.

The Amplitude Guide to Customer Retention: 40+ Resources to Increase Retention

The Amplitude Team

Customer Retention 101

The Amplitude Team

Why retention is the key to sustainable growth Q&A with John Egan, Growth Engineer at Pinterest

William Wickey, Lead Genius

New data shows losing 80% of mobile users is normal, and why the best apps do better

Andrew Chen

Onboarding Teardowns from UserOnboard

Great for inspiration and best practices, this site has lots of reviews of user onboarding experiences from popular apps and websites