Mastering Retention

While demographic data can be informative (and we’ll cover it later in this chapter), the most important way to understand your users is based on their behavior in your product.

Learning what active users are doing in your product can help you understand the value they get from using your product. An important thing to keep in mind is that people can be using your product in various ways and may not all be deriving the same value. This brings us to the concept of behavioral personas.

PRO TIP:
Buyer or customer personas are a common concept in marketing. A persona is a representation of a key customer segment that the marketer wants to target, and can include customer demographics, habits, and goals. This information helps marketers develop the right messaging and marketing channels to reach that segment of potential buyers.



For this playbook, however, we’re going to be talking about behavioral personas—each persona describes a distinct way of using your product. Much like in marketing, understanding these behavioral personas will inform your product design and development for different types of users.



We’ll look at behavioral personas for all 3 stages of the retention lifecycle: current, new, and resurrected.

Example: YouTube behavioral personas

To illustrate, here’s an example of behavioral personas you might be familiar with:

• Creators: The small percentage of people who actually create videos and post them.
• Viewers: The vast majority of YouTube’s traffic: people who are simply watching videos.
• Viewers + Commenters: People who view videos and leave comments.

Each of these groups of users, or behavioral personas, are using YouTube in a distinct way and for a specific reason. Creators use YouTube as a platform to get their content on the web and establish an audience, while Viewers and Commenters use YouTube for entertainment or to follow Creators they like.

Understanding how different groups of users behave in and ultimately derive value from your product helps you:

1. Shape your product to provide the best possible experience for your users.
2. Get a more nuanced understanding of your retention and find areas of your product you can improve.

Each persona may have drastically different retention rates, which you’ll miss out on if you only look at retention for all of your current users lumped together.

If you’re still an early stage startup and can really only focus on one use case for now, your personas can help you decide which group of users is the highest impact to focus on right now. If you have a larger team and more established product, you might notice a new use case you hadn’t thought about before, and start making some changes to improve that experience and broaden your user base.

PRO TIP:
There’s a rule of thumb about online communities, called the ‘1% rule’ or ‘1/9/90 rule)’, that taps into this concept of behavioral personas. According to this rule, only 1% of users actively create content in any internet community or social network. 9% of users may engage with the content, such as commenting, sharing, upvoting, etc. And the remaining 90%?



The vast majority of users are only passively consuming the content. If a platform like Youtube only focused on the 1% of creators and making the best possible experience for them, but neglected the viewing experience, that could have a huge negative impact on their growth.



While these actual ratios may not hold true for all social networks, and certainly not for all types of products, the general takeaway still does. You should never assume that all of your users are looking to get the same benefit out of your product. Make a conscious decision about what type of user you’re optimizing for and their specific use cases.

Either way, finding your behavioral personas will help you understand who your users are, what they’re doing, and the opportunities you have to improve the experience for all of your users.

Case Study 1: From Burbn to Insta

Ever heard of Burbn? It was a location-based app, similar to Foursquare, that included multiple features for checking in at locations, earning points, and posting pictures. Unfortunately, it wasn’t seeing much growth. When the founders studied their user behavior, they found that while most features weren’t being used, there was a small group of users consistently using one aspect of the app: posting and sharing photos.

Seeing that data, they decided to scrap everything and focus solely on photo sharing. They made uploading photos fast and seamless, and in October 2010, they launched Instagram. We all know the rest of the story: how Instagram quickly gained tens of millions of users, sold to Facebook for $1 billion in 2012, and continues to grow, now with over 1 billion monthly active users.

Burbn could easily have been just another failed startup. Instead, its founders found one behavioral persona that was using their product and used it to shape the product into one of the most popular apps today.

Case Study 2: Twitter’s customer support platform

While Twitter was originally created as a social network, it soon became clear that many people were using it for something else entirely: customer support. People were tweeting complaints or problems they were running into, and companies were responding.

Eventually, Twitter realized that this use case was significant enough to warrant a product change to improve the experience for these users and businesses. In early 2016, they released new tools that helped customers quickly move from Tweets to Direct Messages to discuss support issues, as well as provide customer feedback within Twitter. In some cases, Twitter is taking over the role of traditional help desks, support tickets, and email support.

In both of these cases, the companies noticed that a subset of their users had a unique way of using their product and decided to make product changes to support (or completely focus on) that use case to improve the user experience.

In this section, we’ll discuss how to identify your own behavioral personas, assess retention differences, and decide which personas to focus on.

PRO TIP:
For most products, you can think about bucketing users into power, passive, and core personas. This may not apply to every product, so think about whether it makes sense in your case. You can also have multiple personas in each category: for example, two types of Core Users who have distinct use cases:

Power Users are people who use your product with a very high frequency or use a "power" feature that the majority of users don’t take advantage of.
Core Users are people who are using your app at a regular frequency and in the "expected" way
Passive Users are people who might not be using your app in the core way that you designed, but are still coming back at a regular frequency to do something.
We’ve included some real examples of behavioral personas in this chapter, which should give you a better idea of what power, core, or passive users might look like.

To determine whether there are groups of people who are using your product in a specific way or for a certain use case, using both qualitative and quantitative approaches will get you the most complete answer.

Qualitative methods

Start with brainstorming some personas that you think (or know) exist based on your current knowledge of your users. Qualitative data from user interviews and user testing can be really helpful for determining personas.

If you can, ask current users why they use your product and how it fits into their day. You can also study user activity timelines to look for behavioral patterns that jump out. Often, qualitative data provides more context for trends you observe in your user data.

Quantitative methods

It’s important to supplement qualitative knowledge with quantitative methods, which are more scalable and give you a more accurate picture.

Here are a few ways you can discover behavioral personas through your product analytics data:

1. Segment your user base by different user and event properties
2. Bucket users based on the frequency at which they perform certain key events
3. Use a clustering algorithm (like the one that powers Amplitude’s Personas feature) to automatically group users based on similarities in behavior

Example

For an on-demand delivery company, personas could depend on factors like how often people place orders and the amount spent per order. This company used their behavioral data to identify a few major personas.

The company used the first method listed above, segmenting their user base by the event property amount spent per order.

They found the following personas:

• "Individuals": People whose typical order size indicates that they’re just ordering for themselves.
• "Group orders": People whose typical order size is above a certain threshold, indicating that they’re regularly ordering for a larger group, like a family, group of friends, or small company
  

They also used the second method, bucketing users based on frequency of performing certain events. For this company, placing an order is the critical event.

The chart below measures a metric called stickiness (which we’ll discuss more in section 4.3.6).
Out of a 30 day period, it shows the percentage of users who placed an order on at least x days out of 30. In the chart, you can see that a little over 75% of users place an order on 2 or more days out of 30, while only about 20% of users place an order on 7 or more days out of 30.

Based on this data, the on-demand company decided on these personas for order frequency:

• "Occasional orders": People who placed an order on 1-3 days per month.
• "Frequent orders": People who placed orders on > 4 days per month. For these people, using the on-demand service is part of their routine, rather than an occasional convenience.

People who fall into these different categories likely represent distinct demographic groups and are using the on-demand service for different reasons. It’s important for the on-demand company to understand these personas so they can best meet the needs of all their customers.

Amplitude has a feature called Personas that automatically groups users based on what actions they take, and how frequently they take them.

The Personas feature uses a unique clustering algorithm, based on k-means clustering. You can run Personas on any cohort of users, so in this case, you would select your current users cohort.

When you run Personas, it gives you a number of user groups, or clusters. You can see how many users in the time period fall in each group, and also see the percentage overlap with any other cohort you choose. For example, you can see what percentage of users in a cluster are retained 3 weeks later. For more details on how to use Personas in Amplitude, we recommend checking out this guide to personas in our Help Desk.

In the example below, you can see 3 clusters generated from the current users cohort of a business software product. Cluster 1 is the largest, making up 44.5% of all current users—but only 0.766% of users in Cluster 1 are retained in the 3rd week. On the other hand, you can see that users in cluster 2 have very high 3rd week retention: 94% of them come back in the 3rd week.

Personas also allows you to compare the behavior of users in each cluster. You can see tables of events that a given cluster performs more often or less often when compared to the other clusters. When we look at Cluster 2, which has the best retention, you can see that users do the ‘Create Record’ action 60 times on average, which is much more often than Clusters 1 and 3.

The table also shows you the number of standard deviations above (or below) the mean, which gives you an indication of how significant that difference is. In this case, this cluster’s data is a good indicator that users who do ‘Create Record’ above a certain threshold might be more likely to retain long term.

Create cohorts of these behavioral personas for further investigation

Once you’ve identified a few key personas that you’d like to learn more about, you need to create cohorts of these users so that you can do more analysis on them.

In Amplitude’s Personas feature, you can create cohorts directly from Personas to investigate further.

Once you’ve created your lifecycle cohorts (Section 3.3) and identified your behavioral personas (Section 4.2), you should measure the baseline retention for each cohort and dig deeper into the drivers of retention. This section covers a number of product analysis methods that you can use to do that. You don’t need to use all of these methods—think of this chapter as the toolkit from which you can pick and choose the methods that will be most enlightening for your product and your users.

Through these methods, you can answer questions like: Does a certain persona retain better than others or have a higher lifetime value (LTV)? What actions seem to be contributing to those metrics? How could you get more users into that persona?

Your product analysis toolkit includes:
4.3.1 – Measure & compare baseline metrics for each cohort and persona
4.3.2 – Investigate user properties
4.3.3 – Use behavioral cohort analysis to measure the impact of different user actions
4.3.4 – Conversion rate through your critical path funnel
4.3.5 – Find your most common user flows
4.3.6 – Measure stickiness
4.3.7 – Session metrics

Remember, retention is the main metric that you will diagnose in each chapter and try to improve over time. Later, we’ll break down this retention by different user properties and behavioral personas.

First, plot the retention curve of your current users. Remember, you can use N-Day, unbounded, or custom bracket retention, as discussed in Section 3.1. Depending on your business goals, one may make more sense for you to use than the other.

Here are the recommended retention metrics to look at based on your usage interval:

In this chart, you can see the retention curve for current users of a music streaming app. The first event and returning event are set to ‘Play Song’, which is this product’s critical event.

PRO TIP:
A user property is any characteristic that is tied to an individual user. Some common examples are: country, age, language, platform, customer plan type, and paying vs. non-paying user.

Compare the retention curves of behavioral personas

Comparing the retention curves of different behavioral personas will help you decide which personas to focus on—should you commit resources to converting more users to certain "power" personas?

Do some of your core or power behavioral personas retain better or worse than others? And how much of a difference is there over your overall current user retention? By quantifying these retention differences, you’ll have a better idea of which personas to focus on in your retention strategy.

Retention curves make it really easy to see these retention differences at a glance. Below, you can see the daily retention curves for 3 different behavioral personas found in Amplitude. Clearly, Persona 2 (green) has much better retention than the other 2 personas. In a case like this, you’d want to understand whether you can get people in Personas 1 & 3 to behave more like Persona 2 to improve their retention.

Looking at user properties will give you a high-level understanding of who these users are. Measure the breakdowns of key user properties to help you identify trends and groups of users you should study more closely.

User properties in Amplitude

In Amplitude, user properties can get updated throughout the course of a user’s lifecycle in your product. Amplitude tracks a set of default user properties and also allows you to define any custom properties you need.

You can segment every chart by user properties and use them in behavioral cohort definitions.

In Amplitude, you can use the User Composition view to quickly visualize different user properties of a specific group of users. Below, you can see the breakdown of the current users cohort by Region.

Example

One of our customers, an on-demand delivery company, segmented their retention curve for current users by the Platform user property. As you can see, there are some large differences in retention between the different platforms, and people on iOS have the lowest retention of all.

Takeaway
Based on this data, we recommend they look more closely at behavior across their three platforms and see what about the user experience could be improved to get Android and iOS retention levels to match the web.

Segment retention curves by user properties
Once you identify any user properties you want to study, segmenting your retention curve by these properties will help you identify any significant differences that are worth exploring further.

While segmenting your retention curve by user properties can help you uncover useful insights, it doesn’t give you any information about how users’ behavior within your product impacts retention. That’s where behavioral cohorts come in.

A behavioral cohort is a group of users who performed (or didn’t perform) certain actions within a defined time period.

Here’s an example. Facebook found a famous insight that users who added at least 7 friends within their first 10 days were more likely to be retained long term. That’s a behavioral cohort: there’s a behavior (adding at least 7 friends) and a time period (within 10 days of signing up).

PRO TIP:
In user analytics, the broadest definition of a cohort is a group of users who share some common characteristic.
There are two main types of cohorts:

Acquisition cohorts group users by when they signed up for your product. You might break down your cohorts by the day, week, or month they signed up. By measuring the retention of these cohorts, you can see how long people continue to use your product from their initial start point.

Behavioral cohorts group users by behaviors they perform in your product within a given timeframe. These could be any number of discrete actions that a user can perform—sharing a photo, playing a song, buying gold coins, or any combination of these actions. A cohort will be a group of users who did those actions within a timeframe that you specify (for example, within the first 3 days of use). You can then measure how long different cohorts stay active in your app after they perform those actions.

Forward-thinking companies today are using behavioral cohorts to understand how different actions or characteristics of users impact retention.

Creating behavioral cohorts in Amplitude

In Amplitude, it’s easy for anyone to create behavioral cohorts and apply them across different charts to measure the impact of user actions on your metrics. In this example, you can see a cohort of users who started a trial subscription within 7 days of first using the product.

You can add additional events and user properties to any behavioral cohort definition, as well as specify that users have not done a particular action or don’t have a certain user property.

Once you’ve created a behavioral cohort, simply select that cohort in a retention chart to see how well those users retain. You can compare their retention with users who are not in the cohort, or with a different cohort of users.

The chart below shows the retention curve for 3 cohorts of users:

• Started trial – first 7 days
• Did not start trial – first 7 days
• Played at least 3 songs – first 7 days

As you can see, users who started a trial in their first week have significantly better retention than users who didn’t, indicating that starting a trial could potentially contribute to better retention. Users who play at least 3 songs are somewhere in between, but still nowhere close to users who start a trial.

Takeaway
Behavioral cohorts can help you form and test hypotheses about actions that are important for retention. With the graph above, you now have correlation, but not causation. In this case we recommend running a test where you encourage more users to start a trial early in the user experience, such as in the onboarding flow or with an email reminder. Then, you can evaluate whether getting more users to start a trial causes higher overall retention.

A critical path funnel is the series of actions you anticipate users taking in order to complete your critical event. A funnel chart will allow you to visualize the drop-off along each of those steps. You can also compare conversion rates for different groups of users.

Not every product will have a clearly defined path that you want your users to repeat, so this analysis may not be necessary for everyone.

Example
One of the companies we work with, an on-demand delivery company, identified their critical funnel as:

App launched → Select vendor → Add item → Complete order

Here’s the funnel for the company’s current users—each step in the funnel shows the number of users who moved on from the previous step. Of everyone who entered the funnel (by opening the app), 66% did all of the steps and completed an order.

When we segmented the funnel by platform (iOS, Android, or web), we found that users on iOS had a higher conversion rate through the critical funnel than web and Android.

Takeaway
The largest drop-off happens between the App launched → Select vendor step. This is the step we recommend focusing on first if we want to improve the conversion rates for web and Android.

Visualizing user routes with Pathfinder

Creating your own path analysis visualization can be very time-consuming; instead, the Pathfinder feature in Amplitude enables you to explore the actions users take to or from any point in your product.

Pathfinder aggregates the paths users take, so that you can see the percentage of users or sessions that followed each sequence.

You can specify a particular starting action to see all of the events that follow, or select an ending action to see all of the paths that lead up to that event. For example, a common use case is to see what users do before making a purchase or upgrading their plan.

In addition, you can study the behavior of a certain group of users by defining a user segment or choosing a behavioral cohort. This allows you to compare the paths of different groups of users.

Funnels are great for measuring well-defined sequences, but what if you’d like a broader picture of how new users are behaving? It’s impossible to know ahead of time every single route a user might take in your product, and people often defy our expectations of ‘normal user behavior.’

This is where user activity streams that show you sequences of events can be really useful. Depending on your analytics setup, you might be able to get raw logs of all of the events that users do and study some of these sequences to look for patterns. Some analytics platforms (including Amplitude) will give you access to these user timelines out of the box. In addition, you can do some user testing to watch how people navigate through your product.

There are a number of questions you can answer with path analysis, including:

• Comparing the paths of retained users to dormant users
• Seeing what users who drop out of a funnel are doing instead
• Identifying the main paths toward an important event in your product, like creating a new account or making a purchase

For example, you can use path analysis to discover the paths users take to end up at your critical event. In the image above, you can see what users are doing before they purchase a subscription in a music streaming app.

Measuring stickiness provides another dimension of understanding user engagement. While retention measures the rate at which users return over time, stickiness looks at usage frequency—how many times people use your product within a certain time period.

To compare stickiness metrics, you can measure stickiness both for:

• General usage: how many days out of a week or month did users open the app and do anything?
• Your critical event: how many days out of a week or month did users do the critical event?

If your product has a daily or weekly usage interval, you should compare the weekly stickiness metrics of your persona cohorts. If your product has a biweekly or monthly usage interval, you should compare the monthly stickiness metrics.

TERMS TO KNOW:
Stickiness refers to the frequency at which people are using your product. Specifically, stickiness measures the number of days out of a given time period that a user was active, or did a specific event (like your critical event). In Amplitude, we have two options for measuring stickiness:
Weekly Stickiness: The percentage of users who were active or performed a specific event at least N days out of a week.
Monthly Stickiness: The percentage of users who were active or performed a specific event at least N days out of a month.

Example: General usage stickiness in a social mobile game

Stickiness can help you identify your most engaged users. Here’s a chart showing stickiness for the main behavioral personas of one of our customers, a social mobile game. This graph is measuring stickiness for general usage, counting each day that a user opened the app.

As you can see, Persona 4 is the most sticky. Almost 75% of the users in that cohort are opening the app for 7 out of 7 days in a week—we don’t need to tell you that’s really high.

Takeaway
With a chart like this, we recommend focusing in on Persona 4 to learn more about what makes them so engaged. You’ll then want to encourage those behaviors in other users to get more people to match Persona 4.

Session Metrics in Amplitude

Amplitude has default ways of defining user sessions, but you’re free to modify this definition as makes sense for your product. For example, a music streaming service would want a session to last as long as someone was playing music, even if the app is in the background.

Key session metrics you can measure are:

• Length distribution: the distribution of session lengths of all users, shown as a histogram.
• Average length: the average session length per user.
• Average per user: the average number of sessions per user, per day.

Below, you can see a graph of session length distribution. The chart shows that the greatest number of sessions are between 10-30 minutes long.

Yet another way to round out your understanding of user engagement is with session metrics. Loosely defined, a session is the period in which a person is actively using your product. A caveat is that the length of time someone spends in your product may not be a good indicator of engagement for your business.

You should only look at session metrics if it makes sense for your app. For example, an on-demand delivery service or an app that helps you find and book exercise classes wouldn’t necessarily care how much time a user spends in the product—if anything, the process to complete an order should be as fast and seamless as possible. For a social game or music streaming service, however, the amount of time spent in the app directly correlates with how engaged a user is.

Here we see that Power Users (blue) spend much longer in the app, with an average session length around 30 minutes, compared to roughly 16 minutes for Passive Users (green) and Total Current Users (orange).

Summing up

Now that you know how to find your behavioral personas and have these product analytics methods at your disposal, it’s finally time to roll up your sleeves and get to work.

You’ll apply these methods in each of the next 3 chapters, where we’ll walk through the entire Retention Lifecycle Framework. Let’s get started!

This section includes key concepts and methods that you’ll use to understand user behavior and retention at all stages of the retention lifecycle. Refer back to this section as you work through Chapters 5-7 of this playbook to remind yourself of all the methods available to you.

A behavioral persona describes a distinct way of using your product. Identifying your product’s personas will inform your product development for different types of users.

• Qualitative: User interviews and user testing can provide more context for trends you observe in your product analytics data.
• Quantitative: Segment by different user & event properties; bucket users based on the frequency at which they perform key events; or use a clustering algorithm to automatically group users based on similar behaviors.

Comparing the retention curves of different behavioral personas will help you decide which personas to focus on—should you commit resources to converting more users to certain "power" personas?

Make sure you choose the retention method that makes the most sense for your product: N-Day, unbounded, or custom bracket retention.

Measure the breakdowns of key user properties to help you identify trends and groups of users you should study more closely. Some common examples are: country, language, platform, paying vs. non-paying — but make sure you look at ones that are important for your business.

Segment your baseline retention curve by different properties to identify factors that could impact retention either positively or negatively.

A behavioral cohort is a group of users who performed (or didn’t perform) certain actions within a defined time period. Create cohorts for behaviors that you’re interested in, and then measure the retention of those users to see how well that behavior correlates with retention.

You can also apply behavioral cohorts to funnel conversion rates and any of the other analyses in this toolkit.

A critical path funnel is the series of actions you anticipate users taking in order to complete your critical event. Comparing conversion rates for your behavioral personas and by different user properties will help you identify areas for improvement.

Funnels are great for measuring well-defined sequences, but user behavior usually isn’t so linear. Look at the most common paths to or from important actions, and compare the paths of your behavioral personas and cohorts.

Stickiness refers to the frequency at which people are using your product. Specifically, stickiness measures the number of days out of a given time pe-riod that a user was active, or did a specific event (like your critical event).

Key session metrics you can measure are:

• Length distribution: the distribution of session lengths of all users, shown as a histogram.
• Average length: the average session length per user.
• Average per user: the average number of sessions per user, per day.