Why You Need Cohorts to Improve Your Retention
Heyday launched in December 2013 as an automatic journaling app. Using your mobile photos and location data, Heyday automatically creates a personal journal entry out of your day. It’s a beautiful product and the launch was a massive success.
Heyday’s founders so impressed investors that they added $3.5 million in funding pre-launch. They received rave reviews from the tech media and in less than one month post-launch, they had amassed hundreds of thousands of downloads.
To a lot of people in tech, this sounds like success. Gaudy download counts and another big round of funding—you’ve made it.
But to experienced founders like Heyday founder/CEO Siqi Chen, all of that was just the starting point. Before Heyday, Siqi worked on making mobile games insanely sticky at Zynga, so he knew full well that getting a download means only you’ve gotten a foot in the door. The real work begins after that.
Long-term success means not only getting someone to download your app, but also getting them to come by day after day because you’ve built something that they can’t live without.
Possessing that must-have value proposition isn’t something you launch with, it’s something you achieve through close study, experimentation and iteration. To get there, you need to go beyond vanity metrics that only superficially measure growth and retention—like download counts and even daily active users (DAU) / monthly active users (MAU).
You need to dig deeper into your app using a method called cohort analysis. That’s how you’ll identify how well your users are being retained and the primary factors that will drive growth for your app. Here’s how the most experienced and analytical product people like Siqi go beyond your standard cohort analysis to do it.
What is Cohort Analysis?
A cohort is a group of users that share a common characteristic. Cohort analysis is looking at the retention analytics of those users over time. There are 2 main ways to break your users into cohorts for cohort analysis:
- Acquisition cohorts: divide users by when they signed up for your product. With a consumer app, you might break down your cohorts by the day they signed up. SaaS services (like Amplitude) are more likely to track monthly cohorts. By measuring the retention of these cohorts, you can see how long people continue to use your app from their initial start point.
- Behavioral cohorts: divide users by the behaviors they exhibit in your app within a given timeframe. These could be any number of discrete actions that a user can perform — sharing a photo, playing a song, buying gold coins, or any combination of these actions. A cohort will be a group of users who did those actions within a timeframe that you specify (for example, within the first 3 days of app use). You can then look at how long different cohorts stay active in your app after they perform those actions.
Cohort analysis is critical because DAU / MAU counts are highly distorted by growth. If your app is growing rapidly, new user signups will mask where your existing users are dropping off in your DAU / MAU numbers. If you’re only looking at DAU / MAU, you’ll be blind to retention issues that will kill your app if left unaddressed.
How to build a retention process using cohort analyis
The most powerful aspect of cohort analysis is that you’ll not only see that customers leave and when they leave, but you can start to understand why your customers leave your app—so that you can fix it.
At HubSpot, the billion-dollar SaaS company that popularized the idea of inbound marketing, VP of Growth Brian Balfour uses this defined retention process to fight churn and increase growth in the company and products. The process can be broken down into specific stages:
- Goals - Set a goal for the process. Do you want to reduce churn in the short-term? Long term? What is your growth target?
- Exploration - Explore current data to see where changes can be made to reach your goal.
- Hypothesize - Decide what questions to ask and possible outcomes from experiments.
- Brainstorming - Conceive possible experiments to test hypotheses.
- Testing - Run different tests to evaluate hypotheses.
- Analyzing - Analyze test data to see whether goals were met.
- Systematizing - Make any positive changes part of the system.
At each stage, you’re looking at a specific stage in the user lifecycle, and then using different strategies for improving retention in each:
- Early: how do the day 1 (D1) retention rates look. Do you see a drop off to 80% active or down to 10% active? How quickly can you get your user to the ‘a-ha’ moment, when they realize the core value of your product?
- Mid: from D2 to D30, are you able to keep getting the user back and experiencing the core value of the app repeatedly?
- Long-term: retention typically levels out and goes asymptotic. The best apps have a retention curve that looks like a smile, where user reactivation results in a later upward curve.
(via Evernote Smile Graph on TechCrunch)
Through this system, you can continually improve your app and increase retention. You aren’t looking at an aggregated churn number (e.g., 15% monthly), making some changes here and there, and then praying that you see the improvement reflected in your aggregate churn rate. You’re targeting a specific behavior and testing to see whether encouraging users to adopt that behavior improves retention.
And this all starts with cohorts. Here’s how you can use both acquisition and behavioral cohorts to tease out exactly what your users are doing and when they’re doing it. From this data, you can build a systematic approach to learn how users fall in love with your app—and then make it happen again and again.
Acquisition Date Cohorts—Finding Problem Moments in Your App
Say you have a music app. You know that you have a churn problem, but that’s about it. You eyeball your numbers, and you’re seeing about 10% of your users dropping off each day.
The users in the retention chart above are split into daily cohorts — users who signed up on the same day. You can see that 13,473 users signed up for your music app on Oct 27. Day 1 retention was 57.9%, day 7 retention was 19.9%, and day 10 retention was 14.1%. So on the 7th day after first using the app, 1 in 5 users that signed up on Oct 27 were still active users in the app.
Out of all of your new users during this time range (322,902 users), 49.8% are retained on day 1, 15.8% are retained on day 7, and 4.7% are retained on day 30.
The best way to visualize this information is to turn it into a retention curve, which shows your retention for these cohorts over time. When you chart your data like this, it becomes incredibly easy to see when users are leaving your product.
This retention curve immediately tells you something super important: half of all users stop using the app after the first day.
After that initial large drop, there is a second brisk drop to under 20%, before the curve starts to level off after a week, leaving about 5% of original users still active in the app at day 30.
That’s not great (although it is common — some data shows that the average app loses 90% of its users within 30 days) . Early retention is a significant issue. A curve like this indicates that users aren’t getting the core value out of the app quickly enough, so they are leaving. Now you know that you need to improve the early app experience to get the user to your core value as quickly as possible.
Hitting the limits of acquisition cohorts
If your app has the retention curve shown above, you immediately want to figure out what you can do to boost your retention.
The problem is that from just looking at acquisition cohorts, you don’t have any information about how you can improve the user experience to retain your users. You can’t isolate specific behaviors or user properties.
Acquisition cohorts are great for showing you trends and telling you when people are churning, but to understand why they are leaving you need to turn to another type of cohorts: behavioral cohorts.
Behavioral Cohorts — Discover Which Behaviors Drive Retention
From the moment that users sign up with your app, they make hundreds of decisions and exhibit countless little behaviors that all lead towards their decision to stay or go. These behaviors could be anything:
- using core feature X but not using core feature Y
- engaging only with notifications of type Z
- connecting with 1-2 people on the app, not 10+
That means that when you decide to rework your user onboarding to ramp up your D1 and D2 retention, you’ll have a million ideas on how to do it. Ultimately, you may end up choosing the one that was argued for the hardest by the loudest person on the product team.
Rather than choosing what to work on based on anecdote or random choice, behavioral cohorting allows you to answer that question in a systematic, quantitative way. Unlike acquisition cohorting, which is grouping users by sign-up date, behavioral cohorting is grouping users based on the specific actions that they have (or haven’t) taken.
This type of analysis is what made Zynga’s games insanely sticky. It gets at the heart of what every startup wants to know: What makes users stay? If you can figure out what your users find valuable and optimize for this core experience in your app, you can keep more users around for longer.
Finding the right cohorts
For your music app, you can create different user cohorts from the following actions: playing a song, searching for an artist, or creating a playlist.
Say you wanted to see the retention for users that favorited songs in the app. With behavioral cohorts you can look at retention for new users who favorited 3 or more songs:
Now you see that this might be an interesting behavior to explore.
While 50% of all users churn within one day of using the app, only ~15% of users who favorite 3 or more songs churn out after day 1. There could be something about favoriting songs that keeps people around.
Inverting your cohort
To check, you can also look at the inverse — all the users who didn’t favorite 3 or more songs:
Users who didn’t favorite 3 or more songs have worse retention than most, with 55% of these users churning out after that first day.
From even this simple analysis you can see that getting people to favorite songs early in their experience allows them to encounter the core value of the app, meaning that they are more likely to continue as users.
Of course, there is more to a music app than favoriting songs. Behavioral cohorts give you a more granular understanding of why customers churn.
You can use any action that can be performed in your app to create behavioral cohorts. This means that you can correlate any number of different user actions with retention rates.
For example, your music app has a cool feature that lets people join communities based on their favorite genres. So you’re curious: Does that help improve retention, or is it clutter that confuses the user?
Here you can see that initial retention is similar to users who favorite songs, but what’s also obvious is that retention is better for people involved in these communities in the medium-term, i.e. those first few days of using the app beyond day 1.
As users get involved with other people and find more music to play, they start to enjoy the app more and continue using it. You would probably hypothesize this in your retention process, but now you have data to back up your hypothesis in black and white (well, burgundy and mint green).
Finding the right combinations
From here, the next step is obvious. What about user who favorite songs and join communities? Using the Amplitude dashboard, you can filter your actions to combine these two cohorts:
Now you can see that users exhibiting both these behaviors are far more likely to keep using the app in those first few weeks. At the end of the first week, retention is above 60% for the favorite+community cohort, whereas it’s under 10% for users without either of these behaviors.
Success! You now have a great idea of what makes people use your music app. Infinite users forever! Well, not quite.
Causation, not correlation
Just because people who favorite songs and join communities churn less doesn’t mean that driving people towards these behaviors will magically make them stay with the app forever. A call-to-action getting them to join 20 communities immediately upon sign up will probably end badly.
That’s because favoriting songs and joining communities could be merely correlated with user engagement, not causing it.
To determine causation, you’ll need to actually change the onboarding experience to emphasize favoriting and see whether that improves retention. For example, when Twitter had a problem with churn, they experimented with different sign up processes to put different value propositions in front of new users as soon as possible to see which caused an improvement in retention.
In this case, if you have a high enough number of users for such testing, you might A/B test different onboarding flows that emphasize favoriting in the app to see if it increases retention.
Once you have this data from behavioral cohorts you can start to run experiments to test behaviors that may be related to retention. You can see what works and what doesn’t and systematically increase your retention.
When It All Comes Together
For Heyday, Siqi knew they had to build on their initial success by iterating on the product to make it more and more sticky. To get true, sustaining growth, you need to master retention for your app—otherwise, your launch-day bump will be followed by massive drop off and a quick death.
Early on, Siqi was faced with a huge product decision. For each journal entry that the app created for you, it would put together a collage of the photos you’d taken that day. If you tapped on the collage, you would be taken to an editing screen where you could apply filters or re-arrange the collage. However, a vocal group of users said that instead of editing the collage, they just wanted to tap to view it in more detail.
Should the app focus on just showing people their images of the day and make a killer passive experience, or should the app empower users to edit their photos and create their own story out of the day? Whichever way they went would decide the future product direction of the company.
Siqi and his team knew that one of the most critical questions they could ask was which of these behaviors—viewing or editing—correlated with long-term retention in their app.
To find the answer, they created a behavioral cohort of users who heavily edited collages early on in their use of the app and another behavioral cohort of users who were primarily viewing their images in Heyday.
What they learned was absolutely critical in understanding the core value users took away from the app. They discovered that the Heyday users who edited their collages were retained far better than those who were just using Heyday to view photos.
In a pivotal moment in their company, they had just the right data for the job to help them move in the right direction. They doubled down on the editing experience everywhere in the app, resulting in a better core experience for all of their new users.
How to Get Started With Your Own Cohort Analysis
What was once only available to billion-dollar companies like Zynga is now available to every company that wants to improve its retention. Instead of having to go out and hire an in-house data science team to do cohort analysis, all you need to do is drop in an analytics SDK and define some events, and you’re off to the races.
- Take a look at your retention by acquisition cohort. That will show you when users are dropping off.
- Define events for a few of your app’s core user actions and then pull up your behavioral cohorts. Analyze your behavioral cohorts by comparing them, inverting them, and combining them. Use that with your learnings from looking at acquisition cohorts to generate a hypothesis about actions you can emphasize during a specific part of the customer journey to drive retention.
- Make changes to your app—using A/B testing if you have a high enough usage volume—to see whether driving certain actions in your app actually cause users to come back.
- Process your learnings and repeat.
By using analytics solutions such as Amplitude’s behavioral cohorts you can see the specifics of your customers’ behaviors and start to make data-driven decisions to enhance their experience within your app.
By breaking down their app experience into individual customer behaviors, you can find exactly the point at which they either had that ‘a-ha’ moment or missed it, drill down into why it occurred or didn’t, and then make sure that it is there for every single one of your customers in the future.
AJay Arora: Founder NoteAny.com
Alicia Shiu: Thanks AJay! Exactly, behavioral cohorting can help you prioritize which product changes to focus on by showing you what might have the largest impact on retention. Prioritization is key, especially for small/early teams.
Stitchard: A really useful post. Thank you
Carylyne Chan: Product @ keyreply.com
david zangwill: Yes I agree with Carylyne - amazing feature that startups needs to determine feature/market fit - the only downside is that behavioral cohorting on Amplitude is a enterprise feature.. as my company is early stage (tight budget) and needs these insights, I’m considering going with mixpanel instead.
Aditya Vempaty: Thanks for commenting! Totally understand the issue with budgets, but one thing to be aware is that Behavioral cohorting is something that Mixpanel can’t do. This is do to having them not being able to tie the event property together with the people priority. We are happy to chat more to see if we can work something out as well.
tchernetski: Hey, great article! Where can I read how to add custom segment like “Favorite > 3 songs”, I have only standard Amplitude segments there :(
Alicia Shiu: Thanks! “Favorite > 3 songs” is a Behavioral Cohort, which is a feature on our Enterprise plan. You can read about them here: https://amplitude.zendesk.com/hc/en-us/articles/231881448, and if you’re interested in learning more, just reach out to firstname.lastname@example.org!
tchernetski: Thanks, Alicia!
Roninpawn: Another way to attain this kind of data that some may find simpler and more cost effective: You know those emails people send to your company’s public-facing address, sometimes praising you, a lot of times venomously angry about something? Well, what you do is, you aggregate the comments from those emails and other sources into a common database and then cross-analyze the synergy platform for maximum logistical visibility while proactively capitalizing on the emerging paradigms of the millennial value space. Or if all that doesn’t work, just listen to feedback. Trust me, no one’s doing it right now. It’s cutting, fucking, edge.
Mel: Wow, Thanks for wasting my time !
Vladislav Yakov: Thank You
Last Updated: 05/17/18
Release Notes: May 2018
New in Amplitude in May 2018: exclude events from funnels, add descriptions to events in bulk, impro...
How I PM: Sam Goertler, Lead Product Manager at theSkimm
Sam is the Lead Product Manager at theSkimm, and former Senior PM at Asana and this is how she produ...
How to Organize Your Product Team Around Your North Star
Organizing a product team is hard but having a north star makes it easier.