Calm is a simple mindfulness meditation app that brings clarity and peace of mind to their users. With a solid user base on iOS and Android, Calm needed to understand how current features were driving retention and what they could do to improve it. They worked with Amplitude to analyze their user behavior data and discover ways to potentially improve Calm’s retention.
Discovering a retention hook
Calm built a Daily Reminder feature that allowed users to set a reminder for their daily meditation session, but the reminder feature was buried deep on the Settings page of the app. Very few users, less than 1%, were finding and using reminders. Calm hadn’t given the reminders feature much thought for several months, but on a hunch, used Amplitude’s behavioral cohorting feature to measure the retention of users who set a reminder, compared to those who didn’t.
To their surprise, they found an almost 3x increase in retention for user who set Daily Reminders. With such a small sample size of users, they couldn’t know whether this was a causal relationship. It could be that the power users of their app, who would have been well-retained anyways, were the ones digging into the Settings page and finding the Reminders feature.
Experiment: Does prompting users to set Daily Reminders increase retention
So, Calm ran an experiment on a portion of their new users where they made Daily Reminders more prominent. In the test, after a user completes their first meditation session, they are shown a screen encouraging them to set a daily reminder.
Result: Increase retention by 3x
Calm found that people who set a reminder from the new prompt had an equal boost in retention to the users who had previously found the reminder feature on their own, indicating that the relationship between daily reminders was causative, not just correlative.
In addition, 40% of users who saw the prompt went on to set a daily reminder, so the new prompt provides a big boost to overall new user retention. Based on these results, Calm plans to roll out the new reminder prompt to all users in the next app update.
Why Amplitude over Mixpanel
Prior to Amplitude, Calm was using Mixpanel for analytics. According to Tyler Sheaffer, Calm’s lead iOS Engineer, “We were just not tracking stuff because it would be too expensive. And that’s a bad way to think about what to track.”
Calm was initially drawn to Amplitude based on their scalability and advanced behavioral features – said Alex Tew, CEO of Calm, “There’s really no incentive not to track as many things as possible.” Alex quickly saw that there was much more to be gained from switching to Amplitude, saying, “It’s actually a much better product than Mixpanel.”
“I also think there’s something more that you get,” said Alex. “With the continual improvement of the platform, support from our Success Manager, and even the pricing structure – you feel like Amplitude is on your side. They want you to have success and get results, as opposed to just, ‘Hey, here’s your analytics platform.'”
Comments
Udit Sajjanhar: Did this have an effect on Daily Retention, Weekly Retention or Monthly Retention. In My experience I have seen such things to boost retention temporarily for short period and then it falls back. Did Calm see a permanent increase in retention?
Alicia Shiu: Calm was focusing specifically on Daily Retention when they ran the experiment, but it’s had a positive impact on weekly & monthly retention as well. They’re still seeing this significant increase in retention 12+ weeks out, which is really great. It does seem to have a more long-term, potentially even permanent, effect.