How the Freeletics Team Goes From Hypothesis to A/B Testing
In this edition of "How I Amplitude," Junaid Kokan, Product Manager for Freeletics, takes us step-by-step on how he identifies hypotheses in Amplitude all the way through running monetization experiments.
“The idea was to lay down a user journey map right from the install until the user made the decision to purchase. For example, we could see that monetization was front-loaded. The median time was 28 minutes, meaning most monetization happened within the first few minutes of installing the app.”
The Backstory
We review our progress and strategy each year at Freeletics. At the end of 2023, we wanted to grow three pillars: real-time coaching, selling more, and building a strong platform - making sure everything is iterable, scalable, and that we’ve paid down tech debt.
We took this whole strategy and broke it down into team goals. My team focused on real-time coaching and selling more, as that’s where we could add the most value. The third pillar is more ongoing work over the year but doesn’t require specific features to attack on the optimization side.
From there we decided how we would go about achieving these goals.
For our revenue goal, we saw the biggest opportunity was increasing revenue from new users who installed the app.
Some purchases are impulse conversions - they happen within one day. However, we also wanted to look at longtail conversions, too - those with usage of 30 days or more from installation.
Finally, we monitored basket size and how to increase it. But since Freeletics is a subscription business, basket size relates just to first monetization moment.
Analyzing User Behavior
We weren’t sure where to start, so we started with three main steps.
The first step was to analyze the behavior of the user. That behavior could be something generic, for example, how users convert. We would look at how they behave before purchase, what actions they take, and how it affects their longevity.
Second was to talk to users—there’s no better way to know their needs and the reasons why they subscribe. Our analysis told us what was happening, but talking to users told us the why. It’s vital to get both perspectives.
You have to constantly work on customer feedback to build your backlog. No matter when you attack or start a certain feature, you always have this history of what people really need and where you can use those ideas to tackle the product strategy.
The third step was to enhance the user journey. We created a and put down all the channels where users came from to the app and website so you know what behaviors you want to analyze.
Analyzing user behavior involves doing all of these three things in parallel, enhancing them as we understood more about our users.
Using Amplitude to Collect Data
is the easiest tool I’ve found for data analysis and doing things fast. We’ve used other things in the past, like SQL and other SQL-based chart tools. However, with Amplitude, if you know the events flowing into the infrastructure, you know how to create charts after a couple of tries.
When developing the roadmap, I started by creating a document in Confluence but it can just as easily be done using . I documented the status quo, stating how many people use Freeletics, what the is within 30 days, how many converted within that time, and whether any converted within the first one or two days.
The idea was to lay down a user journey map right from the install until the user made the decision to purchase. You have different that show you where the user enters, where in the journey they purchase, and where they drop off. This gave me a good idea of what the highest priorities were and some insights to work from.
For example, we could see that monetization was front-loaded. The median time was 28 minutes, meaning most monetization happened within the first few minutes of installing the app.
We also looked at when users returned to the app between days one to seven after installing and days eight to 30. We could see where people drop off and what we should improve.
Then we looked at where in the app users convert and which paywall they convert from. It was surprising to see that the order the paywalls perform is not the same in which the paywalls are structured as part of the funnel.
For example, let’s say there’s paywall A, B, and C. You might expect paywall A to do better than B in terms of purchases, but in some cases, the order shifted. One explanation could be that the user doesn’t understand the app at that point and instead buys it later. This can show an issue in the app design that we could work on as part of the strategy for impacting sales.
We had tried incentivizing users to purchase with a discount paywall, but we found that the majority of people, some 80-90%, didn’t even see the option as they closed the pop-up.
From Problem to Solution
Armed with these insights, I presented our findings to the team and we brainstormed ideas for how to improve them.
One idea was offering trials to convince users who otherwise wouldn’t convert. Another option was to provide weekly subscriptions to reduce the commitment, enabling them to subscribe later down the funnel.
Before moving forward, we discussed the impact versus effort, including what would make sense to work on and prioritize within the .
We plugged the estimated impact into Excel and modelled different scenarios, for example, what if the drop-off was not 80% but 50%? How could we improve conversions, and what would the results be from a financial perspective? Once we had come up with estimates we prioritized them based on effort, estimates, and resources.
We would also try to A/B test things whenever possible. A/B testing makes it easy to show the impact that your efforts are having. It also helps me as a Product Manager to learn and iterate on what features worked and why in a granular way. The results of these tests accrue over time and can also help you to estimate future test outcomes.
I found the in Amplitude to be useful as part of the funnel chart to see improvement. You can see the significance shown and conclude a test. In the past, I had to run calculators, go to different sites and plug in the numbers, but Amplitude makes this super easy.
See you in the community!
Connect with Junaid and other practitioners in our community. We focus on in-the-weeds, actionable content. You'll find other Amplitude users sharing best practices, and we have plenty of programs to help you connect with others working on similar things.