As an engineering team, you don’t just want to ship features. You want to ship impact. And, if you’re in a startup, the quicker the changes, the better.
But to know what your customers want—and what changes to make—data and analytics are crucial.
That’s why MySwimPro made the switch to Amplitude, said the company’s VP of Engineering Nick Newell in a discussion with Erin Del Mundo, Amplitude Success & Renewals Manager.
While the team had the expertise to build its previous, custom solution, it lacked the number of engineers to maintain it.
“Every hour, every day that [our engineers] are spending on [maintaining the custom solution], they’re not working on growth."
The move paid off: Amplitude helped the MySwimPro team conduct experiments crucial to its business goals without overextending its resources. The team increased the number of experiments it ran by 10x, and the engineers had more time to work on adding value to the product, work that is critical to the business’s growth.
- Balancing rapid experimentation with limited team resources
- Taking big risks as a small team—with data for support
- Using analytics to quickly resolve issues in the user experience
- Avoiding data overload while still getting the insights you need
Read on to learn how Amplitude helps the MySwimPro team drive results, or watch the full webinar, Deliver Impact with Every Release.
Speed up experiment velocity
Nick and the rest of the MySwimPro team knew that in order to grow the platform, they had to experiment. The decision to go with Amplitude boiled down to one key factor: the speed of experimentation. With its previous custom stack, the team was only able to run approximately two experiments every quarter. “If you can only run one experiment every couple months . . . you're not going to cut it,” comments Nick.
Now, with Amplitude, the team is able to run 20 experiments every quarter. Additionally, Amplitude has enabled the team to run experiments on multiple platforms simultaneously—including its iOS app, Android app, and web platform.
Nick explains that even if half of the 20 experiments fail, that means ten are successful, “which is huge because each one of those could be bringing 20%, 30%, 50%, or even 70% improvements on your revenue.”
Take risks
MySwimPro is a small but growing business.
“When you're in that position, you have to take some risks,” says Nick, explaining that the team doesn’t wait for their experiments to reach statistical significance.
A smaller pool of daily and monthly active users means waiting for statistical significance would dramatically slow the rate of experimentation.
“We would be back to doing one experiment every two months,” explains Nick. “We wouldn’t be able to make decisions.”
Amplitude gives the team more data to base its decisions on. Amplitude experiments enable MySwimPro’s engineers to know the number of days until their experiment reaches statistical significance. It also gives them extra charts to inform their decisions, like average revenue per user (ARPU).
The ARPU chart shows the lifetime value of a member. When the team runs an experiment, they’ll look at a primary metric, like revenue, then dig into conversion rates and how much revenue is associated with each variant. The team can look at how the data is trending and decide without waiting for statistical significance.
ARPU is just one of the charts the team uses. “There are many tools in the Amplitude platform. . . . [Amplitude] has given us the data that we need to take a little bit more risk,” says Nick.
Focus on individual users to fix product issues
Hearing about a potential user issue without being able to investigate further is frustrating for an engineering team, says Nick.
Being unable to examine individual user journeys, he adds, means “You’re living in the world of the abstract. This is where we were with Google Analytics.”
“The main limitation [with GA was] that we didn’t get enough data about a single user. If you're interested in really getting deep insights about what your members and your users are doing, you're not going to get that from Google Analytics.”
Now, with Amplitude, the team can examine what happened in the product experience of a specific user for any issues or defects that users report.
“We can look at every single event that has happened to that user prior to when they're experiencing the problem,” says Nick. “It’s transformational.”
For more reasons why leading organizations choose Amplitude over GA, check out the comparison guide.
As a result, the team has seen improvements in the two DevOps Research and Assessment (DORA) metrics they use to measure quality—change failure rate and mean time to restore. “Those two numbers have gone about 10x in a positive direction since we started using Amplitude,” Nick says.
“It used to take us days and days and days to figure out what was going on in the product. Now it takes 20 minutes because we take one example, we drill in, we can see exactly where the problem is, and we can fix it.”
Move the needle on KPIs
Amplitude gave Nick and his team the extensive user data they needed to understand their customers. The Amplitude platform also made it easy for the team to dig into that data in a targeted way to extract key insights and drive their growth.
Initially, Nick was concerned about the potential data overload that digital analytics tools could generate. However, the team learned that they could use the data to their advantage if they were selective about what they focused on. “You will have a lot of data, but I promise you that you can overcome that with focus and discipline,” advises Nick.
To build that focus and discipline in a small team, Nick recommends focusing only on two key performance indicators (KPIs). The MySwimPro team is currently working on growth and its unique value proposition, and they have one KPI for each area:
- Growth KPI: Revenue
- Value proposition KPI: Number of guided workouts.
Nick shares that Amplitude helps the team access the data they need and ignore the noise. As a result, they’ve been able to prioritize their time on work that pushes their two objectives.
Become data-driven
Amplitude has transformed the culture around data at MySwimPro. “It changes the strategy and the conversation . . . when you have good data,” says Nick. He shares that now, during team meetings, it’s common for someone to raise their hand and say: “Let's just run an experiment, and in two weeks, we're going to know.” The team doesn’t have to argue back and forth. They just go with what the data says.
Tune into the full webinar—Deliver impact with every release.
