Insight/Action/Outcome: By democratizing experimentation, Super.com accelerated testing by 4X and drove deeper insights into customer interests—helping them drive a 90% month 1 increase in customer retention.
I’ve spent my career analyzing data at companies from 50 to 5,000 people across various industries. The continual challenge of leading data teams has always felt like the long cycle time of going from an idea to being able to evaluate it in front of our users, and the tendency for data teams to succumb to what ideas leaders want to see rather than the features that users truly want. These ideas have underpinned my focus at Super.com to help our teams get more ideas in front of users and make it easy for anyone to launch an idea.
Super.com is the all-in-one app that puts more money in your pocket! Our users can save big on hotels, access a cash advance, get cash back on purchases, boost their credit score, earn money playing games, and more. Our flagship products are in travel and fintech. We have ~200 employees, have raised ~$200M to date, and have processed >$2B in sales. Our Super+ membership is one of the fastest-growing programs in the US.
We’re trusted by over ten million customers, saving them over $200 million USD to date. Super.com is backed by four-time NBA champion and MVP Steph Curry, who understands the winning mindset required to grow and how small wins like what we give to our customers add up.
It’s all about data
At Super.com, we rely on data to make decisions. Data enables us to grow and save simultaneously. We think of data as our customers speaking to us in numbers because we can track their feedback on our offerings in real time.
To get this, we run experiments to evaluate our team’s ideas for features and benefits to offer our customers. But we wanted a faster way to test, assess, and get capabilities out the door. That’s where Amplitude Analytics and Experiment came in. It’s become a big part of our workflow, with 60% of the company being weekly active users, half of whom build and share their analyses across nearly every team at our company.
Testing, testing… What do customers need?
One of our key initiatives was to start monetizing our savings and earnings products through our Super+ membership program. Customers would pay $15 monthly to unlock new benefits and features. Before offering this feature, we needed to test customer interest, and with Experiment, we discovered their interest was high. Experiments enabled us to determine winning features or copy, then rapidly iterate on these to optimize for our users.
Regardless of your company size, new features or products don't work if users don't like them—but you can’t know that with just a handful of experiments.
So the question is not, “How do we run successful tests?” but “How do we run more experiments in general and make it easier for anyone to launch a test?” The key is to democratize experimentation by lowering the barrier to entry and making it easier to run a wide variety of experiments. The more times we have at bat, the more chances we will have to be successful.
It’s not about running a few successful tests but continually running as many as you can. Only then can you really understand what customers want, as you don’t know what the winner will be before you run them.
With Experiment, anyone from an intern to our CEO can quickly test an idea and understand its validity in a trustworthy manner. The biggest win for me is the faster ability to test, or our velocity gain—in Q1 2024 alone, we ran 4X more tests than before, launching a new experiment every day.
By establishing a single platform for experimentation, we enabled a large group of users to share knowledge and help one another, and leverage a common dataset and metrics to judge success. Having a consistent user interface with clear signals on whether your experiment was successful and error free empowers us to get comfortable launching and evaluating our results more quickly, and find winners faster.
The biggest win for me is the velocity gain—in Q1 2024 alone, we ran 4X more tests than before, launching a new experiment every day.
Customers just keep coming
We are now using Experiment's features to make testing fully self-service for our team. Without engineering, our product managers can easily change our website copy and fonts and then observe the impact on click-through rates or how users engage with different versions of the site.
With these data points at our fingertips, we’ve learned what makes our customers feel rewarded for their membership and built on that. The result? A 90% increase in customer retention.
We can also better ascertain which advertisements are most effective at showcasing our products or Super+ membership to customers. With Experiment, we constantly iterate on our landing pages through channels such as Facebook or Google ads. Thanks to this, we saw our conversion rate increase 35%, driving direct bottom-line impacts from our performance marketing efforts.
Don’t limit experimentation to centralized teams. Think of how to make running a new test so easy that anyone with a great idea would be able to do it regardless of their skill set. The more variety of ideas, the faster you can innovate.
Learn, grow, repeat
Now, we have documented best practices that help us maximize the benefits of our Amplitude data. We look at questions that are important to our continuous and long-term growth, like “What are the key learnings from both our winning or losing experiments?” or “How do we ensure we can scale experiments as our company continues to grow?”
We also launched an internal sportsbook so people can bet on experiment outcomes with internal tokens—earning them bragging rights. It’s not about seeing who’s right but challenging ourselves to learn about and understand our customers better—this is how we can encourage growth within a diverse team. We focused on building a culture of truth seeking and celebrating the true answer, rather than on validating the hypothesis that led you to run the experiment.
We're Amplitude power users. We’ve invested a lot into building data literacy within our company and have squeezed value out of every feature possible. Christopher Murphy, from the Amplitude account team, has been very helpful along the way, flagging new capabilities and getting us the support we need.
We're a startup that works closely with many younger companies. So, we're always extremely keen to give feedback and sit down with Amplitude’s product teams to discuss our experience using the product features.
It's impossible to know where our company and our customers would be without Amplitude—it’s ingrained in our culture and everything we do. Amplitude has enabled us to democratize self-serve insights and experimentation. This has improved our productivity, which is key for us as we continue to scale, move faster, iterate, learn, and drive impact.
Document what worked and what didn’t. Focus on what can be learned from both our successful experiments, and experiments we do not roll out. If we don’t challenge ourselves and each other, we stall our growth.
