Insights/Action/Outcome: Thrive Market would show members product recommendations every time they added an item to their cart, but the Member Services team received comments that the feature was a nuisance. The team wanted to see what would happen if they suppressed that feature, using Analytics to understand the number of users affected by the pop-up. After deciding on the metric they would use to evaluate the experiment (an add-to-cart rate and an add-to-carts-per-customer metric), they found that removing the recommendation engine removed friction, and led to a statistically significant improvement in the total add-to-cart rate.
Data can be a source of tremendous insights, but converting it into actionable information can be intimidating without the right tools or asking the right questions. How do you make data accessible? More importantly, how do you allow different teams to play with data and rearrange it into intelligence that drives innovation?
These are the questions we ask ourselves at Thrive Market. Thrive Market is the health-first membership for conscious living, carrying a hyper-curated catalog featuring organic and sustainable food, supplements, and health and beauty products. Thrive Market’s mission is to make healthy living easy, affordable, and accessible to everyone. For just $5 per month, members enjoy substantial savings and convenient online ordering of their favorite products.
Like most online retailers, we needed to find ways to attract new members, improve retention, and increase order conversion. And being a subscription-based business meant we needed to provide the best member experience to our members.
The push to centralize and standardize data ownership
Long before I arrived at Thrive Market, Thrive Market had the vision to build out an agile product team to test, iterate, and learn quickly. The focus was on running rapid A/B tests and building analytics dashboards to scope product opportunities, find gaps in the member journey, and determine where our members were dropping out of our purchase funnels. Thrive Market hired a group of product analysts to sit in cross-functional pods alongside product managers and engineers. They ramped up testing, which marked a fundamental change in how teams approached data.
Thrive Market adopted Amplitude Analytics to improve data consistency and quality across our web platform and iOS and Android apps. It had been difficult to perform cross-platform testing or collect analytics across platforms, and it was painful to spot instrumentation issues and regressions. Engineers would roll out a new release, and we would catch instrumentation issues later.
The first step in overcoming these hurdles was centralizing and standardizing ownership and instrumentation of data. Previously, engineers, product managers, and analysts owned and architected different functions. Consolidating ownership within the analytics team gave our analysts responsibility for writing requirements and ensuring new instrumentations go through a centralized space. The team also began to build guardrails and standardize events. That cleaned up the data and helped us move forward.
Giving one team exclusive ownership and permission to architect our data created a single source of truth that helped drive a company-wide transformation.
Giving one team exclusive ownership and permission to architect our data created a single source of truth that helped drive a company-wide transformation. Once other teams realized they had access to clean, reliable data, they started to turn to the data more often. But that’s only part of the picture. People also began to see that they could use Analytics to quickly translate this data into stories that can transform our members’ online experience and drive business results.
Building a tech stack to generate insights
I joined Thrive Market in 2021 and quickly became the go-to person for Product Analytics. Thanks to the platform's intuitive interface, anyone in the company can generate easy-to-understand charts, tables, and dashboards to visualize complex data. No one had to mess with SQL queries or comb through spreadsheets to find the right information. Using analytics tools can be intimidating, but Amplitude makes analyzing and visualizing data accessible to people who aren’t data scientists.
When you're a growing company, you tend to accumulate a lot of data debt. I came in with fresh eyes and began to have discussions about how much of this data we wanted to keep, how much of it is bloat that we can eliminate, what data we should collect but don’t—and then try to set up the necessary infrastructure to reflect the results of these decisions.
We recently migrated to Snowflake for several reasons, including optimizing costs, reducing administrative duties, increasing scalability, and running concurrent, isolated workloads. As we moved all of our data sources to Snowflake, Analytics became more of our core analytics tool, which simplified our ability to do ad hoc analysis and made it more seamless. The two platforms sat side by side: Analytics handled the frontend data collection and visualization, and Snowflake contained all of our other data in a mesh of tables, data lakes, data warehouses, and data marts. It can be useful to look at these two platforms independently: Analytics is the primary tool for our product teams, and we use Snowflake to store a lot of our operational data, such as user transaction purchasing, registration, and membership data, as well as marketing information, member services data, operational data, and other pieces that we store in our backend. But we’ve moved all of our Analytics data into our Snowflake instance and are working on more ways to integrate these different data sources together as we mature even more.
Taking new risks with A/B testing
Thrive Market’s marketing and CRM teams use Analytics to monitor campaigns to determine how they increase site visits and conversions. Merchandising teams use it to sell banner ads, and business teams track various KPIs. Even operations team members leverage the platform to track picking, packing, and shipping orders at our warehouse.
It’s great to see this level of engagement throughout the organization, but the most prevalent use case is for A/B testing. We conduct a lot of our experimentation in Optimizely and use Analytics to visualize and interpret the results. We can build a chart or a dashboard to monitor any metric along the member journey—not just two or three metrics we set at the start. For example, all we used to do was see which members of the test groups went to Thrive Market’s landing page. With Analytics, we can drill down further and see whether they made it to a second page or a third, and we can keep going from there.
Typically, A/B testing costs money and ties up valuable resources. You must pay project managers, engineers, and analysts to spend days and weeks spinning up and running these tests, but a simple “yes” or “no” answer is no longer enough. Sure, you get the answer to a hypothesis, but so much valuable learning occurs beyond that initial test.
From A/B test results, it’s easy to dig into sales funnels and generate charts that track 20–30 variables and show where customers come in and drop off. For example, we can see whether iPhone users react the same way as Android and browser users.
One of the most significant testing wins happened when Thrive Market changed how we recruited new members. We’ve always had a team dedicated to membership conversion, and we are constantly experimenting with our membership funnel—everything from the landing page to a member deciding to pay for an annual membership. In Q1 of 2023, we ran close to 50 experiments on this funnel, and we used Analytics dashboards to see which variant drives higher membership conversion and what part of the funnel it increases. These answers help us understand what the driver was and then develop new ideas for new experiments.
Over the years, we have run various promotions to get members through the door and only then invited them to become Thrive Market members. One previous strategy was asking people to sign up for a 30-day trial in hopes they would then subscribe. The theory—and the logical assumption—was that they would try it, like it, and sign up. That tactic resulted in many conversions, but many users never placed an order, and some even canceled.
We decided to take a bigger risk. What if we asked people to subscribe first? What if you had to pay $60 to browse the products on Thrive Market’s website? At first, it didn’t make much sense—why would anybody do that? Still, we tested. Sure enough, conversions increased, and retention improved, too. Setting up tests in Amplitude is so fast and easy that anyone can validate even the wildest scenarios with a few clicks, enabling the team to iterate and enact change rapidly.
Removing friction
Another win came from taking a closer look at product discovery. When a member adds a product to their cart, we used to show them some product recommendations—sort of a, "Hey, you added a product to your cart, consider one of these other products." But we started to hear from the Member Services team that members were writing to complain about that feature being a nuisance. We decided to run a test and see what would happen if we suppressed that feature. They turned to Analytics to determine how many members were affected by the pop-up and understand the potential negative impact of suppressing it.
Next, we decided on the metrics we would use to evaluate the experiment: an add-to-cart rate and an add-to-carts-per-customer metric. We wanted to know, for all of the members who we are suppressing it versus the ones who were not, how many add-to-carts are we losing or gaining? We found that suppressing the feature removed a lot of friction for members, resulting in around a 2% increase in total add-to-carts. It sounds small, but it was a statistically significant improvement for us, which is what we were looking for.
In this case and others, Analytics makes it easy for teams to demonstrate our learnings to Thrive Market’s executive team. We can show them the impact of users moving through funnels regarding engagement, conversion, and spending lift.
More than 75% of how we use Analytics is working within funnel analysis charts to track conversion rates and evaluate the parts of the funnel where customers are dropping out. Whether it’s our membership funnel, add-to-cart funnel, or order-conversion funnel, we're constantly using Analytics to understand user behavior and evaluate the best way to optimize our product for our members. That's very difficult to replicate in another type of business intelligence tool—partly because collecting the type of data necessary and performing that type of analysis is something that Analytics is well-built for.
We're constantly using Analytics to understand user behavior and evaluate the best way to optimize our product for our members. That's very difficult to replicate in another type of business intelligence tool—partly because collecting the type of data necessary and performing that type of analysis is something that Analytics is well-built for.
Approximately the remaining 10% of our use case is monitoring business health and ensuring no data regressions. These seem like basic features, but they have transformed Thrive Market’s business.
An improved member experience
Analytics has given us tools to move faster and ask better questions. Instead of a developer spending days or weeks writing SQL queries and hand-coding a dashboard, an analyst can do it in half an hour or so. Amplitude has sped up the velocity and frequency of testing. There’s no longer a need for months of meetings to figure out what hypotheses to test, making our process much more cost-effective.
But more than just running experiments, we can easily and constantly iterate on our product and understand how those changes drive certain metrics or behaviors. It’s possible with other platforms, but making all of our data accessible at scale is something that Analytics does very well, and we’ve taken advantage of that for product feature launches.
Making all of the data accessible at scale is something that Analytics does very well, and we’ve taken advantage of that for product feature launches.
The real-time, granular visualizations from Analytics allow our teams to dive deep into sales funnels, follow the member’ journey, and optimize their online experience at the push of a button. In doing so, Thrive Market is creating value and driving the business to improve the unique online shopping experience for members.
