Run Better Growth Experiments with Qualitative and Quantitative Data

Dan Stephen

Director of Partnerships

people reacted
6 -minute Read,

Posted on August 10, 2021

Build a data-driven foundation that gives a complete view of your audience so you can make the best growth marketing decisions.

Growth experiments to get qualitative and quantitative data.

Growth marketing isn’t about quick wins; it’s about building a data-driven foundation that enables you to scale your company indefinitely. At that foundation are your customers—and if you want your foundation to be strong, you need to invest first and foremost in satisfying your customers.

To successfully grow your company, you need to understand what drives your customers to take actions (or not take actions) on your site. This requires a complete view of your customers, their decision-making process, and their journey. You can only get this complete view if you have quantitative and qualitative data. Quantitative data stems from analytics and provides the “what” (the actions they’re taking), where qualitative data comes from user research and provides the “why” (their reason for taking those actions).

“It’s really important to have both quantitative and qualitative metrics at every part of the funnel and look at them simultaneously to make sure you’re not just improving business metrics, but you’re improving customer satisfaction.” – Nikunj Kothari, Growth Product Lead, Opendoor

Combining rich product analytics from Amplitude with continuous research from a tool like UserLeap is the best way to get a complete picture of your sales funnel so you can make the best growth marketing decisions.

Use product analytics to identify customer problems

Product analytics helps identify customer issues by tracking and revealing quantitative data. Quantitative data, like retention rates, bounce rates, and sign-up rates, can indicate problems in your marketing efforts if there are inordinately high or low occurrences of these rates.

Quantitative data tells you what the problem is, whether that’s low retention or high bounce rates. Without quantitative data, any idea you have about what your marketing problem might be is just a guess.

To get quantitative data, you need to run product analytics—the kind Amplitude provides.

Identify the problem

To properly utilize product analytics, you must first analyze your funnel at a high level with Amplitude to find improvement opportunities like overall customer lifetime value and top-performing acquisition campaigns. This high-level information highlights glaring red flags such as sharp drop-off points with customers on certain days or underperforming campaigns, information that is valuable to both growth marketers and product teams.

Let’s say, when looking at your funnel, you’ve found user count spiked following a campaign but fell shortly after. By the month’s end, you had a handful more users than the month before, but nothing like the beginning of the month. Upon closer inspection of your n-day retention rate, you notice many of your new sign-ups left by day five. Giving you a specific problem to fix: people dropping off after five days with your app.

Get a closer look at your audience

Next, go beyond typical analytics—page traffic and bounce rate—and look at cohorts, or audience segments. Unlike general traffic information, cohorts can be broken into behavioral groups. Making cohort information more meaningful than general traffic, as the cohorts are tied to real-life behavior and actions.

Your high-level analysis showed a sharp customer drop-off on day five of using your product, so you dig into cohorts and see who is actually dropping off. After looking at your n-day retention rate and cohorts who left by day five, you realize many of these customers weren’t using the social sharing feature in your app. Now your problem of people leaving after five days has a thread to pull at: driving the use of the social sharing feature.

Set conversion drivers for the problem

Once you’ve identified the potential fix for the problem you’re solving, set conversion drivers that counter or solve that problem. A conversion driver is an ideal action you want to monitor, like using social sharing in your app or creating a playlist. Conversion drivers vary from cohort to cohort and campaign to campaign. Think: What is it that I want this particular group to do?

You know users who use social sharing tend to stay longer, so social sharing is your conversion driver for this campaign. Use Amplitude to create a conversion driver for social sharing and then move on to hypothesizing how you might drive more customers to try this feature.

Hypothesize and A/B test

At this point, you know what it is you want to fix, and your conversion drivers are set. Now, you need to brainstorm ways to boost your conversion driver and test those hypotheses. Ask yourself: How can we make these conversions a reality?

If you want people to use the social sharing feature to see if these customers stick around for longer than five days you can A/B test various app layouts that emphasize the social feature, such as a popup tutorial that calls it out or run an ad campaign that showcases the perks of the social sharing feature. From there, closely monitor your conversion driver performance and see which changes have the biggest impact fine-tuning whatever works the best.

It’s worth noting that you need to capture as much data as possible while running these A/B tests — a failed test can provide as much useful data, if not more, than a successful test.

However, this quantitative data reveals only part of the picture. Qualitative data provides the finer details and allows you to ensure your quantitative-driven changes are made with as much information as possible.

Run user research to delve into the why behind the customer problem

After identifying the issues within your customer funnel, the next step to improving growth marketing is pinpointing why that’s the problem. And you can only identify why your problem exists with qualitative data—data provided through firsthand surveys.

Surveys asking your audience directly about a specific problem or growth experiment, like a newly changed call-to-action or button placement, give you a single source of truth: your customer. This takes away any doubt general analytics may leave and allows you to address the problem directly. But remember: quantitative data provides the information necessary to ask the right questions with qualitative research.

So if quantitative data indicated your problem was a low retention rate after day five of user sign-ups, then you could use qualitative data to help identify why users are leaving after five days.

One of the best ways to gather qualitative data is with microsurveys, a quick survey embedded into your site or app, UserLeap provides. From there, you can ask customers to provide feedback about a feature — again, removing any doubt left by your data.

Let’s return to the cohort of people not using social sharing. Using quantitative analytics, we uncovered the correlation between people leaving the app early on and not using the sharing feature. We even tried out various layouts and popups to try driving the use of that feature—all of this possible only because we have quantitative analytics from Amplitude.

Next, using UserLeap, we could utilize a microsurvey to ask people why they use—or don’t use—the social sharing feature. We could even ask people about the new placement of the feature or popup being used to promote it to determine why these changes were or weren’t meaningful. This allows you to get to the heart of the issue and make the feature more appealing to a wider audience, thus improving retention rates.

The usefulness of qualitative research goes beyond individual use cases like the one above. Continuous research can help you uncover your why while you’re still building the hypothesis, as well as helping you to test your hypothesis, allowing you to try out solutions to problems before they’re actually problems.

For example, you can use a sign-up barriers microsurvey to test out various landing pages before they’re even live to see what pages perform the best or worst, and why. Feedback from these micro surveys will help you build the most effective site or app possible before it goes live, so you are making the best first impression.

Qualitative Data Survey Results

Using UserLeap’s powerful, AI-driven text analysis, you can quickly sift through any surveys, whether they’re focused on sign-up barriers or improving a live feature. UserLeap’s text analysis will pull key terms, sentiment, and themes from the answers to help you find common ground or recurring issues. With this information, you can quickly make meaningful improvements that align with your customers’ wants and needs.

Create better growth experiments with quantitative and qualitative data

With both quantitative and qualitative data, you now have the answers to how, what, when, where, and why, enabling you to make informed decisions with your UX experiments, solve customer problems, and better your overall understanding of your audience.

When creating your hypothesis for experimentation, leverage the full breadth of data you now have: ideate and test solutions that meet the underlying customer problems vs. just addressing surface-level business metrics. Combining the powers of Amplitude and UserLeap make this process as painless and seamless as possible. But don’t take our word for it:

“At Opendoor, Amplitude and UserLeap are both an essential part of our PM tech stack. Amplitude shows us when and where we have issues, and UserLeap helps us quickly find out why. Without the qualitative insights provided by UserLeap, we would need to spend weeks deploying many A/B tests to solve our most pressing issues.” – Nikunj Kothari, Growth Product Lead, Opendoor

Whether your site or app is live or in early testing, your customers are your source of truth. Find that source with quantitative and qualitative data and create an app that makes the best first impression imaginable.


Learn how Opendoor sped up experimentation and drove product growth by combining qualitative and quantitative analytics with UserLeap and Amplitude. Watch the recorded webinar now

Dan Stephen

Dan works with the world's leading consulting and agency firms to help them bring product analytics capabilities to their clients digital transformations. Over the years Dan has built deep product and growth expertise in a range of industries including: Consumer Tech, Entertainment, eCommerce and Fintech. Dan is Amplitude's resident Aussie and Trivia Host.

More from Dan

Inside Amplitude