Using Session Replay for Savvier User Analysis That’s 90% Faster

Insight/Action/Outcome: The data team hypothesized cultural differences were causing a lower conversion rate in a regional market. They used Amplitude Experiment to determine whether a change in wording would resonate better with the audience. It did—so much so that their CTA clickthrough rate increased 2X, increasing conversions by 20%.

When you have a complex product, you need to approach it through the lens of a product analyst. What parts of the product perform well? What parts don’t? How do customers want to use the product, and does their journey reflect that? You’re never going to find one solution that solves all your problems. But at travel-planning platform Evaneos, we found a platform with the combination of capabilities that comes pretty close.

A certified B Corp, Evaneos’s mission is to design better trips for travelers, the planet, and local communities. We are transforming the tourism industry by saying no to mass travel. Partnering with 600 local agencies in 160 destinations worldwide, we offer 8,000 customizable, 100% carbon-neutral travel itineraries to fit the individual traveler’s desires. Our products include our online platform and mobile apps, catering to different national markets in multiple languages. We also offer a range of trip categories. A tour for a solo traveler will be very different from a couple’s honeymoon, which is different again from a foodie tour for a group of friends or a kid-friendly safari.

Our products are complex, and our seven-person data team is responsible for understanding how our users interact with our products to improve click and conversion rates, determine the best pricing structure, or replicate something that’s working very well. We work alongside product managers, designers, engineering managers, and developers on various impact teams. I’m a data analyst, and my day-to-day tasks might involve insight discovery through user paths and segmentation, analysis via funnel charts, or experimentation—yet it always includes cross-team collaboration.

I may be a data analyst, but my colleagues and I all have a product analyst mindset. That’s why I’ve trained all Evaneos analysts to use Amplitude.

The analyst’s biggest challenge: coordinating across teams

Evaneos was gearing up to use Amplitude Analytics when I arrived at the company, which was great for me. They already had everyone’s buy-in, and since I had used Amplitude before, I could quickly start implementation and training others. Over the following 18 months, we adopted Amplitude Experiment and, most recently, Session Replay.

Our adoption of these solutions wasn’t to answer a specific question or impact a particular metric. Instead, our goal was to improve workflows and processes to better understand user behavior—and having a unified set of powerful capabilities in a single platform made that job easier.

With so many teams working together, coordination was one of our biggest challenges. It can be hard to track what pages and features are live and what modifications have been made, especially when using a mishmash of point solutions like we once did. Using a separate tool for A/B tests and then analyzing the results with SQL was a pain because we had to change platforms, and we didn’t find the same data everywhere despite using integrations and data pipelines. The process wasn’t good enough; it was time-consuming and inaccessible for anyone who wasn’t a data analyst. As a result, product teams struggled to understand the real impact of their features. Having all these tools within the Amplitude platform enables us to keep track of changes, communicate easily, and gather data in one place for effective analysis.

It replaced another solution we used for our B2C side, and we had nothing to track our B2B audience activities. Now, we have insights into both sides of our business. If we want to know what our users are doing after they see a certain page, it's very easy to retrace their steps. We don't have to set up anything, compared to before when we’d have to take time to add certain tags to target the users we want. Not everyone is data savvy, and adopting Session Replay means that everyone can get closer to our users with fewer steps standing in the way.

Having a combined toolset offers us greater flexibility. From a simple dashboard, people can see whatever key metrics they need, such as click and conversion rates, and dig deeper if necessary. That brings greater efficiency to understanding feature impacts—something we saw soon after adopting Analytics.

Saving time through 90% faster analysis

The most significant benefit of the Amplitude Platform for the data team has been time saved. We once ran an experiment that wasn’t getting the expected results, and we used Session Replay to detect a display that didn’t look quite right. That might sound like a minor cosmetic issue, but remember that it’s our business to inspire people with pictures of beautiful places. In a big-ticket industry like travel, anything less than inspiring runs the risk of a high exit rate.

It previously would’ve taken a day or two to notice something was off and fix it. But using Session Replay, we detected and fixed the issue within a couple of hours, improving the user experience.

We can also complete a funnel analysis in Analytics within a couple of minutes, where we used to spend 15–20 minutes dealing with SQL queries for each question. Some tasks, like tracking every step in the user journey, were so time-consuming with SQL that we wouldn’t do them at all. And sometimes, we just did qualitative analysis because we didn’t have the quantitative data necessary to confirm or reject what we heard in interviews. We do things now that we couldn’t do before, such as dedicating more time to deep.

The impact of real-time experiments

In another example, we noticed that the bookings conversion rate for one market was lower than for others. Looking at the data in Analytics, the team hypothesized this was due to cultural differences between markets, leading some groups to respond better than others. The project team tested this hypothesis with user interviews, followed by A/B testing on different CTAs to confirm what wording resonated best.

The results were very encouraging: We improved our clickthrough rate for a particular CTA by 2X in our target market, leading to a 20% increase in conversions. The sales process is long in the tourism industry, so it’s hard to attach revenue to that change, but these results have shifted how we think about our products. We see that an international travel company like Evaneos needs to think about not only our destinations but the origins of our travelers. Now, we are asking how to best optimize our product for each market where we operate.

What’s also significant from this example is the product team doesn’t have to wait on me to provide metrics like click rates, insights into user journeys, or updates on experiments because all this information is available within Amplitude. Our previous A/B testing tool required us to use SQL to analyze results. It took days to start an experiment, and it was a lot of extra work to understand the test’s progress and analyze the results.

Because Experiment and Analytics are natively integrated, they work together seamlessly, which makes the entire process easier, faster, and more accessible. Amplitude Experiment’s built-in duration calculator takes our product data from Analytics to provide the required sample size and estimate how long it will take to reach statistical significance. Then, we can monitor the test’s progress in real time. Experiment also enables us to place real-time alerts on various metrics. If something goes wrong, we receive a notification via email or Slack, and we can solve the problem immediately. We never had that alerting capability with our previous testing tool.

Finally, we’ve eliminated the extra step of analyzing results using SQL since the results flow to Analytics at the end of an experiment. It’s faster and easier than our previous A/B testing tool, and it’s cheaper, too. By consolidating multiple point solutions on Amplitude, we have had significant cost savings.

We always have at least one test running, and as we continue to do more in Experiment, we expect to use Session Replay heavily to compare what users are doing in one version versus the other—and continue to learn more about our B2B audience. We can even go beyond product teams to work with designers, who can use Session Replay to create better paths for our customers.

Insights accessible to everyone

Data accessibility is a top priority for us. I trained my data analyst colleagues on Amplitude, and I’m moving into training our product managers and product designers, too. In addition to reducing time spent on analysis and facilitating real-time monitoring, Amplitude enables more profound discovery. The data team used to spend our time answering simple questions. Now, we get better, more intriguing questions that require more detailed answers. For the data team, that’s a lot more fun.

The best thing about Amplitude is that it’s a complete platform for everyone, including data analysts, product teams, and technical teams. It has all the capabilities the data team needs to dig into detailed, complex analysis, which means we don’t have to ask as much of the technical team, especially when running experiments.

It’s also a very easy tool for beginners to pick up and use. That enables product teams to get self-service data and insights, empowering them to become more autonomous. Within two weeks of adopting Session Replay, our product teams became more savvy with their analysis. They want to create charts and learn how to extract more information from the data.

Amplitude enables us to discover, analyze, and experiment, and the fact that we can do it all in one platform is the best thing for everyone on the team. It’s enabled Evaneos to go from a traditional analytics structure to a more agile, data-driven approach.