Creating a Self-Serve Data and Experimentation Culture for Better Decision-Making

When it comes to designing an app or website, simplicity is best. Getting users from points A to B in the fewest steps and with the cleanest interface is the mark of an outstanding experience. But even the most straightforward user journey generates a string of event data that needs to be sorted and interpreted. With the right tools, this raw data becomes a treasure trove of information that can take your business to the next level.

I’m the Senior Product Analyst at SmartRecruiters, a software supplier that enables companies to run their talent acquisition program like a high-powered marketing and sales machine. Our B2B talent acquisition suite makes hiring easy and helps businesses find and hire the people they need.

With the right tools, raw data becomes a treasure trove of information that can take your business to the next level.

A recent complement to our flagship B2B product is our B2C solution, Smartr. Smartr is a user-friendly online platform that matches jobseekers with over 4,000 employers. We have two million active users looking for work online, generating hundreds of millions of events.

Making sense of user journeys was impossible

When I joined the company in 2020, making sense of our users’ journeys was next to impossible. We were tapping directly into our event database and trying to visualize things using SQL and Tableau. We wrote queries for every event we analyzed and fed the results into Tableau. It was time-intensive, and we were jumping through hoops to make the data work because there were too many moving parts.

At the time, Smartr was in its infancy—and our team was new, too. As we added more people to work on Smartr, we were simultaneously looking at new tools that could help us improve. We wanted to analyze and visualize more events and build our reporting, but our visualization tools were unsuitable for event tracking analysis. I started looking for the perfect solution to help us better understand user behavior.

Expanding on the right platform

During some basic research into analytics tools, I discovered that the enterprise side of SmartRecruiters was using the free version of Amplitude Analytics. Since we already had it in-house, I decided to try it and signed up for some workshops with Amplitude’s account managers.

Even with the limited features of the free Starter package, it was apparent right away that Analytics was the best tool on the market for event tracking. It integrates seamlessly with our database, and its point-and-click interface, built-in visualization tools, and report templates let us translate raw data into easy-to-understand information. In less than a minute, we can ask and answer questions that once took half a day to formulate, extract, and interpret using SQL queries and Tableau.

I quickly concluded that we weren’t using Analytics to its full capacity and championed our move to one of the platform’s paid tiers so we could take advantage of its full power.

Monitoring new features in real time

As we ramped up its use, Analytics introduced real-time monitoring that lets us test a new feature the second we add it to our website or app. With an existing user ID, we can test a project in real time and see what events are firing without creating a test environment or waiting for users and customers to log in.

Real-time data makes teams and users happier because there are fewer stumbling blocks in development cycles and deciphering customer journeys.

Within minutes of launching a new feature, I can tell my engineers whether it’s firing the desired events, and they can troubleshoot and resolve the issue immediately. It’s a smarter way of working (pardon the pun), and my team and our users are happier because there are fewer stumbling blocks in our development cycles and deciphering customer journeys.

Taking experimentation to new heights

Analytics is powerful, and as we grew the Smartr team, we started looking at doing more with event data. The logical answer was to increase our A/B testing so we could try different versions of our website and app.

To buy a separate experimentation platform or build our own would easily cost a hundred thousand dollars a year, and then we’d have to find a way to integrate the new platform with our data and events.

Instead of looking at a separate solution, we turned to Amplitude Experiment, a robust experimentation and feature management solution that lets us trial functionalities and interface enhancement. Experiment integrates fully with Analytics tools while offering a complete experimentation workflow.

One of the biggest successes with Experiment has been the launch of our Smartr’s bookmark feature. When candidates find an interesting posting, they can flag it, continue their job search, and go back and apply later. We experimented with two new versions of the bookmark icon, a star and a heart, in addition to the existing bookmark icon as a control. and saw that users preferred a heart.

We ran the test by assigning 1/3 of our traffic to each of the variants. We were interested in testing the bookmarking rate, which is the conversion rate of the users being exposed to the different variants and how many of those bookmarked a job. The bookmarking rate for the heart button increased compared to the control option by 46.6% while the bookmarking rate for the star button only by 8.78%. We also wanted to test the application rate, which is the percentage of bookmarked jobs that the candidates ended up applying for. For the heart button, that rate increased by 58% compared to the control, while for the star button the improvement was just 23.4%.

Amplitude confirmed the statistical significance of these improvements after six days of running the experiment, so the decision to change the old bookmarking button to a heart was easy.

Of course, we then asked further questions. How would users react if we changed part of our onboarding experience or created an entirely new one? Would they engage with content more? Apply for more jobs? And how would this impact our KPIs? We found ourselves asking and answering more questions relevant to our audience.

A growing data culture

It’s exciting to see our data culture grow. In the last 12 months, we’ve seen a 950% increase in adoption. And rather than seeing an initial spike after hiring and dropping off, engagement remains high. It’s ushered in a self-service data model, where people feel more confident about the data they see and their ability to make decisions based on it.

I use Analytics daily to create new charts and present them to the team, leading by example. Everyone from product teams to designers and engineers can create charts and dig into the data they need, whenever they need it. While not everyone uses Analytics daily, fewer people come to me with questions or requests for queries, and the team can move faster.

Because of this growth, I was an Amplitude Pioneer Awards finalist in their Data Culture Award category. It feels great that our work is recognized externally as well as internally.

The flexibility to refine Smartr

Any analytics tool will provide insights to help make more data-informed decisions, but the flexibility of Analytics sets it apart. It makes data analysis so easy. Instead of taking time, effort, and resources to build what we need from scratch, everything is ready to use and modify.

My team has refined our product offering and fine-tuned our user experience. We’ve built a thriving community of job seekers leveraging our platform to make their next—or first—career move. We can create new events and instantly track and analyze them using the built-in dashboards. We’ve begun to make decisions based on data and focus on monitoring performance instead of writing queries and managing our analytics tools. It’s easy to see what features users need and follow their paths as they navigate from the homepage to other pages on our site.

When building a new product, you must respond quickly and ensure all event property details are correct. The combination of Analytics and Experiment ensures that the Smartr team continues to build a product that measures up to our enterprise counterpart.