Empowering Data-driven Decisions at Criteo

The only way to improve your product is to understand how your customers use it. To do that, you have to collect and analyze accurate data and relevant feedback. The best way to obtain this information is through honest and open conversations with your customers. But how can you be sure that you’re asking the right questions and getting the right answers in return?

Like other online advertising platforms, Criteo relies on customer feedback to build better products for digital marketers. We offer a holistic suite of advertising solutions to retailers, brands, and publishers of all sizes. Our platform analyzes the behavior of 2.5 billion monthly online shoppers and uses AI to serve hyper-relevant ads to unique audiences who are more likely to engage with your brand.

In a crowded marketplace, we want to empower our clients to be marketing superheroes; we want them to leverage our platform to make their day-to-day jobs easier. But we didn’t always have a way to gauge their user experience with any degree of certainty.

Moving from Managed Services to Self-Service Dashboards

I’m a data scientist and the head of Criteo’s Interfaces and Measurements team. Before I came to Criteo, we were more of a managed services company. Our customers would call and ask us to take care of various aspects of their advertising campaigns. This approach ate up our time and resources and made it impossible to innovate.

That’s why we started building self-service analytics dashboards for our clients. By giving them more control over the data, they could take on more responsibility for everything from their retargeting campaigns to banner ads.

We follow a three-step process when building self-managed dashboards. First, we determine what metrics are most helpful to our users by examining how they interact with our platform. Next, we build an initial dashboard and roll it out to a dozen customers, who trial the new tool for a month or two. After the initial round of feedback, we spend a few more weeks tweaking the dashboards before making them available to all our users.

A New Way to Weigh Client Feedback

When we first began to build these dashboards, we followed our trials with an analytics training module and a phone call asking users to describe their experience. They often told us they were confused and couldn’t understand how to use a new feature, despite having gone through a training session. We couldn’t be sure our customers had even tried the features they claimed to find confusing. Sometimes, it was apparent that they’d only spent a few minutes with the product just before the call.

Even though we asked for critical feedback on new features, we found that some testers were afraid of being too negative if they shared anything but favorable feedback. It was clear that we needed to track their use while they tested a new dashboard to give context to their evaluation.

Move away from anecdotal evidence and start using data to drive decisions. Click To Tweet

Around the same time that I came to Criteo, we onboarded Segment to collect, clean, and control customer data, and Amplitude to analyze how our clients use the platform. Amplitude empowered us to move away from anecdotal evidence and start using data to drive decisions about new platform features.

The first thing our new system helped us do was prioritize client feedback. We can now see which clients used the dashboards frequently, so we know their comments and concerns are very valuable. If we see someone hasn’t spent much time on the dashboards and yet they tell us they don’t like how it looks or they don’t find a feature useful, then we don’t weigh that feedback as heavily. Amplitude helped us identify the most genuine feedback and focus on the real issues.

Testing and Improving Customer Workflows

We then worked to build and refine the seven automated workflows that form the core of our self-service platform, including campaign, coupon, and content creation.

We used Amplitude to determine whether our customers could complete a given workflow, such as creating a banner ad or assembling a retargeting campaign.

Amplitude shows us the percentage of customers who follow a workflow through to the end, and we can monitor where people get stuck along the way. If we see that 90% of our users complete a workflow, we know that it works pretty well. On the other hand, if an individual workflow has only 30% completions, we know that we need to make significant changes.

Amplitude’s detailed analytics allow us to pinpoint our customers’ pain points and retool our workflows accordingly. In just a few months, our campaign creation workflow completion rate went up more than 3x. While we still have a ways to go, we know we’d still be in the dark about what was going wrong if we had continued to rely on anecdotal evidence.

Amplitude also helped improve the way we roll out changes to our product. Our product data scientists can add and subtract features and functionalities and then run before/after tests to gauge our users’ reactions. If our customers embrace a feature, we keep it. But if we see adoption is low, it’s easy to try something else. We can also prioritize workflows that need improvement and forecast future additions to our platform based on current customer behavior.

Transforming Criteo’s Internal Data Governance

So far, we’ve discussed how Amplitude’s detailed insights led to external-facing changes to our platform and customer workflows. But using Amplitude also prompted a transformation of Criteo’s internal data governance practices.

Our team has grown over the years. In the beginning, we had one analyst and one developer using Amplitude, and now we have 60 internal users, including new hires and product managers. As we grew, we had to change how we accessed and spoke about data.

We didn’t have proper documentation, so my colleagues, analysts Eve-Anne Pagani and Myriam Klikel took ownership of the issue and created comprehensive user guides. We also opened a Slack channel where users can ask questions, request support, and participate in our bi-weekly hour-long open discussions.

We have now gathered our taxonomy, data tracking, and event conventions in a central repository accessible to all Amplitude users on our dev, project, and analytics teams. Everyone follows the same guidelines, so there’s no confusion or unknowns around how various teams use Amplitude and what we mean when we talk about the data.

A Culture of Self-Reliance and Innovation

Amplitude makes data accessible to people who aren’t data scientists. Its charts and visualizations present complex information in an easy-to-understand format. Our product managers can generate reports that track their goals without asking our data analysts to provide and interpret their results. Our product managers are free to focus on our customers’ campaigns while our data analysts can continue to provide support and build new features.

Data should be accessible to people who aren't data scientists. Click To Tweet

Amplitude has created a culture of self-reliance at Criteo. It has empowered us to move away from a managed services business model and into a model that offers self-serve online marketing tools to our customers—while still offering support from our product managers as needed. It has also strengthened our internal teams and processes by giving us the tools to track our progress, build better workflows, and collaborate.

We started using Amplitude to gather real-time insights that would help us make meaningful improvements to our products. Now, Criteo empowers our people to make data-driven decisions, setting the stage for future growth and innovation.