---
title: "Putting a Number on AI quality"
description: "The Economist used Amplitude Agent Analytics to move from sampled reviews to measurable AI quality, reaching 96.9% task success and cutting failures by 84%."
canonical: "https://amplitude.com/en-us/blog/putting-a-number-on-ai-quality"
---

# Putting a Number on AI quality

The Economist used Amplitude Agent Analytics to move from sampled reviews to measurable AI quality, reaching 96.9% task success and cutting failures by 84%.

Source: https://amplitude.com/en-us/blog/putting-a-number-on-ai-quality

---

[Fionn O'Raghallaigh](/blog/author/fionn-oraghallaigh)

[AI Group Product Manager, The Economist](/blog/author/fionn-oraghallaigh)

[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Famplitude.com%2Fblog%2F%2Fblog%2Fputting-a-number-on-ai-quality)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Famplitude.com%2Fblog%2F%2Fblog%2Fputting-a-number-on-ai-quality)[](https://twitter.com/intent/tweet?url=https%3A%2F%2Famplitude.com%2Fblog%2F%2Fblog%2Fputting-a-number-on-ai-quality\&text=Putting%20a%20number%20on%20AI%20quality)[](mailto:?subject=Checkout%20this%20Amplitude%20Article\&body=Check%20this%20out%3A%20https%3A%2F%2Famplitude.com%2Fblog%2F%2Fblog%2Fputting-a-number-on-ai-quality)

[The Economist Group](https://www.economistgroup.com/) is best known for its newspaper that has existed since 1843. In 1946, the [Economist Intelligence Unit](https://www.eiu.com/n/) was set up to answer questions Economist readers were asking. Today EIU helps businesses, financial firms and governments to understand how the world is changing and how that creates opportunities to be seized and risks to be managed. EIU’s team of analysts cover nearly every country in the world with analysis, forecasts and indicators to back decisions. They create a lot of content. And data. It can be hard to navigate.

Enter Lens. In March, we added the multi-turn AI research assistant Lens to [Viewpoint](https://www.eiu.com/n/solutions/viewpoint/), EIU’s website. It allows our analyst, strategist, and risk customers to, for example, ask about the fiscal outlook or compare political risk across the five biggest economies in South America and get an answer sourced entirely from EIU content.

Getting Lens to a good enough quality was hard work. We worked with our analysts on evaluations. We went back and forth until we were happy. By the time we launched, we knew Lens was good. But a feature in the wild is a different beast and pinpointing the areas to improve was the next challenge.

## From sessions to signal

Product analytics showed us what users did. It couldn't tell us whether the AI answer was any good. A user who asked three questions in a session might have been getting excellent answers and going deeper. Or they might have been rephrasing the same question because the first two answers were wrong. The data looked the same either way. We opened sessions and read them. But not fast enough to act on it. First impressions matter.

Lens sessions landed into [Amplitude Agent Analytics](https://amplitude.com/agent-analytics) with signals attached out of the box: intent classification, outcome tracking, quality scores. Getting there took one engineer and some back and forth, but once it was running, reading a trace stopped being archaeology. I could open a session, see each turn, see what the agent retrieved and how it responded, and see how it scored.

Before that, our data insights lead, our platform engineers and I were drawing conclusions from handfuls of sessions. A few examples, a hunch. Now we're looking at the same evidence across all of them.

## Filtering by behavior

Traces

9,240 sessions **312** of 9,240 · Power users

Power users×

Filter

Pull every support ticket tagged billingPower user

Draft a release note for v4.12New user

Compare YoY active-user growthDormant

Build a cohort of accounts that activatedPower user

Generate a weekly retention emailNew user

Why did session length drop 18%Dormant

Summarize churn risk by segmentDormant

Explain the drop in trial conversionsNew user

Find accounts stuck in onboardingDormant

Agent name

Cohort›

Has tool errors

Select cohort

Power users

New users

Dormant

Cohort definition

Power users

Include users who —

did **Ran an analysis** ≥ 5 times

in the last **14 days**

\&active **3 of the last 4 weeks**

\= **8,412** users · 3.4% of active Apply

##### The capability I lean on most is addictive. Pull every session matching a behavioral signal and drill in.

Early on, Agent Analytics' request complexity classification surfaced a concern about how Lens handled ambiguous questions. I filtered to those sessions, found two areas that worried me, and we now track both programmatically. That whole loop, from hunch to confirmed pattern to standing metric, took an afternoon.

It also catches things that look like problems and are not. Around half of our session outcomes were labelled "Clarification Requested," which initially read as a defect. Drilling in showed the agent was ending answers with a follow-up question, a deliberate design choice. But when we looked at what users did next, the pattern was clear enough. They weren't going deeper, they were going elsewhere. We tweaked the prompt. Without the ability to interrogate the label, we might have spent a sprint fixing behavior that was working as intended.

## A scorecard instead of a vibe

We all know AI outputs are inherently non-deterministic, which is a long way of saying that viewing a handful of sessions and calling it quality assurance doesn't scale. We wanted a number.

Amplitude's standard signals gave us a floor on day one. Custom evals let us go further, scoring sessions against our own definition of good, including rubrics built from ground-truth Q\&A pairs our analysts wrote.

By Amplitude's measure, Lens now holds a 96.9% task success rate, and weekly task failures fell 84% as we worked through the issues the data surfaced. Those gains came from ordinary engineering. The measurement is what tells us where to point it.

Evals alone tell you whether an answer was correct. Quality signals connected to real product usage tell you whether the product is getting better at its job.

## What comes next

Agent interactions in Amplitude arrive as decomposed events, the same shape as any other product event. That opens the door we care most about, connecting Lens quality to engagement across Viewpoint as a whole. Does a strong Lens experience deepen how teams use the platform?As we add structured data and chart visualisation to Lens, and as other parts of The Economist Group explore whether the architecture fits their own products, that connected view is how we will judge the work.

None of this is about watching individual users. It is about holding our AI products to the same standard of evidence we hold our analysis to. Our clients pay us for rigor. The tools we use to improve their experience should have some too.

About the author

Fionn O'Raghallaigh

AI Group Product Manager, The Economist

[More from Fionn](/blog/author/fionn-oraghallaigh)

Fionn leads AI product at the Economist Intelligence Unit. Before moving into product, he was a journalist. Knowing what makes an answer trustworthy, useful and worth someone's time turns out to be good preparation for building AI products.

[More from Fionn](/blog/author/fionn-oraghallaigh)

Topics

[AI](/blog/tag/artificial-intelligence)

[Agents](/blog/tag/agents)

[Amplitude Agent Analytics](/blog/tag/amplitude-agent-analytics)

[Amplitude Analytics](/blog/tag/amplitude-analytics)

[Media and Entertainment](/blog/tag/media-and-entertainment)

#### Recommended Reading

[Read ](/blog/holiday-checkout-friction)

[Product](/blog/holiday-checkout-friction)

###### [Find Your Holiday Checkout Friction Before Peak Traffic Does](/blog/holiday-checkout-friction)

[Oct 26, 2026](/blog/holiday-checkout-friction)

[8 min read](/blog/holiday-checkout-friction)

[Read ](/blog/amplitude-github-learning-loop)

[Product](/blog/amplitude-github-learning-loop)

###### [Amplitude + GitHub Agents: Closing the Product Learning Loop](/blog/amplitude-github-learning-loop)

[Oct 19, 2026](/blog/amplitude-github-learning-loop)

[4 min read](/blog/amplitude-github-learning-loop)

[Read ](/blog/lessons-from-consolidating-our-mcp-server)

[Insights](/blog/lessons-from-consolidating-our-mcp-server)

###### [Lessons from consolidating our MCP server](/blog/lessons-from-consolidating-our-mcp-server)

[Oct 6, 2026](/blog/lessons-from-consolidating-our-mcp-server)

[10 min read](/blog/lessons-from-consolidating-our-mcp-server)

[Read ](/blog/what-ai-engineers-do)

[Insights](/blog/what-ai-engineers-do)

###### [What AI engineers do in the age of agentic code](/blog/what-ai-engineers-do)

[Oct 1, 2026](/blog/what-ai-engineers-do)

[14 min read](/blog/what-ai-engineers-do)