How to Connect AI Evals to Your Product Retention Metrics

This blog was co-authored by Sandhya Hegde, Cofounder at Calibre, an applied AI research firm.

If you’ve shipped an agentic feature to your product in the last year, you’ve probably asked some version of this: Are agents making my product stickier, or just cannibalizing the parts that already work? Do users with high eval scores actually retain better? How do I understand the impact of agent improvements at the user and business level, not just inside a session?

We had those questions too, so we ran an analysis on Amplitude’s Global Agent for 20K+ users who were regular Amplitude veterans (effectively power users) before we launched Global Agent. We wanted to understand how positive agent sessions (based on eval scores) affected their relationship with Amplitude and find predictors of better long-term retention.

The core insight: Scoring vs retention

For established users of an AI product, eval scores and retention are basically orthogonal. Bad sessions don’t push them out. Good sessions don’t keep them in.

For new users of an AI product, that relationship is the opposite: Eval scores are highly predictive of adoption and future retention. The single sharpest value signal is whether the user had a positive experience and saved the agent’s output in the first session. Below are the three patterns that stood out in our data:

1. Our most retained users are also having the most negative agent experiences.

Pure correlation, not causation. Our power users push the agent harder than anyone else, hit dead ends constantly, and come back the next day with new queries anyway. They’re not retained because the agent fails them. They’re retained despite it because the agent is part of how they work, and friction is the cost of living on the jagged frontier.

2. A positive first-week experience was worth a 3x retention multiplier.

This was the sharpest signal. Users whose first week/session with Global Agent was strictly positive retained 3x better over long periods of time than users whose first week tripped a single failure flag, even when both groups were equally active everywhere else in Amplitude. Same product, same user quality, different first impression, and completely different downstream behavior.

3. Early positive agent experiences pulled overall Amplitude usage up too.

The agent isn’t a sandbox (though it technically runs in one). A great first session with Global Agent improved weekly retention to the rest of Amplitude’s platform, not just to the agent itself. Bad first sessions, on the other hand, didn’t push users off Amplitude. They just pushed them away from the agent specifically.

The implication: If your eval-cohort is showing strong retention and you’re feeling smug that low eval scores aren’t hurting retention, don’t. They probably are. You’re just not looking at the full picture.

The first clue: Tangled eval cohort retention curves

When we set out to correlate eval scores and retention, we started simple. We plotted weekly retention by eval cohort across all users based on session modes (refer to our agent analytics modes post for definitions of Clean Success, Graceful Recovery, Silent Fail, and Dead End). To our surprise, we got nothing. All four eval cohorts were within 5% of each other.

At first glance, there are two ways we can interpret this:

• Bad sessions are a great demand signal. Users kept trying because they really needed an answer (not because they were masochists... or maybe they are).
• The eval categorization and scoring don’t matter. It’s either incorrect or irrelevant.

It seemed dangerous to embrace either. They would easily become great excuses to not care about the quality of our agent’s output in each and every session.

So we went back to the drawing board of user psychology. What makes users love/hate an AI product? Change their old workflows and become sticky with a new surface? Give it a chance to work? Was it the agent’s sheer power to be able to handle complex tasks or the speed with which it automated simple ones?

We wanted to find the strongest signal that made a difference.

The data immediately revealed a story when we split the question into two cohorts (established users and new users of an AI product) and asked each question separately. The tangled chart becomes clear, and the answers are nearly opposite.

The signal: First-time UX eval score predicts long-term retention

At this point, we started focusing on first-time Global Agent users only. Note: These were all already power users of Amplitude.

We took Q1 2026 first-time Global Agent users and split them by first-session outcome:

• positive (clean success flags) vs.
• negative (any failure flag tripped, including thumbs-down feedback)

The retention curves separate immediately and stay separated week over week.

We then layered in the strictest “user got real value” signal in the data: whether they saved the agent’s output (a chart, cohort, dashboard, etc.) and committed it to their workspace.

That’s a 3.57x gradient at Week 1, holding at 3x by Week 4. Monotonic, persistent, and specific to Global Agent.

What this means for your agents

The first session is the whole adoption decision. New users meeting your agent are making a binary call: Is this useful enough to come back to? Value moments like save decisions happen within an hour of the first session, and what happens in that hour determines the next four weeks.

Eval and analytics need to share a schema. None of this analysis would have been possible without session-level eval flags fired into the same event stream as product actions, with consistent property naming. Cross-cutting queries like “users with a clean session AND a save AND a return message in week 2” only work if all three events live in the same place, scoped to the same user.