AI Agent Evaluation Metrics: From Quality Scores to Retention

AI agent evaluation metrics are quantitative measures of how well an AI agent performs its job. They score whether the agent reached the right result, how much time and money it spent, how users reacted to the output, and whether the agent moved business results like retention, conversion, and revenue.

The four groups, with examples:

• Quality: task completion, tool call accuracy, faithfulness, trajectory, and safety.
• Efficiency: latency, tokens, steps, and cost per successful task.
• User behavior: rephrases, abandonment, negative feedback, escalations, and return usage.
• Business outcomes: agent-assisted conversion, retention, support deflection, and revenue per agent session.

LLM agent evaluation builds on standard AI evaluations but grades multi-step paths, not single answers.

Quality scores aren't enough because they grade the agent's output, not the user's response to it. An agent can pass its eval suite while users quietly give up.

Picture a hypothetical travel booking agent that returns accurate flight options on every test case. In production, users keep rephrasing to ask for cheaper options because the agent buries the lowest fare, and many abandon the booking. The eval dashboard stays green while bookings slip.

The user's reaction matters because it signals what they're likely to do next. Cheng and Jiang's 2020 study of 1,064 US chatbot users found that "user satisfaction positively affected both the continued use intention of chatbot services and customer loyalty." The study covers chatbots and shows association, not cause, but satisfied users were far more likely to say they'd keep using the service.

Outcome metrics have their own blind spot. Retention moves slowly, and a drop won't tell you which tool call broke. Treat quality metrics as your fast signal and user outcomes as the test of whether those scores matter.

Quality metrics show whether the agent got the task right, through the right steps, within the rules. They are the core of most LLM agent evaluation suites and cover task completion, tool use, factual grounding, trajectory, reliability, and safety.

• Task completion rate: the share of tasks the agent finishes correctly. Measure it with code checks against an expected end state, or an LLM judge when the answer is open-ended. Amplitude published a task-based evaluation of its Global Agent across four analytics question types: descriptive, diagnostic, predictive, and prescriptive.
• Tool call and argument accuracy: whether the agent picked the right tool and passed valid parameters. Check each call against a schema and the expected tool for that test case.
• Faithfulness and hallucination rate: the share of claims supported by retrieved sources. Score it with a judge model that compares the answer to the retrieved context.
• Trajectory quality: whether the steps were sensible and free of loops or redundant calls. Compare the trace to a reference path, or have a judge grade it.
• Reliability (pass^k): whether the agent succeeds every time, not once. Run each task several times and count only tasks that pass every run.
• Safety and policy adherence: the share of runs that stay within policy. Use red-team test sets, policy-specific judges, and human review for edge cases.

Reliability is the metric teams most often skip. In the 2024 τ-bench study from Sierra researchers, agents built on GPT-4o succeeded on fewer than half of simulated retail and airline tasks. Fewer than 25% of retail tasks succeeded across all eight repeated trials. Those are 2024-era models, but the gap between one-run success and consistent success still applies.

Efficiency metrics track how much time, compute, and money the agent spends to finish a task. The core set is end-to-end latency, token usage, number of tool calls or steps, and cost.

NVIDIA's technical blog on agent evaluation says to "track tokens, tool calls, and end-to-end latency per successful task." The formula looks like this: Cost per Successful Task = Total Agent Cost / Successful Tasks.

Consider two hypothetical support agents. One costs less per call but fails and retries; the other costs more per call but finishes in fewer steps. Per successful task, the second can win.

Latency also crosses into user behavior. A correct answer that arrives after the user has left counts as a failure to them.

User behavior metrics show what people do after the agent responds: rephrase the request, leave, flag the answer, ask for a human, or come back later. They're the closest real-time signal of whether users trusted the result, and they catch failures that eval suites miss.

• Rephrase rate: the share of turns where the user restates the same request. A spike often means the agent misread intent.
• Mid-conversation abandonment: sessions that end before the task completes.
• Negative feedback rate: thumbs-down ratings or flagged responses per conversation.
• Escalation rate: how often a conversation ends with a human handoff or a support ticket.
• Seven-day return rate: the share of agent users who come back to the agent within seven days.

These signals come from product events, not from the agent trace, so measuring them takes behavioral data alongside your traces. A trace shows the agent's steps. It can't show that the user opened a ticket an hour later.

Business outcome metrics tie an agent to retention, conversion, and revenue by comparing what agent users do over weeks and months with what a comparable group does. They're the slowest metrics to move and the ones that decide whether an agent gets more investment or gets switched off.

• Agent-assisted conversion: the conversion rate for sessions where the agent helped, compared with similar sessions without it.
• Retention of agent users versus a holdout: N-Day retention for each group over the same window.
• Support deflection: tickets avoided because the agent resolved the issue, confirmed by no follow-up ticket.
• Revenue per agent session: revenue tied to sessions that include an agent interaction.

Watch for selection bias. Users who try an agent may already be your most engaged customers. A holdout group that never sees the agent gives you a fairer comparison than a simple user versus non-user split.

With Agent Analytics, agent conversations become Amplitude events, so you can query them in the same funnels, cohorts, and retention analyses you already use.

You measure AI agent performance with three kinds of graders, run at two points: offline before release and online in production. Code-based checks handle anything with a verifiable answer, like a valid API call or a correct database state. LLM-as-a-judge scores open-ended outputs at scale, and human review calibrates the judges and covers edge cases.

Offline evaluation before you ship

Offline evaluation runs the agent against a fixed set of tasks with known expected results. Anthropic's guide to evals for AI agents suggests starting with 20 to 50 tasks rather than waiting for a perfect suite. Draw them from real failures and the requests your users send most often.

A regression caught here costs you a test run. The same regression caught in production costs you users. Rerunning evals on every prompt, tool, or model change keeps regressions out of production.

Online evaluation in production

Online evaluation scores real sessions after launch, because live traffic rarely matches your test set. Anthropic's engineering team notes that "production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures."

The strongest online method treats agent versions like any other product change. Roll a new prompt or model out to a share of users with Feature Experimentation, keep a holdout, and compare task completion, rephrase rate, and retention across variants. If the new version hurts a key metric, roll it back.

Track one metric from each group plus a North Star Metric that ties them together: task completion rate, cost per successful task, rephrase rate, retention of agent users, and weekly successful agent tasks.

The North Star works because it blends groups. A task only counts if the agent got it right and the user accepted it. That stops a team from chasing completion scores that users ignore.

Connecting agent quality to product outcomes takes eval scores and traces in the same place as your behavioral data. Then you can segment users by the quality of their agent experience and compare what they do next.

Attach a quality score to each conversation, build cohorts by quality level, and compare retention and conversion across them. Our guide to AI agent analytics covers the instrumentation.

Amplitude Agent Analytics ties agent traces and evals to conversion, retention, and revenue, so you can compare what users do after good and bad agent sessions. Amplitude's Dashboard Agent can monitor the dashboards that track those metrics and surface anomalies.

One honest caveat: dedicated LLM observability tools go deeper on span-level debugging for engineers. Many teams run one alongside a behavioral platform and send scores across.

Pick the five scorecard metrics, instrument the user events behind them, and run your next prompt or model change as an experiment. Amplitude Agent Analytics puts quality, behavior, and outcome metrics side by side.