Agent Analytics Overview

Agent Analytics measures the quality of your AI agents and connects it to the business outcomes you already track in Amplitude: conversion, retention, and revenue. Every agent event carries your product user_id and lives in the same project as your product events, so questions like "do users with successful agent sessions retain better?" are a single chart, not a data engineering project.

Observability tools tell you what your agent did. Agent Analytics tells you whether it worked for the user and what it did to your business.

What you can answer

• Quality: What share of sessions complete the user's task? Where does quality regress after a prompt or model change?
• Behavior: Do users who adopt the agent convert and retain better than those who don't? Which user segments struggle?
• Cost: What does each agent, model, and topic cost, and is the spend going to sessions that succeed?
• Diagnosis: For any failing session, what exactly happened, turn by turn, tool call by tool call?

Core concepts

Agent Analytics organizes data into a three-level hierarchy, with agents as a dimension across all of it:

• Session: one unit of work the user hands the agent (a conversation thread, a support ticket, a voice call, a background run). Identified by [Agent] Session ID, which your app supplies from the ID it already has.
• Turn: one exchange inside a session: user message, tool calls, AI response.
• Span: one operation inside a turn: an LLM call, a tool invocation, a retrieval step.
• Agent: the AI component doing the work. Every event carries [Agent] Agent ID. Agents can delegate to child agents; sessions record the root agent and the delegation depth, so multi-agent systems are first-class. An agent is a dimension on events, not a fourth level of the hierarchy.

How quality is measured

Agent Analytics measures quality in three complementary ways:

• Signals: Amplitude automatically evaluates every closed session against a built-in set of signals: task completion, response quality, user intent, session safety, user friction, negative feedback, and data quality. Results land on the session's [Agent] Session Record event, each with a written rationale. No setup required.
• Evaluators: Custom judges you define and calibrate in the Amplitude UI for the criteria that matter to your product: policy compliance, groundedness, tone, anything you can describe. Each active evaluator produces an [Agent] Evaluator Result event per session. You can also compute evaluations on your own infrastructure and send them in as Session Enrichment.
• Scores: Your users' and reviewers' voice: thumbs ratings, CSAT, or your own LLM-as-judge verdicts, sent from your application as [Agent] Score events, each one a discrete judgment about a specific message or session. Amplitude never generates Scores; they are always yours. Explicit user feedback overrides the detected negative feedback signal, so the user always gets the last word.

Scores and Session Enrichment sound similar but play different roles: a Score is one named verdict about one target ("this response got a thumbs down"), while Session Enrichment is the full set of session-level quality labels you compute yourself, standing in for Amplitude's signals when enrichment runs on your infrastructure instead of Amplitude's. Feedback is a Score; do-it-yourself signals are Enrichment.

Signals give you coverage on day one, Evaluators give you precision on your own criteria, and Scores keep both honest against real user judgment.

Instrumentation at a glance

Where it shows up in Amplitude

• Monitor dashboards for quality, cost, latency, and volume by agent.
• Sessions for browsing, filtering, and inspecting individual sessions turn by turn.
• Evaluators for building, calibrating, and running custom judges against curated datasets with ground truth labels.
• Everything else in Amplitude: agent events work in charts, funnels, cohorts, and Session Replay links like any other events. Build the funnel from first agent session to conversion, or the cohort of users with three failed sessions this month, directly.

Manage access with RBAC

• View Agent Analytics Objects: view sessions, evaluators, and related objects. A role needs this permission to use Agent Analytics at all.
• Manage Inactive Evals and Runs: create, update, and delete draft evaluators, and launch dry runs.
• Activate Evals: mark evaluators as active and archive live evaluators.

Next steps

• Quickstart: first events in about five minutes.
• Setup: plan production instrumentation, privacy modes, and session lifecycle.
• Analyze results: explore sessions, run evaluators, connect to product analytics.
• Taxonomy: the full event and property reference.