Analyze agent results

Once events are flowing, Agent Analytics gives you three ways in: dashboards for the big picture, sessions for individual diagnosis, and evaluators for measuring the criteria you define. Everything also works in standard Amplitude charts, funnels, and cohorts.

Monitor

Open Monitor from the Agent Analytics left navigation. Monitor summarizes agent health across the selected project and time range. The page charts agent activity and session volume, session quality from the always-on signals, and cost and latency trends. Use these charts to spot a shift before you open individual sessions, and apply the agent filter to focus on one agent or compare agents.

Explore sessions

The Sessions view lists every agent session with its signal results, cost, turns, and duration. Filter by agent, by any signal outcome (for example, sessions where Task Completed is false), by user, by time range, or by whether you've manually reviewed a session.

Opening a session shows the full transcript-level detail including every turn with its user message, tool calls with inputs and outputs, AI responses with model, tokens, cost, latency, and the session's signal results with their rationales. What's visible depends on your privacy mode. For example: in metadata_only mode, the structure and metrics are visible but not the content. Clicking the Info tab displays the signals. Clicking the Review tab on a session displays customer evaluators along with manual tags and comments.

As you inspect sessions, you can mark the ones you've reviewed. The Manually Reviewed filter can then narrow the list to reviewed or not-reviewed sessions. A reviewer can work through a backlog without re-reading sessions they've already confirmed. Reviewed sessions and the notes you leave on them feed the datasets you use to calibrate evaluators.

From a session you can jump to the user's other sessions, their Session Replay (when linked), or to their full event stream in User Lookup.

Topics

The Topics page groups your closed sessions by what they were about, so you can see the main things users bring to your agent without defining categories yourself. Amplitude reads each session's content, clusters similar sessions together, and gives each cluster an auto-generated label. Clustering runs periodically, so it fills in as sessions arrive (usually within a day or two) and keeps updating.

Amplitude discovers topics project-wide, building one shared topic map across every agent in the project. This means that the Topics page doesn't filter by a single agent. Every session belongs to both a topic and an agent. To view one agent's topics, filter the Sessions view by that agent and a topic. If you need fully separate topic maps per agent, instrument each agent into its own project.

The page has two parts:

• Topic clusters (the map). A scatter plot where each point is a session, and sessions about similar things sit closer together, forming clusters. Each cluster is a labeled topic, with a ranked list alongside showing how large each one is. Zoom into a cluster to focus it, or click a point to open that session. It's a bird's-eye view of what people bring to your agent and how big each theme is.
• Scorecard. A per-topic breakdown of how many sessions fall in each topic and how they score on the quality signals, so you can see which themes are doing well and which are struggling.

How quality is measured

Agent Analytics measures quality in three layers. All three are queryable in any Amplitude chart.

Signals: automatic, on every session

After a session closes, Amplitude evaluates it against the built-in signals: task completion, response quality, user intent, session safety, user friction, negative feedback, and data quality. Results land as properties on the session's [Agent] Session Record event, each with a written rationale and, where applicable, evidence quotes.

To trend a signal, chart [Agent] Session Record grouped by the signal property, for example the share of records where [Agent] Task Completed is true. To build a cohort, use the same property: users with three or more sessions where Task Completed is false in the last 30 days.

Task completion and response quality are booleans. User intent and session safety are classifications. Friction, negative feedback, and data quality are booleans with companion properties listing the specific patterns detected.

Evaluators: your criteria, calibrated

Custom evaluators are checks you define in the Amplitude UI for criteria specific to your product, and they come in two kinds. LLM evaluators judge each session against a rubric you write in plain language: policy compliance, groundedness against your knowledge base, tone, refund-eligibility accuracy, anything you can describe. Code evaluators apply deterministic pattern rules with no LLM, so they're fast, exact, and cheap. Either kind has an output type: binary (detected or not), classification (a label from your set), or score (a numeric value).

The workflow:

1. Curate a dataset. Collect up to 100 representative sessions. Amplitude suggests filtering by similar sessions to create a dataset or selecting the auto-create dataset options that are based on common filters.
2. Label ground truth. Mark the expected outcome on each dataset session. Ground truth is what calibration measures against.
3. Draft and dry-run. Write the evaluator prompt, pick the judge model, and run it against the dataset. Compare results to ground truth, then iterate on the prompt until agreement is high (around 90% is a good bar). Two practices make evaluators reliable: prefer binary yes/no questions over numeric scales (LLM judges are inconsistent on ranges), and start with a small, fast model, since well-scoped binary checks usually calibrate at a fraction of the cost of a frontier model. Evaluators run on your own model API key, including dry runs and backfills, so the model, provider, and spend are yours to control.
4. Activate. An active evaluator runs on every closed session going forward, producing one [Agent] Evaluator Result event per session with the result, the judge model, the evaluator version, and the cost of the evaluation. Session dimensions (agent, cost, turns, topic) are echoed onto each result, so you can segment evaluator outcomes directly.

Active evaluators score new sessions as they arrive. To score historical sessions, run the evaluator against a dataset. Note that custom eval-related events count against your Amplitude event volume, unlike core agent events, which are metered as agent sessions.

Scores: the user's voice

Scores are feedback your application sends: a thumbs control, a CSAT survey, your own offline LLM judge, or human review. They arrive as [Agent] Score events targeting a message or a session, with an evaluation source of user, ai, or reviewer. Amplitude never generates Score events.

Two things make Scores powerful:

• A Score named user-feedback overrides the detected negative feedback signal for its session. When you have real user feedback, it wins over inference.
• Scores share the project with everything else, so "sessions the model judged fine but the user thumbed down" is one chart: Session Records with Response Quality true, joined by session to Scores with value 0.

Datasets

A dataset is a curated set of up to 100 sessions with ground-truth labels that you use to calibrate and back-test evaluators before you activate them. Build one from the Sessions view by filtering to the sessions you care about and adding them, or start from a dataset Amplitude suggests for an agent. The Datasets page in the left navigation lists your datasets and how far you've reviewed each one.

Open a dataset to label ground truth on each session, add comments or tags, and track review progress. To work with the data outside Amplitude, export a dataset two ways:

• Summary export: the dataset's metadata and the ground-truth label for each session.
• Full export: the summary plus the full session detail for each session, the same content you see in a session's detail view.

Datasets are the input to evaluator calibration: draft an evaluator, dry-run it against a dataset, and compare its output to your ground-truth labels. Refer to Evaluators for that workflow and Runs for the log of every scoring run.

Runs

The Runs page is the log of every evaluator run. Each time an evaluator scores a set of sessions it creates a run, and this page lists them so you can track scoring as it happens and review what has already been scored.

There are two kinds of runs: dry runs, which you use to test and calibrate an evaluator, and one-off runs, which score whatever sessions you pick. For each run, you can see its status, which evaluators and agents it used, how many sessions it covered, what it cost, and when it started. You can narrow the list down by agent, evaluator, type, or status, click into any run for more detail, or start a new one from the Start Run dialog. The Start Run modal allows you to choose the sessions, the date range, or the dataset to run the evaluator on.

Connect to product analytics

This is the point of putting agent data in Amplitude. Some patterns to start with:

• Funnels: First Agent Session to your conversion event. Or segment an existing signup funnel by whether the user touched the agent.
• Cohorts: Users with repeated task failures, for targeting with Guides or measuring churn impact. Users with high agent engagement, for expansion analysis.
• Retention: Retention curves segmented by [Agent] Task Completed on the user's first session.
• Session Replay: When linked at instrumentation time, jump from any agent session to the recording of the user's visit.
• Cost accountability: Session Cost USD on the Session Record, grouped by Topic Summary or agent, tells you where spend goes and whether those sessions succeed. Refer to how cost is calculated for where the numbers come from and why the provider bill remains the financial source of truth.

Agent events are ordinary Amplitude events. This means anything you can do with product events, you can do with them, including exporting them and querying them alongside everything else.