Agent Analytics FAQ and troubleshooting

What is Agent Analytics?

What is Agent Analytics?

Agent Analytics is Amplitude's analytics product for AI agents. It captures every agent conversation and workflow (sessions, turns, traces, tool calls, token costs) and automatically scores each session on quality signals like task completion, helpfulness, user friction, and safety. Because everything lands as standard Amplitude events, you can connect agent quality directly to the product metrics you already track.

Can it connect agent quality to business outcomes like retention or conversion?

Yes, this is the core of the product. Evaluation results are emitted as standard Amplitude events, so you can build funnels, retention curves, and cohorts on them. For example, you can compare retention for users whose agent sessions completed successfully versus those who hit errors, or cohort users by the topics they ask about and see how each cohort converts. Most teams monitor their agents across several disconnected systems that a business user can't join together; Agent Analytics puts quality and outcomes in one place.

Is Agent Analytics only for chatbots?

No. It supports conversational agents, voice agents, background or autonomous "doer" agents, multimodal agents (including image inputs and outputs), and multi-agent systems. Multi-agent workflows are tracked with per-agent identifiers, and you can analyze cross-agent flows with Amplitude's journey charts.

How is it different from my observability and eval tools?

How does Agent Analytics fit alongside your existing tools (LangSmith, Langfuse, Braintrust, Datadog)?

You don't have to choose; most teams run all three layers, because they answer three different questions. Observability platforms tell you whether your agent is running: uptime, latency, errors. Offline eval tools tell you whether a change is safe to ship: synthetic test suites, regression testing before release. Agent Analytics tells you whether the agent is working: did it complete the user's task in production, was the user satisfied, and did the interaction drive the outcome you built the agent for, connected to the conversion and retention data you already have. Test changes with your eval stack, monitor infrastructure with your observability stack, and use Agent Analytics to learn what real sessions are doing to your business after release.

Why can't I just track agent events with regular product analytics?

Traditional engagement metrics break for agents: a long conversation can mean a delighted user or a deeply frustrated one, and identical usage patterns can represent opposite outcomes. Agent Analytics adds what product analytics can't see (conversation content, traces, and LLM-scored quality signals) and joins them with behavioral data.

Getting started and instrumentation

Do I have to use your SDK?

No. There are several paths, and they all produce the same event taxonomy:

• Amplitude AI SDK: lowest friction, full enrichment out of the box.
• HTTP API: send agent events directly from any stack.
• OpenTelemetry: already tracing with OTel? Route your GenAI spans to Amplitude as a second destination; no re-instrumentation of your app.
• Warehouse import: bring historical conversations from Databricks, BigQuery, or similar. Note that imported data must be mapped to the [Agent] event taxonomy first (event types, session and turn IDs, and the required properties per event); raw conversation tables can't be sent as-is.

If you use LangChain, the callback handler works with any model provider. For the full wire contract when emitting events yourself, refer to Send agent events without the AI SDK on the setup page.

Can I use the @amplitude/ai SDK in the browser? What about Lovable, Superblocks, or other AI app builders?

No. The AI SDK is Node.js and Python only; importing it in a browser bundle or an edge runtime (for example Cloudflare Workers) fails on Node-only modules. The supported path for browsers, edge runtimes, and AI app builders is to emit the [Agent] taxonomy directly with the standard @amplitude/analytics-browser SDK. The setup page carries the full contract and a paste-ready prompt for builder agents.

Should I build my own client-side idle timer to end sessions?

Almost never. There are two clocks, and only one is yours. Amplitude's clock always runs: if a session receives no events for 30 minutes (configurable, or -1 to disable), the server closes it and runs enrichment, and an explicit Session End from your app closes it immediately. If you also build your own short timer in the client (say, ending the session after two quiet minutes and rotating the session ID), you chop real conversations into fragments: a user who pauses to read a long answer comes back to a "new" session. Enrichment then judges half-conversations, task completion looks artificially low, and, since Agent Analytics is billed per session, one conversation bills as several. Close explicitly when the job genuinely ends, and let the 30-minute server clock catch abandonment. Only mirror an idle timer client-side if your product truly defines sessions that way, and keep it generous.

What is an agent session, and how is it different from an Amplitude session?

They are two different concepts, defined by two different properties:

• An agent session is defined by the [Agent] Session ID event property. It represents one job the user hands the agent, from start to finish: a conversation thread, a support ticket, a voice call, or a background run. You set it from an ID you already track (thread ID, ticket ID, call ID, job ID); everything sharing that ID, including sub-agent activity, rolls into the same agent session.
• A regular Amplitude session is defined by the standard Session ID property ($session_id). It's behavioral: the user's app or web visit, which powers Session Replay and product reports, and closes after inactivity.

The two are related but independent. A user might have three agent conversations in one product visit, or one agent workflow that runs long after they've left your app. Agent sessions don't interfere with your existing session definitions. To link them, forward the standard session ID to your backend and pass it on the agent session (browserSessionId), so agent events join the user's product session.

How do agent sessions end? Can I configure the timeout?

Sessions end one of two ways:

• Explicitly (recommended): call trackSessionEnd() (Node) or track_session_end() (Python) when the job finishes, such as a closed ticket or a completed run. Quality-signal evaluation runs immediately.
• By idle timeout: if you don't end the session, it closes automatically after 30 minutes of inactivity, measured from the last agent event received for the session. This is configurable per session with idleTimeoutMinutes (Node) or idle_timeout_minutes (Python); today, also include an idle_timeout_minutes key in the agent's context so the override reliably reaches the server (refer to where to set the override). Raise it for jobs with long natural gaps (for example, 240 for a support ticket worked over hours), or set it to -1 to disable the idle close entirely and keep the session open until you end it explicitly (with a 90-day backstop).

When the same user returns with a new goal, start a new session with a new session ID rather than continuing the old one.

How do I link agent sessions to my existing Amplitude users?

Send the same User ID (or Device ID) you use in your product analytics; that's what joins agent activity to the rest of the user's journey. Session IDs don't participate in identity resolution. Anonymous or pre-signup users are merged later through Amplitude's standard identity resolution once they identify. Amplitude recommends deciding your ID strategy on day one of instrumentation, since it's the most common setup issue.

How do I instrument a thumbs up / thumbs down control?

Refer to Instrument thumbs up / thumbs down feedback on the setup page. The short version: send an [Agent] Score event with the name user-feedback, value 1 or 0, targeting the AI message being rated, with source user. Using the name user-feedback also makes your explicit feedback override the detected negative feedback signal for that session.

Can I import historical conversations?

Yes, through the HTTP import API or warehouse import. Note that evaluators score sessions going forward; refer to the evaluators section below for how to score historical data.

Signals and evaluators

What gets scored automatically?

Every session is scored out of the box on seven built-in signals, with no setup required. Results land as properties on the session's [Agent] Session Record event:

• Task Completed: did the agent actually fulfill the user's request by the end of the session? Looks at successful tool output, explicit confirmation, and user acknowledgment, and credits recovery if early errors were followed by success. Ignores tone and formatting.
• Response Quality: were the agent's responses accurate, clear, and well-structured, independent of whether the task got done? Brief-but-accurate counts as helpful; judges the text, not the underlying data.
• Session Safety: classifies the session as legitimate usage vs. problematic: Normal (the default), Off Topic, Prompt Injection, Abuse, Probing, Unsafe Output (agent produced harmful content), or Data Leak (agent exposed PII, credentials, or internal info).
• User Intent: what type of request was the user making? Information Request, Analysis & Synthesis, Task Execution, Content Creation, Advice & Recommendation, or Off-Topic / Social, classified by the dominant intent across the session, not the greeting.
• User Friction: detects behavioral anti-patterns that suggest a rough experience: retry storms (the same request 3+ times), clarification loops, cascading tool failures, early abandonment, mid-conversation pivots, scope creep, context loss, and workarounds. A single reasonable clarifying question or a topic-change follow-up doesn't count.
• Has Negative Feedback: detects only explicit dissatisfaction voiced by the user: "that's wrong," frustration or hostility aimed at the agent, complaint-style corrections, giving up or demanding a human, or explicit low ratings. Deliberately excludes behavioral signals (those belong to User Friction) and normal iteration like rephrasing.
• Data Quality Issues: code-based (no LLM) detection of mechanical response problems: empty responses, errors surfaced to the user, truncated output, repeated content, and refusals.

The six LLM-based signals run as a single batched judge call per session, not seven separate calls, and each result comes with a label, a rationale, and a supporting evidence quote from the conversation so you can see why a session was flagged. Signals are directional indicators, not ground truth: the judge is instructed to be conservative and only flag issues clearly evident in the conversation. Signals become event properties, so you can chart and segment on them immediately.

What's the difference between signals and evaluators?

Signals are the fixed, built-in scores that run on every session automatically. Evaluators are custom checks you define for anything domain-specific ("did the agent quote the correct policy?", "did it stay in character?") using LLM-as-a-judge with your own criteria.

What is the recommended model to use for an evaluator?

Use a small, fast model and a binary (yes/no) question. In practice:

• Binary beats numeric. LLM judges are inconsistent on numeric ranges; a clear yes/no question calibrated against human labels is far more reliable. Built-in signals use binary judgments for the same reason.
• Small models are usually enough. For well-scoped binary checks, lightweight models (for example GPT-4o mini, Claude Haiku, Gemini Flash class) typically reach high agreement with human labels at a fraction of the cost, which matters when an evaluator runs on every production session. Reserve larger models for genuinely nuanced judgments, and validate with a dry run before activating either way.
• Don't debate the model choice; test it. Take a set of real sessions and have a human mark the expected result on each (the ground truth). Then dry-run the evaluator on those same sessions and count how often it agrees with the human labels. If agreement is around 90%, the evaluator is trustworthy enough to activate, whatever size the model is. If a small model reaches that bar, use the small model; if it falls short, improve the prompt or step up to a stronger model and measure again.

Custom evaluators run on your own model API key (bring your own key), so you control the model, the provider, and the spend. This applies to calibration runs as well, including dry runs and one-off runs over past sessions, not just live scoring.

Do evaluators run retroactively on past sessions?

Active evaluators score new sessions as they arrive. To score historical sessions, run the evaluator from the Runs page over specific sessions, a date range (sampling supported), or a dataset. Fully automatic retroactive scoring is on the roadmap.

How do I know I can trust the scores?

Calibrate before you activate: assemble a dataset of real sessions, dry-run the evaluator, label a sample with human ground truth, and compare. Activate when agreement is high; iterate on the prompt when it isn't. Evaluator versions are tracked so you can refine over time.

What kinds of evaluators can I create, and when should I use each?

Evaluators come in two kinds:

• LLM evaluators judge each session against a rubric you write in plain language. Use them for subjective or nuanced criteria that need reading comprehension: policy compliance, groundedness against a knowledge base, tone, or whether a decision like a refund was correct. They send conversation turns to a model, so they cost tokens per session and their judgment is not perfectly deterministic.
• Code evaluators apply deterministic pattern rules with no LLM. Use them for exact, mechanical checks that don't need judgment: matching a phrase or format, flagging a banned term, or classifying by keyword. They're fast, cheap, and repeatable. Three methods are available: regex (a yes/no pattern match), keyword classifier (label a session by which keyword patterns it matches), and pattern scorer (a score based on match density).

Either kind produces one of three output types:

• Binary: detected or not.
• Classification: a label from a set you define.
• Score: a numeric value.

For LLM evaluators, a binary output calibrated against human labels is usually the most reliable choice; reach for classification or a score only when you genuinely need a label or a number. Code evaluators map directly to an output type: regex is binary, keyword classifier is classification, and pattern scorer is a score.

Users barely click thumbs up/down. Is that a problem?

No. Explicit feedback is famously sparse and unreliable. Agent Analytics doesn't depend on it: built-in signals score every session, and implicit behavioral signals (copy, retry, abandonment, expressed frustration) are captured automatically. Explicit ratings, when you have them, are ingested too, and a user-feedback Score overrides the detected negative feedback signal for its session, so the user always gets the last word.

Pricing

How is Agent Analytics priced?

Agent Analytics is billed on agent sessions, not events, metered per month or per year depending on your contract. Every plan includes a free session allotment, with additional volume available as you scale. Contact your account team or refer to the pricing page for details.

Will agent events count against my Amplitude event volume?

Mostly no. Core agent events (user messages, AI responses, and tool calls) are metered separately as agent sessions and do not count against your existing event volume limits. The exception is custom eval-related events, which do count against your Amplitude event volume.

What about the cost of running evaluations?

The built-in quality signals are included in the base price. Custom LLM-as-a-judge evaluators run on your own model API key, so inference costs go to your provider at your rates, and you choose the model.

How does LLM cost tracking work?

Costs are computed from the token counts on your traces multiplied against a continuously maintained open-source model-price catalog (Pydantic's genai-prices project), so using standard model names matters. Heavily cached or batch-discounted workloads may diverge from your provider bill; treat provider billing as the financial source of truth and Agent Analytics cost data as the analytical view (per-agent, per-user, per-feature attribution your bill can't give you). Models the catalog can't price (nonstandard names, gateway aliases, fine-tuned ft: models) get no automatic cost; pass the cost explicitly for those.

Privacy and security

What happens to conversation content? Where does it go?

You control exactly what leaves your systems, at three levels:

• Metadata only (metadata_only): events and metrics, no message text at all.
• Enriched metadata (customer_enriched): structured labels and evaluation results you compute, no raw content.
• Full content (full): message content, with PII redaction configurable through the SDK. When enabled, emails, phone numbers, credit cards, SSNs, IP addresses, and base64-encoded image data are scrubbed client-side, before data leaves your environment. Phone and SSN detection is tuned for US formats; add custom regex patterns or plug in your own redaction (for example, Presidio) for international locales or anything domain-specific.

Stored content is governed by your organization's standard Amplitude security controls, and access is restricted by role-based permissions. Refer to the setup page for the property-level matrix of what each mode sends.

Does Amplitude train AI models on your data?

No. Your conversation data is used solely to deliver Agent Analytics to you, powering your sessions, signals, and evaluations. Amplitude does not use your data to train AI models.

What about GDPR / HIPAA / compliance?

Amplitude's standard compliance posture and DPAs apply to Agent Analytics data. For regulated content, the metadata-only and enriched-metadata modes let you get quality analytics without raw conversation text ever leaving your systems.

Can you limit who on your team sees conversation content?

Yes. Access to Agent Analytics, and to conversation content specifically, is controlled through Amplitude's Role-Based Access Control (RBAC) feature. Admins grant and manage these permissions at the role level.

What agents does it support?

Does it work with any model or framework?

Yes, the event schema is provider-agnostic. OpenAI, Anthropic, Google (including Vertex), Bedrock, open-source models, LangChain, Vercel AI SDK, custom stacks: if you can emit events, it's supported. Cost tracking works best with standard model identifiers.

What about third-party or hosted agents you don't control (support bots, platforms like Intercom Fin or Sierra)?

Supported through webhook/API instrumentation: forward the conversation data the platform exposes and you get session analytics, quality signals, and outcome attribution. You'll have less depth than with first-party instrumentation (token-level cost and internal tool calls typically aren't exposed by hosted platforms).

Voice agents?

Yes. Instrument server-side and forward the transcript/analysis webhooks your voice platform provides.

Multi-agent systems?

Yes. Each agent gets its own identifier and version; sub-agents that share a session ID roll up into one session, and journeys across agents are analyzable in standard Amplitude charts.

Working with the rest of Amplitude

Can I chart agent data in regular Amplitude charts?

Yes. Built-in signals land on the [Agent] Session Record event and custom evaluator results on [Agent] Evaluator Result, so everything works in the standard chart builder (segmentation, funnels, retention, dashboards) right alongside your product events.

Should agent data live in the same project as my product analytics?

Yes, agent data can live in the same project as your product analytics; that's the simplest way to analyze them together. If you prefer to keep agent data in a separate project, Portfolio is also supported, so identity resolves across both. Your CSM or solutions engineer can help you choose the right setup.

Troubleshooting

I instrumented, but no events are arriving

Work down this list; the first item is by far the most common cause.

1. Your LLM calls run outside a session context. This is the classic patch() trap: patched calls that fire outside an active session context are silently dropped, with no event and no error, and the LLM call itself succeeds normally, so nothing looks broken. This includes AMPLITUDE_AI_AUTO_PATCH=true, which patches at process start but establishes no session. It also catches background jobs, startup warm-up calls, and promises that escape the request scope even when you use the Express middleware. Fix: wrap calls in session.run(), use the request middleware for request-scoped work, or enter a context manually.
2. Events are buffered and never flushed. Call flush() on shutdown, and per invocation in serverless environments, or short-lived processes exit before delivery.
3. Wrong API key or project. The SDK uses your standard project API key (Settings, then Projects). Confirm you're watching Live Events in the same project the key belongs to.
4. Local check before Amplitude: construct the client with debug: true; every tracked event prints to stderr. If nothing prints, the problem is instrumentation (usually item 1). If events print but never arrive, the problem is delivery (items 2 and 3).

Events arrive, but Session Records never appear

Session Records and Evaluator Results are only produced for closed sessions.

• If you never end the session explicitly, it closes after the idle timeout: 30 minutes of inactivity by default. Enrichment appears after that.
• If your job has long natural gaps, a default timeout closes the session mid-job; raise the idle timeout when opening the session, or disable it with -1 and close explicitly.
• Check [Agent] Close Reason on the records you do have: a high share of timeout means your app isn't closing sessions explicitly.
• If enrichment is disabled for the agent, its sessions are closed on a schedule but produce no Session Record at all.
• A Session End that arrives after the timeout already closed the session is silently ignored, so a surprisingly high timeout share can mean your closes fire too late (for example, only on page close), not that users abandon.
• There is exactly one Session Record per session, and late events after any close are stored but never enriched, so a record that seems to be "missing the last turns" usually means those turns arrived after the close. If a session is missing its record entirely and its last activity is well past the timeout window, contact support with the session ID.

Verification reports failing gates, but I'm early in setup

Expected. The verification gates check the full instrumentation: real user ID, real session ID, agent ID per component. If you're running with placeholder IDs or the minimal setup, gates for identity intentionally fail. That's the ladder telling you what's left, not an error. Refer to the ID ladder in the quickstart.

I see metrics but no message content

Content properties ($llm_message.text, system prompt, tool payloads, score comments) are gated by your privacy mode. In metadata_only and customer_enriched, content is never sent by design; you still get tokens, cost, latency, models, and session grouping. In full mode with PII redaction on, redacted patterns appear as placeholders like [email] and [phone].

If you instrument without the AI SDK and messages don't render in the session thread view, check the property shape: $llm_message must be an object, { "text": "..." }. A plain string is silently ignored.

Events stopped arriving after I added session-end handling (browser)

If you called setTransport('beacon') anywhere (a common move inside a page-hide or session-end handler), the browser SDK applies it globally and permanently: every subsequent event fires fire-and-forget with no retry and drops silently. Remove the call and use the default transport.

Signal results look uninformative

Two common causes:

• Privacy mode. Signals run in every mode, but in metadata_only the content-dependent ones (task completion, response quality) have little to read and produce low-information results. Structural signals (data quality, cost, latency, behavioral friction) remain useful. If you need content-based quality measurement without sending content, compute it on your side and send Session Enrichment.
• Sessions closing mid-job. A timeout-closed half-session evaluates as an incomplete conversation. Fix the idle timeout first, then re-judge signal quality.

Where did [Agent] Topic Classification go?

It's deprecated and no longer fires. Topic and classification outputs now land on [Agent] Evaluator Result (classification output type) and in [Agent] Topic Summary on the Session Record. Migrate any charts built on the old event.

Session Cost USD is missing from some Session Records

Session cost is all-or-nothing by design: if any event in the session had token usage but no cost, the total is omitted rather than reported partially, because a partial number would silently understate spend. The usual cause is an unpriceable model somewhere in the session: a nonstandard model name, a gateway alias like openrouter/auto, or a fine-tuned ft: model. Fix the model identifier or pass the cost explicitly on those calls; sessions after the fix will carry totals. Also note that on SDK releases before the pricing fix, unpriceable models recorded $0 instead, which quietly undercounts rather than omitting. And embedding costs never contribute to session totals.

Properties appear on events where they don't belong

If you see, for example, [Agent] Provider or [Agent] Temperature on Tool Call or User Message events, the SDK did not put them there: each event builds its properties fresh, and those properties are only written by AI Response and Embedding tracking. The usual cause is caller-supplied eventProperties, in particular a process-global default set through ToolCallTracker.setAmplitude({ eventProperties }), which is applied to every tool event from then on. Audit what your instrumentation passes as eventProperties; for custom segmentation dimensions, prefer context, which maps to the registered [Agent] Context property.

How fast should things appear?

• SDK events in Live Events: typically under a minute after flush().
• Session Record and Evaluator Results: within roughly 15 to 20 minutes after the session closes (session processing is batched), where "closes" means your explicit Session End, or the idle timeout (default 30 minutes of inactivity) for sessions that never close explicitly.