Track AI assistant conversations, token cost, latency, and feedback

To track your AI assistant's conversations, token usage, cost, latency, model, and user feedback in Amplitude, instrument it with Agent Analytics. Wrap your LLM client with the Amplitude AI SDK, run each conversation in a session, and send a score when a user rates a response. Amplitude groups the messages, tool calls, and responses into sessions tied to your users, then scores each session. For defaults with no customization, complete the Agent Analytics quickstart instead. If you're still evaluating, Analyze agent results shows what Agent Analytics produces.

What you can track

Each signal lands as an event property you can chart, filter, and group by. The AI SDK fills in all of them except feedback, which your app sends when the user rates a response.

For every property, refer to the Agent Analytics taxonomy. For how Amplitude prices model calls, refer to How cost is calculated.

Minimal setup with the AI SDK

This sequence covers all six signals for a Node or Python app that calls OpenAI. Other providers and runtimes follow the same pattern. Refer to Choose an instrumentation path.

1. Install the SDK: npm install @amplitude/ai @amplitude/analytics-node (Node) or python3 -m pip install amplitude-ai (Python).
2. Initialize it with your project API key and wrap your LLM client. The wrapped client records tokens, cost, latency, and model on every response.
3. Run each conversation inside a session with your user ID and a stable session ID.
4. When a user rates a response, send a user-feedback score that targets the response's message ID.
5. Flush before the process exits, then verify your instrumentation.

import { AmplitudeAI, OpenAI } from "@amplitude/ai";

const ai = new AmplitudeAI({ apiKey: process.env.AMPLITUDE_AI_API_KEY });
const openai = new OpenAI({
  amplitude: ai,
  apiKey: process.env.OPENAI_API_KEY,
});
const agent = ai.agent("support-bot");

await agent.session({ userId, sessionId }).run(async (s) => {
  s.trackUserMessage(message);
  return openai.chat.completions.create({ model: "gpt-4o", messages });
});

// Later, when the user clicks thumbs down on a response
ai.score({
  userId,
  sessionId,
  name: "user-feedback",
  value: 0,
  targetId: aiMessageId,
  targetType: "message",
  source: "user",
});

await ai.flush();

To let your AI coding agent add this instrumentation for you, or to place agents and flushes on a long-lived server, refer to Instrument your agent with the AI SDK. For thumbs, CSAT, and score naming, refer to Collect user feedback. For OpenTelemetry attribute names, refer to the OTLP reference.

How setup works

1. Pick the Amplitude project the data flows into, and get its project API key.
2. Decide your session ID, user ID, and privacy mode. Refer to Before you instrument.
3. Instrument your code with one path. Refer to Choose an instrumentation path.
4. Check your events. Refer to Verify.

Each turn of an instrumented session produces an [Agent] User Message, one [Agent] Tool Call per tool invocation, and an [Agent] AI Response. When the work ends, send [Agent] Session End. It's optional and recommended. If you omit it, Amplitude closes the session with its idle-timeout rules. After the session closes, Amplitude adds an [Agent] Session Record with the session's enrichment results. The timeline below shows this sequence. Click any event to inspect its shape.

Before you instrument

Decide these before you write code. Each one is hard to change after rollout.

• Session ID and user ID: The ID that represents one unit of work, and the stable user ID you already use for product analytics. Refer to Agent Analytics sessions and user identity.
• Privacy mode: Whether prompt and response text leaves your infrastructure. Refer to Choose a privacy mode.

Choose a privacy mode

Three modes control what leaves your infrastructure. full (the default) sends message content, metadata_only sends none, and customer_enriched sends none and carries quality labels you compute yourself. Every mode sends tokens, cost, latency, model names, and session grouping. For a full comparison and how to apply each mode on your path, refer to Agent Analytics privacy modes.

Choose an instrumentation path

Pick one path. Every path produces the same [Agent] events, so sessions, enrichment, and charts work the same way.

Connect with the SDK

Install the AI SDK for Node or Python, then let your AI coding agent instrument the app or wire it in by hand. Go to Instrument your agent with the AI SDK.

Send OpenTelemetry traces directly

Point your OTLP exporter or Collector at Amplitude's endpoint and set a conversation attribute on your spans. Go to Send OpenTelemetry traces to Agent Analytics.

Send agent events without the AI SDK

Emit [Agent] events with a standard Amplitude SDK or the HTTP API when the AI SDK doesn't fit your runtime. Go to Send agent events without the AI SDK.

Verify

Send one real interaction and check nine gates on its events in Live Events, then run the production checklist before rollout. Refer to Verify your Agent Analytics instrumentation. If something's missing, refer to Troubleshoot Agent Analytics setup.

Extend your setup

These steps are optional:

• Session timeouts: Keep sessions open longer than 30 minutes for jobs that wait on humans. Refer to Configure session timeouts.
• User feedback: Send thumbs up, thumbs down, and CSAT ratings as [Agent] Score events. Refer to Collect user feedback.
• Session Replay: Link each agent session to the recording of the visit it happened in. Refer to Link agent sessions to Session Replay.
• Multi-agent systems: Record delegation from a parent agent to child agents. Refer to Instrument multi-agent systems.
• Custom evaluators: Add your own model provider key to run evaluators and calibration runs. Refer to Set up custom evaluators.