Run an agent experiment

An agent experiment compares two or more versions of an agent on real traffic: a new system prompt against the current one, a cheaper model against the default, a different routing rule, or a different tool strategy. Your flag or experimentation tool assigns each user, session, turn, or span to an arm. The AI SDK records that arm on every [Agent] event as [Agent] Experiments, and any Agent Analytics chart then splits by arm with no further instrumentation: cost, latency, tool errors, task completion, and the rest.

Amplitude doesn't assign arms. Keep Amplitude Experiment, Statsig, LaunchDarkly, or whatever flag client the application already calls, and hand its result to the SDK. The SDK never sends an exposure event; the flag tool keeps doing that.

This page shows the three shapes most agent experiments take, in one agent loop. For the property contract, limits, and the OpenTelemetry, HTTP, and warehouse paths, refer to Record experiment variants in Agent Analytics.

Prerequisites

• Node @amplitude/ai 0.25.0 or later, or Python amplitude-ai 1.22.0 or later. For setup, refer to Instrument an agent with the AI SDK.
• A flag or experiment in your flag tool, with the user randomized where the behavior changes. The examples use a flag named planner-prompt for the prompt, router for the per-turn router, and search-strategy for the tool.

Choose the unit before you write code

The unit on each experiment entry records what the flag tool randomized, so the readout compares like with like. Pick it from the behavior you're changing:

Mixing units inside one key confuses the readout. If a session-level flag also changes per turn, give the per-turn decision its own key.

Shape 1: a prompt or model experiment at session start

Evaluate the flag once, when the conversation starts, and pass the result on the session. Every event in the session, including the provider calls the SDK wraps, carries the arm.

If the flag runs in Amplitude Experiment, pass the fetchV2() or fetch_v2() result through the helper and name the experiment flags in keys. Configuration flags in the same result stay out of the charts.

import { AmplitudeAI, experimentsFromAmplitude } from '@amplitude/ai';
import { Experiment } from '@amplitude/experiment-node-server';

const ai = new AmplitudeAI({ apiKey: process.env.AMPLITUDE_API_KEY! });
const experiment = Experiment.initializeRemote(process.env.AMPLITUDE_DEPLOYMENT_KEY!);
const agent = ai.agent('support-bot');

export async function handleConversation(userId: string, sessionId: string) {
  const variants = await experiment.fetchV2({ user_id: userId });
  const prompt = variants['planner-prompt']?.value === 'concise' ? CONCISE_PROMPT : DETAILED_PROMPT;

  await agent
    .session({
      userId,
      sessionId,
      experiments: experimentsFromAmplitude(variants, { keys: ['planner-prompt'] }),
    })
    .run(async (s) => {
      // Wrapped provider calls, tools, and tracking calls in here carry planner-prompt:<arm>.
      await runTurns(s, userId, prompt);
    });
}

For any other flag tool, build the entry from the value it returned: experiments: [{ key: 'planner-prompt', variant, provider: 'launchdarkly' }]. For an OpenFeature provider, register AmplitudeAIHook once and every flag you list is recorded when the application evaluates it inside a session. Refer to Use the helpers for your flag tool.

Shape 2: a router experiment on every turn

A router decides per user message, so the arm can change inside one session. Evaluate the flag on each turn and record it with unit: 'turn' through addExperiments() (Node) or add_experiments() (Python). The call replaces the entry with the same key, so events from that turn on carry the new arm, and the previous turn's events keep theirs.

async function runTurns(s: Session, userId: string, prompt: string) {
  for await (const message of incomingMessages()) {
    const route = await flags.getVariant('router', { userId, turn: message.id });
    s.addExperiments([{ key: 'router', variant: route, unit: 'turn', provider: 'statsig' }]);

    s.trackUserMessage(message.text);
    const model = route === 'fast' ? 'gpt-4o-mini' : 'gpt-4o';
    const completion = await openai.chat.completions.create({
      model,
      messages: [{ role: 'system', content: prompt }, ...history, { role: 'user', content: message.text }],
    });
    // [Agent] AI Response from the wrapped client carries router:<route> for this turn.
  }
}

The session-level planner-prompt entry from shape 1 stays on every event. Both experiments appear in [Agent] Experiments, for example ["planner-prompt:concise", "router:fast"].

Shape 3: a tool strategy experiment on one span

When the experiment changes how a single operation runs, pass the arm on that tracking call only. A call-level entry applies to that event and overrides a session entry with the same key. Use unit: 'span'.

const strategy = await flags.getVariant('search-strategy', { userId });
const started = Date.now();
const hits = await search(query, strategy);

s.trackToolCall('search_docs', Date.now() - started, true, {
  input: { query, strategy },
  output: { hits: hits.length },
  experiments: [{ key: 'search-strategy', variant: strategy, unit: 'span' }],
});

The [Agent] Tool Call event carries all three arms. Other events in the turn carry only the session and turn entries.

Read the result

1. Open any chart built on [Agent] events: [Agent] Session Record for task completion and friction, [Agent] AI Response for cost and latency, [Agent] Tool Call for tool errors.
2. Group by [Agent] Experiments. Each key:variant value becomes its own series, so a session in planner-prompt:concise and router:fast counts in both.
3. Filter on [Agent] Experiments contains router: to keep one experiment in view, or on [Agent] Experiment Key when only one experiment ran.
4. Compare arms on the metric you changed the agent for, and on at least one guardrail: cost per session, latency, and Negative Feedback are the usual three.

Agent Analytics reports what each arm did. For a statistical verdict with exposure counts, use the analysis view of the flag tool that assigned the arms, for example Analysis view in Amplitude Experiment.

Check the instrumentation before you launch

• Run the agent once per arm and confirm every event in the session carries [Agent] Experiments. MockAmplitudeAI.summary() prints an Experiments fill line with the count of events that carry it; a partial fill means an event fired before the assignment was recorded.
• Confirm the flag tool still sends its own exposure event. The SDK records the arm on agent events only.
• Keep arm names stable for the life of the experiment. The chart groups on the exact key:variant string.
• If the application uses the Amplitude AI MCP server or AGENTS.md with a coding agent, scan_project lists the flag reads it found and validate_file warns when a helper is called without naming the experiment flags.