Set up custom evaluators in Agent Analytics

Custom evaluators score your agent sessions against quality criteria you define. Calibration runs compare an evaluator's judgments against your labeled examples to tune its accuracy. LLM evaluators and their calibration runs use a model provider key you supply (bring your own key, or BYOK). You can connect a provider directly or use models behind an OpenAI-compatible gateway. Code evaluators and Amplitude's built-in signals don't need a key.

Use custom evaluators when the built-in signals, such as task completion and response quality, don't capture a criterion specific to your product. Each active evaluator produces one [Agent] Evaluator Result event per closed session. Refer to [Agent] Evaluator Result.

After you add a key, refer to Create and calibrate evaluators to write, calibrate, and activate an evaluator.

Add a model provider key

You must be an organization admin to manage model provider keys.

Add a key under Settings > AI Controls > Model Providers for any of the supported providers:

• OpenAI
• Azure OpenAI
• Anthropic
• Google Gemini
• Fireworks AI
• Amazon Bedrock
• Google Vertex AI
• OpenAI-Compatible Gateway

Amplitude encrypts keys at rest and doesn't show them again after you save them. Keep your own copy if you need to reference or rotate a key later. For more on AI settings, refer to AI controls.

Connect an OpenAI-compatible gateway

Use a gateway when your organization manages model access through a service such as LiteLLM. Agent Analytics can discover the gateway's models, including custom model names, for use as evaluator judges. Gateway credentials support Agent Analytics custom LLM evaluators and their calibration runs only.

Your gateway must accept OpenAI-compatible Chat Completions requests at /chat/completions relative to its base URL. The URL must be reachable from Amplitude's servers.

1. Go to Settings > AI Controls > Model Providers.
2. Find OpenAI-Compatible Gateway and select Connect Provider Key.
3. Enter the gateway's key in API key.
4. Enter the API base URL in Base URL, such as https://gateway.example.com/v1. Include the gateway's API path, but leave off /models and /chat/completions.
5. If the gateway requires extra authentication headers, add them under Custom headers (optional). Amplitude sends these headers alongside the API key.
6. Select Save.

If you previously saved the gateway under OpenAI with a custom base URL, save it under OpenAI-Compatible Gateway to use Chat Completions.

Choose a gateway model

Open Evaluators in Agent Analytics, then create or edit a custom LLM evaluator. Choose a judge model from the model picker.

Amplitude fetches the gateway's model list from /models relative to the base URL, using the saved API key and custom headers. For example, https://gateway.example.com/v1 uses https://gateway.example.com/v1/models.

If your model isn't in the list, use the Gateway model ID text field below the model picker in the evaluator editor. Enter the model name your gateway expects, such as my-eval-model. This also works when the gateway doesn't provide /models.

The field shows an openai: prefix that Amplitude adds automatically. Enter only the gateway's model name, unless that name itself starts with openai:.

Save the evaluator as a draft and run a calibration to check that the model works before activation. Saving a gateway key alone doesn't confirm that a model can run.

If your organization also has an Azure OpenAI credential for Agent Analytics, Azure takes precedence and the picker doesn't offer gateway models. Ask your organization admin to review the credential setup. A gateway takes precedence over a direct OpenAI key for OpenAI-routed evaluator calls. Gateway calls don't support web search.

Example with LiteLLM

In your LiteLLM configuration, define a model alias for your evaluator. Replace <your-upstream-model-id> with a model your provider account can use:

model_list:
  - model_name: my-eval-model
    litellm_params:
      model: openai/<your-upstream-model-id>
      api_key: os.environ/OPENAI_API_KEY

Deploy LiteLLM with this configuration and authentication enabled. Connect its API base URL and a LiteLLM key with access to my-eval-model in Amplitude. Keep the upstream provider key in LiteLLM.

Choose my-eval-model from the model picker, or enter it in Gateway model ID. LiteLLM routes that alias to the configured upstream model.

Troubleshoot a gateway connection