--- title: Set up custom evaluators in Agent Analytics description: "Connect a model provider or an OpenAI-compatible gateway to run custom LLM evaluators and calibration runs in Agent Analytics." product: general lang: en last_updated: 2026-10-08 token_estimate: 1599 --- # Set up custom evaluators in Agent Analytics > For AI agents: a documentation index is available at [/docs/llms.txt](/docs/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`. Custom evaluators score your agent sessions against quality criteria you define. Calibration runs compare an evaluator's judgments against your labeled examples to tune its accuracy. LLM evaluators and their calibration runs use a model provider key you supply (bring your own key, or BYOK). You can connect a provider directly or use models behind an OpenAI-compatible gateway. Code evaluators and Amplitude's built-in signals don't need a key. Use custom evaluators when the built-in signals, such as task completion and response quality, don't capture a criterion specific to your product. Each active evaluator produces one `[Agent] Evaluator Result` event per closed session. Refer to [\[Agent\] Evaluator Result](https://amplitude.com/docs/amplitude-ai/agent-analytics/taxonomy#agent-evaluator-result). After you add a key, refer to [Create and calibrate evaluators](https://amplitude.com/docs/amplitude-ai/agent-analytics/evaluators) to write, calibrate, and activate an evaluator. ## Plan availability BYOK for LLMs, Calibration Runs, Custom Code-Based Evaluators, and Custom LLM Evaluators are included on every plan, with an add-on available on the Plus, Growth, and Enterprise plans. Go to [Amplitude pricing](https://amplitude.com/pricing) for plan details. _Availability reflects the latest Amplitude plans. If your organization is on an older plan, your access may differ. Check your plan under [Plans & Billing in your organization settings](/docs/admin/account-management/account-settings#organizational-settings), or contact your Amplitude account representative._ ## Add a model provider key You must be an organization admin to manage model provider keys. Add a key under _Settings > AI Controls > Model Providers_ for any of the supported providers: - OpenAI - Azure OpenAI - Anthropic - Google Gemini - Fireworks AI - Amazon Bedrock - Google Vertex AI - OpenAI-Compatible Gateway Amplitude encrypts keys at rest and doesn't show them again after you save them. Keep your own copy if you need to reference or rotate a key later. For more on AI settings, refer to [AI controls](https://amplitude.com/docs/amplitude-ai/ai-controls). ## Connect an OpenAI-compatible gateway Use a gateway when your organization manages model access through a service such as LiteLLM. Agent Analytics can discover the gateway's models, including custom model names, for use as evaluator judges. Gateway credentials support Agent Analytics custom LLM evaluators and their calibration runs only. Your gateway must accept OpenAI-compatible Chat Completions requests at `/chat/completions` relative to its base URL. The URL must be reachable from Amplitude's servers. 1. Go to _Settings > AI Controls > Model Providers_. 2. Find _OpenAI-Compatible Gateway_ and select **Connect Provider Key**. 3. Enter the gateway's key in **API key**. 4. Enter the API base URL in **Base URL**, such as `https://gateway.example.com/v1`. Include the gateway's API path, but leave off `/models` and `/chat/completions`. 5. If the gateway requires extra authentication headers, add them under **Custom headers (optional)**. Amplitude sends these headers alongside the API key. 6. Select **Save**. If you previously saved the gateway under _OpenAI_ with a custom base URL, save it under _OpenAI-Compatible Gateway_ to use Chat Completions. ### Choose a gateway model Open **Evaluators** in Agent Analytics, then create or edit a custom LLM evaluator. Choose a judge model from the model picker. Amplitude fetches the gateway's model list from `/models` relative to the base URL, using the saved API key and custom headers. For example, `https://gateway.example.com/v1` uses `https://gateway.example.com/v1/models`. If your model isn't in the list, use the **Gateway model ID** text field below the model picker in the evaluator editor. Enter the model name your gateway expects, such as `my-eval-model`. This also works when the gateway doesn't provide `/models`. The field shows an `openai:` prefix that Amplitude adds automatically. Enter only the gateway's model name, unless that name itself starts with `openai:`. Save the evaluator as a draft and [run a calibration](https://amplitude.com/docs/amplitude-ai/agent-analytics/evaluators#calibrate-against-your-reviews) to check that the model works before activation. Saving a gateway key alone doesn't confirm that a model can run. If your organization also has an Azure OpenAI credential for Agent Analytics, Azure takes precedence and the picker doesn't offer gateway models. Ask your organization admin to review the credential setup. A gateway takes precedence over a direct OpenAI key for OpenAI-routed evaluator calls. Gateway calls don't support web search. ### Example with LiteLLM In your [LiteLLM configuration](https://docs.litellm.ai/docs/proxy/configs), define a model alias for your evaluator. Replace `` with a model your provider account can use: ```yaml model_list: - model_name: my-eval-model litellm_params: model: openai/ api_key: os.environ/OPENAI_API_KEY ``` Deploy LiteLLM with this configuration and authentication enabled. Connect its API base URL and a LiteLLM key with access to `my-eval-model` in Amplitude. Keep the upstream provider key in LiteLLM. Choose `my-eval-model` from the model picker, or enter it in **Gateway model ID**. LiteLLM routes that alias to the configured upstream model. ## Troubleshoot a gateway connection | Symptom | What to check | | --- | --- | | The model list is empty or missing a model. | Check that the gateway serves `/models` and that the saved key can access the model. Amplitude caches discovery results, so changes might not appear immediately. Enter the exact name in **Gateway model ID** to use an unlisted model. | | The gateway returns `401` or `403`. | Check the API key, model permissions, and any required custom headers. | | The gateway returns `404` for a request. | Check the base URL and API path. Confirm that you saved the key under _OpenAI-Compatible Gateway_ and that the gateway supports `/chat/completions`. If the error names a missing model, check its exact model ID. |