---
title: Set up custom evaluators in Agent Analytics
description: Add your own model provider key (BYOK) so Agent Analytics can run custom evaluators and calibration runs against your quality criteria.
product: general
lang: en
token_estimate: 524
---
# Set up custom evaluators in Agent Analytics

> For AI agents: a documentation index is available at [/docs/llms.txt](/docs/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Custom evaluators score your agent sessions against quality criteria you define, and calibration runs compare an evaluator's judgments against your own labeled examples to tune its accuracy. Both make model calls with a model provider key you supply (bring your own key, or BYOK). Amplitude's built-in signals run without any setup and don't need a key.

Use custom evaluators when the built-in signals, such as task completion and response quality, don't capture a criterion specific to your product. Each active evaluator produces one `[Agent] Evaluator Result` event per closed session. Refer to [\[Agent\] Evaluator Result](https://amplitude.com/docs/amplitude-ai/agent-analytics/taxonomy#agent-evaluator-result).

## Plan availability

BYOK for LLMs, Calibration Runs, Custom Code-Based Evaluators, and Custom LLM Evaluators are included on every plan, with an add-on available on the Plus, Growth, and Enterprise plans. Go to [Amplitude pricing](https://amplitude.com/pricing) for plan details.

_Availability reflects the latest Amplitude plans. If your organization is on an older plan, your access may differ. Check your plan under [Plans & Billing in your organization settings](/docs/admin/account-management/account-settings#organizational-settings), or contact your Amplitude account representative._

## Add a model provider key

Add a key under _Settings > AI Controls > Model Providers_ for any of the supported providers:

- OpenAI
- Azure OpenAI
- Anthropic
- Google Gemini
- Fireworks AI
- Amazon Bedrock
- Google Vertex AI

Custom evaluators and calibration runs use this key to make model calls on your behalf. Amplitude encrypts keys at rest and doesn't show them again after you save them, so keep your own copy if you need to reference or rotate it later. For more on AI settings, refer to [AI controls](https://amplitude.com/docs/amplitude-ai/ai-controls).