# Amplitude Documentation — Experiment
# Generated: 2026-09-08T23:39:57Z
# Pages included: 67 of 94
# Source: https://amplitude.com/docs/llms.txt
# Product feed: llms-experiment.txt
> Feature and web experimentation, rollout workflows, experiment SDKs, and experiment APIs.
> Token budget (150,000 tokens) reached. Use Docs MCP or `/docs/api/content/<slug>` for pages not included here.
> For AI agents: a documentation index is available at [/docs/llms.txt](/docs/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.
================================================================================
# Feature Experiment
URL: https://amplitude.com/docs/feature-experiment
Updated: 2026-01-08
================================================================================

# Feature Experiment

Test, analyze, and ship product changes with confidence. Amplitude Experiment unites feature flags and experimentation so you can release features safely and measure their impact across web, mobile, and backend.

**Outcome:** [Roll out features without the risk](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts)

Release behind a flag, ramp to the audience you choose, and turn it off the moment something looks wrong.

**Outcome:** [Prove which change wins](https://amplitude.com/docs/feature-experiment/experiment-quick-start)

Run A/B, multivariate, and bandit tests so the data, not the loudest opinion, makes the call.

**Outcome:** [Test the website without an engineer](https://amplitude.com/docs/feature-experiment/feature-vs-web-experimentation)

Edit pages with a visual editor and ship the winning variant without a code deploy.

**Outcome:** [Run experiments anywhere users are](https://amplitude.com/docs/sdks/experiment-sdks)

Use Experiment SDKs to flag and test features across web, mobile, and backend.

**Outcome:** [Automate your release workflow](https://amplitude.com/docs/apis/experiment/experiment-management-api)

Manage flags, experiments, and deployments from CI, scripts, or your own tools with the Experiment APIs.

**Outcome:** [See Experiment in action](https://app.amplitude.com/experiment/demo/470648/config/69916/activity)

Click through a hands-on demo before you instrument your own product.

## Ship with feature flags

Use flags to control release scope, target audiences, and rollback paths before you measure impact.

- [Create a feature flag or experiment](https://amplitude.com/docs/feature-experiment/workflow/create) to define the change you want to ship.
- [Configure variants](https://amplitude.com/docs/feature-experiment/workflow/configure) to set treatment values, payloads, and rollout behavior.
- [Define the audience](https://amplitude.com/docs/feature-experiment/workflow/define-audience) to target users by properties, cohorts, or segments.
- [Manage rollouts](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts) to release gradually and reduce launch risk.

## Run trustworthy experiments

Build each experiment around clear goals, clean exposure tracking, and a decision workflow.

- [Define experiment goals](https://amplitude.com/docs/feature-experiment/workflow/define-goals) to choose primary and secondary metrics.
- [Track exposure](https://amplitude.com/docs/feature-experiment/track-exposure) so analysis uses the users who encountered the variant.
- [Estimate experiment duration](https://amplitude.com/docs/feature-experiment/workflow/experiment-estimate-duration) to set expectations before launch.
- [Make a decision](https://amplitude.com/docs/feature-experiment/workflow/make-decision-experiment) to ship, iterate, or stop the experiment.

### [Getting Started with Amplitude Experiment](https://academy.amplitude.com/path/getting-started-with-amplitude-experiment-learning-path)

Earn a badge when you learn the Amplitude Experiment workflow and understand how get set up for experiments that produce trustworthy results.

================================================================================
# Overview
URL: https://amplitude.com/docs/feature-experiment/overview
================================================================================

# Overview

Amplitude Experiment runs feature flag rollouts, A/B tests, and web experiments so you can validate product changes before shipping them to all users. Configure variants, allocate a percentage of traffic, pick the metrics to evaluate, and let Experiment measure the impact using either a sequential testing model or a T-test.

Amplitude Experiment supports three use cases:

- **Product experimentation**: Improve key KPIs by running experiments and A/B tests to onboard new users, reduce friction for checkout experiences, roll out new features, and more.
- **Progressive feature delivery:** Pre-plan and stage new features for beta testers, a percentage of your users, or specific target audiences.
- **Dynamic in-product experiences:** Deploy and adapt custom experiences at scale.

Experiment offers two categories:

- **Feature Experiments**: Use feature flags to display or hide functionality or A/B options from your customers.
- **Web Experiments**: Use a Web editor to make direct changes to your website.

## How feature flags and the Web editor differ

Feature experimentation uses feature flags to create your experimental variants. Flags are switches that let you modify your product's experience without changing code. Use flags to set up experiments in your product or to stage and roll out new features to your users. Your code uses the [Amplitude Experiment SDK](https://amplitude.com/docs/sdks/experiment-sdks) or [REST API](https://amplitude.com/docs/apis) to communicate with Amplitude Experiment. For more information on feature flags, go to [Feature Flags](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts).

> **Note:**
>
> Amplitude Experiment defaults to a sequential testing statistical model in all experiments, but you can opt for a [T-test](https://amplitude.com/docs/feature-experiment/experiment-theory/analyze-with-t-test) instead.

Web experimentation uses a visual editor to create your experimental variants. The editor works best for A/B or multi-armed bandit experimentation. With the visual editor, you can select and alter web elements, including content and element properties. Web Experiment lets less technical users, or users with fewer permissions in your system, create experiments without engineering resources.

Web experiments use pages to control where your experiment's variants apply on your website. Pages let you scope experiments to specific URLs without affecting unrelated parts of your site.

### Functional availability

For in-depth information about what functionality is available for Feature, Web, or both types of experimentation, go to [Differences Between Feature and Web Experimentation](https://amplitude.com/docs/feature-experiment/feature-and-web-experiment-functional-comparison).

## Creating an experiment

Many experimentation programs fail in the first step because nobody can articulate the problem that experimentation might solve. If you can't explain in clear, simple language why you're running an experiment, you can't realistically hope to learn anything useful from it.

Before doing anything else, spend some time creating a strong mission statement for your experiment. The statement answers two questions: What's the problem, and how can running an experiment help you solve it? The effort you put in at this stage pays off later.

After you create the mission statement, you're ready to configure your experiment. Create a new deployment (or choose one you created earlier) for the experiment, and install the SDK you plan to use.

### Create a hypothesis

Next, focus on the mission statement you created for your experiment. The mission statement serves as the foundation for your experiment's hypothesis. A hypothesis is a prediction of how your experiment is likely to be the correct choice. The hypothesis tells you whether your experiment succeeds or fails.

The problem statement is only the first part of a hypothesis. A hypothesis has two other parts: a proposed solution and a predicted result.

A proposed solution is a description of the changes you want to make to fix your problem. For example: "Consolidate two steps of the onboarding process into one." A predicted result is your expected outcome from the experiment. For example: "Decrease onboarding churn by 20%."

An example of a hypothesis statement:

_"User churn in our onboarding funnel is significantly higher than industry average. Product data suggests our funnel may be too confusing; we believe consolidating steps two and three in the funnel can fix the problem. As a result of this change, we expect onboarding churn to decrease by 20%."_

Every hypothesis statement is unique to your needs, particularly around the problem definition stage. For example, your question may be more exploratory in nature: _"Why are so many users dropping out of our onboarding funnel?"_ Or, you may be more interested in testing different solutions to a problem you already understand, such as _"We've come up with several potential UI changes to rectify a known user pain point. Which one works best?"_

Use this basic template as a starting point for most hypotheses, especially if you're new to experimentation.

### Pick a metric

Focus on the last sentence in the hypothesis statement. The sentence includes a specific measurement of how you expect user behavior to change: onboarding churn decreases by 20%. The measurement determines whether your experiment is a success: either you hit this number, or you don't.

To know whether you hit the number, you need a way to measure the decrease in churn. You need a metric. In Experiment, any event you log can serve as an experiment metric. For the example experiment above, use the event your product uses to track drop-off in your funnels as your metric.

### Create a variant

You could roll out the new onboarding process to all your users and observe the result. If you do that, you can't know whether the product changes you made caused any improvement in your onboarding churn rate. And if that rate drops after you roll out the new funnel, the cause could be poor design choices, random chance, or some external influence you didn't consider. You have no way to find the specific cause.

For this reason, you need to create at least one treatment variant for your experiment. A treatment variant is a different user experience that you display to a percentage of your users. In the onboarding example, the variant is the new, streamlined version of the onboarding process. Some of your users experience the new version, while others continue with the current process (known as the control). The differences in how your users respond to each variant determine the experiment's success.

When creating variants, minimize the number of changes in each variant. Ideally, each variant contains a single change. A single change lets you understand which changes produce positive results. Also, make sure the variants differ noticeably from each other. Distinct variants confirm that users in each segment experience different experiences and that your changes drive any differences in behavior between the segments.

### Decide who receives the variant

Next, define a bucketing unit, which determines what group of people sees the same variant. The most common bucketing unit is "user." However, if you're a B2B business or you use the collaboration feature, you might want to use a bucketing unit such as "organization" or `company_id`, so every user within the same organization experiences the same variant. Consistent bucketing helps reduce product-related confusion caused by disparate user interfaces if people are sitting next to their coworkers. Another reason to bucket by `company_id` is to reduce load on your customer support team. The support team can more easily track which accounts have which features enabled. Either way, make sure the Stable Unit Treatment Value Assumption ([SUTVA](https://blogs.iq.harvard.edu/violations_of_s#:~:text=Methods%20for%20causal%20inference%2C%20in,treatments%20of%20others%20around%20him)) holds for whatever bucketing unit you choose to best ensure inference.

> **Note:**
>
> If your organization has purchased the [Accounts add-on,](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users.

### Allocate users

After you create your variants, decide how many of your users receive them. You can roll your experiment out to your entire user base, or to a fraction of them. Specify how many users in your experiment receive the control experience and how many receive your variants. You can define specific user segments to include or exclude from your experiment, and you can even choose a specific experience for individual user or device IDs.

### Activate your experiment

You're ready to roll out your experiment to your users. Select **Start Experiment**, and your experiment goes live.

### Analyze your results

After your experiment goes live, you can generate and view your results at any time. Experiment tells you when your experiment reaches statistical significance, which is when the experiment has enough results and data that you can trust the information you're gathering is accurate. Experiment provides all the data it collects so that you can analyze and interpret your results.

To learn more about how to design, roll out, and learn from experiments, refer to these [articles on the experimentation workflow](https://amplitude.com/docs/feature-experiment/workflow/create).

> **Note:**
>
> Consider using experiment briefs to better communicate and streamline your experimentation processes among your team. Experiment briefs help create transparency and align experimentation goals. [Read more about experiment briefs and how to use them in this blog](https://amplitude.com/blog/experiment-brief).

> **Note:**
>
> The [Website Conversion Agent](https://amplitude.com/docs/amplitude-ai/website-conversion-agent) can help you identify high-impact pages, generate experiment strategies, and create draft experiments, all through a guided AI workflow.

================================================================================
# Web Experiment
URL: https://amplitude.com/docs/web-experiment
================================================================================

# Web Experiment

Turn every page on your website into a testing ground for what works. Amplitude Web Experiment pairs a point-and-click editor with rigorous analytics so product, marketing, and growth teams can ship changes that move the numbers, without waiting on engineering.

**Outcome:** [Ship tests without an engineering ticket](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment)

Launch experiments from a visual editor so marketing and growth move at their own speed.

**Outcome:** [Show each visitor the right variant](https://amplitude.com/docs/web-experiment/targeting)

Target by URL, behavior, and user property so the test reaches the audience that matters.

**Outcome:** [Test copy, design, and layout](https://amplitude.com/docs/web-experiment/actions)

Edit text, swap images, and restyle elements on the page itself instead of mocking up a new one.

**Outcome:** [Try a different page entirely](https://amplitude.com/docs/web-experiment/pages)

Run page-level experiments when one tweak isn't enough and you need to redesign the whole flow.

**Outcome:** [Prove the variant moved the metric](https://amplitude.com/docs/web-experiment/post-experiment)

Tie results to revenue, signups, or conversion so you know which variant actually won.

**Outcome:** [Personalize without rebuilding the page](https://amplitude.com/docs/web-experiment/out-of-the-box-widgets)

Drop in prebuilt banners and modals to tailor the visitor experience in minutes.

## Build web experiments

Use the visual editor to create page changes, redirects, and widgets without a full release cycle.

- [Set up a web experiment](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment) to create the experiment, choose pages, and define goals.
- [Edit page elements](https://amplitude.com/docs/web-experiment/actions) to change text, images, styles, and layout.
- [Run page experiments](https://amplitude.com/docs/web-experiment/pages) to test full-page changes across matching URLs.
- [Use URL redirect testing](https://amplitude.com/docs/web-experiment/url-redirect-testing) when each variant needs a separate destination.

## Target and measure visitors

Connect web experiments to Amplitude data so each test reaches the right audience and reports on the right outcomes.

- [Configure targeting](https://amplitude.com/docs/web-experiment/targeting) to include visitors by URL, behavior, and user properties.
- [Track web experiments](https://amplitude.com/docs/web-experiment/tracking) to send exposure and interaction data to Amplitude.
- [Analyze results](https://amplitude.com/docs/web-experiment/post-experiment) to compare variants and make launch decisions.
- [Review performance guidance](https://amplitude.com/docs/web-experiment/performance) to reduce page impact from experiment code.

### [Getting Started with Amplitude Web Experimentation](https://academy.amplitude.com/getting-started-with-amplitude-web-experimentation)

Learn how to use Web Experimentation to fuel conversion rates in your web experiences and single-page apps.

================================================================================
# Experiment JavaScript SDK
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-javascript
================================================================================

# Experiment JavaScript SDK

- **Package:** `@amplitude/experiment-js-client` ([npm](https://www.npmjs.com/package/@amplitude/experiment-js-client))
- **Latest version:** 1.23.0

Official documentation for Amplitude Experiment's Client-side JavaScript SDK implementation.

## Install

Install the Experiment JavaScript Client SDK with one of the three following methods:

> **Info:** Unified SDK
>
> Install the [Browser Unified SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-unified-sdk) to access the Experiment SDK along with other Amplitude products (Analytics, Session Replay). The Unified SDK provides a single entry point for all Amplitude features and simplifies the integration process by handling the initialization and configuration of all components.

#### npm

```bash
# Install Experiment SDK only
npm install --save @amplitude/experiment-js-client

# Or install Unified SDK to get access to all Amplitude products
npm install @amplitude/unified
```

#### yarn

```bash
# Install Experiment SDK only
yarn add @amplitude/experiment-js-client

# Or install Unified SDK to get access to all Amplitude products
yarn add @amplitude/unified
```

#### script

```html
<script src="https://unpkg.com/@amplitude/experiment-js-client@1.15.5/dist/experiment.umd.js"></script>
<script>
    // TODO: Replace DEPLOYMENT_KEY with your own deployment key.
    // If you're using a 3rd party for analytics, configure an exposure
    // tracking provider.
    window.experiment = Experiment.initializeWithAmplitudeAnalytics('DEPLOYMENT_KEY');
</script>
```

> **Tip:** Quick start
>
> The right way to initialize the Experiment SDK depends on whether you use an Amplitude SDK for analytics or a third party (for example, Segment).
>
> #### Amplitude
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```typescript
> import { Experiment } from '@amplitude/experiment-js-client';
>
> // (1) Initialize the experiment client with Amplitude Analytics.
> const experiment = Experiment.initializeWithAmplitudeAnalytics(
>     'DEPLOYMENT_KEY'
> );
>
> // (2) Fetch variants and await the promise result.
> await experiment.fetch();
>
> // (3) Lookup a flag's variant.
> const variant = experiment.variant('FLAG_KEY');
> if (variant.value === 'on') {
>     // Flag is on
> } else {
>     // Flag is off
> }
> ```
>
> #### Unified SDK
>
> 1. [Initialize the Unified SDK](#initialize)
> 2. [Fetch variants](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```typescript
> import { initAll, experiment } from '@amplitude/unified';
>
> // (1) Initialize the Unified SDK with your API key
> // Note: if deploymentKey is not provided, it will fall back to the api key
> initAll('YOUR_API_KEY', {
>     experiment: {
>         deploymentKey: 'DEPLOYMENT_KEY'
>     }
> });
>
> // (2) Fetch variants and await the promise result.
> await experiment()?.fetch();
>
> // (3) Lookup a flag's variant.
> const variant = experiment()?.variant('FLAG_KEY');
> if (variant.value === 'on') {
>     // Flag is on
> } else {
>     // Flag is off
> }
> ```
>
> #### Third party
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants for a user](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```typescript
> import { Experiment } from '@amplitude/experiment-js-client';
>
> // (1) Initialize the experiment client and implement a
> //     custom exposure tracking provider.
> const experiment = Experiment.initialize(
>     'DEPLOYMENT_KEY',
>     {
>         exposureTrackingProvider: {
>             track: (exposure) => {
>                 // TODO: Implement exposure tracking
>                 // analytics.track('$exposure', exposure)
>             }
>         }
>     }
> );
>
> // (2) Fetch variants with the user and await the promise result.
> const user = {
>     user_id: 'user@company.com',
>     device_id: 'abcdefg',
>     user_properties: {
>         premium: true,
>     },
> }
> await experiment.fetch(user);
>
> // (3) Lookup a flag's variant.
> const variant = experiment.variant('FLAG_KEY');
> if (variant.value === 'on') {
>     // Flag is on
> } else {
>     // Flag is off
> }
> ```

## Initialize

Initialize the SDK in your application on startup. The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) argument you pass into the `apiKey` parameter must live in the same Amplitude project to which you send events.

#### Amplitude

```typescript
initializeWithAmplitudeAnalytics(apiKey: string, config?: ExperimentConfig): ExperimentClient
```

#### Unified SDK

```typescript
initAll(apiKey: string, config?: UnifiedConfig): void
```

#### Third-party

```typescript
initialize(apiKey: string, config?: ExperimentConfig): ExperimentClient
```

| Parameter | Description |
| --- | --- |
| `apikey` | _Required_. The deployment key which authorizes fetch requests and determines which flags to evaluate for the user |
| `config` | The client configuration to customize SDK client behavior. |

The initializer returns a singleton instance, so subsequent initializations for the same instance name always return the initial instance. To create multiple instances, use the `instanceName` configuration.

#### Amplitude

```typescript
import { Experiment } from '@amplitude/experiment-js-client';

const experiment = initializeWithAmplitudeAnalytics('DEPLOYMENT_KEY');
```

#### Unified SDK

```typescript
import { initAll, experiment } from '@amplitude/unified';

// Note: if deploymentKey is not provided, it will fall back to the api key
initAll('YOUR_API_KEY', {
    experiment: {
        deploymentKey: 'DEPLOYMENT_KEY'
    }
});
```

#### Third-party

```typescript
import { Experiment } from '@amplitude/experiment-js-client';

const experiment = Experiment.initialize(
    'DEPLOYMENT_KEY',
    {
        exposureTrackingProvider: {
            track: (exposure) => {
                // TODO: Implement exposure tracking
                // analytics.track('$exposure', exposure)
            }
        }
    }
);
```

### Configuration

Configure the SDK client during initialization.

#### Configuration

| Name | Description | Default Value |
| --- | --- | --- |
| `debug` | **Deprecated.** When `true`, sets `logLevel` to `Debug`. Use `logLevel` instead. | `false` |
| `logLevel` | The minimum log level to output. The SDK ignores messages below this level. Options: `LogLevel.Disable`, `LogLevel.Error`, `LogLevel.Warn`, `LogLevel.Info`, `LogLevel.Debug`, `LogLevel.Verbose`. Go to [Custom logging](#custom-logging) for details. | `LogLevel.Error` |
| `loggerProvider` | Custom logger implementation. Must implement the `Logger` interface. Go to [Custom logging](#custom-logging). | `null` (uses default ConsoleLogger) |
| `fallbackVariant` | The default variant to fall back if a variant for the provided key doesn't exist. | `{}` |
| `initialVariants` | An initial set of variants to access. This field helps bootstrap the client SDK with values rendered by the server using server-side rendering (SSR). | `{}` |
| `source` | The primary source of variants. Set the value to `Source.InitialVariants` and configured `initialVariants` to bootstrap the SDK for SSR or testing purposes. | `Source.LocalStorage` |
| `serverZone` | Select the Amplitude data center to get flags and variants from, `us` or `eu`. | `us` |
| `serverUrl` | The host to fetch remote evaluation variants from. For hitting the EU data center, use `serverZone`. | `https://api.lab.amplitude.com` |
| `flagsServerUrl` | The host to fetch local evaluation flags from. For hitting the EU data center, use `serverZone`. | `https://flag.lab.amplitude.com` |
| `fetchTimeoutMillis` | The timeout for fetching variants in milliseconds. | `10000` |
| `retryFetchOnFailure` | Whether to retry variant fetches in the background if the request doesn't succeed. | `true` |
| `automaticExposureTracking` | If true, calling [`variant()`](#variant) tracks an exposure event through the configured `exposureTrackingProvider`. If no exposure tracking provider is set, this configuration option does nothing. | `true` |
| `fetchOnStart` | If true or undefined, always [fetch](#fetch) remote evaluation variants on [start](#start). If false, never fetch on start. | `true` |
| `pollOnStart` | Poll for local evaluation flag configuration updates every minute on [start](#start). | `true` |
| `automaticFetchOnAmplitudeIdentityChange` | Only matters if you use the `initializeWithAmplitudeAnalytics` initialization function to integrate with the Amplitude Analytics SDK. If `true` any change to the user ID, device ID or user properties from analytics triggers the experiment SDK to fetch variants and update its cache. | `false` |
| `userProvider` | An interface used to provide the user object to `fetch()` when called. | `null` |
| `exposureTrackingProvider` | Implement and configure this interface to track exposure events through the experiment SDK, either automatically or explicitly. | `null` |
| `instanceName` | Custom instance name for experiment SDK instance. **The value of this field is case-sensitive.** | `null` |
| `initialFlags` | A JSON string representing an initial array of flag configurations to use for local evaluation. | `undefined` |
| `httpClient` | (Advanced) Use your own HTTP client implementation to handle network requests made by the SDK. | Default HTTP client |

### Integrations

If you use either Amplitude or Segment Analytics SDKs to track events into Amplitude, set up an integration on initialization. Integrations automatically implement [provider](#providers) interfaces to enable a more streamlined developer experience by making it easier to **manage user identity** and **track exposures events**.

#### Amplitude integration

The Amplitude Experiment SDK is set up to integrate with the Amplitude Analytics SDK.

```typescript
import * as amplitude from '@amplitude/analytics-browser';
import { Experiment } from '@amplitude/experiment-js-client';

amplitude.init('API_KEY');
const experiment = Experiment.initializeWithAmplitudeAnalytics('DEPLOYMENT_KEY');
```

Using the integration initializer configures implementations of the [user provider](#user-provider) and [exposure tracking provider](#exposure-tracking-provider) interfaces to pull user data from the Amplitude Analytics SDK and track exposure events.

**Supported Versions**

All versions of the next-generation [Amplitude analytics Browser](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2) SDK support this integration.

| Legacy Analytics SDK Version | Experiment SDK Version |
| --- | --- |
| `8.18.1+` | `1.4.1+` |

#### Segment integration

Experiment's integration with Segment Analytics requires manual configuration. Then, configure the Experiment SDK on initialization with an instance of the exposure tracking provider. Make sure this happens _after_ the analytics SDK loads an initializes.

```typescript
analytics.ready(() => {
    const experiment =  Experiment.initialize('DEPLOYMENT_KEY', {
        exposureTrackingProvider: {
            track: (exposure) => {
                analytics.track('$exposure', exposure)
            }
        }
    });
});
```

When [fetching variants](#fetch), pass the segment anonymous ID and user ID for the device ID and user ID, respectively.

```typescript
await experiment.fetch({
    user_id: analytics.user().id(),
    device_id: analytics.user().analyticsId(),
});
```

#### mParticle integration

Experiment's integration with mParticle requires manual integration. The values you use for `user_id` and `device_id` depend on your specific configuration.

In accordance with your event forwarding settings to Amplitude, the event type `Other` may not be the right classification. Make sure that destination settings forward the event type you use to Amplitude.

```typescript
const identityRequest = {
    userIdentities: {
        email: "joe_slow@gmail.com",
        customerid: "abcdxyz"
    }
};
mParticle.Identity.login(identityRequest, () => {
    console.log("Identity callback");
});

const experiment = Experiment.initialize("DEPLOYMENT_KEY", {
    exposureTrackingProvider: {
        track: (exposure) => {
            window.mParticle.logEvent('$exposure', window.mParticle.EventType.Other, exposure);
        }
    },
    userProvider: {
        getUser: () => {
            const user_id = window.mParticle.Identity.getCurrentUser().getUserIdentities().userIdentities.customerid;
            const device_id = window.mParticle.getDeviceId();
            if (user_id != null) {
                return {
                    user_id: user_id,
                    device_id: device_id
                };
                else
                    return {
                        device_id: device_id
                    };
            }
        }
    }
});
```

## Manage user consent

The Experiment SDK doesn't expose a consent API. There's no `consentOptions` configuration, `ConsentStatus` type, or `setConsentStatus()` method.

To support GDPR-style consent flows, gate Amplitude initialization and variant fetches on the signal from your consent management platform (CMP). Don't initialize the analytics SDK or call [`fetch()`](#fetch) or [`start()`](#start) until the user grants consent.

```typescript
import * as amplitude from '@amplitude/analytics-browser';
import { Experiment } from '@amplitude/experiment-js-client';

// Call this only after your CMP reports that the user granted consent.
async function onConsentGranted() {
    amplitude.init('API_KEY');
    const experiment = Experiment.initializeWithAmplitudeAnalytics('DEPLOYMENT_KEY');
    await experiment.fetch();
}
```

If a user withdraws consent, stop calling `fetch()` and clear any stored variants with [`clear()`](#clear).

## Fetch

Fetches variants for a [user](https://amplitude.com/docs/feature-experiment/data-model#users) and stores the results in the client for fast access. The function [remote evaluates](https://amplitude.com/docs/feature-experiment/remote-evaluation) the user for flags associated with the deployment used to initialize the SDK client.

> **Tip:** Fetch on user identity change
>
> If you want the most up-to-date variants for the user, it's recommended that you call `fetch()` whenever the user state changes in a meaningful way. For example, if the user logs in and receives a user ID, or has a user property set which may affect flag or experiment targeting rules.
>
> Pass new **user properties** explicitly to `fetch()` instead of relying on user enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation). Remote user-property sync through a separate system has no timing guarantees for `fetch()`, which can create a race condition.

```typescript
fetch(user?: ExperimentUser, options?: FetchOptions): Promise<Client>
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to evaluate. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. |
| `options` | optional | Explicit flag keys to fetch. |

#### Account-level bucketing and analysis (v1.5.6+)

If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users. Reach out to your representative to gain access to this beta feature.

Include groups in the user sent with the fetch request (recommended), or identify groups with the user through a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or through [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).

```typescript
await fetch({
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
    groups: {'org name': ['Amplitude']}
});
```

To pass freeform group properties, refer to this example:

```typescript
await fetch({
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
    group_properties: {'org name': ['Amplitude']}
});
```

```typescript
const user = {
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
};
await experiment.fetch(user);
```

If you're using an [integration](#integrations) or a custom [user provider](#user-provider) then you can fetch without inputting the user.

```typescript
await experiment.fetch();
```

If `fetch()` times out (default 10 seconds) or fails for any reason, the SDK client returns and retries in the background with back-off. You may configure the timeout or disable retries in the [configuration options](#configuration) during SDK client initialization.

## Start

> **Info:** Fetch vs start
>
> Use `start` if you're using client-side [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation). If you're only using [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), call [fetch](#fetch) instead of `start`.

Start the SDK by getting flag configurations from the server and fetching remote evaluation variants for the user. The SDK is ready when the returned promise resolves.

```typescript
start(user?: ExperimentUser): Promise<void>
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to fetch variants. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. Also sets the user in the SDK for reuse. |

Call `start()` when your application is initializing, after user information is available to evaluate or [fetch](#fetch) variants. The promise resolves after loading local evaluation flag configurations and fetching remote evaluation variants.

Configure the behavior of `start()` by setting `fetchOnStart` in the SDK configuration on initialization to improve performance based on the needs of your application.

- If your application never relies on remote evaluation, set `fetchOnStart` to `false` to avoid increased startup latency caused by remote evaluation.
- If your application relies on remote evaluation, but not right at startup, you may set `fetchOnStart` to `false` and call `fetch()` and await the promise separately.

#### Amplitude

```typescript
await experiment.start();
```

#### Third party

```typescript
const user = {
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        premium: true
    }
};
await experiment.start(user);
```

## Variant

Access a [variant](https://amplitude.com/docs/feature-experiment/data-model#variants) for a [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) from the SDK client's local store.

> **Info:** Automatic exposure tracking
>
> When you use an [integration](#integrations) or set a custom [exposure tracking provider](#exposure-tracking-provider), `variant()` automatically tracks an exposure event through the tracking provider. To disable this functionality, [configure](#configuration) `automaticExposureTracking` to be `false`, and track exposures manually using [`exposure()`](#exposure).

```typescript
variant(key: string, fallback?: string | Variant): Variant
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) to access the variant for. |
| `fallback` | optional | The value to return if no variant was found for the given `flagKey`. |

When determining which variant a user has been bucketed into, you'll want to compare the variant `value` to a well-known string.

```typescript
const variant = experiment.variant('<FLAG_KEY>');
if (variant.value === 'on') {
    // Flag is on
} else {
    // Flag is off
}
```

> **Info:** Access a variant's payload
>
> A variant may also be configured with a dynamic [payload](https://amplitude.com/docs/feature-experiment/data-model#variants) of arbitrary data. Access the `payload` field from the variant object after checking the variant's `value`.
>
> ```typescript
> const variant = experiment.variant('<FLAG_KEY>');
> if (variant.value === 'on') {
>     const payload = variant.payload;
> }
> ```

A `null` variant `value` means that the user hasn't been bucketed into a variant. You may use the built in **fallback** parameter to provide a variant to return if the store doesn't contain a variant for the given flag key.

```typescript
const variant = experiment.variant('<FLAG_KEY>', { value: 'control' });
if (variant.value === 'control') {
    // Control
} else if (variant.value === 'treatment') {
    // Treatment
}
```

## All

Access all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) stored by the SDK client.

```typescript
all(): Variants
```

## Clear

Clear all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) in the cache and storage.

```typescript
clear(): void
```

You can call `clear` after user logout to clear the variants in cache and storage.

```typescript
experiment.clear();
```

## Exposure

Manually track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for the current variant of the given flag key through configured [integration](#integrations) or custom [exposure tracking provider](#exposure-tracking-provider). Generally used in conjunction with setting the `automaticExposureTracking` [configuration](#configuration) optional to `false`.

```typescript
exposure(key: string): void
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) variant to track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for. |

```typescript
const variant = experiment.variant('<FLAG_KEY>');

// Do other things...

experiment.exposure('<FLAG_KEY>');
if (variant.value === 'control') {
    // Control
} else if (variant.value === 'treatment') {
    // Treatment
}
```

## Providers

> **Tip:** Integrations
>
> If you use Amplitude or Segment analytics SDKs along side the Experiment Client SDK, Amplitude recommends you use an [integration](#integrations) instead of implementing custom providers.

Provider implementations enable a more streamlined developer experience by making it easier to manage user identity and track exposures events.

### User provider

The SDK client uses the user provider to access the most up-to-date user information only when needed (for example, when [`fetch()`](#fetch) is called). The user provider is optional, but helps if you have a user information store already set up in your application. With a user provider, you don't need to manage two separate user info stores in parallel. Separate stores can create divergent user state if the application user store is updated and experiment isn't (or vice versa).

```typescript
interface ExperimentUserProvider {
  getUser(): ExperimentUser;
}
```

To use your custom user provider, set the `userProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```typescript
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    userProvider: new CustomUserProvider(),
});
```

### Exposure tracking provider

Amplitude highly recommends implementing an exposure tracking provider. [Exposure tracking](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) increases the accuracy and reliability of experiment results and improves visibility into which flags and experiments a user is exposed to.

```typescript
export interface ExposureTrackingProvider {
  track(exposure: Exposure): void;
}
```

The implementation of `track()` should track an event of type `$exposure` (a.k.a name) with two event properties, `flag_key` and `variant`, corresponding to the two fields on the `Exposure` object argument. Finally, the event tracked must eventually end up in Amplitude Analytics for the same project that the \[deployment\] used to [initialize](#initialize) the SDK client lives within, and for the same user that variants were [fetched](#fetch) for.

To use your custom user provider, set the `exposureTrackingProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```typescript
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    exposureTrackingProvider: new CustomExposureTrackingProvider(),
});
```

## Bootstrapping

You may want to bootstrap the experiment client with an initial set of flags or variants when variants come from an external source (for example, not from calling `fetch()` on the SDK client). Use cases include [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation), [server-side rendering](https://amplitude.com/docs/feature-experiment/advanced-techniques/server-side-rendering), or integration testing on specific variants.

### Bootstrapping variants

To bootstrap the client with a predefined set of variants, set the flags and variants in the `initialVariants` [configuration](#configuration) object, then set the `source` to `Source.InitialVariants` so that the SDK client prefers the bootstrapped variants over any previously fetched & stored variants for the same flags.

```typescript
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    // Map flag keys to variant objects. The variant object may either be
    // pre-evaluation (SSR) or input manually in for testing.
    initialVariants: {
        "<FLAG_KEY>": {
            "value": "<VARIANT>"
         }
    },
    source: Source.InitialVariants,
});
```

### Bootstrapping flag configurations

You may choose to bootstrap the SDK with an initial set of local evaluation flag configurations using the `initialFlags` configuration. The SDK evaluates these flag configurations when [variant](#variant) is called, unless an updated flag config or variant is loaded with [start](#start) or [fetch](#fetch).

To download initial flags, use the [evaluation flags API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api#flags-api)

```typescript
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    initialFlags: "<FLAGS_JSON>",
});
```

## HTTP client

You can provide a custom HTTP client implementation to handle network requests made by the SDK. This is useful for environments with specific networking requirements or when you need to customize request handling.

```typescript
export interface SimpleResponse {
  status: number;
  body: string;
}

export interface HttpClient {
  request(
    requestUrl: string,
    method: string,
    headers: Record<string, string>,
    data: string,
    timeoutMillis?: number,
  ): Promise<SimpleResponse>;
}
```

To use your custom HTTP client, set the `httpClient` [configuration](#configuration) option with an instance of your implementation on SDK initialization.

```typescript
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    httpClient: new CustomHttpClient(),
});
```

## Custom logging

Control log verbosity with the `logLevel` configuration, or implement the `Logger` interface to integrate your own logging solution.

### Log levels

- `LogLevel.Disable` - No logging
- `LogLevel.Error` - Errors only (default)
- `LogLevel.Warn` - Errors and warnings
- `LogLevel.Info` - Errors, warnings, and informational messages
- `LogLevel.Debug` - Errors, warnings, info, and debug messages
- `LogLevel.Verbose` - All messages including verbose details

```typescript
import { Experiment, LogLevel } from '@amplitude/experiment-js-client';

// Only log errors
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Error
});

// Log errors and warnings
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Warn
});

// Log everything (verbose)
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Verbose
});
```

### Custom logger

Implement the `Logger` interface to use your own logging solution:

```typescript
import { Experiment, Logger, LogLevel } from '@amplitude/experiment-js-client';

// Implement the Logger interface
class CustomLogger implements Logger {
  error(message?: any, ...optionalParams: any[]): void {
    // Send errors to your logging service
    myLoggingService.error(message, ...optionalParams);
  }

  warn(message?: any, ...optionalParams: any[]): void {
    myLoggingService.warn(message, ...optionalParams);
  }

  info(message?: any, ...optionalParams: any[]): void {
    myLoggingService.info(message, ...optionalParams);
  }

  debug(message?: any, ...optionalParams: any[]): void {
    myLoggingService.debug(message, ...optionalParams);
  }

  verbose(message?: any, ...optionalParams: any[]): void {
    myLoggingService.verbose(message, ...optionalParams);
  }
}

// Initialize with custom logger
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  loggerProvider: new CustomLogger(),
  logLevel: LogLevel.Warn
});
```

### Debug flag (deprecated)

The `debug` configuration flag is deprecated. Use `logLevel` instead.

```typescript
// Deprecated: Sets logLevel to Debug
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  debug: true
});

// Preferred: Use logLevel instead
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Debug
});
```

================================================================================
# Run your first experiment
URL: https://amplitude.com/docs/quick-guides/run-your-first-experiment
================================================================================

# Run your first experiment

## Why experiment in Amplitude

### One place, no stitching

Behavioral data and experiment results live in the same place, so there is no stitching required.

### Target with real cohorts

Use your existing Amplitude cohorts directly as targeting audiences.

### Metrics you already trust

Results auto-connect to your existing metrics: funnels, retention, revenue.

## Run your first experiment

Five steps from idea to shipped decision.

1. **Choose your type and define your hypothesis**

   Pick [Web Experiment](https://amplitude.com/docs/web-experiment) for a visual, no-code change or [Feature Experiment](https://amplitude.com/docs/feature-experiment/overview) for a code-controlled change behind a feature flag. Then write the hypothesis down: if I change _X_, then _metric Y_ improves because _reason Z_.

2. **Set your audience**

   Target an existing Amplitude cohort, or [define targeting rules](https://amplitude.com/docs/feature-experiment/workflow/define-audience) in the experiment itself.

3. **Build your variants and pick your goal metric**

   Control is the current experience. Treatment is your change. Connect the result to an existing Amplitude event so the [goal metric](https://amplitude.com/docs/feature-experiment/workflow/define-goals) uses data you already trust.

4. **Set traffic split and launch**

   Use a 50/50 split and run for at least two weeks, so the experiment collects enough traffic to reach significance. After launch, monitor the _Experiment Results_ view.

5. **Ship or kill**

   A significant positive result means you roll out to 100%. No significant result means you [iterate on the hypothesis](https://amplitude.com/docs/feature-experiment/workflow/make-decision-experiment).

   [Open in Amplitude →](https://app.amplitude.com/experiment/create)

## When to use Experiment

Use [Experiment](https://amplitude.com/docs/feature-experiment/overview) when you have a specific change in mind and need proof it works before you ship it to everyone. Two neighboring tools solve different problems.

### [Feature flags alone](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts)

Use a plain flag rollout when you want to control who gets a feature, or kill-switch it, without measuring a controlled comparison.

### [Guides and Surveys](https://amplitude.com/docs/guides-and-surveys/get-started)

Use Guides and Surveys when you want to nudge users or collect feedback in-app, not test a variant against a control.

## Pick your first experiment

Each use case is a full setup, end to end. Start with the one closest to a question your team is asking right now.

### Test your onboarding flow

Trigger on signup and measure whether more users reach your activation event.

- **Trigger**: Signup.
- **Goal metric**: Activation event.

### Optimize a conversion moment

Change the pricing or upgrade page with the visual editor, no code required.

- **Trigger**: Pricing or upgrade page visit.
- **Goal metric**: Upgrade or purchase event.
- **Type**: Web Experiment, no code.

### Validate a new feature before full rollout

Release behind a flag to a percentage of users and compare engagement before you go wide.

- **Trigger**: Feature flag targeting a percentage of users.
- **Goal metric**: Feature engagement event.
- **Type**: Feature Experiment, SDK.

================================================================================
# Set up a web experiment
URL: https://amplitude.com/docs/web-experiment/set-up-a-web-experiment
================================================================================

# Set up a web experiment

Web Experiment lets you create an A/B or [multi-armed bandit experiment](https://amplitude.com/docs/feature-experiment/workflow/multi-armed-bandit-experiments) without new code. Open your site in the [Visual Editor](#the-visual-editor), choose the elements you want to experiment with, and change their content or properties directly.

Web Experiments use [Pages](https://amplitude.com/docs/web-experiment/pages) to control where your experiment variants apply on your website. Pages let you scope experiments to specific URLs without affecting unrelated parts of your site.

## Before you begin

Before you set up a web experiment, [implement](https://amplitude.com/docs/web-experiment/implementation) the Web Experiment script on your site.

Creating and running a web experiment differs from [Feature Experiment](https://amplitude.com/docs/feature-experiment/workflow/create), though some steps overlap.

## Set up a web experiment

##### To set up a web experiment

1. In Amplitude Experiment, navigate to _Experiments > Create Experiment > Web Experiment_.
2. In the New Experiment modal, name your experiment.
3. Enter the URL of a page this experiment targets and select the project from the drop-down. Amplitude uses this URL to create your first [Page](https://amplitude.com/docs/web-experiment/pages). You must instrument Web Experiment on this URL for the experiment to work.

   If the script is on the page you specified, Amplitude Experiment opens the page in the [Visual Editor](#the-visual-editor) as a new variant in your experiment.

   You have two options for the treatment variant action: [element changes](https://amplitude.com/docs/web-experiment/actions#element-changes) or [URL redirect](https://amplitude.com/docs/web-experiment/actions#url-redirect).

> **Warning:**
>
> If the script isn't on the page you specify, or if you have an ad blocker or other privacy extension enabled, Amplitude Experiment can't open the Visual Editor. Amplitude opens the Site Setup panel and prompts you to [implement](https://amplitude.com/docs/web-experiment/implementation) the script.

4. To change text, colors, or other elements of the page's UI, click **Element Changes**.
5. Click the element you want to change.
6. The editing toolbar opens beside the selected element with quick actions such as editing the element's content, or [move element](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#move).

   Click the expand icon to open the drawer and edit CSS style properties.

   Changes apply automatically to the current variant.
7. Repeat this process for each element you want to change for your experiment.
8. (_Optional_), click **+** to add another variant.
9. When you're done, click **Continue**.

After you set the experiment, define the experiment's [goals](https://amplitude.com/docs/feature-experiment/workflow/define-goals).

##### To define your experiment's goals

1. In the **Pages** tab, configure which [Pages](https://amplitude.com/docs/web-experiment/pages) your experiment targets. Create new Pages or reuse existing saved Pages. If you're only targeting the page you set on creation, skip this step.
2. From the Include pages where dropdown, specify how Amplitude Experiment identifies these pages.

   Use the same pattern to exclude the experiment from pages you select. Learn more about [managing Pages](https://amplitude.com/docs/web-experiment/pages) for precise experiment targeting.
3. Target the users you want to include in this experiment. Go to [audience targeting](https://amplitude.com/docs/web-experiment/targeting#audience-targeting) for more information. Web Experiment audience targeting works differently than Feature Experimentation.
4. Specify any [additional options](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences) in the Advanced tab.
5. Click **Save and Close** to finish creating your Web Experiment.

> **Tip:** Create a new run of an existing experiment
>
> To re-run an experiment, go to [New Experiment Run](https://amplitude.com/docs/feature-experiment/troubleshooting/new-experiment-run).

## Test and preview your web experiment

Before you run your web experiment, Amplitude recommends you test and preview each variant.

##### To test your web experiment

1. Click **Test & Preview**. This puts your experiment in test instrumentation mode, but doesn't begin the experiment. Only users who open the page with the preview link experience your changes.
2. In the modal, click **Preview** to open a new tab that applies the changes you made for that variant.
3. Click the chain link icon to copy the URL to share with others.

Test each variant at least once. Test on more than one page if your experiment targets multiple pages.

If your changes aren't visible, wait up to 60 seconds for caches to refresh. If the changes don't appear correctly after that time, check your configuration for possible issues. To diagnose why an experiment isn't firing, use the [Amplitude Chrome extension](https://amplitude.com/docs/data/chrome-extension-debug#web-experiments) to inspect audience and page targeting, check exposure status, and preview variants.

> **Warning:** Ad blockers
>
> Ad blocking plugins or extensions may prevent you from testing and previewing your experiment.

## The Visual Editor

The Visual Editor loads the site at the URL you specified on experiment creation and adds an overlay. The Visual Editor gives you access to every element of your site so you can modify the site for your experiment.

> **Note:**
>
> When you modify your site in the Visual Editor, your live site remains unchanged until you launch the experiment. No changes in the Visual Editor appear on your site in real time.

When you click an element, the editing toolbar opens beside the selected element with quick actions such as editing the element's content or [move element](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#move). Changes in the visual editor apply automatically to the current variant.

To undo or redo changes, click the undo and redo buttons in the visual editor navigation bar, or use the keyboard shortcuts `Cmd+Z` (undo) and `Cmd+Shift+Z` (redo) on macOS, or `Ctrl+Z` (undo) and `Ctrl+Shift+Z` (redo) on Windows and Linux.

This toolbar consists of the following sections and tools:

- Selector
- Styles
- CSS
- HTML
- Move
- Navigation

### Selector

The selector is a unique identifier for the selected element on the current page. Web Experiment disables the selector by default. Update the selector if you're running the experiment on multiple pages, because the generated selector is only unique for the current page. As an alternative, edit the selector to target multiple elements.

#### Stability best practices

- Visual Editor behavior: The editor selects the shortest unique path, which may include dynamically generated class names such as `sm-contentWrapper_ab12c_3`. These class values often include a hash that changes between builds, causing the selector to stop matching.
- When classes have stable substrings: Prefer an attribute selector that matches the stable portion instead of the full hashed class.

  ```css
  /* Instead of a fragile exact class */
  .sm-contentWrapper_ab12c_3

  /* Use a contains match on the class attribute */
  div[class*="contentWrapper"]
  ```

  This continues to match elements like \`

`even if a later build generates`sm-contentWrapper\_d9f7e\_1\`.

- Most reliable long-term approach: Add a stable attribute (for example, `data-*`) or a unique id to the element you want to modify, and target that.

  ```html
  <div data-test-id="experiment-content"></div>
  ```

  ```html
  [data-test-id="experiment-content"] /* or */ #experiment-content
  ```

#### Target elements in shadow DOM

Web Experiment applies variant changes to elements inside **open** shadow DOM trees. The experiment script resolves selectors through shadow boundaries when it applies element changes at runtime.

Web Experiment doesn't support targeting elements inside **closed** shadow roots.

> **Note:**
>
> Shadow DOM styles are scoped to the shadow tree. Text, visibility, and display changes apply within the shadow boundary and don't leak styles to the rest of the page.

### Styles

The Styles tab contains frequently used CSS properties, including font size, font color, text alignment, padding, margin, background color, display, and visibility.

### CSS

The CSS tab lets you define any CSS property and value, which Amplitude applies inline to the selected element.

### HTML

The HTML tab lets you edit the HTML contents of the selected element.

### Move

Move the selected element up or down in the DOM tree to adjust its placement relative to its current node. The Move option appears in the primary tool menu bar. The Move control updates the underlying HTML of your page rather than moving elements by a predefined amount using CSS.

##### To move elements in the page

1. Click the element you want to move.
2. In the editor menu bar, click the **Up** or **Down** arrow buttons to move the element. The element moves past one surrounding HTML element at a time.
3. Continue clicking the buttons until you move the element into the position you want.

Note the following as you move elements on the page:

- If changes don't apply, confirm you chose the correct selector for the intended element.
- Confirm that CSS styles on your page don't conflict with the updated positioning.
- Confirm that JavaScript doesn't reset your changes after you apply them.
- Moving an element ignores invisible elements in your DOM.

In addition to moving elements up or down, click **Rearrange** to enter preview mode. Preview mode lets you explore different element placements on the page.

### Navigation mode

Navigation mode lets you navigate between pages in your experiment without exiting the editor.

Navigation mode is useful for:

- Experiments spanning multiple pages, such as multi-step forms, checkout flows, or onboarding sequences.
- Editing variants across multiple pages within the same experiment.

##### To enter Navigation mode

1. Open your experiment and access the visual editor.
2. In the primary toolbar at the bottom of the page, click **Navigate**.
3. Navigate your site.

   - When you click an element on the page, the element performs its intended function rather than acting as a selection for the experiment.
   - If you navigate to a page not included in your experiment and attempt to make edits, the Visual Editor updates the page targeting rule to include the page URL and displays a confirmation message.
4. When you land on the page you want to edit, click the pencil icon to toggle the Visual Editor back to **Edit mode**.
5. Toggle between Edit and Navigation mode as needed to complete your experiment's configuration.

### Mobile Simulator

Mobile Simulator lets you preview and edit your experiment variants as they appear on mobile devices. Use the simulator to test responsive layouts, mobile-specific styling, and touch-friendly designs without leaving the Visual Editor.

> **Warning:** Site restrictions
>
> Mobile Simulator loads your site in an iframe. Sites that set `X-Frame-Options: DENY` or `Content-Security-Policy: frame-ancestors 'none'` block this behavior, and the simulator can't display the page.

##### To enter Mobile Simulator

1. Open your experiment and access the visual editor.
2. In the primary toolbar at the bottom of the page, select **Mobile** in the device toggle.
3. Alternately, click the element's pencil icon to open the side panel that contains the upload action.

### Image upload

Upload an image directly in the Visual Editor to replace an existing image element instead of self-hosting the file and pasting a URL. Amplitude stores the image and applies the returned CDN URL to the element as a standard Visual Editor action, so the change behaves like any other element change in your variant. Any paid plan supports image upload.

Start an image upload from either of two places:

- The element toolbar, when you select an image element.
- The **Insert** panel in the top menu bar (formerly labeled **Widgets**).

##### To upload an image

1. In the Visual Editor, select the image element you want to replace, or open the **Insert** panel to add a new image.
2. Select the image file you want to upload from your computer.
3. Amplitude uploads the file to Amplitude-hosted storage and applies the resulting CDN URL to the element.

> **Note:** Content Security Policy
>
> If your site enforces a strict Content Security Policy, allowlist the Amplitude image CDN domain in your `img-src` directive so uploaded images load on your pages.

================================================================================
# Experiment Quick Start
URL: https://amplitude.com/docs/feature-experiment/experiment-quick-start
================================================================================

# Experiment Quick Start

Experiment is a workflow-driven behavioral experimentation platform that speeds up creating variants of features and websites for testing.

With Experiment, you can modify and configure product experiences for unique audiences through:

- **Product experimentation**: Run experiments and A/B tests to onboard new users, reduce friction for checkout experiences, roll out new features, and more.
- **Progressive feature delivery**: Pre-plan and stage new features for beta testers, a percentage of your users, or specific target audiences.
- **Dynamic in-product experiences**: Deploy and adapt custom experiences at scale.
- **Web experimentation**: Perform A/B testing directly on your website.

Experiment supports experimentation through either Feature Experiment or Web Experiment:

- Feature Experiment runs experiments through [feature flags](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts). Feature flags are switches that let you modify your product's experience without changing code.
- Web Experiment runs experiments through the [Visual Editor](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor). The Visual Editor places a dynamic, customizable layer over your existing website. Through the Visual Editor, you can customize any element of your website without changing the site's underlying code or structure.

For more information about the use cases for each type of experimentation, go to [Feature and Web Experiment use cases](https://amplitude.com/docs/feature-experiment/feature-vs-web-experimentation).

This page splits the quick start into Feature Experiment and Web Experiment. Select the tab for the experiment type you want to set up.

> **Warning:**
>
> This quick start guide contains only the basic information needed to implement Experiment. Review the entire set of [Experiment documentation](https://amplitude.com/docs/feature-experiment/overview) to understand the full complexity of the product.

#### Feature Experiment

Experiments and feature flags use the [Amplitude Experiment SDK](https://amplitude.com/docs/sdks/experiment-sdks) or [REST API](https://amplitude.com/docs/apis) to communicate with Amplitude Experiment.

Setting up an experiment is a multi-stage process with these procedures:

1. Install and set up SDKs and deployments.
2. Create feature flags.
3. Select the deployment and create a payload.
4. Create variations and send payloads.
5. Design experiments.

## Before you begin

Before you start using experiments:

- Confirm you have developer write access to the [Experiment SDK](https://amplitude.com/docs/sdks/experiment-sdks).
- Confirm you have access to the Experiment feature.
- Confirm you have developer access to the application where you integrate your feature flags.

## Set up the SDK

Install the Amplitude SDK with the Experiment client. For example:

```bash
npm install @amplitude/analytics-browser @amplitude/experiment-js-client
```

```ts
import * as amplitude from "@amplitude/analytics-browser";
import { Experiment } from "@amplitude/experiment-js-client";
amplitude.init("AMPLITUDE_API_KEY");
const experiment = Experiment.initialize("DEPLOYMENT_API_KEY");
await experiment.start();
```

## Set up a deployment

Experiment uses the same projects as Amplitude Analytics. As a best practice, create one project for each product and each environment. Because [flags](#flags-and-experiments), [experiments](#flags-and-experiments), and [deployments](#deployments) only exist within a single project, you must duplicate these objects across projects within the same product.

In Amplitude Experiment, a deployment serves a group of flags or experiments for use in an application. Each [project](#projects) has a deployment using the project API key as the deployment key, available by default. On creation, Experiment assigns a randomly generated deployment key to each deployment. Experiment uses the deployment key to identify the deployment and authorize requests to the evaluation servers.

> **Note:** Client vs. server deployments
>
> Deployments are either client or server deployments. Use client-side deployments to initialize client-side SDKs, and server-side deployments to initialize server-side SDKs or authorize requests to the Evaluation API.

Deployments belong to Amplitude Analytics projects, and a project can have multiple deployments. Amplitude recommends that you name deployments after the platform (client-side) or service (server-side) to which Experiment serves variants (for example: `android`, `ios`, `web`). The default project API key deployment is useful for getting started. Use explicit deployments for each platform or service in larger organizations or teams that may share the same Amplitude project across multiple platforms for the same application. Each deployment receives a unique key for use in your application.

##### To create a deployment

1. Go to _Experiments > Deployments_.
2. Select **Create Deployment**.
3. Select the project you want from the dropdown list.
4. Name your deployment.
5. Select whether your deployment is for **Client** or **Server**.
6. Select **Create a Deployment**. For full details, go to [Configure your experiment](https://amplitude.com/docs/feature-experiment/workflow/configure).

## Create a new flag

A flag lets you enable or disable a function or feature in your product without deploying new code each time. Flags drive both experiments and feature rollouts. Flags work well for launching experiments and ending them after you collect enough data, and for rolling out new features (and rolling them back, if needed).

##### To create a new feature flag

1. Go to _Experiment > Feature Flags_.
2. Select **Create Feature Flag**.
3. In the Create Flag section, select the project you want from the dropdown list, then give your flag a name. Amplitude Experiment generates the flag key from the name you choose. The flag key is an identifier for the flag used in your codebase.
4. Specify the [evaluation mode](https://amplitude.com/docs/feature-experiment/local-evaluation) for your experiment. Select either **Remote** or **Local**.
5. Specify the **bucketing unit** you want to use for this experiment.

> **Tip:**
>
> The best bucketing unit is typically the user. In some B2B use cases, you might want to use company ID or city as the bucketing unit. For example, bucketing by company ID ensures all users in a particular company have the same user experience. Confirm the [Stable Unit Treatment Value Assumption](https://blogs.iq.harvard.edu/violations_of_s#:~:text=Methods%20for%20causal%20inference%2C%20in,treatments%20of%20others%20around%20him) holds for whichever unit you choose.

6. Select **Create**. Experiment opens a blank template for your flag.
7. Choose the deployment you want from the Deployment dropdown menu.
8. (_Optional_) Select **Advanced Settings** to modify the [bucketing salt](https://amplitude.com/docs/feature-experiment/implementation#consistent-bucketing) options.

> **Note:**
>
> If you change the bucketing salt, users can switch between variants in your experiment. For that reason, Amplitude recommends not changing the bucketing salt unless you know what you’re doing. For more information, go to [How randomization works in Amplitude Experiment](https://amplitude.com/docs/feature-experiment/under-the-hood/experiment-randomization).

## Create variations

A variant exists within a flag or an experiment, and represents a variable experience for a user. Variants comprise the A/B changes you want to test. All feature flags must contain at least one variant. You can add as many variants as you want to a flag.

##### To add a variant

1. Go to your _Experiment > Feature Flags_ and select your flag.
2. In the Variants section, select the **Plus** icon to create a variant.
3. Enter the name, value, and description of your variant.
4. Select **Apply**.

> **Note:**
>
> You can send a payload with your variant. A payload is a JSON-coded set of variables that can remotely change flags and experiments without a manual code change. Because you can send a payload with your control, you don’t need to create a variant for the control itself.
>
> Add JSON content to the Payload field when creating a variant. Payload content resembles:
>
> ```json
> {
>   "layout": "cards",
>   "titlePosition": "above",
>   "gradient": false,
>   "showDescription": true,
>   "cardCount": 3
> }
> ```

## Add targeting to the flag

In the Assignment section, define the user segments you want to experience your new feature. Defining a user segment limits your rollout to users in specific geographical locations, certain demographic groups, or who meet certain usage thresholds in your product (for example, power users). For more information on segmenting, go to [Define your audience](https://amplitude.com/docs/feature-experiment/workflow/define-audience).

##### To add targeting

1. Specify the percentage of users who receive the variant.
2. To define a user segment, go to the Rule Based User Segments section and select **Segment 1**. Then follow the same steps you use to build a [user segment](https://amplitude.com/docs/analytics/charts/build-charts-modify-user-segment) in Amplitude Analytics.

## Finalize the flag

After you set up the flag, associate it with a deployment, set up your variants or payloads, and target your users, finalize the feature flag. Finalizing the flag activates the flag and makes it available.

##### To finalize a feature flag

1. Go to your feature flag.
2. Select **Activate flag**.

## Design the experiment

You can create an experiment directly or convert an existing flag to an experiment.

When designing your experiment:

- Set metrics for the experiment.
- Set up any further variations and payloads.

Adding goals (or metrics) lets you track the success rate of your experiment. All experiments should have at least one metric. Tell Amplitude Experiment what you want your primary metric to be, and define any secondary metrics. The primary metric determines whether Amplitude accepts or rejects your hypothesis, and therefore whether your experiment succeeded or failed.

##### To add metrics

1. Open your experiment and go to the Metrics section.
2. Select your recommended metric from the Metric dropdown or create a custom metric.
3. Select the metric type using "should" or "should not":

   - "Should": A Success metric states the goal changes by the goal amount and direction.
   - "Should not": A Guardrail metric states the goal doesn’t change by the goal amount and direction.
4. Specify whether you expect the metric to increase or decrease.
5. (_Optional_) Set the minimally acceptable goal for the experiment, also known as the [minimum detectable effect](https://amplitude.com/docs/feature-experiment/experiment-theory/experiment-set-mde). The minimum detectable effect is the minimum difference between the control and the variant for the experiment to count as a positive result.
6. To add secondary metrics, select **Add Metric** and repeat this process for each additional metric you want to include.

##### To add more variations and payloads

To create more variations and payloads, repeat the steps in [Create variations](#create-variations) in your flag.

## Start your experiment

After you finish designing your experiment, select **Start Experiment** to begin.

## Code examples

The following code examples show the code for a feature flag and a JSON payload.

#### Feature flag

```js
import { useState, useEffect } from 'react';
import { getBlogLayoutFlag } from '../services/featureFlags'; // Adjust to wherever you fetch your Amplitude flag
import type { BlogPost } from '../types';

type LayoutFlag = {
  layout: 'cards' | 'list' | 'carousel';
  titlePosition: 'above' | 'below' | 'center';
  gradient: boolean;
  showDescription: boolean;
  cardCount: number;
};

export default function BlogPostLayoutClient({ posts }: { posts: BlogPost[] }) {
  const [layoutFlag, setLayoutFlag] = useState<LayoutFlag | null>(null);

  useEffect(() => {
    getBlogLayoutFlag().then((flag) => {
      console.log(':magic_wand: Received Flag from Amplitude:', flag);
      if (flag) {
        setLayoutFlag(flag);
      } else {
        console.log(':warning: No flag returned, falling back to default layout');
        setLayoutFlag({
          layout: 'cards',
          titlePosition: 'above',
          gradient: false,
          showDescription: true,
          cardCount: 3,
        });
      }
    });
  }, []);

  if (!layoutFlag) {
    // You might render a loader here
    return null;
  }

  // Render your posts according to layoutFlag...
  return (
    <div>
      {/* e.g. layoutFlag.layout === 'cards' ? <CardGrid posts={posts} /> : ... */}
    </div>
  );
}

```

#### Payload

```js
// services/featureFlags.ts
import { experiment } from '@amplitude/experiment-js';  // adjust import to your SDK
import type { LayoutFlag } from '../types';              // reuse the same LayoutFlag type

export const getBlogLayoutFlag = async (): Promise<LayoutFlag> => {
  try {
    // In dev, clear any stale flags
    if (process.env.NODE_ENV === 'development') {
      localStorage.clear();
      console.warn('Cleared localStorage in dev mode');
    }

    // Initialize the experiment SDK
    await experiment.start();

    // Grab the variant for our blog layout test
    const variant = experiment.variant('blog_post_layout');
    console.log(':movie_camera: Full Variant Object:', variant);

    // Some payloads come in `payload`, some in `value`
    const value = variant?.payload ?? variant?.value;
    console.log('Cleaned Flag Payload:', value);

    // If there's no usable object, fall back to defaults
    if (!value || typeof value !== 'object' || Object.keys(value).length === 0) {
      console.warn('No valid layout flag found, using fallback layout');
      return {
        layout: 'carousel',
        titlePosition: 'above',
        gradient: false,
        showDescription: true,
        cardCount: 3,
      };
    }

    // Otherwise assume it's Amplitude's LayoutFlag shape
    return value as LayoutFlag;

  } catch (error) {
    console.error('Error fetching blog layout flag:', error);

    // On error, also fall back
    return {
      layout: 'carousel',
      titlePosition: 'above',
      gradient: false,
      showDescription: true,
      cardCount: 3,
    };
  }
};
```

#### Web Experiment

Web Experiment requires you to implement the Web Experiment script on your site before you begin.

Paste the script into the `<head>` element of your site, as high as possible to avoid flickering.

The script tracks [impression events](https://amplitude.com/docs/web-experiment/tracking#impression-events) with the [Browser SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2) already installed on your site, or with a [third-party analytics SDK](#integrate-with-a-third-party-cdp).

## Add the experiment script

Replace `API_KEY` with your project's API key in one of the following synchronous scripts, depending on your region:

#### US Data Center

```html
<script src="https://cdn.amplitude.com/script/API_KEY.experiment.js"></script>
```

#### EU Data Center

```html
<script src="https://cdn.eu.amplitude.com/script/API_KEY.experiment.js"></script>
```

> **Note:** Security headers
>
> Your site may need the following security header adjustments to work with Web Experiment.
>
> #### Content Security Policy
>
> If your site defines the `script-src` content policy directive, add `*.amplitude.com` and `unsafe-inline` to the policy values. These changes enable loading the Web Experiment script and visual editor on your site.
>
> ```text
> Content-Security-Policy: script-src *.amplitude.com unsafe-inline;
> ```
>
> #### Cross-Origin-Opener-Policy
>
> If your site sets the `Cross-Origin-Opener-Policy` header, you can either remove the header or set it to `unsafe-none`. This setting allows the visual editor to load on your site.
>
> ```text
> Cross-Origin-Opener-Policy: unsafe-none
> ```

## Set up a web experiment

1. In Amplitude Experiment, go to _Experiments > Create Experiment > Web Experiment_.
2. In the New Experiment modal, give your experiment a name. Enter the URL of a page this experiment targets. Amplitude must be instrumented on that page, and select the appropriate project from the dropdown. Amplitude uses this URL to create your first [Page](https://amplitude.com/docs/web-experiment/pages).
3. If the script is present on the page you specified, Amplitude Experiment opens the page in the [Visual Editor](#the-visual-editor) as a new variant in your experiment.

   You have two options for the treatment variant action: [element changes](https://amplitude.com/docs/web-experiment/actions#element-changes) or [URL redirect](https://amplitude.com/docs/web-experiment/actions#url-redirect).

> **Warning:**
>
> If the script isn’t present on the page you specify, or if you have an ad blocker or other privacy extension enabled, Amplitude Experiment can’t open the Visual Editor. Amplitude Experiment opens the Site Setup panel and prompts you to [implement](https://amplitude.com/docs/web-experiment/implementation) the script.

4. To change text, colors, or other elements of the page’s UI, select _Element Changes_.
5. Select the element you want to change.
6. The editing toolbar opens beside the selected element with quick actions such as editing the element’s content or [moving the element](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#move).

   Selecting the expand icon opens the drawer, which lets you edit CSS style properties. When you’re done, select _Apply_.
7. Repeat this process for each element you want to change for your experiment.
8. If needed, select _+_ to add another variant.
9. When you’re done, select _Continue_.
10. Next, [define your experiment's goals](https://amplitude.com/docs/feature-experiment/workflow/define-goals).
11. In the _Pages_ tab, configure which [Pages](https://amplitude.com/docs/web-experiment/pages) your experiment targets. You can create new Pages or reuse existing saved Pages. If you target only the page you set on creation, you can skip this step. From the _Include pages where_ dropdown, specify how you want Amplitude Experiment to identify these pages.

    Use the same pattern to exclude the experiment from the pages you select. Learn more about [managing Pages](https://amplitude.com/docs/web-experiment/pages) for precise experiment targeting.
12. Next, target the users you want to include in this experiment. If you’re familiar with feature experiment targeting, note that Web Experiment [audience targeting](https://amplitude.com/docs/web-experiment/targeting#audience-targeting) works differently.
13. The _Advanced_ tab provides several [additional options](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences) for your experiment.
14. When you’re ready, select _Save and Close_ to finish creating your Web Experiment.

> **Tip:** Create a new run of an existing experiment
>
> If you have an experiment that you need to re-run, refer to [New Experiment Run](https://amplitude.com/docs/feature-experiment/troubleshooting/new-experiment-run).

### Test and preview your web experiment

Before running your web experiment, Amplitude recommends that you test and preview each variant. When you’re ready:

1. Select _Test & Preview_. This action puts your experiment in test instrumentation mode, but doesn’t start your experiment. Only users who open the page with the preview link can view your changes.
2. In the modal, select _Preview_ to open a new tab that applies the changes you made for that variant. Select the chain link icon to copy the URL so you can share it with others.

Test each variant at least one time, preferably more. Test each variant on more than one page if your experiment targets multiple pages.

If your changes aren’t apparent, you may need to wait up to 60 seconds for caches to refresh. If the changes don’t appear correctly after that time, the configuration may have an issue.

> **Warning:** Ad blockers
>
> Ad blocking plugins or extensions may prevent you from testing and previewing your experiment.

## Add actions

Actions define how variants modify your site. Actions relate to variants rather than a specific page, and apply to specific [Pages](https://amplitude.com/docs/web-experiment/pages) to control exactly where they take effect.

Experiment applies variant actions during evaluation. Evaluation runs on the initial page load and any time state pushes to or pops from the session history. History state changes also cause the SDK to revert all applied element change and custom code actions before reevaluating and reapplying actions with the updated page in mind.

Actions include:

- [Element changes](https://amplitude.com/docs/web-experiment/actions#element-changes)
- [URL redirects](https://amplitude.com/docs/web-experiment/actions#url-redirect)
- [Custom code](https://amplitude.com/docs/web-experiment/actions#custom-code)

For more information about each action, refer to the preceding links.

### Action examples

> **Tip:**
>
> Generative AI like ChatGPT or equivalents can create HTML and CSS for simple elements. ChatGPT generated the following modal and banner examples, which Amplitude then modified.

#### Insert an element

To insert an element onto your page, follow this pattern.

1. Write the HTML and CSS for the element you want to add to the page.
2. Identify the selector of the parent element you want to insert your new element into. The parent element is often the `body`.
3. Paste the following JavaScript code, and update `PARENT_SELECTOR` with the parent element selector from step 2.

   ```js
   utils.waitForElement("PARENT_SELECTOR").then(function (e) {
     e.appendChild(html);
     utils.remove = function () {
       html.remove();
     };
   });
   ```

To insert your element into the parent element at a specific position, use `insertBefore()` instead of `appendChild()`.

#### Add a banner

This example adds a discount code banner to the top of the page.

#### JS

```js
utils.waitForElement("body").then(function (e) {
  e.insertBefore(html, e.firstChild);
  utils.remove = function () {
    html.remove();
  };
});
```

#### CSS

```css
.announcement-banner {
  background-color: #fafafa;
  color: #333;
  padding: 10px;
  text-align: center;
  font-family: Arial, sans-serif;
  border-bottom: solid #e5e5e5;
  border-bottom-width: 1px;
}

.announcement-banner p {
  margin: 0;
  font-size: 16px;
}
```

#### HTML

```html
<div class="announcement-banner">
  <p>🎉 Big Sale: Get 25% off on all items! Use code <strong>SAVE25</strong></p>
</div>
```

## Work with pages

In a Web Experiment, Pages control where your experiment variants apply on your site. Pages scope experiments to specific URLs, which lets you run tests on targeted pages without affecting unrelated parts of your site.

A Page defines the conditions under which a web experiment applies to your site, and includes:

- A unique name.
- URL targeting conditions.
- A Visual Editor URL to help preview the experiment.

### Create a page

When you create a new Web Experiment, specify a page by:

- **Manual URL input**: Enter a specific URL to define the page.
- **Import a saved page**: Select a page from a previous experiment.

After you add the page, continue with experiment setup, or go directly to the Visual Editor.

### Update a page or create another

To update a page definition, navigate to the Pages tab of the Experiment Setup flow, or select the pencil icon on the Pages section of the Settings tab. From there, rename the page, update its Visual Editor URL, or update the page targeting rules.

#### Page targeting rules

- **URL Matches**

  - **Description**: Match the page URL, ignore query parameters or hash fragments.
  - **Examples**:

    - `https://example.com/pricing`
    - ✅ https://example.com/pricing#details
    - ❌https://example.com/pricing/enterprise
- **URL Matches Exactly**

  - **Description**: Match the full page URL exactly.
  - **Examples**:

    - `https://example.com/pricing?utm_source=facebook`
    - ❌https://example.com/pricing
    - ❌ https://example.com/pricing?utm\_source=tiktok
- **URL Matches Pattern**

  - **Description**: Match the full page URL, including wildcards (`*`).
  - **Examples**:

    - `https://example.com/blog/*`
    - ✅ https://example.com/blog/my-first-post
    - ✅ https://example.com/blog/my-second-post#get-started
- **URL Contains**

  - **Description**: Match the full page URL, where the URL contains a specific substring.
  - **Examples**:

    - `/blog/my-first`
    - ✅ https://example.com/blog/my-first-post
    - ❌ https://example.com/blog/my-second-post
- **URL Starts With**

  - **Description**: Match the full page URL, where the URL starts with an exact substring.
  - **Examples**:

    - `https://example.com/blog`
    - ✅ https://example.com/blog/my-first-post
    - ❌ https://example.com/pricing
- **URL Ends With**

  - **Description**: Match the full page URL, where the URL ends with an exact substring.
  - **Examples**:

    - `/blog/my-first-post`
    - ✅ https://example.com/blog/my-first-post
    - ❌ https://example.com/blog/my-first-post#get-started
- **URL Matches Regex**

  - **Description**: Match the full page URL with a regular expression you define.
  - **Examples**:

    - [Learn Regex](https://www.regular-expressions.info/quickstart.html)
    - [Test Regex](https://regex101.com/)

### Manage page scope for variants

In a web experiment, you can scope each variant to a specific page so the variant’s changes apply only where you intend. This rule applies to all variant types.

#### Visual editor

When you use the Visual Editor to make changes, for example text edits or style updates, those changes associate with the page you select during the preview session. For each change, specify the page or pages it applies to.

Page scope lets you:

- Assign updates or changes to a specific page.
- Avoid applying the same change across all views.
- Maintain better isolation and clarity across your experiment setup.

> **Tip:** Double-check the page scope
>
> Check the page scope for each change to ensure you don’t introduce cross-page conflicts or unintended edits.

#### Custom code

When you add custom code or URL redirects as variants, you can explicitly define which page or pages the variant applies to.

This flexibility lets you create a single experiment that includes custom code with different behaviors, depending on the active page.

## Targeting

Web Experiments target both pages and audiences. Amplitude evaluates page targeting first, and then audience targeting. Both targeting methods evaluate locally in the browser when the page first loads.

Web Experiments use [Pages](https://amplitude.com/docs/web-experiment/pages) to precisely control where experiment variants apply on your website. Pages define the conditions under which a web experiment applies, including targeting conditions to match specific URLs and visual editor URLs for previewing experiments.

### Audience targeting

By default, a new Web Experiment targets all users. Audience targeting lets you target specific users for your experiment. Users who aren’t targeted in the experiment receive the default experience and don’t count toward analysis.

If any segments match, Amplitude buckets that user into a variant based on the configured rollout and variant distribution. For a segment to match, the user must meet all conditions you set. For more details, go to [Web Experiment targeting](https://amplitude.com/docs/web-experiment/targeting).

### Web Experiment performance

Web Experiment intentionally minimizes its impact on page performance.

### Script size

The Web Experiment script is dynamic and includes all your experiment configurations to avoid multiple synchronous downloads. The script size starts with a base size and scales with each experiment.

|  | Uncompressed | Compressed |
| --- | --- | --- |
| Base script | 79KB | 20KB |
| Per-flag size | ~1KB | ~100B |

To avoid continually increasing script sizes, deactivate or archive experiments when they’re complete.

> **Note:** Custom code impact on flag size
>
> Custom code increases the size of a flag’s code because of the size of the custom code itself.

### Caching

Web Experiment uses two caching layers: CDN and browser. These layers help provide more reliable script delivery to your site.

#### CDN cache

Amplitude caches the Web Experiment script on a CDN. When a user requests the script, the user’s browser loads the script from the CDN if another user loaded it in the same geographic area. The CDN cache has a max age of one minute, and serves stale content while the script reloads for up to one hour. The script serves a stale response if the origin returns an error for the maximum amount of time possible.

The cache control response header that configures CDN caching is:

`max-age=60,stale-while-revalidate=3600,stale-if-error=31536000`

#### Browser cache

The browser cache serves the Web Experiment script without making a network request for 60 seconds, or for the maximum amount of time if the server returns an error. The browser cache serves the script from memory (0ms latency) if a user loads multiple pages on your site, or reloads the same page within a one minute window.

The cache control response header that configures browser caching is:

`max-age=60,stale-while-revalidate=3600`

================================================================================
# Implement Web Experiment
URL: https://amplitude.com/docs/web-experiment/implementation
================================================================================

# Implement Web Experiment

Web Experiment requires a standalone script that you add to your website. Paste the script into the `<head>` element of your site, as high as possible to avoid flickering.

The script tracks [impression events](https://amplitude.com/docs/web-experiment/tracking#impression-events) with the [Browser SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2) already installed on your site, or a [third-party analytics SDK](#integrate-with-a-third-party-cdp).

> **Note:** Cookie consent
>
> If your site requires cookie consent before storing data on a visitor's device, configure consent gating before you add the script. Refer to [Cookies and consent management](https://amplitude.com/docs/web-experiment/cookies-and-consent-management).

## Add the experiment script

Replace `API_KEY` with your project's API key in one of the synchronous scripts below for your region.

#### US Data Center

```html
<script src="https://cdn.amplitude.com/script/API_KEY.experiment.js"></script>
```

#### EU Data Center

```html
<script src="https://cdn.eu.amplitude.com/script/API_KEY.experiment.js"></script>
```

> **Note:** Security headers
>
> Your site may need the following security header adjustments to work with Web Experiment.
>
> #### Content Security Policy
>
> If your site defines the `script-src` content policy directive, add `*.amplitude.com` and `unsafe-inline` to the policy values. These changes enable loading the Web Experiment script and visual editor on your site.
>
> ```text
> Content-Security-Policy: script-src *.amplitude.com unsafe-inline;
> ```
>
> #### Cross-Origin-Opener-Policy
>
> If your site sets the `Cross-Origin-Opener-Policy` header, either remove the header or set it to `unsafe-none`. Setting `unsafe-none` allows the visual editor to load on your site.
>
> ```text
> Cross-Origin-Opener-Policy: unsafe-none
> ```

### Async script with anti-flicker snippet

The synchronous script above provides the best experience for your users. To load the script asynchronously, include the following anti-flicker snippet. The snippet masks elements on the page until Amplitude applies all changes. Replace `API_KEY` with your project's API key and set the timeout to control when the anti-flicker mask clears.

#### US Data Center

```html
<script>
  (function (d, h) {
    // TODO: Replace API_KEY with your API key.
    var apiKey = "API_KEY";
    // TODO: Set a timeout in milliseconds for the anti-flicker.
    var timeout = 1000;
    // Hides the page and loads the script. Shows page if script fails to load,
    // otherwise the script shows the page.
    var id = "amp-exp-css";
    try {
      if (!d.getElementById(id)) {
        var st = d.createElement("style");
        st.id = id;
        st.innerText =
          "* { visibility: hidden !important; background-image: none !important; }";
        h.appendChild(st);
        window.setTimeout(function () {
          st.remove();
        }, timeout);
        var sc = d.createElement("script");
        sc.src =
          "https://cdn.amplitude.com/script/" + apiKey + ".experiment.js";
        sc.async = true;
        sc.onerror = function () {
          st.remove();
        };
        h.insertBefore(sc, d.currentScript || h.lastChild);
      }
    } catch (e) {
      console.error(e);
    }
  })(document, document.head);
</script>
```

#### EU Data Center

```html
<script>
  (function (d, h) {
    // TODO: Replace API_KEY with your API key.
    var apiKey = "API_KEY";
    // TODO: Set a timeout in milliseconds for the anti-flicker.
    var timeout = 1000;
    // Hides the page and loads the script. Shows page if script fails to load,
    // otherwise the script shows the page.
    var id = "amp-exp-css";
    try {
      if (!d.getElementById(id)) {
        var st = d.createElement("style");
        st.id = id;
        st.innerText =
          "* { visibility: hidden !important; background-image: none !important; }";
        h.appendChild(st);
        window.setTimeout(function () {
          st.remove();
        }, timeout);
        var sc = d.createElement("script");
        sc.src =
          "https://cdn.eu.amplitude.com/script/" + apiKey + ".experiment.js";
        sc.async = true;
        sc.onerror = function () {
          st.remove();
        };
        h.insertBefore(sc, d.currentScript || h.lastChild);
      }
    } catch (e) {
      console.error(e);
    }
  })(document, document.head);
</script>
```

### Integrate with a third-party customer data platform

If you use a customer data platform (CDP) other than Amplitude to send events, set up an integration to provide user identity information and track events. Without an integration, the script assumes Amplitude Browser SDK is installed on the same site.

The Web Experiment script supports common CDP integrations through an `integration` query parameter in the script URL.

#### Segment integration

Web experimentation supports Segment by default. Add `integration=segment` as a query parameter to the script URL. For example, in Amplitude's US region:

```html
<!-- Replace API_KEY with your project's API key -->
<script src="https://cdn.amplitude.com/script/API_KEY.experiment.js?integration=segment"></script>
```

#### Tealium integration

If you send events through [Tealium](https://amplitude.com/docs/data/source-catalog/tealium) using Tealium iQ or Tealium AudienceStream & Universal Data Hub, you don't need to set up an integration. Tealium loads the Amplitude Analytics SDK onto the site, which integrates directly with the Web Experiment script.

#### Custom integrations

Implement the `IntegrationPlugin` interface and set the `experimentIntegration` window variable to add a custom integration. Place the plugin script before the Web Experiment script tag.

- `getUser(): object`: Return the [experiment user](https://amplitude.com/docs/feature-experiment/data-model#users) object.
- `track(): boolean`: Track the event through a third party. Return `true` if tracking succeeded. Returning `false` causes Amplitude to persist the event and retry at an interval.
- `setup(): Promise<void>`: (Optional) Set up the integration asynchronously. Returns a promise that resolves when the integration can return user information from `getUser()`.

```html
<script>
  window.experimentIntegration = {
    getUser: () => {
      // TODO: Return user
      return {
        user_id: "user",
        device_id: "device",
      };
    },
    track: (e) => {
      // TODO: Track event
      analytics.track(e.eventType, e.eventProperties);
      return true;
    },
  };
</script>
// TODO: Add the Web Experiment script tag
```

[Contact support](https://amplitude.zendesk.com/hc/en-us/requests/new) for help with a custom integration for your CDP.

## Content management systems

Amplitude Web Experiment supports any content management system (CMS) that supports custom scripts. Amplitude provides plugins for Wordpress and Shopify.

### Wordpress

Amplitude's [Wordpress plugin](https://amplitude.com/docs/data/amplitude-wordpress-plugin) enables Amplitude Analytics, Experiment, and Session Replay.

### Shopify

[Amplitude's Shopify App](https://apps.shopify.com/amplitude) enables Amplitude Analytics, Experiment, and Session Replay on your Shopify site.

> **Warning:** Shopify and flickering
>
> The method Shopify uses to load Amplitude's Shopify app causes flickering. To avoid flickering, add the [asynchronous web script with the anti-flicker snippet](#async-script-with-anti-flicker-snippet) to your `theme.liquid` file.

## Tag managers

Tag managers, such as Google Tag Manager, load scripts asynchronously, which causes flickering. Tag managers can help you start using the visual editor to create variants while you work on adding the Web Experiment script directly to the page. Amplitude recommends against using tag managers in production.

### Google Tag Manager (GTM)

> **Warning:** Causes flicker
>
> Implementing Web Experiment with a tag manager causes flicker. Use a tag manager only when getting started, if adding the script to the site isn't possible.

Use a [custom HTML tag](https://support.google.com/tagmanager/answer/6107167) to add the script through GTM.

================================================================================
# Cookies and consent management (Web Experiment)
URL: https://amplitude.com/docs/web-experiment/cookies-and-consent-management
================================================================================

# Cookies and consent management (Web Experiment)

Consent gating runs Web Experiment without touching the visitor's device until they answer your cookie banner. While consent is undecided, the script still evaluates experiments and applies variants with no flicker, but it holds every cookie and browser storage write in memory, buffers impression events, and doesn't contact any third-party origin. When the visitor grants consent, the script flushes the held data to the device and sends the buffered impressions. When the visitor denies consent, the script discards the held data and erases anything it stored in earlier sessions.

Consent gating covers the Web Experiment script only. To gate Amplitude Analytics event collection on consent, use the Browser SDK's opt-out and deferred initialization instead. Refer to [Cookies and consent management (Browser SDK)](https://amplitude.com/docs/sdks/analytics/browser/cookies-and-consent-management).

## Enable consent gating

Set `consentOptions` on the `window.experimentConfig` object, above the Web Experiment script tag:

```html
<script>
  window.experimentConfig = {
    consentOptions: {
      consentRequired: true,
    },
  };
</script>
<!-- Web Experiment script tag -->
```

| Option | Default | Behavior |
| --- | --- | --- |
| `consentRequired` | `false` | When `true`, the script gates device storage and third-party requests on consent. When `false`, the consent feature is off and the script behaves as if it doesn't exist. |
| `consentStatus` | `'pending'` | The initial consent status, for visitors whose decision you already know when the page renders. One of `'granted'`, `'pending'`, or `'denied'`. |

The script supports three status values:

- **`pending`**: The visitor hasn't decided. The script runs entirely in memory.
- **`granted`**: The visitor consented. The script runs normally.
- **`denied`**: The visitor refused. The script erases its stored data, and the client doesn't start on this page load.

`pending` is only meaningful as an initial status. If your consent platform already knows the visitor's decision when the page renders, set `consentStatus` to `'granted'` or `'denied'` in the config so the script starts in the right state.

## Update consent status at runtime

When the visitor answers the banner, call `setConsentStatus` from your consent management platform's callback:

```js
window.webExperiment?.setConsentStatus("granted"); // or 'denied'
```

The script exposes `setConsentStatus` on `window.webExperiment` as soon as it loads, before the client finishes initializing, so a consent callback that fires early still lands. Guard the call with optional chaining in case the callback runs before the script tag itself loads.

A runtime status always wins over the `consentStatus` config value. Runtime transitions to `pending` log a warning and have no effect.

## What happens in each state

### While consent is pending

The script evaluates experiments and applies variants immediately, so visitors get a flicker-free experience whether or not they've answered the banner. Everything that normally lands on the device stays in memory instead:

- Cookie and browser storage writes go to an in-memory buffer.
- Impression events buffer instead of sending.
- The script doesn't load the cross-subdomain [behavioral targeting](https://amplitude.com/docs/web-experiment/targeting#real-time-behavioral-targeting-local) relay frame.

If the visitor leaves the page without deciding, the buffered data disappears with the page. Nothing persists.

### When the visitor grants consent

A `granted` status resolves the gates in place, without a page reload:

- Buffered storage writes flush to the device.
- Buffered impression events send once.
- The behavioral targeting relay frame loads.

If the page loaded with consent denied, the client didn't start, so a later grant (for example, a visitor who reopens your preference center and opts in) starts it fresh on the same page.

### When the visitor denies or revokes consent

A `denied` status, whether set at load or as a mid-session revocation, erases everything the script has stored on the device: the keys listed in [Data the script stores](#data-the-script-stores-on-the-device), deleted from this origin's storage and from every cookie scope the script writes to. The script permanently discards impression events buffered before the denial. If the visitor opts back in later on the same page, the discarded impressions don't send.

After a mid-session revocation the current page keeps its applied variants, running in memory with device writes suppressed, until the next navigation.

## Data the script stores on the device

With consent granted (or with consent gating off), the script persists the following keys. `<slice>` is the first 10 characters of your project API key.

| Purpose | Storage | Keys |
| --- | --- | --- |
| Visitor identity | localStorage, cookie | `EXP_<slice>`, `EXP_<slice>_identity` |
| First-seen date, landing page | localStorage, sessionStorage | `EXP_<slice>_DEFAULT_USER_PROVIDER` |
| Behavioral targeting | localStorage, cookie | `EXP_<slice>_rtbt_events`, `EXP_<slice>_rtbt_session` |
| Marketing attribution | localStorage, cookie | `EXP_MKTG_<slice>`, `AMP_MKTG_ORIGINAL_<slice>` |
| Redirect impressions | sessionStorage, cookie | `EXP_<slice>_REDIRECT` |
| Variant and flag caches | sessionStorage | `amp-exp-*` |
| Exposure queue and dedupe | localStorage, sessionStorage | `EXP_unsent_*`, `EXP_sent_*` |
| Refusal marker | cookie | `EXP_<slice>_erased` (written on denial, survives erasure) |

## Notes

- Consent gating works with the [anti-flicker snippet](https://amplitude.com/docs/web-experiment/implementation#async-script-with-anti-flicker-snippet). Variants apply during `pending`, so pages don't flicker while the visitor decides.
- Denying Web Experiment consent doesn't opt the visitor out of Amplitude Analytics. Manage analytics consent separately with the [Browser SDK](https://amplitude.com/docs/sdks/analytics/browser/cookies-and-consent-management).
- [Preview and test](https://amplitude.com/docs/web-experiment/url-redirect-testing#preview-and-test) links work regardless of consent status.

================================================================================
# Web Experiment event tracking
URL: https://amplitude.com/docs/web-experiment/tracking
================================================================================

# Web Experiment event tracking

Web Experiment uses impression events for analysis and billing. The Web Experiment script tracks impression events through the [integration](https://amplitude.com/docs/web-experiment/implementation#integrate-with-a-third-party-customer-data-platform). Experiment analysis requires impression event tracking.

## Impression events

The impression event is the same as the Feature Experiment [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events), but has a different event type: `[Experiment] Impression`. Event properties contain the flag key and the variant of the flag or experiment that the user encountered.

When Amplitude ingests an impression event, Amplitude uses the flag key and variant to set or unset user properties on the user associated with the event. Setting user properties is essential for experiment analysis queries on primary and secondary success metrics.

### Impression transformation

Web Experiment sends impression events in one form, and Amplitude transforms them into Amplitude-standard impressions upon ingestion. Amplitude modifies the event type and event properties for consistency with other Amplitude properties. Amplitude sets or unsets experiment user properties for accurate experiment analysis. If you're tracking impressions through a 3rd party customer data platform (CDP), the CDP records the event in its pre-transformation state.

| Property Type | Pre-transformation | Post-transformation |
| --- | --- | --- |
| Event Type | `$impression` | `[Experiment] Impression` |
| Event Property | `flag_key` | `[Experiment] Flag Key` |
| Event Property | `variant` | `[Experiment] Variant` |
| Event Property | `experiment_key` | `[Experiment] Experiment Key` |

## Estimate monthly impressions

Amplitude tracks one impression event per experiment when Web Experiment applies a variant action to a page. To estimate the number of impressions per month, consider:

1. `M` = Your volume of monthly tracked users (MTU).
2. `E` = The number of experiments you run per month.
3. `P` = The average number of page views per user, per month.

Impressions Estimate = `M * E * P`

This estimate provides an upper bound. Target specific pages and audiences, or roll out to a subset of users to reduce the total number of impressions.

================================================================================
# Post-experiment steps
URL: https://amplitude.com/docs/web-experiment/post-experiment
================================================================================

# Post-experiment steps

Web experiments in Amplitude Experiment help you test hypotheses, validate ideas, and make data-informed product decisions. After you identify a winning variant, Amplitude recommends moving that variant to your production code base instead of keeping the experiment live at 100% traffic allocation.

## When the experiment concludes

When you're ready to end the experiment and select a winner:

1. **Analyze the results and confirm a winner**. Confirm the experiment reached statistical significance and that the winning variant aligns with business goals.
2. **Implement the winner in code**. Work with your engineering team to replicate the winning experience in your production code base. For more information, go to [Benefits to migrating your winning variant](#benefits-to-migrating-your-winning-variant).
3. **Deactivate or archive the experiment in Amplitude**. Disable the experiment to remove unnecessary logic and prevent accidental reactivation or analysis confusion.
4. **Document the outcome**. Capture experiment details like the goal, key learning, decision made, and implementation follow-up in an internal knowledge base.

### Activate a feature flag

If your change needs rollback capability or an incremental rollout, use a feature flag. Feature flags enable ongoing control without the overhead of experiment logic and metadata.

## Benefits to migrating your winning variant

Moving your winning variant to your production code base provides the following benefits:

- **Performance and user experience**: Running web experiments at 100% adds avoidable client-side overhead to your pages. The overhead increases page load execution time and can negatively impact performance, especially at scale. For more information about how Amplitude optimizes for performance, go to [Web Experiment Performance](https://amplitude.com/docs/web-experiment/performance).
- **Technical debt**: Long-running experiments add clutter to dashboards and experiment environments. Leaving experiments active after a decision causes unnecessary configuration overhead and increases the risk of user-facing errors.
- **Platform cost and impression volume**: Each experiment evaluation counts toward your monthly impression volume in Experiment. When you run a test at 100% after it no longer provides learning, the experiment still evaluates on each page load. Over time, these evaluations increase your costs and create budgeting inefficiencies.

================================================================================
# Proxy Web Experiment
URL: https://amplitude.com/docs/web-experiment/proxy
================================================================================

# Proxy Web Experiment

You might prefer not to load third-party experimentation scripts directly from vendor-hosted domains, or you might want to perform client-side experiment evaluation. Common reasons include security policies, privacy controls, and tighter infrastructure ownership.

A proxy-based experimentation architecture lets teams keep full control over script delivery, evaluation, and exposure tracking while using a centralized experimentation platform.

This document outlines common proxy patterns and the implementation details that support them.

## Script delivery proxy

Customers host a lightweight endpoint that proxies requests for the experimentation JavaScript SDK. End users load the script from a customer-owned domain rather than a third-party domain.

This pattern lets you:

- Reduce third-party script exposure.
- Align with internal security or compliance requirements.

### CDN configuration

- **Origin**: `https://cdn.amplitude.com`
- **Path rewrite**: `/script/{API_KEY}.experiment.js → /script/{API_KEY}.experiment.js`
- **Cache TTL**: 1–5 minutes (balances freshness with performance)
- **Cache key**: Full URL including `API_KEY`

**Before (direct to Amplitude):**

```html
<script src="https://cdn.amplitude.com/script/abc123def456.experiment.js"></script>
```

**After (through your proxy):**

```html
<script src="https://experiments.acme.com/script/abc123def456.experiment.js"></script>
```

## Remote evaluation proxy

The Web Experiment script calls a single endpoint for remote evaluation. An experiment requires remote evaluation when its targeting rules reference at least one property that isn't available locally on the client.

This pattern lets you:

- Centralize control over evaluation logic.
- Enrich requests with server-side context.

### Expose your proxy evaluation endpoint

Create and expose an HTTPS endpoint for the Web Experiment SDK to call for remote evaluation:

_Example URLs:_

- **Full path**: `https://experiments.acme.com/sdk/v2/flags?delivery_method=web`
- **Base URL**: `https://experiments.acme.com/`

The SDK uses this base URL for all evaluation requests.

### Configure the Web Experiment script

Configure the Web Experiment script or SDK to call your new endpoint instead of the Amplitude-hosted URLs (`flag.lab.amplitude.com` or `flag.lab.eu.amplitude.com`). Add the following code above the Web Experiment script.

```html
<script>
  window.experimentConfig = {
    flagsServerUrl: "https://experiments.acme.com/",
  };
</script>
```

### Implement proxy logic

For each incoming browser request, the proxy must:

_Accept request_

- **Method:** GET.
- **Query parameters:** `delivery_method=web`.

_Forward to Amplitude_

- **US projects:** `https://flag.lab.amplitude.com/sdk/v2/flags?delivery_method=web`.
- **EU projects:** `https://flag.lab.eu.amplitude.com/sdk/v2/flags?delivery_method=web`.

_Inject headers_

| Header name | Value | Notes |
| --- | --- | --- |
| Authorization | `Api-Key <AMPLITUDE_PROJECT_API_KEY>` | Store and manage only on the server. Never expose to the browser. |
| X-Amp-Exp-User | `<base64-encoded user object>` | Forward from browser as-is. Contains `{"user_id": "...", "device_id": "..."}` |

_Return response_: Return Amplitude's JSON response unchanged to the browser.

---

#### Example request flow

_Browser to proxy_:

```plaintext
GET https://experiments.acme.com/sdk/v2/flags?delivery_method=web
X-Amp-Exp-User: eyJ1c2VyX2lkIjoidXNlciIsImRldmljZV9pZCI6ImRldmljZSJ9
```

_Proxy to Amplitude_:

```plaintext
GET https://flag.lab.amplitude.com/sdk/v2/flags?delivery_method=web
Authorization: Api-Key <PROJECT_API_KEY>
X-Amp-Exp-User: eyJ1c2VyX2lkIjoidXNlciIsImRldmljZV9pZCI6ImRldmljZSJ9
```

_Proxy to browser_: Returns Amplitude's evaluation response.

#### Response format

The endpoint returns a JSON array of experiment flag objects pre-evaluated for the user.

_Flag object schema_:

| Field | Description |
| --- | --- |
| key | Unique flag key that identifies the experiment. |
| segments | Targeting and bucketing rules pre-evaluated with the user's properties. Amplitude includes conditions that are true and removes others. |
| variants | Available variants including key, metadata, and web experiment actions (mutations, custom code, URL redirects). |
| metadata | Experiment metadata: version, deployment status, and re-run indicators. |

---

#### Example response

```json
[
  {
    "key": "example",
    "metadata": {
      "deliveryMethod": "web",
      "deployed": true,
      "evaluationMode": "local",
      "experimentKey": "exp-1",
      "exposureEvent": "$impression",
      "flagType": "experiment",
      "flagVersion": 5
    },
    "segments": [
      {
        "bucket": {
          "allocations": [
            {
              "distributions": [
                { "range": [0, 21474837], "variant": "control" },
                { "range": [21474836, 42949673], "variant": "treatment" }
              ],
              "range": [0, 100]
            }
          ],
          "salt": "G0U0BTSK",
          "selector": ["context", "user", "web_exp_id"]
        },
        "conditions": [
          [
            {
              "op": "is",
              "selector": ["context", "user", "device_category"],
              "values": ["desktop"]
            }
          ]
        ],
        "metadata": { "segmentName": "Segment 1" },
        "variant": "off"
      },
      {
        "metadata": { "segmentName": "All Other Users" },
        "variant": "off"
      }
    ],
    "variants": {
      "control": {
        "key": "control",
        "payload": [{ "action": "mutate", "data": { "mutations": [] } }],
        "value": "control"
      },
      "off": { "key": "off", "metadata": { "default": true } },
      "treatment": {
        "key": "treatment",
        "payload": [
          {
            "action": "mutate",
            "data": {
              "mutations": [
                {
                  "action": "set",
                  "attribute": "html",
                  "metadata": {
                    "scope": ["89e01141-6e35-4bc0-bb66-0e1bc0fd4823"],
                    "type": "text"
                  },
                  "selector": ".md\\:max-w-max:nth-child(1)",
                  "value": "updated text"
                }
              ]
            }
          }
        ],
        "value": "treatment"
      }
    }
  }
]
```

---

## Impression (exposure) event forwarding

Customer-controlled backend systems send impression (exposure) events to the experimentation platform rather than the browser sending them directly.

This pattern lets you:

- Unify event pipelines.
- Simplify compliance with data governance policies.

If you already use Amplitude's Analytics SDK, update the tracking endpoint for the browser SDK through `serverUrl`.

If you aren't using Amplitude's Analytics SDK, add the following impression event forwarding script above the Web Experiment script.

```html
<script>
  // TODO: Replace this object with your tracker implementation.
  const customTracker = {
    track: (eventType, eventProperties) => {
      // TODO: Send eventType and eventProperties to your analytics tool.
      console.log({ eventType, eventProperties });
    },
  };

  window.experimentIntegration = {
    getUser: () => {
      // TODO: Return user.
      return {
        user_id: "user",
        device_id: "device",
      };
    },

    track: (e) => {
      // TODO: Track event
      customTracker.track(e.eventType, e.eventProperties);
      return true;
    },
  };
</script>
```

================================================================================
# Set project-level user permissions in Amplitude Experiment
URL: https://amplitude.com/docs/feature-experiment/project-level-permissions
================================================================================

# Set project-level user permissions in Amplitude Experiment

Experiment project-level permissions let Amplitude admins manage access to Experiment separately from [Analytics permissions](https://amplitude.com/docs/admin/account-management/user-roles-permissions). Use project-level permissions when you want to:

- Prevent analytics team members from releasing features through Experiment.
- Prevent product development team members from affecting data taxonomy or key dashboards and charts in Analytics.
- Allow all team members to keep higher permission levels in their primary apps.

> **Note:**
>
> Project-level user permissions in Experiment are only available to Growth or Enterprise customers.

## Set project-level user permissions in Experiment

1. In Experiment, select **Permissions**. The Experiment Permissions page opens to the Joined Users tab.
2. In the Search field, type the name or email of the user. Then select the checkbox next to the user's name. The actions above the table become selectable.
3. Select **Manage Project Access** to search for the project where you want to adjust permissions.
4. From the dropdown that shows the current permission level for the selected user, select the updated access level. Then select **Next**.
5. To confirm the changes, select **Submit**.

## Flag-level access controls

Flag-level access controls let you decide which Experiment users can change specific flags or experiments.

When you turn on flag-level access controls, users in your organization can't save changes to a restricted flag or experiment unless you designate them as an editor for that item.

### Default access for new flags and experiments

Set the default access for new flags and experiments to a restricted list of editors, or to all users in your organization. An organization-wide setting controls this default. Go to _Experiment > Permissions > Organization Settings_ to change the setting.

Only users with the admin role can change this setting.

The default makes new flags and experiments editable by all users in your organization. After you create a new flag or experiment, you can manually restrict access to that item.

If you change the default so that new flags and experiments are viewable instead of editable, only editors can change new flags and experiments. Remove this restriction after you create the flag or experiment.

If you create a flag or experiment through the [Management API](https://amplitude.com/docs/apis/experiment/experiment-management-api), the item defaults to editable regardless of the organization setting.

### Manage access to flags and experiments

To edit the list of approved editors, navigate to _\[flag or experiment\] > More Actions > Manage Access_. From this page, add individual users, or specify that all users in your organization can edit the flag.

After you grant a user editor permissions to your flag, Amplitude Experiment checks permissions and verifies that the user's role has edit access. For example, if you assign a user the viewer role and later add the user as an editor to your flag, the user can't save changes until you give the user a role with editing privileges.

Users get a notification when you add them as an editor to a flag or experiment. To control your notification settings, go to _Personal settings > Notifications > Updates about my experiments_.

### Bypass access restrictions

Use one of the following methods to change a restricted flag or experiment when no editor users are available:

1. Admin users can edit restricted flags and experiments, even when admins aren't on the list of editors.
2. Use the management API to edit all flags and experiments, regardless of the item's restricted access.

## Permissions matrix

The following tables describe the permissions included with each permission level.

> **Note:** Role-based Access Controls (RBAC)
>
> For Enterprise organizations that use Role-based Access Controls (RBAC), refer to the available [Experiment Roles and Permissions](https://amplitude.com/docs/admin/account-management/role-based-access-controls-rbac#rbac-permission-reference).

|  | Viewer | Member | Manager (Project) | Admin (Org) |
| --- | --- | --- | --- | --- |
| Deployments | Read | Read/Write | Read/Write | Read/Write |
| Activate | Read | Read/Write | Read/Write | Read/Write |
| Variants | Read | Read/Write | Read/Write | Read/Write |
| Allocation | Read | Read/Write | Read/Write | Read/Write |
| Analysis | Read | Read/Write | Read/Write | Read/Write |
| Metrics | Read | Read/Write | Read/Write | Read/Write |

| **Experiments and Flags** | Viewer | Member | Manager (Project) | Admin (Org) |
| --- | --- | --- | --- | --- |
| Read | Y | Y | Y | Y |
| Create |  | Y | Y | Y |
| Edit |  | Y | Y | Y |
| Delete |  | Y | Y | Y |

| **Deployments** | Viewer | Member | Manager (Project) | Admin (Org) |
| --- | --- | --- | --- | --- |
| Read | Y | Y | Y | Y |
| Create |  | Y | Y | Y |
| Edit |  | Y | Y | Y |
| Delete |  | Y | Y | Y |

| **Mutual Exclusion Groups** | Viewer | Member | Manager (Project) | Admin (Org) |
| --- | --- | --- | --- | --- |
| Read | Y | Y | Y | Y |
| Create |  | Y | Y | Y |
| Edit |  | Y | Y | Y |
| Delete |  | Y | Y | Y |

| **Users** | Viewer | Member | Manager (Project) | Admin (Org) |
| --- | --- | --- | --- | --- |
| Add user to a project |  |  | Y | Y |
| Edit project role |  |  | Y | Y |
| Add user to organization |  |  |  | Y |
| Edit organization role |  |  |  | Y |

| **Other** | Viewer | Member | Manager (Project) | Admin (Org) |
| --- | --- | --- | --- | --- |
| View Project API Key | Y | Y | Y | Y |

================================================================================
# Define your experiment's audience
URL: https://amplitude.com/docs/feature-experiment/workflow/define-audience
================================================================================

# Define your experiment's audience

After you define the events that make up your experiment, set up the audience eligible for the experiment, also known as targeting. You can open eligibility to all users or target specific groups.

> **Note:**
>
> Users must still trigger the exposure event before they receive the experiment. The targeting section only defines the potential audience.

Targeting groups limits experiment exposure to users in specific geographical locations, demographic groups, or users who meet certain usage thresholds in your product, such as power users. Segments are the unique characteristics of your audience you want to target, such as geolocation or device type. All Amplitude user properties and cohorts can define user segments. For any user property in a rule-based segment, Amplitude uses the most recent value the user property received. For more information on segments, go to [Segment Overview](https://amplitude.com/docs/data/destination-catalog/segment).

You can include any number of user segments. If a user belongs to more than one segment in an experiment, Amplitude assigns the user to the first segment they match.

> **Note:**
>
> Caching expiration differs based on the properties that define user segments. User properties always include the latest values your product sends, and default to the most recent value when Amplitude hasn't received new data. Cohorts sync every hour.

Defining your audience also requires that you specify your distribution and rollout:

- **Distribution**: the percentage split between the control and the experiment group. The default distribution is 50%, so half the audience receives the control and half receives the experiment. You can customize the percentage to put more users in one group. Distribution applies to your targeted audience, not your total user base.
- **Rollout**: the percentage of your targeted audience that receives the experiment. At 100%, every user in your targeted audience receives either the control or the treatment. Experiment displays the potential number of users who receive the experiment.

### Set the target audience for your experiment

1. Create a new experiment or open an existing one.
2. In the Targeting section, click the **edit** icon.
3. In the Targeting tab, select the bucketing unit you want. Bucketing is typically by `User`. You can also bucket by [groups](https://amplitude.com/docs/analytics/account-level-reporting#how-group-level-reporting-works).
4. Select the users eligible for this experiment. Targeting defaults to **All Users**. Add a segment to switch to **Targeted Users**. Targeting all users means every user who triggers the exposure event receives your experiment. Targeted users lets you specify segments of users that receive your experiment.
5. Select the bucketing you want to target. The default is `Amplitude ID`. You can also select options such as:

   - Device type.
   - Language.
   - Region.
   - Postal code.
6. To customize the distribution percentage, click **Switch to custom**, then specify the percentages you want.
7. In the All users field, specify the percentage of users that receives the experiment.
8. Click **Save & Close** or **Testers**.

After you specify your audience, specify your testers.

### Testers

Test your experiment to confirm all aspects work and that the implementation succeeds. Go to [Test and Launch Your Experiment](https://amplitude.com/docs/feature-experiment/workflow/experiment-test) for more details about testers.

### Boolean logic in segments

Amplitude applies Boolean `AND` logic to conditions within the same segment. Amplitude evaluates these conditions as `if, else if`. For example, consider two segments:

Segment 1: users in India who access your site through the web. Segment 2: users on Android devices who access your website through the mobile browser.

With Boolean logic, a user must meet both conditions in a segment to qualify. If a user accesses your website but isn't in India, the user doesn't qualify for Segment 1. If a user accesses your site through an iOS device, the user doesn't qualify for Segment 2.

The `if, else if` evaluation means Amplitude first checks whether a user meets the conditions for Segment 1. If not, Amplitude checks the conditions for Segment 2.

================================================================================
# Configure your experiment's delivery
URL: https://amplitude.com/docs/feature-experiment/workflow/configure-delivery
================================================================================

# Configure your experiment's delivery

After you design your experiment, prepare it for testing and launch. Specify the evaluation mode and bucketing unit (if you haven't already), select the deployments that host this experiment, verify your variants, and identify the team members who participate in QA testing.

Members of your engineering team should oversee this stage of the experiment process.

> **Note:**
>
> If you're launching a feature flag and not an experiment, skip the experiment design steps. You still need to complete the process in this article.

To configure your experiment's delivery, follow these steps:

1. In the Bucketing Options section on the Advanced tab, specify the [evaluation mode](https://amplitude.com/docs/feature-experiment/local-evaluation) for your experiment: _Remote_ (Amplitude servers evaluate the experiment) or _Local_.
2. In the advanced settings on the Targeting tab, specify the **bucketing unit** for this experiment.

   Set the bucketing unit to _User_ in most cases. In some B2B use cases, use company ID or city instead. For example, bucketing by company ID gives all users in a company the same experience. Confirm that the [Stable Unit Treatment Value Assumption](https://blogs.iq.harvard.edu/violations_of_s#:~:text=Methods%20for%20causal%20inference%2C%20in,treatments%20of%20others%20around%20him) holds for the unit you choose.

   When you're ready, select **Continue**.
3. On the Delivery tab, select the deployments to use for this experiment or flag. In Amplitude Experiment, a deployment serves a group of flags or experiments for code execution. To learn more about deployments, refer to [configuring your experiment](https://amplitude.com/docs/feature-experiment/workflow/configure). After you select the deployments you need, select **Continue**.
4. Verify your variants. Select **Continue** when you're ready.
5. Add the user, device, or cohort IDs of your QA testers to confirm your implementation works. Add the IDs for each variant. Assign each tester to only one variant, as if Amplitude had bucketed them into your experiment.

   When you're done, select **Save and Close**.

================================================================================
# Notifications through Slack, Microsoft Teams, or Webhook
URL: https://amplitude.com/docs/feature-experiment/notifications
================================================================================

# Notifications through Slack, Microsoft Teams, or Webhook

Set up notification alerts for your Web and Feature experiments or for individual feature flags. Amplitude offers two notification delivery paths:

- **Email or Slack direct messages**: Notifications for Experiments (Web and Feature) about to start or end, when Amplitude detects a sample-ratio mismatch (SRM), or when the experiment's recommendation metric reaches statsig.
- **Slack channels, Microsoft Teams channels, webhooks, or email**: Notifications for changes to an active flag configuration that generate a new history version, or when you activate or deactivate flags.

This page covers notification alerts delivered through Slack channels, Microsoft Teams channels, webhooks, and email. Amplitude delivers only alerts created through the Experiment Alerts function this way.

Amplitude delivers all other experiment alerts through email or Slack direct message. For more information on those notification types, go to [Account Settings Notifications](https://amplitude.com/docs/feature-experiment/workflow/experiment-learnings#enable-notifications), [Integrate Slack](https://amplitude.com/docs/analytics/integrate-slack), or [Integrate Microsoft Teams](https://amplitude.com/docs/analytics/integrate-microsoft-teams) to manage your workspace integrations.

If you use Google Chat, send an [email](https://support.google.com/chat/answer/14929313?hl=en) or [webhook](https://docs.cloud.google.com/workflows/docs/notify-google-chat).

To integrate your Amplitude flags and experiments with Sentry, refer to [Sentry's documentation](https://docs.sentry.io/organization/integrations/feature-flag/generic/).

> **Note:**
>
> You must have [Member permissions](https://amplitude.com/docs/admin/account-management/user-roles-permissions) to create, edit, or delete notification alerts. You don't need permissions to receive notifications if you're a member of the dedicated Slack channel or webhook. However, you must have Viewer permissions to open the notification for more details. Contact your Admin if you need different permissions.

## Connect a Slack workspace to your experiment notifications

1. Go to _Experiment_ and then click _Experiments_. You can also access alerts by going to _Settings > Personal settings > Experiment_ and clicking **Add alert**, or _Settings > Organization settings > Experiment_ and clicking **Add alert**.
2. Click **Alerts** in the top right.
3. Click **Connect To Slack**.
4. Click **Allow** to confirm that you want to connect Amplitude to Slack.

> **Note:**
>
> If a Slack channel ID appears instead of a Slack channel name, confirm that you connected Amplitude to your Slack workspace. If the connection exists, confirm that you're a member of the Slack channel. To request access, contact the person who created the alert (displayed in the **Created** column of the table).

## Connect Microsoft Teams to your experiment notifications

To deliver experiment alerts to a Microsoft Teams channel, your organization must connect Amplitude to Microsoft Teams. If the connection doesn't exist, the alert creation modal shows a **Connect to Microsoft Teams** action that starts the connection flow.

For setup steps, go to [Integrate Microsoft Teams with Amplitude](https://amplitude.com/docs/analytics/integrate-microsoft-teams).

## Change alerts

Change alerts notify you when flag or experiment configurations change. Amplitude generates an alert any time a change creates a new history version, or when you activate or deactivate flags.

### Experiment activities that generate alerts

Specify which experiment activities trigger alerts:

- Flags created, updated, or deleted.
- Targeting rule changes (for example, updates to target segments or bucketing settings).
- Variant changes.
- Flag activation or deactivation.

## Set up an alert

After you connect your Slack workspace or Microsoft Teams organization to Amplitude, you can create notifications for your experiments.

To send notifications through webhooks, provide the URL and a valid [signing key](https://docs.knock.app/developer-tools/outbound-webhooks/overview#verifying-the-signature) when you create the alert.

Webhook schema:

```json
{
  "flagId": number
  "flagName": string (This is the flag key and not the flag name. If you want the flag name, use `flag.name`)
  "scope": "exp_deployment" | "project" | "exp_tags"
  "scopeParam": number | undefined
  "scopeParamName": string | undefined
  "action": "created" | "deleted" | "updated"
  "modifiedBy": string
  "flag": JSON
  "oldFlag": JSON
  }
```

> **Note:** JSON Schemas
>
> View the JSON schema for the `flag` and `oldFlag` parameters in the [Experiment Management API Flag Endpoints](https://amplitude.com/docs/apis/experiment/experiment-management-api-flags#get-details) documentation.

### Create an alert

1. From the Amplitude home page, go to the _Experiment_ section and click _Experiments_. You can also access alerts by going to _Settings > Personal settings > Experiment_ and clicking **Add alert**, or _Settings > Organization settings > Experiment_ and clicking **Add alert**.
2. Click **Alerts** in the top right.
3. Specify the project.
4. Set the scope. Select one of:

   - **All In the Project**: Notifications for all experiments and flags in the project.
   - **By Deployment**: Notifications for all experiments and flags in the deployment. Specify the deployment by label. For more information, go to [Deployments](https://amplitude.com/docs/feature-experiment/data-model#deployments). Web Experiments and Guides & Surveys experiments use the Project API Key deployment.
   - **By Tag**: Notifications only for experiments and flags tagged with specific labels.
5. Choose the delivery channel:

   - For Slack, click the dropdown to choose the channel for your alerts.
   - For Microsoft Teams, click the dropdown to choose the team and channel for your alerts.
   - For a webhook, enter the URL and your signing key.
   - For email, enter an email address.
6. Name your alert.
7. Click **Create Alert**.

================================================================================
# Set the MDE for your experiment
URL: https://amplitude.com/docs/feature-experiment/experiment-theory/experiment-set-mde
================================================================================

# Set the MDE for your experiment

Before you run an experiment, set a Minimum Detectable Effect (MDE) to estimate how you measure success. Think of MDE as the minimum change you want to find by running your experiment. No standard calculation exists for the MDE, so setting one requires judgment. In Amplitude Experiment, the default MDE is 2%. Because the MDE links directly to your business needs, be thoughtful during each experiment's [design phase](https://amplitude.com/docs/feature-experiment/workflow/define-goals). When you set the MDE, consider the primary metric and any associated risks.

## MDE and the metric goal type

When you create your experiment, you select between two metric goal types: success or guardrail. The following case study shows how the goal type can change the MDE.

The marketing director of a small arts organization uses Amplitude Experiment to plan updates to a ticketing management system. With no data science team, the director decides which experiments to run and how to run them. The planned updates are:

- Add a "quick checkout" option on event pages to increase conversion from page visits to ticket sales for logged-in users.
- Add a new payment option during checkout for all users.

The goal of the first update is to increase conversion rates, so a success metric fits. The metric tells the marketing director whether the new button sits in the right place and stays visible enough to meet the conversion rate goal. The marketing director's next fiscal quarter goal is to increase ticket sale revenue by 3%. These company goals shape the success metric and set the test direction to _increase_ and the MDE to 3%.

The second update meets financial requirements. As a required change to the checkout process, a guardrail metric helps confirm that the new payment method doesn't decrease completed sales. Over the last four fiscal quarters, an average of 1% of users abandoned checkout after starting the process. The guardrail metric direction is _decrease_ and the MDE is 1%.

> **Note:**
>
> If you run a [T-test](https://amplitude.com/docs/feature-experiment/workflow/experiment-estimate-duration), Amplitude's duration estimator can also help set the MDE. Review the recommended MDE that Amplitude provides, or change the MDE until the duration estimate is reasonable.

## MDE and the primary metric

In Amplitude, the MDE is relative to the control mean of the primary metric. For example, if the conversion rate for the control group is 10%, an MDE of 2% (0.2) means Amplitude detects a change when the rate moves outside the range 9.8% to 10.2%.

In the ticketing case study, the primary metric of ticket sales may require a different MDE if:

- The hypothesis testing experiment runs during an annual discount on ticket prices.
- The number of available events, which correlates positively to ticket sales, is much smaller than previous fiscal quarters.
- The experiment runs during a global pandemic that prohibits large in-person gatherings.

Consider your business needs and circumstances when you plan an experiment and set the MDE. One goal of any experiment is to cause as little harm as possible.

> **Note:**
>
> You can also set the MDE when you analyze your experiment results.

## MDE and associated risk

Experiments don't produce risk-free results, and running them can take time and require large sample sets. This can mean higher costs and greater potential for adverse effects on users. The MDE has an inverse relationship to sample size: the smaller or more "sensitive" the MDE, the larger the sample size you need to reach statistical significance.

Use these questions to assess risk:

- Are the estimated costs or run time of an experiment worth the expected outcome?
- What are the possible negative side effects for users in the experiment, and is the outcome worth potential losses?
- Do you need an experiment at all, or should you consider other options, such as a feature release?
- What's the smallest percentage change that satisfies you? For example, would you roll out the experiment if you saw a lift of 2%, 3%, or 5%?
- If your experiment produces positive outcomes, such as an increase in annual subscribers from 100 to 105, is that change large enough to present to leadership?

## Common questions

These questions cover Amplitude Experiment's duration estimate. For setup guidance and pre-launch planning, refer to [Estimate the duration of your experiments](https://amplitude.com/docs/feature-experiment/workflow/experiment-estimate-duration).

### How does the duration estimate work?

The experiment duration estimate predicts how long your experiment needs to run to generate statistically significant results. The duration estimate works only with the primary metric and sequential testing, and doesn't support Experiment Results.

Amplitude Experiment uses the means, variances, and exposures of your control and variants to forecast expected behavior and calculate the number of days your experiment takes to reach statistical significance. The prediction improves as more data arrives. If any of these inputs change significantly during the experiment, the accuracy of the prediction is likely to decrease.

### What's the difference between the duration estimate and the duration estimator?

Amplitude calculates the duration estimate using sequential testing while the experiment runs. The duration estimator uses the [T-test](https://amplitude.com/docs/feature-experiment/workflow/experiment-estimate-duration).

### Why isn't the duration estimate showing?

The duration estimate displays when your experiment meets all the following criteria:

- The metric hasn't yet reached statistical significance.
- The end date of the analysis window is in the past.
- The experiment has enough observations.
- The experiment status is rolled out or rolled back.
- None of the following statistical conditions hold:

  - The absolute lift is outside the confidence interval.
  - The confidence interval flips (lower confidence interval > upper confidence interval). This can happen when the mean for the treatment or control fluctuates while the experiment runs, or when rollout weights or targeting segments change.
  - The standard error is very small.
  - The variance is negative.
  - The conversion rate is greater than 1 or less than 0 (where applicable).

If the estimate doesn't show, one or more of these criteria isn't met.

### Is there a cap for the duration estimate?

Yes, the cap is 40 days. The reasons:

- The duration estimate uses real-time simulations, where latency scales with the number of days simulated.
- Means and standard deviations usually don't change much over time, especially for experiments with longer running times.
- Short-term predictions are easier to make accurately than long-term predictions. (Weather forecasts beyond ten days change frequently as the date approaches for the same reason.)
- Most experiments shouldn't take 40 days to complete.

### How does Amplitude Experiment determine the number of exposures per day?

Amplitude Experiment assumes a constant number of exposures per day. Amplitude calculates this value by dividing the cumulative exposures as of today by the number of days the experiment has run so far.

### What types of errors are there?

The duration estimate is still an estimate. Don't take it as truth. You may encounter:

- Irreducible error: error inherent to the estimation process. You can't correct for it. Each simulation reaches statistical significance at a different time, which is the main reason to run multiple simulations. The time it takes for an experiment to reach statistical significance is itself a random variable. The time depends on the p-value, which depends on the data the experiment collects. Even if you know the control mean, control standard deviation, treatment mean, and treatment standard deviation, and you force a normal distribution and independence on everything, Experiment can't reduce error all the way to zero.
- Incorrect estimates: when Amplitude Experiment generates a duration estimate, Amplitude estimates the control population mean, control population standard deviation, and other quantities from the sample. These estimates are as good as they can be, but they still leave room for error.
- Drift: for example, if today the control mean is 5 and ten days from now it's 15, the control mean shows drift. A common example is seasonality. Drift in any input degrades the estimate, because the model assumes no drift during hypothesis testing.

### What does "Threshold reached" mean?

If your experiment displays "Threshold reached" with "0 days left" in the duration estimator, the confidence interval doesn't contain the MDE (the threshold).

This message isn't necessarily bad when your recommendation metric is a guardrail, because the effect size is smaller than the allowed amount.

The message is a bad sign when your recommendation metric is a success metric, because the effect size is smaller than what you hoped for. End the experiment in this case: even if you reach statistical significance, the lift is smaller than what's practically significant.

### What does "Statistical significance may never reach" mean?

When the duration estimator shows 40 or more days to complete an experiment, Amplitude may assume that the experiment isn't likely to reach statistical significance after running for two weeks. In those cases, Experiment shows this message.

================================================================================
# Set up and run mutually exclusive experiments
URL: https://amplitude.com/docs/feature-experiment/advanced-techniques/mutually-exclusive-experiments
================================================================================

# Set up and run mutually exclusive experiments

When you run several experiments at the same time, you may want to keep users who receive one experiment from seeing a second, related experiment. These experiments might solve the same problem in different ways, and you don't want to confuse users who see both. The [interaction effect](https://dictionary.apa.org/interaction-effect) can also distort your results.

Experiment lets you set two or more experiments as mutually exclusive. Mutually exclusive experiments don't share users: users who receive experiment A don't receive experiment B, and the reverse also holds.

To learn more about the underlying implementation, refer to [flag dependencies](https://amplitude.com/docs/feature-experiment/under-the-hood/flag-dependencies).

## When to use mutually exclusive experiments

Use mutually exclusive experiments in these situations:

- Simultaneous experiments that occur in the same area of your product and share the same goal.
- Simultaneous experiments that occur in the same funnel and share the same goal.

You can also run these experiments one after the other instead of simultaneously.

## Create a mutual exclusion group

Note these guidelines when you use mutual exclusion groups:

- Distribute traffic evenly between your slots. A slot is the percentage of the group that you want to receive an experiment. You can add multiple slots to your group to distribute traffic across the group.
- Don't add a running experiment to a mutual exclusion group. Adding a running experiment can severely compromise data integrity by removing users from the active experiments. Add experiments to a mutual exclusion group before they start running.
- Don't remove a running experiment from a mutual exclusion group. Removing a running experiment can compromise data integrity by exposing users to the other experiments in the group.
- Don't delete a mutual exclusion group with running experiments, for the same reason. Delete the mutual exclusion group after all experiments in the group conclude.

### To create a mutual exclusion group

1. In the Experiment feature, go to _Experiments > MutEx and Holdouts tab_.
2. If your project has no groups, click **Add a new mutual exclusion group** to create a new group. If you already have groups, click **Create Group**, then select **Mutual Exclusion Group**.
3. Enter a name and description for your group, then enter delivery settings such as evaluation mode, bucketing, and key.
4. Create the number of slots you want and assign experiments to each slot. Experiments in different slots are mutually exclusive. You can add a maximum of 20 slots to a mutual exclusion group.

> **Note:**
>
> After you create the group, you can't change the number of slots it contains or the traffic allocation percentages. This restriction ensures consistent bucketing and a consistent user experience.

5. (_Optional_) Specify individuals or cohorts to add to your mutual exclusion group from the Individuals or Cohorts tabs. This step is helpful when you want to test an experiment in a mutual exclusion group to confirm that Amplitude assigns a specific user to a specific experiment. Don't add the same users or cohorts to more than one slot, because the first slot determines inclusion.
6. Set the traffic allocation percentages for each experiment. By default, Amplitude distributes traffic evenly between them, but you can edit these percentages manually.
7. Click **Add Group**.

## Advanced use cases

- To increase traffic allocation to an experiment, change the slot percentage when you create the group. If the group already exists, assign a single experiment to multiple slots within the same group.
- Adding an experiment to multiple mutual exclusion groups further limits the experiment's traffic, because Amplitude evaluates each user for each mutual exclusion group the user belongs to.

  For example, consider the following two mutual exclusion groups:

  - Mutual exclusion group 1 has experiment A in slot 1 and experiment B in slot 2, where each experiment receives 50% of the traffic.
  - Mutual exclusion group 2 has experiment A in slot 1 and experiment C in slot 2, where each experiment receives 50% of the traffic.

  In this case, experiment A receives `0.5 * 0.5 = 0.25`, or 25% of the total traffic.

  Instead of adding an experiment to multiple mutual exclusion groups, create one group that includes all the relevant experiments. In this example, that group contains experiments A, B, and C.
- Adding an experiment to both a holdout group and a mutual exclusion group further limits the experiment's traffic, because Amplitude evaluates each user for both groups. For example, consider the following holdout group and mutual exclusion group:

  - The holdout group contains experiment A, with a holdout percentage of 5%.
  - The mutual exclusion group contains experiment A in slot 1 and experiment B in slot 2, where each experiment receives 50% of the traffic.

  In this case, experiment A receives `0.95 * 0.5 = 0.475`, or 47.5% of the total traffic.

To learn more, refer to [working with holdout groups in Amplitude Experiment](https://amplitude.com/docs/feature-experiment/advanced-techniques/holdout-groups-exclude-users).

## Common questions

> **Note:**
>
> You can only add experiments running with local evaluation to mutual-exclusion groups running with local evaluation.
>
> Mutual exclusion groups can contain a mix of web experiments and feature experiments.

### Is it best to create a mutual-exclusion group before launching an experiment?

Yes. Avoid configuration changes that affect targeting after the experiment starts. If you add an active experiment to a mutual-exclusion group, Amplitude Experiment may reassign users.

### How do the slot % allocations affect user targeting?

When you create mutual-exclusion groups, you set an **allocation percentage**: the probability that Amplitude assigns a user to an experiment. For example, an allocation percentage of 25% means users have a 25% chance of joining the experiment. Allocation percentages can total 100% or less. If less than 100%, unused traffic receives the "off" treatment of the experiment.

### When does Amplitude Experiment apply mutual exclusion?

Amplitude Experiment follows a specific order of operations when assigning users to experiments:

`individual user qualification → mutual exclusion → sticky bucketing → target segment`

Because Amplitude Experiment evaluates individual user qualification _before_ mutual-exclusion groups, users targeted under **Individual Users** can see multiple experiments even when those experiments share a mutual-exclusion group. The same rule applies to individual inclusions for cohorts: Amplitude Experiment also applies them before mutual-exclusion groups.

### Why does a user see more than one experiment when mutual exclusion is configured?

The same user can see more than one experiment, even when the experiments use mutual exclusion. Possible reasons:

- The order of operations described above (individual user qualification runs before mutual exclusion).
- How [Amplitude tracks unique users](https://amplitude.com/docs/data/sources/instrument-track-unique-users). For example, a user might anonymously use more than one device before logging in. Until Amplitude identifies and merges the user into the existing ID, Amplitude treats the session as a different user available for assignment.
- [Variant jumping](https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping), where a user sees two or more variants for a single flag or experiment.

================================================================================
# Holdout groups
URL: https://amplitude.com/docs/feature-experiment/advanced-techniques/holdout-groups-exclude-users
================================================================================

# Holdout groups

Sometimes you need to prevent a percentage of users from viewing an experiment. This matters most when you measure the long-term, combined effects of multiple experiments. Statistical significance in one experiment may not reflect the true, cumulative impact of your experimentation program.

To exclude users from your experiments, create a holdout group. Holdout groups measure the long-term impact of your rolled-out variants and the lift of your experimentation program as a whole. A holdout group retains a set of users from previous experiments and shows them the winning combinations of past results. This group then gives you true measures of success for your previous experiments.

Your holdout groups experience previous experiments. You can set up dependencies that control which experiments the holdout group sees. For more information, go to [Flag dependencies](https://amplitude.com/docs/feature-experiment/under-the-hood/flag-dependencies#holdout-groups).

## Create a holdout group

> **Note:**
>
> Holdout groups can contain a mix of web experiments and feature experiments.

When you use holdout groups, note the following:

- **Set the holdout percentage between 1% and 10%**: Withholding a significant part of your total traffic can extend the time your experiments need to reach a conclusion.
- **Add experiments to a holdout group before they start running**: Adding running experiments to a holdout group can compromise the integrity of your data because Amplitude unassigns users from the active experiments.
- **Don't remove a running experiment from a holdout group**: Removing a running experiment compromises the integrity of your data because Amplitude assigns users to the active experiments.
- **Don't delete a holdout group with running experiments**: To protect your running experiment's data, delete the holdout group after all experiments in the group conclude.

##### To create a holdout group

1. In the Experiment functionality, go to _Experiments > Mutex and Holdouts tab_.
2. To add a new holdout group to your project, click **Add a new holdout group**.

   - If you have existing groups, click **Create A New Group**, and then select **Holdout Group**.
3. In the Holdout group settings modal, enter the name, description, and holdout percentage for the group. You can also view and change advanced settings such as the evaluation mode and bucketing key of your group.

> **Note:**
>
> You can't change the holdout percentage after you create a group. This restriction ensures consistent bucketing and a consistent user experience.

4. Click **Add Experiment** to add experiments to your holdout group.
5. (_Optional_) Specify individuals or cohorts to include in or exclude from your holdout group. From either the Individuals or Cohorts tabs, add a user or cohort under _Include in holdout_ or _Exclude from holdout_. This option ensures that Amplitude always holds out or never holds out specific users from the holdout group.

Don't add the same users or cohorts to both the _Include a holdout_ and _Exclude from holdout_ slots. The _Include a holdout_ slot determines inclusion.

6. Click **Add Group** to finish the process.

## Manage holdout groups

Manage your holdout groups from the Experiment Groups tab or from within an experiment:

1. In the Mutex and Holdout tab, scroll down the table until you find the group you want to edit.
2. Click the **edit** icon.
3. Make your changes in the Holdout group settings modal and click **Save**.

If you're within an experiment that's part of a holdout group, follow these steps:

1. Click the name of the group you want to edit.
2. Make your changes in the Holdout group settings modal and click **Save**.

## Analyze a holdout group

Analyze your holdout groups using an Experiment Results chart.

##### To create a pre-populated Experiment Results chart

1. Navigate to the Experiments page and click the **Mutex and Holdout** tab.
2. Find the holdout group you want to analyze and click the **Analyze** icon.
3. Click **Open in Chart**.
4. A new Experiment Results chart opens, with the following fields complete:

   - Exposure event
   - Segments for `Holdout` and `On`
   - Statistical method set to [T-test](https://amplitude.com/docs/feature-experiment/experiment-theory/analyze-with-t-test) (_Samples per variant needed_ set to 10,000)
   - Analysis date range
5. From here, select the primary metric and start analyzing the impact of your holdout group.

## Advanced use cases

### Streamline multiple experiments and holdout groups

Adding an experiment to multiple holdout groups can limit the experiment's traffic. Experiment evaluates each user for each holdout group they belong to.

For example, you have the following two holdout groups:

- **Holdout group 1**: This group contains Experiment A and Experiment B, with a holdout percentage of 5%.
- **Holdout group 2**: The second group contains Experiment A and Experiment C, also with a holdout percentage of 5%.

Because Experiment A is part of both holdout groups (1 and 2), it receives the majority of the total traffic, or `0.95 * 0.95 = 0.9025` (90.25%).

Instead of adding an experiment to multiple holdout groups, create a single group with all the relevant experiments. A single group distributes traffic more evenly across experiments.

In the example above, you create one holdout group that contains all three experiments (A, B, and C).

### Manage experiments with holdout groups and mutual exclusion

Adding an experiment to a holdout group and a mutual exclusion group further limits the amount of traffic to the experiment. Experiment evaluates each user for both the holdout group and the mutual exclusion group.

For example, consider the following holdout group and mutual exclusion group:

- **The holdout group** has a holdout percentage of 5% and contains Experiment A.
- **The mutual exclusion group** directs half the traffic to Experiment A in slot 1, and the other half to Experiment B in slot 2.

In this scenario, Experiment A receives about half of the total traffic, or `0.95 * 0.5 = 0.475` (47.5%).

You can use holdout groups with mutual exclusion, but watch for the potential traffic limits as you plan and roll out your experiments.

For more information, refer to [mutual exclusion groups](https://amplitude.com/docs/feature-experiment/advanced-techniques/mutually-exclusive-experiments).

================================================================================
# Estimate the duration of your experiments
URL: https://amplitude.com/docs/feature-experiment/workflow/experiment-estimate-duration
================================================================================

# Estimate the duration of your experiments

The Duration Estimator helps you determine which experiment ideas are viable before you start building. Use the Duration Estimator to avoid running tests that may never reach statistical significance, and to prioritize experiments that can deliver results in a reasonable timeframe.

> **Note:**
>
> The Duration Estimator supports T-tests. Sequential testing, Bayesian methods, and multi-armed bandit methods aren't available in this workflow.

## Open the duration estimator

In your experiment setup, select **Estimate Duration** to open the Duration Estimator.

When you first open the Duration Estimator, you see an empty state. After you add your traffic event and success metric, the Duration Estimator calculates how long your test needs to run.

## Set up your estimate

### Step 1: Add your traffic event

Select **+ Add Event**, and choose the event that represents traffic where you run your experiment.

For example, if you test your homepage, select `Page Viewed`, and add a filter for your homepage URL.

The Duration Estimator pulls the last 29-30 days of traffic data from Analytics, and shows Users per day in the results panel.

If you don't have the right event, select **Enter Manually** to input your own total daily traffic estimate. Traffic is total traffic, not for each variant.

### Step 2: Add your success metric

Select **+ Add Metric**, and choose the conversion metric you want to improve with this experiment.

A success metric is the visitor action you're trying to change with your experiment. Think about what you want more visitors to do because of your changes.

Common success metrics:

- Conversions: Visitor completes a key action, such as signed up, purchased, enrolled, or subscribed.
- Form completions: Visitor submits a form or completes a flow.

How to choose:

- Ask: `What action do you want more visitors to take?`
- Look for metrics that match that action.
- If you see similar metrics, choose a conversion metric (which Amplitude marks as `Conversion of...`) or a metric with an official blue badge.

> **Tip:**
>
> Choose a metric your team uses often.

For example, if you test your homepage hero banner and want more visitors to enroll in a course, select `Conversion of registration: course enrolled`.

The Duration Estimator calculates your current conversion rate from the last 29-30 days of Analytics data, and shows the rate in the results panel (for example, `78.8% -> 82.8%`).

If you don't see the metric you need:

- Search with the search bar at the top of the dropdown.
- Select **Create Metric** at the bottom to build a new one.
- Select **Enter Manually** to input your own baseline conversion rate.

### Step 3: Set your minimum detectable effect (MDE)

The relative MDE is the smallest improvement you want to detect. The default is 5%, which means you test whether you can improve your baseline by 5%.

For example, if your baseline conversion is 78.8% and you set a 5% MDE, you test whether you can reach 82.8%.

How to think about MDE:

- Big, bold changes (like redesigning a hero banner above the fold): expect a large lift, such as 8%.
- Small, subtle changes (like changing button text below the fold): expect a small lift, such as 2%.
- Smaller MDEs require much longer test durations.

If you don't have historical data, select **Enter Manually** to input your own baseline conversion rate.

## Understand your results

After you add your traffic and success metric, the **Estimated Duration** panel shows:

- Duration: How many days you need to run the test (for example, `~130 days`).
- Users per day: Daily traffic the Duration Estimator pulls from Analytics.
- Lift: Your baseline conversion rate to target conversion rate with your MDE percentage.
- Summary: Plain-language explanation you can share with stakeholders.

If your duration is very long, you see a **Long Duration** warning badge. Use the Duration Scenarios table to explore different scenarios.

## Use the Duration Scenarios table to prioritize

The Duration Scenarios table is the most important part of the Duration Estimator. The table shows how your choices affect test duration, so you can make better decisions about what to test and when.

### How to read the table

Rows (confidence level):

- Low (85%): Less certainty your results are real, but faster results.
- Medium (90%): Balanced approach (default setting).
- High (95%): More certainty your results are real, but takes longer.
- Custom %: Enter your own confidence level.

Columns (lift size/MDE):

- 2%: Small, subtle changes (takes longest to detect).
- 5%: Medium-sized changes (default setting).
- 8%: Large, bold changes (fastest to detect).
- Custom %: Enter your own MDE.

The table highlights your selected combination and shows durations for all other scenarios.

### How to think about confidence level

Your confidence level is the risk you're willing to take with your results. Choose based on what's at stake.

95% confidence: Use when the cost of being wrong is high.

- Revenue-critical tests (checkout flows, pricing, and subscriptions).
- High-impact placements (homepage hero, above-the-fold content, and navigation).
- Sensitive or costly bets (brand-new features, compliance-heavy areas, and high-cost builds).

90% confidence: Use when you want balance between speed and reliability (default).

- Medium-stakes decisions where time matters, but the cost of being wrong is manageable.
- Engagement-focused outcomes (click-throughs, mid-funnel steps).
- Iterative improvements in areas with prior evidence.

85% confidence: Use when you need a directional signal.

- Early validation (MVPs, prototypes you follow up on).
- Low-stakes tests (low-traffic pages, below-the-fold changes).
- Well-understood areas where a topline read is enough.

### How to think about MDE (lift size)

MDE reflects the expected impact of your experiment idea. Ask: `How much lift do I realistically expect this change to drive?`

Large MDE (8%+): Use for bold changes with dramatic impact.

- Prominent new CTAs at the top of the homepage.
- Major redesigns of key flows.
- Revenue-driving promotions.
- Because the effect is big, the test resolves quickly.

Medium MDE (3-5%): Use for meaningful but not dramatic improvements.

- UX enhancements.
- Layout adjustments.
- Copy changes.
- This is the most balanced choice for everyday experimentation.

Small MDE (1-2%): Use for subtle tweaks, or when tiny gains are valuable.

- Microcopy changes.
- Slight color adjustments.
- Incremental funnel optimizations.
- These require the most time and traffic, but can add up in mature, high-volume products.

### Use the Duration Scenarios table for prioritization

#### Scenario 1: Your test takes too long

If your estimate shows `~130 days` at 5% MDE and 90% confidence, review the table:

- At 8% MDE (larger change), duration drops to `~51 days`.
- At 85% confidence (lower certainty), duration drops to `~102 days`.

Decision framework:

- Can you test a bigger, bolder idea to get results faster?
- Are the stakes low enough to justify 85% confidence for a quicker read?
- Or is this a high-stakes test where 90-95% confidence is worth the wait?

#### Scenario 2: Compare multiple test ideas

You have three test ideas in your backlog:

- Homepage hero redesign (expected 8% lift): `~51 days` at 90% confidence.
- CTA button text change (expected 5% lift): `~130 days` at 90% confidence.
- Footer link color change (expected 2% lift): `~632 days` at 90% confidence.

Decision: The hero redesign is viable and can deliver results quickly. The CTA change may be worth running if you lower to 85% confidence (`~102 days`). The footer change takes over a year, so it isn't worth testing now.

#### Scenario 3: Balance your testing portfolio

Use the Duration Scenarios table to create a balanced mix:

- High-impact tests (8% MDE, 90-95% confidence): One to two major tests for each quarter that resolve in two to four weeks.
- Everyday optimizations (3-5% MDE, 90% confidence): Regular tests that deliver steady improvements in three to six weeks.
- Quick validation (5-8% MDE, 85% confidence): Fast directional reads on new ideas before heavier investment.

#### Scenario 4: Low-traffic pages

If you test a low-traffic page and durations are very long across all scenarios, you may need to:

- Test on a higher-traffic page.
- Wait until you accumulate more traffic.
- Test something with a larger expected impact.

The Duration Scenarios table makes these trade-offs visible, so you can prioritize experiments that fit your traffic and timeline constraints.

## Adjust advanced settings (optional)

Select **Advanced Settings** to access additional controls:

- **Confidence level**: Low (85%), medium (90%), or high (95%).
- **Statistical power**: Probability of detecting a true effect (default 80%).
- **Rollout**: Percentage of visitors you expose to the experiment (default 100%).
- **Number of variants**: Total variants including control.
- **Distribution**: How traffic splits between variants (default evenly).
- **Statistical method**: T-test.

Most teams don't need to adjust these settings. The defaults work well for standard A/B tests.

## Tips for reducing test duration

If your estimated duration is longer than your timeline allows, use these options.

### Test a bigger idea (increase MDE)

The biggest factor in test duration is the size of the change you want to detect. Larger changes produce larger lifts and resolve faster.

For example, moving from 5% MDE to 8% MDE can reduce duration from `~130 days` to `~51 days`.

Ask:

- Can you test a more impactful variation instead of a subtle tweak?
- Instead of changing button color, can you redesign the entire CTA section?
- Instead of tweaking microcopy, can you rewrite the entire headline?

Large-impact ideas resolve faster. Small-impact ideas take longer, but can add up in mature, high-volume products.

### Lower your confidence level (when stakes allow)

Dropping from 90% to 85% confidence reduces duration, but increases false-positive risk (calling a winner when there isn't one).

For example, at 85% confidence, the same 5% MDE test takes `~102 days` instead of `~130 days`.

Ask:

- What's the cost of being wrong?
- Is this a low-stakes test (below-the-fold change, well-understood area)?
- Can you validate results with a follow-up test if needed?

Don't lower confidence for:

- Revenue-critical tests.
- High-impact placements.
- Brand-new features, or unknown customer segments.

### Choose a higher-traffic page or event

Low traffic is a common reason tests take too long.

Ask: Can you run this test on a higher-traffic page, or choose a more frequent conversion event?

### Increase rollout percentage

If you only expose 50% of visitors to the experiment, increasing to 100% can reduce duration by about half.

### Reduce number of variants

Testing four variations takes much longer than testing two. Consider multiple sequential tests instead of one large multi-variant test.

### Decide if the test is worth running

Sometimes a test isn't feasible. If the Duration Scenarios table shows hundreds of days across all scenarios, the test probably isn't worth building.

## Common mistakes to avoid

- Defaulting to 95% confidence every time. This makes sense for high-stakes tests, but can slow low-stakes experiments.
- Chasing only small lifts. Looking for 1-2% MDE improvements can require large traffic and long run times.
- Skipping a duration check. Even with the right settings, some experiments can't reach significance with available traffic.

The Duration Estimator helps you make this call before you spend time and resources on a test that may never reach significance.

## Common questions

### Why does my estimate say **Long Duration**

Your test takes a long time to reach statistical significance, often because of low traffic or small MDE. Use the Duration Scenarios table to explore faster alternatives.

### What if I don't have 30 days of historical data

Update the timeframe, or select **Enter Manually** to input your own traffic and conversion estimates. Results are most accurate with at least a few weeks of stable data.

### Can I change the MDE after I see the estimate

Yes. Adjust the MDE percentage in the success metric section, and the estimate updates automatically. Use the MDE control to explore different scenarios before committing to your test design.

### What does `Last 29 days offset by 1` mean

This label shows the data timeframe the Duration Estimator uses for calculations. `Offset by 1` means the calculation excludes today because today's data is incomplete, and looks at the previous 29 complete days.

### Should I always aim for high (95%) confidence

No. Many experiments run at medium (90%) confidence, which balances speed and accuracy. Use high confidence when stakes are high, or when you need maximum certainty before a decision.

### How do I choose a success metric

Start by asking `What action do you want more visitors to take?` Then choose a metric that matches that action. Conversion metrics (which Amplitude marks as `Conversion of...`) are often the best choice.

If you're still unsure, search for metrics related to your goal, or select **Enter Manually** to input your own baseline.

### Where can I learn about live experiment duration estimates

Review [Experiment duration estimates](https://amplitude.com/docs/feature-experiment/experiment-theory/experiment-set-mde#common-questions) to understand the duration estimate that Experiment shows while an experiment runs.

================================================================================
# Create a feature flag
URL: https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts
================================================================================

# Create a feature flag

In Amplitude Experiment, a flag enables or disables a function or feature in your product without redeploying code. Flags drive both experiments and feature rollouts. Use flags to launch experiments and end them after you collect enough data, or to roll out new features and roll them back quickly when needed.

Amplitude offers this feature to users on all plans.

This article explains how to create a flag for a feature rollout. For information on how to use flags in your experiments, refer to [rolling out your experiment to your users](https://amplitude.com/docs/feature-experiment/workflow/experiment-test).

> **Tip:** Migrate your flags from Optimizely
>
> Migrate your flags from Optimizely into Amplitude. Contact your Amplitude representative or [Amplitude Support](https://gethelp.amplitude.com) to start the process.

## Create a new flag

Before you create a flag, create a deployment and either install the [SDK](https://amplitude.com/docs/sdks/experiment-sdks) or configure the [evaluation REST API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api). Then follow these steps:

1. Navigate to _Experiment > Feature Flags_ in the left sidebar and click **Create A Feature Flag**.
2. In the Create Flag modal, choose the project for this flag from the **Projects** drop-down menu. Enter a name for your flag. Experiment generates the flag key from the name. The flag key identifies the flag in your codebase.
3. Specify the [evaluation mode](https://amplitude.com/docs/feature-experiment/local-evaluation) for your experiment: **Remote** (Amplitude evaluates the flag on Amplitude's servers) or **Local**. Then specify the **bucketing unit** for this experiment.

> **Tip:**
>
> The best bucketing unit is usually the user. In some B2B use cases, you might use company ID or city as the bucketing unit. For example, bucketing by company ID ensures that all users in a company have the same experience. Confirm the [Stable Unit Treatment Value Assumption](https://blogs.iq.harvard.edu/violations_of_s#:~:text=Methods%20for%20causal%20inference%2C%20in,treatments%20of%20others%20around%20him) holds for whichever unit you choose.

4. Click **Create**. Experiment opens a blank template for your flag.
5. Choose the deployment for your experiment from the **Deployment** drop-down menu. For more information about deployments, refer to [configuring Amplitude Experiment](https://amplitude.com/docs/feature-experiment/workflow/configure).
6. Click **Advanced Settings** to change the bucketing salt.

> **Note:**
>
> Changing the bucketing salt can cause users to switch between variants in your experiment. Amplitude recommends that you don't change the bucketing salt without guidance. For more information, refer to [How randomization works in Amplitude Experiment](https://amplitude.com/docs/feature-experiment/under-the-hood/experiment-randomization).

7. In the Settings section, click the **Add** icon in the Variants section to add a variant. Flags must have at least one variant. A variant is the new feature or product experience you roll out to users. You can add as many variants as you need to a feature flag.
8. Enter a name, value, and description for your variant. Amplitude Experiment generates the variant value from the name. The variant value is a string you use as a flag in your codebase. Click **Apply**.

> **Note:**
>
> Don't name your variants **OFF**. Amplitude Experiment reserves this name for fallbacks (for example, the user segment not included in your experiments).

## Roll out a new feature

After you create the flag, use it to roll out a new feature.

1. In the Assignment panel, define the user segments that receive your new feature and specify the rollout percentage.

   Define a user segment to limit your rollout to users in a specific geographic location, users in certain demographic groups, or users who meet certain usage thresholds (for example, power users).

   To define a user segment, scroll to the Rule Based User Segments section and click **Segment 1**. Follow the same steps you use to build a user segment in Amplitude Analytics.

   All Amplitude user properties and cohorts are available for defining user segments. You can include any number of user segments.
2. Set the **rollout percentage** for this feature. This is the percentage of users in the flag's user segments who receive the new feature. To give everyone in the user segment access, set this value to 100%.
3. Set the percentage for each variant to define how many users receive each one. Percentages must sum to 100%. For example, if you set variant A to 20% and variant B to 80%, four times as many users receive variant B as receive variant A.
4. Set separate rules for users not covered by any segment you created. For example, to limit the rollout to the cohorts you targeted, scroll to the All Other Users section and set the rollout percentage to **zero**.
5. Save your flag and QA it before you set it to **Active**. For more information, refer to [QAing before launching an experiment](https://amplitude.com/docs/feature-experiment/workflow/experiment-test). Your feature is live for the user segments you selected.

================================================================================
# Create a new experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/create
================================================================================

# Create a new experiment

The decisions you make in the [design](https://amplitude.com/docs/feature-experiment/workflow/define-goals) phase define your experiment's success. Define your experiment's purpose and goals before you start to gain useful, actionable insights.

For example, you want to run a hypothesis testing experiment with a direction setting of "increase" and a minimum goal (MDE) of 2%. This setting means you expect the metric to increase by at least 2%. If you change the experiment type to _Do No Harm_, you expect the metric to "not increase by 2%". A good use case for a Do No Harm experiment is launching a service agreement in your app and then testing for a lack of change in user retention.

To create a new experiment, [install an SDK](https://amplitude.com/docs/sdks/experiment-sdks) or call the [evaluation API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api).

Before you launch, estimate run time and viability with the [Duration Estimator](https://amplitude.com/docs/feature-experiment/workflow/experiment-estimate-duration).

## Create a feature experiment

1. Go to _Create > Experiment_, and select **Feature**.
2. Complete the following fields:

   - **Name**: enter the name of the experiment for future reference.
   - **Project**: select the project in which this experiment operates.
   - **Experiment type**: select from the following:

     - **A/B test**: test one or more variants with a goal of improving a metric. Run A/B tests using hypothesis testing or do-no-harm methodologies. For more information, refer to [Define your experiment's goals](https://amplitude.com/docs/feature-experiment/workflow/define-goals).
     - **[Multi-armed bandit](https://amplitude.com/docs/feature-experiment/workflow/multi-armed-bandit-experiments)**: Amplitude allocates an increasing amount of traffic to the winning variant, based on the primary metric, until it reaches 100% allocation.
   - For web experiments, enter the _Targeted Page URL_ where this experiment runs.
3. Optionally, complete the following fields:

   - **Key**: keys are unique to experiments and identify which experiments a user participates in. You can edit keys until you run the experiment.
   - **Evaluation mode**: select whether the experiment runs locally or on Amplitude's Experiment servers. For more information, refer to [Local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation) and [Remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation).
   - **Bucketing unit**: select the unit Amplitude uses to assign variants, either `User` or `Group`.
4. Select **Create**.

After Amplitude creates the experiment, configure additional aspects of the experiment. Go to [Define your experiment's goals](https://amplitude.com/docs/feature-experiment/workflow/define-goals) for next steps.

================================================================================
# Define your experiment's goals
URL: https://amplitude.com/docs/feature-experiment/workflow/define-goals
================================================================================

# Define your experiment's goals

An experiment can't tell you anything without metrics to track against. Add metrics to your experiment in the Goals section of the experiment design panel.

Define your primary metric and any secondary metrics.

A primary metric determines whether to accept or reject your hypothesis, and whether your experiment succeeded or failed. Choose the right primary metric to evaluate experiment success. If you're new to A/B testing, follow these guidelines to choose a primary metric:

- Identify the single user action that tells you if your [variant](https://amplitude.com/docs/feature-experiment/workflow/add-variants) is successful.
- Measure an event that the change in your variant directly affects.
- Pick an event that fully captures the user behavior you're trying to affect.

Experiment supports multiple metrics for each experiment. Secondary metrics aren't required, but they improve the quality of your analysis and help you evaluate whether to roll out the experiment.

> **Tip:** Revenue metrics
>
> A common mistake is defaulting to a revenue metric when your variant changes something unrelated to revenue. If your variant changes how your product page looks and functions, choose a metric on that page as your primary metric instead of a revenue metric that may sit several steps down the funnel.

### Set up metrics for your experiment

1. Open an existing experiment or [create an experiment](https://amplitude.com/docs/feature-experiment/workflow/create), then scroll to the Metrics section and select the **edit** icon.
2. Select **Add metric**, then choose the metric you want from the drop-down list. Alternatively, select **Create a custom metric** to define your own.
3. Specify whether the metric **should** or **should not** **Increase** or **Decrease**, and by what percentage.
4. (_Optional_) For primary metrics, set the minimally acceptable goal. This is the smallest relative distance between the control and the variant needed to determine experiment success or failure.
5. To add secondary metrics, select **Add Metric** and repeat the process.

After you add your metrics, set your [variants](https://amplitude.com/docs/feature-experiment/workflow/add-variants).

### Examples of success and guardrail metrics

**Success metrics** measure the primary outcomes you want to improve:

- **Conversion metrics:** Purchase completion rate, sign-up conversion, add-to-cart rate.
- **Engagement metrics:** Daily active users, average session duration, feature adoption rate.
- **Revenue metrics:** Average order value, revenue for each user, subscription upgrades.
- **Retention metrics:** Day 7 retention rate, return user rate.

**Guardrail metrics** monitor important metrics that shouldn't degrade during the experiment:

- **Performance metrics:** Page load time, API response time, app crash rate.
- **Quality metrics:** Error rate, failed transaction rate, support ticket volume.
- **Core engagement:** Usage of key features unrelated to the experiment, overall session count.
- **Business health:** Subscription cancellation rate, refund rate, negative review rate.

For example, when testing a new checkout flow, your success metric might be "Purchase completion rate (Increase)," and your guardrail metrics could include "Checkout page load time (No increase)" and "Payment error rate (No increase)."

### Duration estimator

The duration estimator calculates the time and sample size you need to achieve significant results, based on your metric settings. Amplitude Experiment pre-populates industry defaults from historical data. You can adjust the confidence level, statistical power, minimum detectable effect, standard deviation, and test type.

## Create a custom metric

Create a new metric if no standard metric meets your needs.

### Steps to create a custom metric

1. Select **Create a custom metric**.
2. Name the metric and add a description.
3. Define the metric's type. A metric can be one of the following types:

   - Unique conversions
   - Event totals
   - Formula
   - Funnel conversions
   - Return on retention
   - Sum of property value
   - Average of property value.
4. Set the events you want by selecting **Add Event**, then choose your events.
5. Set any key properties you want.
6. Select **Save and Close**.

> **Note:**
>
> By default, the Retention metric doesn't support [CUPED](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences), exposure attribution settings, or calendar day windows. The Retention metric calculates exposure attribution settings using any exposure and the nth day value based on 24-hour window increments, for up to two months.

## Define the exposure event

In your experiment, open the Design Experiment panel or the Analysis Settings and choose the **exposure event**. When a user triggers this event, Amplitude Experiment buckets them into the experiment. The Amplitude exposure event is the most accurate and reliable way to track user exposures to your experiment's variants, so use it when possible.

Amplitude sends the `Amplitude exposure` event when your app calls `.variant()`. The event sets the user properties Amplitude Experiment uses for its analyses. When you use the Amplitude exposure event, the event triggers at the correct time.

You can select a custom exposure event instead. Select **Custom Exposure**, then **Select event**. Custom exposure events carry a greater risk of triggering at the wrong time, which can cause a [sample ratio mismatch](https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch).

For more information, refer to the [exposure events](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) article.

## Use Aggregated Metrics in experiments

In addition to event-based metrics, you can use [Aggregated Metrics (fka Warehouse Metrics)](https://amplitude.com/docs/data/warehouse-metrics) as goals in your experiments. Aggregated Metrics are precomputed metrics imported directly from your data warehouse into Amplitude, which keeps your source of truth consistent with your experimental analysis.

Aggregated Metrics are valuable when your experiment goals involve business metrics that are difficult to calculate from behavioral events alone, such as:

- **Revenue and financial metrics**: Average order value, credits remaining, or subscription revenue that require calculations across multiple data sources.
- **Customer health metrics**: Customer lifetime value (LTV), health scores, or churn risk predictions modeled in your data warehouse.
- **State metrics**: Current subscription tier, activation status, or account-level attributes that track user state rather than discrete events.

When you add metrics to your experiment, Aggregated Metrics appear in the metrics picker with a warehouse icon. Use them as primary or secondary metrics, just like event-based metrics. Amplitude displays when each Aggregated Metric last synced and when the next sync is scheduled, so you can confirm your experiment results reflect the most current data.

For more information about setting up and using Aggregated Metrics, refer to the [Aggregated Metrics (fka Warehouse Metrics) Overview](https://amplitude.com/docs/data/warehouse-metrics).

================================================================================
# Exposure tracking in Amplitude Experiment
URL: https://amplitude.com/docs/feature-experiment/track-exposure
Updated: 2024-08-28
================================================================================

# Exposure tracking in Amplitude Experiment

When running an experiment, track each user's [exposure](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) to your feature flag's variant experience. Without exposure tracking, results aren't reliable.

> **Note:**
>
> Exposure tracking is optional for feature flags that don't require analysis.

## Analytics REST API

In this example, the [Analytics REST API v2.0](https://amplitude.com/docs/apis/analytics/http-v2) sends an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) to Amplitude with `curl`.

When the request succeeds, a user appears in the Exposures chart in Amplitude Experiment.

The flag is now active in your deployment, and your experiment evaluated a user and served them the variant.

## SDKs

As with fetching variants, you can simplify exposure tracking using a client-side [Experiment SDK](https://amplitude.com/docs/sdks/experiment-sdks) in your app. Client-side Amplitude Experiment SDKs [automatically track exposures](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#automatic-exposure-tracking) through your installed analytics SDK whenever the Experiment SDK accesses a variant from the variant store.

================================================================================
# Add variants to your experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/add-variants
================================================================================

# Add variants to your experiment

Your experiment needs at least one variant. Experiment compares a variant experience with the control experience, which is usually your product's current user experience (UX). Experiment measures the performance of the variant against a known quantity: the performance of your app today.

## Add and manage variants

Experiment creates your initial variant automatically. By default, this variant uses the name `treatment`. You can rename the variant to describe the experiment you're running.

##### To add more variants

1. In the Experiment section, create a new experiment or open an existing one.
2. Click **Add a variant**.
3. Enter a name for your variant and include a value. By default, the value is a hyphenated (slug) version of the variant name.

> **Tip:** Use variant value in your code
>
> When you implement the experiment in your codebase, use the value of the variant to reference it. Experiment SDKs return variant values in lower case with no spaces.

4. Add a description of the variant. Be specific so other people can understand what your experiment does.
5. Add an optional **payload**. A payload is a JSON object that changes a variant's experience dynamically without requiring more code.

   For example, if you're testing a new splash screen on a marketing webpage, early results might suggest different content could improve performance. Instead of editing your codebase, include the changes in a payload, and Experiment applies them automatically.

   Paste or type your code into the window.
6. Click **Apply**.

You can add any number of variants to an experiment, but adding too many can [make it harder for your experiment to reach statistical significance](https://amplitude.com/docs/feature-experiment/advanced-techniques/bonferroni-correction). Keep your experiments to the minimum number of variants needed.

You can drag and drop your variants in any order. The variant with the `control` label is always the control variant, regardless of its position in the list.

## Distribute traffic to your variants

Unless you specify otherwise, Experiment splits traffic evenly between your variants. To send more traffic to specific variations, customize your variant distribution. In your experiment, go to _Targeting > Distribution_ to change the distribution percentage.

## Stratified sampling and experiment bias

To spread traffic differently for each user segment you've included:

- Segment 1: `Country = USA || 80% treatment, 20% control`.
- Segment 2: `Country = Canada || 50% treatment, 50% control`.

This approach can introduce bias into your experiment results. In general, use uniform allocation ratios across all user segments in an experiment.

Non-uniform allocation ratios often happen inadvertently when users change their rollouts and variants while an experiment runs.

Amplitude Experiment offers stratified sampling (non-uniform allocation ratios) if you need it. Switch the Allow rollout controls per segment toggle to **On**. This option appears only if you've selected [**Targeted Users**](https://amplitude.com/docs/feature-experiment/workflow/define-audience).

This switch appears only for experiments, not for feature flags. Amplitude disables the switch while your experiment is active.

Click **Continue** to move to the rollout phase.

## Rollout percentage

The next step sets the rollout percentage for this experiment. The rollout percentage is the percentage of users in the experiment's user segments who take part in the experiment. Find the rollout percentage in the Rollout section of the experiment design panel.

Enter the percentage of your audience eligible for bucketing into the experiment. If you roll your experiment out to less than 100% of your users, the remaining users see your default product experience and aren't included in any experiment calculations.

Experiment evaluates users in rule-based user segments before users not covered by a user segment. Experiment evaluates individual user or device IDs before both.

Next, [finalize your experiment's statistical settings](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences).

================================================================================
# Cohort Targeting
URL: https://amplitude.com/docs/feature-experiment/cohort-targeting
================================================================================

# Cohort Targeting

A cohort is a static or dynamic set of users defined in Amplitude. For experiment use cases, cohorts are useful for advanced audience targeting. Cohorts aren't always the best solution for targeting, so it's important to understand how cohort targeting works with [local](https://amplitude.com/docs/feature-experiment/local-evaluation) or [remote](https://amplitude.com/docs/feature-experiment/remote-evaluation) evaluation.

Experiment cohort targeting only supports targeting user cohorts.

## Remote evaluation

When you target a cohort in a remote evaluation flag, Amplitude syncs the cohort to the Amplitude Experiment destination. For dynamic cohorts, this sync runs hourly by default. Dynamic cohorts targeted in remote evaluation don't update in real time. For example, if you target a cohort of users who performed a `Sign Up` event, Amplitude targets those users within an hour of the event.

Cohorts targeted for remote evaluation can have a propagation delay on the initial sync or after a large change, depending on the size of the difference. For example, the first sync of a 10-million-user cohort takes longer than later syncs.

Remote evaluation supports cohorts up to 10 million users.

**Remote evaluation cohort targeting use cases**

- You're targeting users based on user behavior or properties that aren't available in Experiment targeting segments.
- The targeting delay from cohort sync intervals doesn't interrupt your process.

Don't use remote evaluation cohort targeting if:

- You need to target users in real time.

## Local evaluation

Local evaluation flags and experiments deployed to up-to-date server-side SDKs can also target cohorts. When you target a cohort in a local evaluation flag, Amplitude syncs the cohort to the Experiment Local Evaluation destination. For dynamic cohorts, this sync runs hourly. Dynamic cohorts targeted in local evaluation don't update in real time. For example, if you target a cohort of users who performed a `Sign Up` event, Amplitude targets those users within an hour of the event, not immediately after.

Local evaluation supports cohorts up to 50 million users.

> **Note:** Cohorts only support User IDs
>
> Local evaluation cohorts only sync user IDs to the SDKs. To target cohorts in local evaluation flags, include a user ID in the user object passed to the evaluate function.

### SDK support

Server-side SDKs can target cohorts when you configure them to do so. Client-side SDKs don't support local evaluation cohort targeting.

On initialization, configure the cohort sync configuration with the project API and secret key to enable local evaluation cohort downloading and targeting.

| SDK | Cohort Targeting | Version |
| --- | --- | --- |
| [Node.js](https://amplitude.com/docs/sdks/experiment-sdks/experiment-node-js) | ✅ | `1.13.4+` |
| [Ruby](https://amplitude.com/docs/sdks/experiment-sdks/experiment-ruby) | ✅ | `1.5.0+` |
| [JVM](https://amplitude.com/docs/sdks/experiment-sdks/experiment-jvm) | ✅ | `1.4.0+` |
| [Go](https://amplitude.com/docs/sdks/experiment-sdks/experiment-go) | ✅ | `1.7.0+` |
| [Python](https://amplitude.com/docs/sdks/experiment-sdks/experiment-python) | ✅ | `1.4.0+` |
| [PHP](https://amplitude.com/docs/sdks/experiment-sdks/experiment-php) | ❌ | N/A |

#### Node.js

```js
const experiment = Experiment.initializeLocal('DEPLOYMENT_KEY', {
  // (Recommended) Enable local evaluation cohort targeting.
  cohortSyncConfig: {
    apiKey: 'API_KEY',
    secretKey: 'SECRET_KEY'
  }
});
```

#### Java

```java
// (1) Initialize the local evaluation client with a server deployment key.
LocalEvaluationClient experiment = Experiment.initializeLocal("<DEPLOYMENT_KEY>",
    // (Recommended) Enable local evaluation cohort targeting.
    LocalEvaluationConfig.builder()
        .cohortSyncConfig(new CohortSyncConfig("<API_KEY>", "<SECRET_KEY>"))
        .build());
```

#### Golang

```go
client := local.Initialize("DEPLOYMENT_KEY", &local.Config{
  // (Recommended) Enable local evaluation cohort targeting.
  CohortSyncConfig: &local.CohortSyncConfig {
    ApiKey: "API_KEY",
    SecretKey: "SECRET_KEY"
  }
})
```

#### Python

```python
experiment = Experiment.initialize_local("DEPLOYMENT_KEY", LocalEvaluationConfig(
  # (Recommended) Enable local evaluation cohort targeting.
  cohort_sync_config=CohortSyncConfig(api_key="API_KEY", secret_key="SECRET_KEY")
))
```

#### Ruby

```ruby
experiment = AmplitudeExperiment.initialize_local('DEPLOYMENT_KEY',
  # (Recommended) Enable local evaluation cohort targeting.
  AmplitudeExperiment::LocalEvaluationConfig.new(
    cohort_sync_config: AmplitudeExperiment::CohortSyncConfig.new(
      api_key: 'API_KEY',
      secret_key: 'SECRET_KEY'
    )
  )
)
```

## Troubleshooting

Cohort targeting can be challenging to troubleshoot because dynamic cohorts and cohort syncs are asynchronous. If your experiment isn't targeting users who should be in the targeted cohort, try the following:

- For local evaluation, check that the SDK version supports local evaluation cohort targeting, and that you set the cohort sync config on initialization.
- Check that the cohort has the required sync (_Amplitude Experiment_ for remote evaluation, _Experiment Local Evaluation_ for local evaluation).
- Check that the cohort contains the expected user. If the user is in the current cohort, check the sync history of the cohort. The user may have been added to the cohort in a sync after the evaluation occurred.
- Check that the user info passed to `fetch` or `evaluate` is correct.

================================================================================
# Managing flags and experiments with approvals
URL: https://amplitude.com/docs/feature-experiment/managing-flags-and-experiments-with-approvals
================================================================================

# Managing flags and experiments with approvals

Improve the governance of your experimentation program and reduce the risk of unintended changes. Require approvals for critical changes to experiment configuration.

Experiment Approvals is only available to Growth and Enterprise customers.

## Configure approvals

To turn on approvals, go to _Organization Settings > Experiment > Approvals_.

Only users with manager or admin roles can modify approvals settings.

Add the individual projects that require approvals.

For each project, specify one of the following options:

1. **Peer approvals**: any user with access can respond to pending approval requests.
2. **Specific approvers**: only designated users can respond to pending approval requests.

Admin users can also respond to pending approvals.

## Request and respond to approval requests

### Require approvals to activate flags and experiments

After you enable approvals for a project, Amplitude recommends this approval process:

1. When you start or schedule an experiment, or activate or schedule a feature flag, select one or more approvers to notify. Include the scheduled start time, stop time, or both in that request.
2. The experiment shows a "Pending Approval" status until an approver approves the request.
3. Approvers review the experiment and either approve or reject the requested changes.
4. Amplitude notifies the original requestor of the response.

While the approval is pending, users can make additional changes to the flag or experiment configuration, enter or exit testing mode, or cancel the request at any time.

After a scheduled flag or experiment receives approval, Amplitude starts it at the scheduled start time and stops delivery at the scheduled stop time, if you set one. After Amplitude starts it, later critical changes still require approval.

### Require approvals for critical changes to live flags and experiments

When a flag or experiment is active, the following updates also require approval:

| Field | Type of change |
| --- | --- |
| Target segments | Adding and removing segments, modifying conditions or bucketing. |
| Variants | Any changes, including adding, renaming, or removing variants. |
| Variant distribution | Any changes. |
| Exposure event | Any changes. |
| Bucketing salt | Any changes. |
| Sticky bucketing | Enabling or disabling. |

When reviewing these approval requests, users can view the full list of changes in the approval banner.

When an approval is pending, Amplitude locks the flag or experiment. The lock prevents other users with access from making changes until the approval request completes or cancels.

================================================================================
# Stale flag management
URL: https://amplitude.com/docs/feature-experiment/stale-flag-management
================================================================================

# Stale flag management

Stale flag management helps you maintain a clean codebase by identifying feature flags that Amplitude fully rolled out or rolled back for a set period.

## How flags and experiments become stale

A feature flag or experiment becomes stale when it meets one of these conditions:

- **Rolled out**: Amplitude rolled out the flag or experiment to 100% of users for more than 30 days.
- **Rolled back**: The flag or experiment is inactive or set to 0% rollout for more than 30 days.

After a flag or experiment becomes stale, you can remove the flag from your codebase and archive it in Amplitude.

## Viewing stale flags and experiments

### Stale status badge

In the list of your flags or experiments, a status badge appears next to each stale flag. Hover over the stale badge to view when the flag became stale and update the stale date if needed.

### Filtering in the table view

In the Feature Flags or Experiments list, the stale status appears as a secondary status badge next to the primary status. Select **Stale** to filter the table to show only stale flags.

## Working with stale flags

Hover over the Stale badge on a flag to open a pop-up that shows when the flag became stale and its rollout status. From the pop-up, you can archive the flag or unmark the flag as stale.

### Default stale date

By default, Amplitude marks a flag or experiment stale 30 days after the last update. For example, if you set a flag's rollout to 100% on March 1 and make no other updates, Amplitude marks the flag stale on March 31. If you deselect the stale value, Amplitude treats the flag as permanent and doesn't mark the flag stale again.

### Updating stale dates

If you change the rollout after you set a stale date, you can update the stale date. For example:

- On January 3, you roll out a flag to 100% and set the stale date to February 3.
- On January 5, you add a segment with 0% rollout to the flag.
- When you save, Amplitude prompts you to update the stale date.

## Notifications

You can receive notifications when flags or experiments become stale. If you own a flag or experiment, Amplitude sends you a daily message listing any flag or experiment that became stale in the previous 24 hours.

You can receive notifications through:

- Email.
- Slack.

Configure [notification preferences](https://amplitude.com/docs/feature-experiment/notifications) in your organization settings.

## Bulk archiving stale flags and experiments

After a flag or experiment becomes stale, remove it from your list:

1. **Remove the flag from your codebase**: Remove the feature flag code from your application. If you deployed the flag across multiple services or codebases, remove it from each location.
2. **Archive the flag or experiment in Amplitude**: Archive the flag or experiment in Amplitude to remove it from your active flags list.

##### To remove a stale item from Amplitude

1. Select the flag or experiment you want to remove.
2. Click **Archive**.

### Handling multiple variants

If a flag or experiment has multiple variants and one variant has a 100% rollout, keep that variant in your application and remove the others. This step keeps the winning variant active.

## Using Amplitude AI or MCP to manage stale flags

You can use Amplitude AI or Amplitude MCP to highlight stale flags and either update the stale date or archive the flag. In the agent window, ask Amplitude to generate a list of your stale flags. After Amplitude returns the list, continue the conversation to archive or modify your stale flags.

================================================================================
# URL redirect testing
URL: https://amplitude.com/docs/web-experiment/url-redirect-testing
================================================================================

# URL redirect testing

Marketers use A/B testing to compare messaging, calls to action, and landing pages so they can increase conversions and improve user experience. A/B testing often requires developer support, which isn't always available.

Amplitude's URL redirect testing feature lets you design, deploy, and analyze A/B tests that redirect visitors to another URL, without extensive developer involvement. URL redirect testing helps you measure whether a redirect improves conversion or user experience.

> **Note:**
>
> The [Website Conversion Agent](https://amplitude.com/docs/amplitude-ai/website-conversion-agent) can analyze your pages, propose conversion optimization strategies, and create draft experiments for you automatically.

URL redirect testing works well if you're building different versions of your page or site on a CMS like WordPress. In these cases, your different URLs and their associated pages are the variants that Amplitude Experiment tracks.

You can use URL redirect testing with both standard A/B tests and [multi-armed bandits](https://amplitude.com/docs/feature-experiment/workflow/multi-armed-bandit-experiments).

To use URL redirect testing, you must [implement](https://amplitude.com/docs/web-experiment/implementation) the Web Experiment script on your site. Add the script to the `<head>` section of your site. Install the Amplitude Analytics SDK on your site for event tracking.

## Set up URL redirect testing

To set up a URL redirect test:

1. In Amplitude Experiment, navigate to the Experiments page and click **Create Experiment** and then click **Web**.
2. In the New Experiment modal, name your experiment.
3. Enter the URL of a page this experiment targets and select the appropriate project from the drop-down. Amplitude uses this URL to create your first [Page](https://amplitude.com/docs/web-experiment/pages). You must instrument Web Experiment on this URL for the experiment to work.

   If the script is present on the page you specified, Experiment opens the page in the [Visual Editor](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor) as a new variant in your experiment.
4. Click the Treatment **three-dot** menu item, select **Edit**, and then under Action select **URL Redirect**.
5. In the URL Redirect panel, add each URL you want to test as a separate variant and click **Apply**.
6. Click **Apply and Exit** to leave the editing view.
7. Configure which [Pages](https://amplitude.com/docs/web-experiment/pages) your experiment should target. You can create new Pages or reuse existing saved Pages.
8. Target the users you want to include in this experiment. Go to [audience targeting](https://amplitude.com/docs/web-experiment/targeting#audience-targeting) for more information. Note that Web Experiment audience targeting works differently than Feature Experimentation.
9. Define your experiment's [Metrics](https://amplitude.com/docs/feature-experiment/workflow/define-goals).
10. Specify any [additional options](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences) in the Advanced tab.
11. Click **Save and Close** to finish creating your Web Experiment.

## Preview and test

Before running your web experiment, test and preview each variant.

To test your web experiment:

1. Click **Test & Preview**. This puts your experiment in test instrumentation mode, but it doesn't begin the experiment. Only users who open the page with the preview link experience your changes.
2. In the modal, click **Preview** to open a new tab that applies the changes you made for that variant.
3. Click the chain link icon to copy the URL to share it with others.

Test each variant at least one time, testing on more than one page if your experiment targets multiple pages.

If your changes aren't visible, you may need to wait up to 60 seconds for caches to refresh. If the changes don't appear correctly after that time, check your configuration for possible issues.

> **Warning:** Ad blockers
>
> Ad blocking plugins or extensions may prevent you from testing and previewing your experiment.

## Redirect impression tracking across domains

When a redirect stays on the same hostname, the Web Experiment script stores the redirect impression in `sessionStorage` and tracks it after the destination page loads. This requires no configuration.

When a redirect targets a different hostname, for example `www.example.com` to `app.example.com` or a different domain entirely, `sessionStorage` doesn't carry over. The script supports two transports for the impression data, which you configure with `redirectConfig` on the `window.experimentConfig` object. Set the config above the Web Experiment script tag:

```html
<script>
  window.experimentConfig = {
    redirectConfig: {
      encodeRedirectInCookie: true,
      encodeRedirectInUrl: false,
    },
  };
</script>
<!-- Web Experiment script tag -->
```

| Option | Default | Behavior |
| --- | --- | --- |
| `encodeRedirectInCookie` | `true` | Stores redirect impression data in a cookie scoped to the root domain (`SameSite=Lax`) so the destination page can track the impression. Works across subdomains of the same root domain, for example `www.example.com` to `app.example.com`. The cookie expires after one minute. Cookies don't cross root domains, so this option doesn't cover redirects to a different site. |
| `encodeRedirectInUrl` | `false` | Base64-encodes the redirect impression data and appends it as an `AMP_REDIRECT` query parameter on the destination URL. The script on the destination page tracks the impression and removes the parameter from the URL. Use this option for redirects to a different root domain, or when the browser blocks cookies. |

Both options apply only to redirects that change hostname. If you enable both, the destination page prefers the URL parameter, then falls back to the cookie.

> **Note:** Script required on the destination
>
> For cross-domain impression tracking to work, the destination page must run the Web Experiment script for the same project. The impression fires when the destination page loads.

> **Warning:** Visible query parameter
>
> With `encodeRedirectInUrl` enabled, users briefly notice the `AMP_REDIRECT` parameter in the destination URL before the script removes it. Make sure the destination page tolerates unknown query parameters.

## Configuration limits

Visual experimentation and Amplitude's low-code implementation apply the following limits on experiment configuration:

- **Evaluation mode**: Limited to _local_ to optimize test performance and minimize latency impact to end-users.
- **Bucketing Unit**: Limited to _User_ since evaluation mode is limited to _local_.
- **Keys**: Limited to _deviceID_ since evaluation mode is limited to _local_.
- **Audience**: Limited to _all users_, since redirect logic triggers before tracking loads on the site.
- **Deployment**: Limited to the project API key to simplify setup requirements.

## Related resources

For more information about URL redirects in Web Experiment, refer to [Web Experiment actions](https://amplitude.com/docs/web-experiment/actions#url-redirect).

================================================================================
# Test and launch your experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/experiment-test
================================================================================

# Test and launch your experiment

Before any users view your experiment, confirm that the variants look and function as intended. Because Experiment lets you assign specific variants by user ID, device ID, or cohort, you can confirm that Amplitude serves test devices the correct variants when they enter your experiment.

To exclude users from an experiment or flag, add them to the `OFF` variant.

On the Overview page for your experiment, review the Overview, Delivery, Variants, and Targeting sections. Confirm each section matches your plan.

Click **Test Instrumentation** to send the experiment's variants to the testers you designated when you [configured the experiment's audience](https://amplitude.com/docs/feature-experiment/workflow/define-audience).

> **Note:** Test Instrumentation and targeting
>
> When you test your instrumentation, Amplitude ignores the target segments you configured in the experiment. Test instrumentation sends variants only to the Testers.

## Launch your experiment

When you're satisfied that your experiment works as intended, click **Start Experiment** to start delivering variants immediately. To start later, stop later, or both, expand **Start Experiment** and click **Schedule**.

You can set an analysis end date or accept the default Experiment analysis range. The analysis end date controls which dates appear in results. It doesn't stop variant delivery. To stop delivery at a set time, [schedule a stop](#schedule-your-experiment).

After your experiment runs, you can [make a decision on your experiment](https://amplitude.com/docs/feature-experiment/workflow/make-decision-experiment) when it reaches statistical significance or its analysis end date.

This scheduling flow also applies to [Web Experiments](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment).

## Schedule your experiment

Schedule a start, a stop, or both so Amplitude begins and ends variant delivery at the times you choose.

Use a scheduled stop when you want Amplitude to stop assigning new users to variants at a future time. Use the analysis end date when you want to bound the results window. Those settings are independent.

1. Expand **Start Experiment** and click **Schedule**.
2. Set a start time, an end time, or both. Schedule a start at least one hour in advance. The end time must be after the start time, and both times must be in the future.
3. Click **Schedule**.

If the experiment is already scheduled or running, click **Edit Schedule** to change or remove the start or stop times.

When you schedule a stop and the analysis end date is empty, Amplitude fills the analysis end date with the stop time. If you already set an analysis end date, Amplitude leaves it unchanged. You can still edit the analysis date range after you schedule a stop, including dates after the stop time.

After the stop time, Amplitude stops assigning new users to variants. The experiment shows **Completed (Pending decision)** until you [make a decision](https://amplitude.com/docs/feature-experiment/workflow/make-decision-experiment).

For a feature flag, expand **Activate flag** and click **Schedule**. Amplitude turns the flag on at the start time and off at the end time.

A scheduled stop doesn't rebalance mutual exclusion or holdout groups. After the experiment auto-stops, its mutual exclusion slot stays unused, and a holdout carve-out continues to hold users back from the experiment.

> **Note:** Scheduled times and variant delivery
>
> Amplitude exposes users to variants as soon as the scheduled time arrives. Users who requested a variant in the hour before the start time may continue to experience the pre-start response until their cached response expires, up to one hour later. A scheduled stop has the same cache window: users who received a variant in the hour before the stop time may continue to experience that variant until the cache expires.
>
> Unlike a manual flag or experiment update, a scheduled start or stop doesn't trigger CDN cache invalidation, so the existing 60-minute cache TTL applies to those earlier requests. For details, refer to [Cache invalidation](https://amplitude.com/docs/feature-experiment/under-the-hood/performance-and-caching#cache-invalidation).
>
> For example, for an experiment that starts at 2:00 PM, a user who visited at 1:30 PM continues to experience the pre-start response until 2:30 PM, when their cached response expires. A user who arrives at 2:00 PM experiences a variant right away.

## QA after rollout

After rollout, you can track how many users Amplitude exposed to each variant on a daily basis.

Go to _Experiments > your experiment > Activity tab > Diagnostics_ to view how many users Amplitude exposed to each variant.

This view is a useful way to QA the assignment process. If one variant enrolls significantly more or significantly fewer users than expected, the difference may indicate an issue to investigate.

If you spot outliers or anomalies that concern you, click into the chart or information to investigate the potential causes. To learn more about understanding anomalies, refer to this article on [Root Cause Analysis](https://amplitude.com/docs/analytics/root-cause-analysis).

For a deeper validation of your experiment's instrumentation and assignment logic, [run an A/A test](https://amplitude.com/docs/feature-experiment/aa-testing) before launching a full A/B test.

================================================================================
# Make a decision on your experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/make-decision-experiment
================================================================================

# Make a decision on your experiment

After your experiment reaches statistical significance, its analysis end date, or a scheduled stop, decide what to do next. This article covers how to end your experiment and the options available.

## End your experiment

End the experiment when you reach its analysis end date or when it reaches statistical significance. To end it, click **Complete Experiment**.

If you scheduled a stop, Amplitude already stopped assigning new users to variants when the stop time passed. The experiment shows **Completed (Pending decision)**. Click **Complete Experiment** to roll out, roll back, or continue.

Then choose one of the following:

- **Roll out** the winning variant.
- **Roll back** everything and return to a pre-experiment state.
- **Continue** the experiment.

You can revisit this decision later.

## What happens when your experiment ends

When you complete your experiment, Amplitude stops collecting new data for analysis. The experiment stays in your experiment list but no longer assigns new users to variants. Historical data and analysis stay available for review.

If you roll out or roll back, Amplitude applies the changes you select. If you continue the experiment, it stays active until the new end date you specify.

### Roll out variants

When you roll out the winning variant to all users, Amplitude:

- Sets the rollout percentage to 100%.
- Changes the distribution weight to 100 for the winning variant and 0 for all other variants.
- Disables [sticky bucketing](https://amplitude.com/docs/feature-experiment/advanced-techniques/sticky-bucketing) (sets it to `false`).

If you roll out to a custom percentage of users, Amplitude doesn't apply these automatic changes. You must apply changes manually after confirming your rollout decision.

You can also roll out to "only the targeted users." This option helps avoid overgeneralizing your experiment's results. For example, you target users in the United States and find a 5% improvement. That result doesn't mean a rollout to all users produces a 5% lift outside the United States. You might experience a 5% lift for U.S. users but a -2% lift for all other users.

### Roll back an experiment

When you roll back your experiment, Amplitude:

- Turns the flag off.
- Sets percentage rollouts to 0%.

### Continue your experiment

To continue running your experiment, enter a new end date to gather more data and click **Start Experiment**.

To restart your experiment with fresh results (for example, after instrumentation issues affected data quality), [create a new experiment run](https://amplitude.com/docs/feature-experiment/troubleshooting/new-experiment-run). A new run excludes previous user data from monitoring and analysis.

## Clean up feature flags

After you decide on your experiment, clean up your feature flags to avoid unnecessary overhead and confusion.

### Deactivate or archive your experiment

After you roll out, roll back, or continue your experiment, deactivate or archive it in Amplitude. Archiving removes unnecessary logic and prevents accidental reactivation or analysis confusion.

To deactivate or archive your experiment:

1. Navigate to your experiment in Amplitude Experiment.
2. Click the menu next to **Complete Experiment** or **Turn off flag**.
3. Select **Archive** to archive the experiment.

Archived experiments stay in your Experiment List with an archived label. You can unarchive them when needed.

### Remove experiment code

When you roll out the winning variant, work with your engineering team to implement the winning experience directly in your production code base. Implementing the variant in code removes the need to keep the experiment active with 100% traffic allocation.

After your engineering team implements the chosen variant in code, remove the experiment logic from your codebase. Removing the logic reduces technical debt and improves performance.

> **Tip:** Use feature flags for ongoing control
>
> If your change needs rollback capability or an incremental rollout, use a [feature flag](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts) instead of keeping the experiment active. Feature flags provide ongoing control without the overhead of experiment logic and metadata.

================================================================================
# Run an A/A test
URL: https://amplitude.com/docs/feature-experiment/aa-testing
================================================================================

# Run an A/A test

A/A testing is a quality assurance technique where you run an experiment with two identical variations to validate your experimentation setup. Unlike an [A/B test](https://amplitude.com/docs/feature-experiment/key-terms), an A/A test splits traffic between two groups that get the same experience.

## Why run A/A tests

A/A tests help you verify that your experimentation infrastructure works correctly before you run real experiments. A/A tests are valuable when:

- **Setting up experimentation for the first time.** Validate that your SDK integration, traffic allocation, and metric tracking function properly.
- **Implementing new metrics.** Confirm that new success metrics behave as expected and don't show false positives.
- **Diagnosing unexpected results.** Rule out implementation issues when an A/B test shows surprising outcomes.
- **Testing at scale.** Verify that your randomization works correctly across large traffic volumes.

## Common issues an A/A test detects

A/A tests can reveal several implementation problems:

- **Improper randomization.** Amplitude doesn't evenly distribute users between variants, which creates [sample ratio mismatches](https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch), or users jump between variants and get different experiences across sessions or page loads. For more on causes and fixes, refer to [Variant jumping](https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping).
- **Tracking inconsistencies.** Events fire differently between variants because of timing issues or conditional logic errors.
- **Sample pollution.** Pre-exposure bias occurs when users get content before Amplitude allocates them to a variant.
- **Statistical configuration.** Incorrect significance thresholds or multiple testing without proper correction.

## When to run A/A tests

A/A tests are diagnostic tools, not routine practice. Run them strategically when you need to validate your setup.

### Good reasons to run an A/A test

- **Initial setup.** First time implementing Amplitude Experiment. Verify that everything works correctly.
- **Major infrastructure changes.** After you update your SDK version, change tracking implementation, or migrate to a new platform.
- **Troubleshooting unexpected results.** An A/B test shows surprising outcomes and you suspect technical issues rather than real user behavior differences.
- **New team members.** When you onboard engineers to experimentation, an A/A test provides hands-on validation of the implementation.

### Don't run A/A tests for

- Routine experiments when you've already validated your setup.
- Every new experiment. Running an A/A test for every experiment wastes traffic and delays real insights.
- Validation without a specific concern.

## Run an A/A test in Web Experiment

### Custom code approach

1. Create a new experiment in Web Experiment.
2. Set up two variants (control and treatment) with a 50/50 traffic split.
3. In both variants, apply identical changes:

   - **Option 1:** Add a `console.log` statement through custom code: `console.log('A/A test variant');`.
   - **Option 2:** Insert an HTML comment that doesn't affect the user experience: \`\`.
4. Configure your success metrics (the conversion events you want to validate).
5. Run the experiment for at least one full business cycle.
6. Analyze results to confirm no significant differences between variants.

### Visual Editor approach

1. Create a new experiment in Web Experiment.
2. Set up two variants (control and treatment) with a 50/50 traffic split.
3. For each variant, open the Visual Editor:

   1. Select the variant to open the Visual Editor on your site.
   2. Use the element selector to select any element on the page (for example, a header, button, or div).
   3. Select **More**, then find the element selector field in the right panel.
   4. Change the selector to an invalid one that doesn't match anything on your page (for example, `#nonexistent-element-aa-test`).
4. Configure your success metrics (the conversion events you want to validate).
5. Run the experiment for at least one full business cycle.
6. Analyze results to confirm no significant differences between variants.

This approach ensures the experiment loads and tracks properly without changing anything visible to users.

## Run an A/A test in Feature Experiment

1. Create a new flag-based experiment with two variants.
2. Ensure both variants return the same feature flag value or configuration.
3. Implement the flag in your code, but keep the experience identical regardless of variant.
4. Track the same conversion events for both groups.
5. Monitor the experiment dashboard for several days to validate proper traffic distribution and metric tracking.

## Interpret your A/A test results

Expect random variation in A/A tests. The key indicator of a healthy A/A test is no statistically significant difference between variants.

### Successful A/A test

No statistically significant difference exists between variants, Amplitude splits traffic evenly, and metrics are comparable. You can proceed with confidence to run real A/B tests.

### Failed A/A test

Any statistically significant result indicates a potential implementation issue that needs investigation. If you see significant differences, check:

- Your traffic allocation percentages in the experiment dashboard.
- Event tracking implementation for conditional logic that might affect variants differently.
- That the experiment fires before any page content that might influence results.
- That Amplitude calculates your metrics consistently across both variants.

## Best practices

- **Run A/A tests during representative traffic periods.** Avoid holidays or unusual traffic patterns that might skew results.
- **Use the same sample size you'd use for real experiments.** Matching sample size validates your statistical power calculations.
- **Test your most important metrics.** Focus on the conversion events that matter most to your business.
- **Document your setup.** A/A tests serve as baseline validation you can reference when you troubleshoot future experiments.
- **Don't over-interpret noise.** Expect random variation. Focus on statistical significance rather than the size of percentage differences between variants.
- **Expect some variant jumping.** A small percentage of users may get different variants across sessions. Variant jumping is normal and shouldn't cause statistically significant differences in a properly configured test.

## When to skip A/A tests

A/A tests are valuable validation tools, but you don't need to run them for every experiment. Skip A/A tests when:

- You have a mature experimentation program with proven infrastructure.
- You're running similar experiments to ones you've validated before.
- Your last A/A test was recent and showed clean results.
- Time-to-insight is critical and you have high confidence in your setup.

If you get unexpected results in your A/B tests or make significant changes to your implementation, an A/A test can quickly rule out technical issues and save debugging time.

================================================================================
# Dimensional Analysis
URL: https://amplitude.com/docs/feature-experiment/dimensional-analysis
================================================================================

# Dimensional Analysis

To exclude QA users or internal traffic that doesn't represent your customer base and may skew results, use Dimensional Analysis. Amplitude's Dimensional Analysis capabilities let you exclude defined user groups from analysis on a per-experiment basis.

## Define your testers

In Feature Experiment, use the _Inclusions_ section to define your test users.

Add users by `User ID` or `Device ID`, or by using a cohort. Amplitude assigns each test user a variant, and that variant determines the experience when the user interacts with the experiment.

## Filter test users

To remove QA users or internal traffic from analysis, filter them out before reviewing results.

Select the _All exposed users_ dropdown and choose _Exclude testers_.

If you select multiple targeting segments, analyze each segment individually, because one segment may show a lift that others don't. Select the segment name in the _All Users_ dropdown. Selecting a segment also filters test users from your analysis.

Investigating specific user segments can reveal additional insight. Experiments that aren't statistically significant overall can contain a small group of users for which the result is statistically significant. Likewise, for statistically significant results, small user segments can drive the performance of the experiment.

================================================================================
# Advanced metric use cases
URL: https://amplitude.com/docs/feature-experiment/advanced-techniques/advanced-metric-use-cases
================================================================================

# Advanced metric use cases

This article covers advanced use cases you may encounter when analyzing experiment results.

## Case 1: Create a funnel analysis based on your experiment's metrics

Imagine a conversion funnel with five steps, where step three represents the exposure event for your experiment. To reduce noise and increase the likelihood of reaching statistical significance, Amplitude Experiment counts metric events only after the exposure event. If the exposure event is step three of the funnel, and you include the whole funnel as a metric, the funnel conversion count is zero. To measure the actual conversion rate of your funnel, make steps three through five a standalone metric in your experiment.

You may need further analysis of your experiment's conversion rates in a funnel analysis.

To use your experiment's metrics in a [Funnel Analysis chart](https://amplitude.com/docs/analytics/charts/funnel-analysis/funnel-analysis-get-the-most), follow these steps:

1. Add the events for your funnel analysis in the Events module.
2. In the _Measured as_ module, choose the **Conversion** time window, then specify the counting method (unique users or totals).
3. Select your analysis unit or group type (for example, **Any Users**) in the Segment By module.
4. Create a user segment for each variant of your experiment.
5. Select **Performed** to add filters with your experiment's flag key and variant. If you restarted your experiment, add an experiment key filter.
6. Set the date range for _any time since_ to match the start date of your experiment.

The Funnel Analysis chart results may differ slightly from your experiment results. Funnel analyses and experiments handle users who [variant jump](https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping) differently.

For example, a funnel analysis includes all users who meet the filter requirements. The funnel analysis then computes the conversion rate of the funnel based on those filtered users. The funnel analysis may include a user even if the user reached the exposure event after completing the funnel.

### Analyze your experiment data using other Amplitude Analytics metrics

Amplitude Analytics offers metrics that Amplitude Experiment doesn't. Use the steps in the previous section to analyze time to convert or [return on or after retention](https://amplitude.com/docs/analytics/charts/retention-analysis/retention-analysis-build).

Refer to this [Help Center article on funnel analysis FAQs](https://help.amplitude.com/hc/en-us/articles/360054203872) for more details.

## Case 2: Analyze your experiment's results based on a subset of users

Imagine your experiment targets all users, but you want to examine the experiment's effect on a subset of users, such as exposed users in the United States only. Adding a filter on the country property doesn't generate the results you expect.

When you create a metric, Amplitude computes that metric on all exposed users. If you add a filter for users in the United States to the metric event, the numerator includes the filter but the denominator doesn't.

To filter a subset of users in your experiment results, follow these steps:

1. In your experiment, go to _Activity > Analysis > external link icon_.
2. In the Variants Performed By section, select **Filter by** to add a filter for the Country property.

> **Note:**
>
> This method filters both the numerator and the denominator of the mean values, so you can correctly analyze the target subset of users exposed to your experiment.

Avoid analyzing your experiment's results based on just one subset. You may encounter a false positive when looking for true statistically significant results.

When you run a [multiple hypothesis test](https://amplitude.com/docs/feature-experiment/advanced-techniques/bonferroni-correction) in this situation, you run a separate hypothesis test for each segment. You may find a positive lift with one subset and a negative decline with another subset. The decision to roll out or roll back in these situations isn't always clear. One option is to roll out only to the group that shows positive lift.

## Case 3: Threshold metrics

Sometimes, you want to define success as a user doing an event multiple times. For example, your users must make a purchase three (3) consecutive times to count as a conversion. To achieve this, create a funnel counting by uniques with three (3) purchase events.

================================================================================
# Learn from your experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/experiment-learnings
================================================================================

# Learn from your experiment

You've designed your experiment, rolled it out to your users, and given them enough time to interact with your new variants. Now, check if your hypothesis was correct.

The Activity tab tells you at a glance whether your experiment produced statistically significant results and what those results are. Experiment automatically uses the information you provided during the design and rollout phases, which avoids repeated effort. The Activity tab breaks results out by variant and provides a detailed tabular breakdown.

> **Note:** Note
>
> Depending on your experiment, some Activity tab cards may not appear.

## Filter card

On the Filter card, set criteria that update the analysis on the page. Filter your experiment results with the following:

- Date.
- Segment.
- Property.

### Date filter

The date filter defaults to your experiment's analysis start and end dates. Adjust the range to scope experiment results to those specific dates. A [scheduled stop](https://amplitude.com/docs/feature-experiment/workflow/experiment-test#schedule-your-experiment) stops variant delivery. It doesn't replace this analysis filter.

### Segment filter

The segment filter lets you select predefined segments, or create one ad-hoc. Predefined segments include:

- Experiment

  - All exposed users. Users who saw a variant.
  - Testers. Users added as testers during experiment configuration.
  - Exclude testers. Excludes users added as testers during experiment configuration.
  - Exclude users who variant jumped. Excludes users who saw more than one variant.
  - Exclude testers and variant jumpers. Excludes users added as testers and users who saw more than one variant.
- Amplitude

  - New user. Users who triggered at least one new user event during the selected date range.
  - Mobile web. Users who triggered events on the web from a mobile device.
  - Desktop web. Users who triggered events on the web from a desktop device.

> **Note:** Support for segments
>
> The Testers, Exclude Testers, and variant jumpers segments are available on feature experiments that use [Remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation).
>
> The Exclude users who variant jumped segment and exclude testers and variant jumpers segment are available on experiment types other than [multi-armed bandit](https://amplitude.com/docs/feature-experiment/workflow/multi-armed-bandit-experiments).

These segments update in real time.

Click the **segment icon** and then click **Create Segment** to open the Segment builder, then define a new segment. Segments you create in one experiment are available across all other experiments and appear in the All Saved Segments category.

### Property filter

Filter your experiment results based on user properties. For example, create a filter that excludes users from a specific country or geographic region, or users that have a specific account type on your platform.

Click **Add filter** to build a property filter.

## Data Quality card

> **Note:** Availability
>
> Data Quality is available to organizations with access to Experiment who have recommendations enabled.

Amplitude doesn't generate p-values or confidence intervals for experiments using binary metrics (for example, unique conversions) until each variant has 100 users and 25 conversions. Experiments using non-binary metrics need only to reach 100 users per variant.

When you expand a category, or click _Guide_, the Data Quality Guide opens in a side panel where you can address or dismiss issues.

## Summary card

> **Note:** Availability
>
> Summary is available to organizations with access to Experiment who have recommendations enabled.

The Summary card describes your experiment's hypothesis and tells you if it reached statistical significance.

> **Note:** Statistical significance and Amplitude
>
> Amplitude considers an experiment statistically significant (stat sig) when Amplitude can confidently say the results are unlikely to have occurred by random chance. More technically, statistical significance is when Amplitude rejects the null hypothesis. That may sound subjective, but it's grounded in statistics. Statistical significance relies on a variant's p-value, which represents the likelihood that your results occurred by chance. A lower p-value means your results are probably not random, and there's evidence to support your hypothesis. If this value drops below a threshold, Amplitude considers the experiment statistically significant.

The Summary card displays a badge labeled _Significant_ if the experiment reached statistical significance, and a badge labeled _Not Significant_ if it didn't. This card can display several badges at the same time:

- **Inconclusive**: The test was inconclusive for the primary metric.
- **Above Goal** or **Below Goal**: The primary metric's mean was either above or below its goal, depending on the direction of the test (increase = above, decrease = below).
- **Above Control** or **Below Control**: The primary metric's mean was either above or below the control's mean, depending on the direction of the test (increase = above, decrease = below). These badges only apply to stat sig results.

## Analysis card

The top of the Analysis card shows an overview that explains how your experiment performed, broken down by metric and variant. Below the overview, a collection of experiment results charts, which you can analyze by metric, display information about:

- Confidence intervals.
- Cumulative exposure.
- Event totals.
- Mean value over time.

For more information, refer to [Dig deeper into experimentation data with Experiment Results](https://amplitude.com/docs/analytics/charts/experiment-results/experiment-results-dig-deeper#interpret-your-results).

> **Tip:** Chart filtering
>
> The Experiment Results chart on the Activity tab responds to the selections you make in the [Filter card](#filter-card).

Click **Open in Chart** to open a copy of the Experiment Results in a new chart.

> **Note:**
>
> If you run an A/B/n test, Amplitude Experiment displays the confidence interval and p-value for the control against each treatment individually. To view the comparison between two non-control treatments, either change the control variant or open the test in Analytics and create a chart using the two treatments you want.

If needed, adjust the experiment's confidence level. The default is 95%. You can also [choose between a sequential test and a T-test](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences).

> **Note:**
>
> Lowering your experiment's confidence level makes it more likely that your experiment reaches statistical significance. However, this increases the likelihood of a false positive.

### Group by

This feature is in Early Access. This feature may continue to evolve. This documentation may not yet reflect the latest updates.

> **Note:**
>
> Group-bys in Experiment charts may slow query performance. For more information, refer to [Limitations](#limitations).

When you run an experiment, you often want to know if the experiment affected different users differently, that is, whether heterogeneous treatment effects exist. One way to find out is to filter for `Platform = iOS`, then `Platform = Android`, then `Platform = Web`. Grouping results by `Platform` achieves the same result with fewer clicks. For more information, refer to [Group-bys: How Amplitude prunes and orders chart results](https://amplitude.com/docs/analytics/charts/group-by).

Group-by settings on the dashboard are temporary. Refreshing the dashboard resets any group-bys you define.

## Diagnostics card

The Diagnostics card provides information about how your experiment is delivering. The Diagnostics card shows charts about:

- Assignment events (cumulative and non-cumulative).
- Exposure events (cumulative and non-cumulative).
- Assignment to exposure conversion.
- [Variant jumping](https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping).
- Anonymous exposures (cumulative and non-cumulative).
- [Exposures without Assignments](https://amplitude.com/docs/feature-experiment/troubleshooting/exposures-without-assignments) (cumulative and non-cumulative).

For more control, open any of these charts in the Analysis Chart modality.

## Enable notifications

You can receive notifications about your experiments in a dedicated Slack channel or through a webhook. Go to [Integrate Slack](https://amplitude.com/docs/analytics/integrate-slack) and then [Experiment Notifications](https://amplitude.com/docs/feature-experiment/notifications) to set up these notification alerts.

You can set up a notification for the following events:

- **Experiment about to start:** Amplitude sends this notification when your scheduled experiment is about to start.
- **Experiment end reached:** Amplitude sends this notification when your experiment is complete.
- **SRM detected:** Amplitude sends this notification if it identifies a [sample ratio mismatch](https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch) issue.
- **Long-running experiments:** Amplitude sends this notification when your long-running experiment is complete.
- **Statsig for the recommendation metric is reached:** Amplitude sends this notification when your experiment's recommendation metric reaches stat sig.

Amplitude Experiment sends a notification to the editors of the experiment.

## Next steps

No experiment is a failure. Even if you didn't get the results you hoped for, you can still learn from the process. Use your results as a starting point for asking questions about the changes you made, the outcomes you saw, what your customers expect from your product, and how you can deliver on those expectations.

In general, the next step is to decide whether to run another experiment that supports your hypothesis to gather more evidence, or to implement the variant that delivered the best results. You can also export your experiment to the Experiment Analysis in Amplitude Analytics for a deeper review, where you can segment users and generate more insights.

================================================================================
# New Experiment Run
URL: https://amplitude.com/docs/feature-experiment/troubleshooting/new-experiment-run
================================================================================

# New Experiment Run

Create a new run of an existing experiment after you fix instrumentation issues that affected data quality. A new run excludes previous user data from your experiment's monitoring and analysis.

## Create a new run

To create a new run of an existing experiment:

1. Open a running or completed experiment.
2. Open the menu next to **Turn off flag** (for completed experiments) or **Complete experiment** (for running experiments) and select **New run**.
3. Select a new analysis range. Optionally, configure how Amplitude handles users in the new run and what happens to existing feature flags.

## Configuration changes on a new run

Amplitude applies the following changes to your experiment configuration when you create a new run:

| Property | How it changes on a new run |
| --- | --- |
| [Experiment key](#experiment-key) | Amplitude assigns a new value. |
| Start date | Amplitude sets the start date to the restart date. |
| End date | (Optional) Amplitude sets the end date to your selected value. |
| Bucketing salt | (Optional) If you select this option, Amplitude randomizes the salt to a new value. |
| Sticky bucketing | If you select the option to re-randomize users and sticky bucketing is on, Amplitude turns sticky bucketing off. |
| Decision | If you rolled out or rolled back your experiment, Amplitude erases the decision. |

## Experiment key

By default, Amplitude delimits your experiment runs by time. You can also differentiate runs with the experiment key property on the default exposure event. The experiment key prevents your new run from including stale evaluated users. After you create the new experiment run, enable the setting under the exposure event control to use the experiment key.

To use the experiment key:

1. Your experiment must use Amplitude's default exposure tracking.
2. Your client SDK version must support experiment restarts.

| SDK | Minimum version |
| --- | --- |
| JavaScript | v1.10.2 |
| Android | v1.10.0 |
| iOS | v1.11.0 |
| React Native | v1.2.0 |

If you use the [Evaluation API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api), the response body contains the experiment key of the actively running experiment.

```json
{
    "<flag_key>": {
        "key": "<variant_value>",
        "payload": <variant_payload>,
        "experiment_key": "exp-1",
    },
    // ...
}
```

The experiment key is also available when you fetch variants with Experiment SDKs. For example, the JavaScript SDK [Variant](https://amplitude.com/docs/sdks/experiment-sdks/experiment-javascript#variant) object contains the existing `value` and `payload` properties along with a new `expKey` property.

================================================================================
# Configure your experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/configure
================================================================================

# Configure your experiment

Configuring Feature Experiment is a two-stage process:

- Create a deployment.
- Install the SDK you want to use.

Configuring Experiment isn't the same as [creating an experiment](https://amplitude.com/docs/feature-experiment/workflow/create). Configure Experiment and confirm it's working before you create any experiments. This page covers Feature Experiment. To configure Web Experiment, go to [Implement Web Experiment](https://amplitude.com/docs/web-experiment/implementation).

## Create a deployment

In Experiment, a deployment serves a group of flags or experiments for code execution. After you create a deployment, Experiment generates an access key that routes your flags and experiments.

Deployments live under Amplitude Analytics projects. A project can have multiple deployments, but each deployment attaches to a single project.

To create a deployment, follow these steps:

1. Go to _Experiment > Deployments_, then click **Create Deployment**.
2. Choose the Amplitude Analytics project to associate with the deployment. To create deployments in multiple projects, select all the relevant projects from the drop-down list.
3. Enter a descriptive name for your deployment.
4. Specify its type:

   - **Client-side**: These deployments run on a client device, such as a web browser or mobile app. The deployment key for client deployments is publicly viewable and works with client-side SDKs.
   - **Server-side**: These deployments run on a server you control, such as a web server or batch processing system. Keep the deployment key for server deployments secret and use it with server-side SDKs. Server-side keys can access the REST API for flag evaluation. If you only need to evaluate flags through the REST API rather than run a full experiment, create a server-side deployment.
5. Click **Create a Deployment**. Experiment creates your deployment and generates keys to copy and use.

### Deployment keys

Each deployment generates a unique key that ensures data integrity. The key associates all data your experiment collects with that deployment, which keeps results and analytics accurate. For more information about keys, go to [Keys and Tokens](https://amplitude.com/docs/apis/keys-and-tokens#keys-overview).

- **Client-side keys**: The deployment key for client deployments is publicly viewable.
- **Server-side keys**: Keep the deployment key for server deployments secret and use it only with server-side SDKs. Server-side keys access the API for flag evaluation.

## Install the SDK

If you don't use the API, install an [Experiment SDK](https://amplitude.com/docs/sdks/experiment-sdks) next. All SDKs send a request to Amplitude Experiment to determine what flag configurations to serve a user. Note the following differences between client-side and server-side SDKs.

> **Warning:**
>
> Engineers should install SDKs. Otherwise, you risk issues with your data repositories.

**Client-side** SDKs run in the end-user application deployment. Client-side SDKs:

- Assume a single-user deployment.
- Use client-side deployment keys, which are public and visible to end users.
- Fetch variants up front for a given user.
- Store variants locally on the client for offline mode.

**Server-side** SDKs run in a server deployment. Server-side SDKs:

- Assume a multi-user deployment.
- Use server-side deployment keys, which you should keep private.
- Fetch variants on each request.

## The user context

When assigning variants, the evaluation engine applies targeting rules to a user context object, which represents an individual user's identity. In client-side SDKs, the SDK sets this object on initialization and passes it to the server on every variant request. In server-side SDKs, the user may change, and the SDK sets the user on every request.

When targeting individual users to assign variants, Experiment matches on any of the listed user identifiers, such as `user_id` and `device_id`. With rule-based user segments, users match on any predefined property (country, platform, and so on) or on custom properties in the `user_properties` object. Read more about [defining experiment users](https://amplitude.com/docs/feature-experiment/data-model#users).

> **Note:**
>
> Use the same user identifiers for Amplitude Experiment that you use to send data to Analytics. This way, identities resolve correctly and Analytics associates generated data with the same user.

================================================================================
# Contentful
URL: https://amplitude.com/docs/feature-experiment/contentful
================================================================================

# Contentful

The Contentful plugin for Amplitude Experiment lets you create content variations in Contentful, use Experiment to control which variant each user sees, and track the performance of those variants.

## Features

- Run A/B tests in Amplitude Experiment and author content in Contentful.
- Read properties from Amplitude Experiment (refreshed every 5 seconds) and build content around those properties.

## Requirements

To use the plugin, ensure you have the following:

- Access to an Amplitude plan with Amplitude Experiment enabled.
- Your Amplitude Org URL value. This value appears in the URL you use to access Amplitude: `https://app.amplitude.com/experiment/<ORG_URL>/dashboard`.
- A Management API key, which you can find in the Experiment side bar.

## Installation and use

Complete the following steps in Contentful and Amplitude to add and activate the Contentful plugin for Amplitude Experiment.

##### Install the plugin

1. Install the [plugin](https://www.contentful.com/marketplace/app/amplitude-experiment/) from the Contentful marketplace.
2. In the plugin configuration, enter the data center, Org URL, and Management API key you created in Experiment.
3. Click **Install to selected environments**.
4. Click **Save** to complete the plugin setup.

After you enable the plugin, a `Variant Container` content model appears on the Content Model tab.

### Add a variant container to a content model

The variant container in Contentful works with Amplitude Experiment to decide which variant of the experiment each user sees.

For best results, Amplitude recommends using a **Reference** content type to hold experiments.

##### To add a variant container

1. Open the content model of the page to which you want to add the variant container.
2. Click **Add field**. Select a **Reference** field.
3. Provide a name for the field. For best results, enter a name that matches the purpose of the field. For example, `Hero Banner` or `Demo CTA`.
4. Select **One reference** as the Type.
5. Click **Add and configure**.
6. On the Name and field ID tab of the field configuration, select **Accept only specified entry type**, and then select **Variant container**. This setting keeps the Contentful API response consistent when Contentful serves page content.
7. Click **Confirm** to create the field, then click **Save** to update the content model.

### Add content to your experiment

After you configure the variant container and reference field, open the page where you want to add an experiment.

##### To add content to your experiment

1. In the Content editor, select the page.
2. Find the field you created in the previous step and click the associated **Add content** selector.
3. Select **Variant Container** as the content to add.
4. In the field configuration, enter the **Flag Key** of the experiment. The Flag Key field shows matching keys as you type.
5. After you select the Flag Key, any variants associated with that key appear in the **Variants** section.
6. For each variant, select either **Link an existing entry** or **Create new content type** to populate the variant.
7. Click **Publish** to publish the variant container.
8. Click **Publish** to publish the updated page with the experiment enabled.

### Integrate with your front end

Contentful returns JSON when a user requests the page:

```json
{
  "__typename": "PageLanding",
  "sys": {
    "id": "2cayfg7wVF5WezADCHgSgL",
    "spaceId": "4y4crvvoco9a"
  },
  "internalName": "Homepage",
  "heroBanner": {
    "__typename": "VariationContainer",
    "experiment": {
      "id": "183980",
      "key": "contentful-hero-banner",
      "name": "contentful-hero-banner",
      "tags": [],
      "state": "running",
      "deleted": false,
      "enabled": true,
      "endDate": null,
      "decision": null,
      "variants": [
        {
          "key": "control"
        },
        {
          "key": "treatment"
        }
      ],
      "projectId": "289220",
      "startDate": "2024-02-22",
      "deployments": ["14"],
      "description": "",
      "bucketingKey": "amplitude_id",
      "bucketingSalt": "28fWNw1M",
      "bucketingUnit": "User",
      "decisionReason": null,
      "evaluationMode": "remote",
      "experimentType": "hypothesis-testing",
      "rolloutWeights": {
        "control": 1,
        "treatment": 1
      },
      "targetSegments": [],
      "stickyBucketing": false,
      "rolledOutVariant": null,
      "rolloutPercentage": 0
    },
    "experimentId": "contentful-hero-banner",
    "meta": {
      "control": "6rmYK8YKYtTkKcFRY9Pf2w",
      "treatment": "kwDjI2f2vKE2GvQoeqq1d"
    },
    "variationsCollection": {
      "items": [
        {
          "__typename": "Hero",
          "sys": {
            "id": "6rmYK8YKYtTkKcFRY9Pf2w",
            "spaceId": "4y4crvvoco9a"
          },
          "preHeadline": "Organic Products",
          "headline": "100% Fresh Food",
          "cta": "Shop Now",
          "description": "Whatever you want from your local stores, brought right to your door. \t\t\t\t\t\t\t",
          "image": {
            "__typename": "Asset",
            "sys": {
              "id": "6PkraSxWWd96AU6FTYVssd"
            },
            "title": "Fresh food",
            "description": "",
            "width": 2560,
            "height": 960,
            "url": "https://images.ctfassets.net/4y4crvvoco9a/6PkraSxWWd96AU6FTYVssd/69b8d7f7cabb9f578097d50f2bf8aa70/Hero-3-1-scaled.jpg",
            "contentType": "image/jpeg"
          }
        },
        {
          "__typename": "Hero",
          "sys": {
            "id": "kwDjI2f2vKE2GvQoeqq1d",
            "spaceId": "4y4crvvoco9a"
          },
          "preHeadline": "Exclusive Offer",
          "headline": "Loyalty Program",
          "cta": "Get Free Shipping",
          "description": "We missed you! Finish your order today and get free shipping when you join our loyalty program.",
          "image": {
            "__typename": "Asset",
            "sys": {
              "id": "6WFOK0460CwrW7aChl1QjM"
            },
            "title": "Loyalty Green",
            "description": "",
            "width": 1920,
            "height": 720,
            "url": "https://images.ctfassets.net/4y4crvvoco9a/6WFOK0460CwrW7aChl1QjM/e52fb2129b848203f6006ff9881309d9/Hero-_-loyalty-green.jpg",
            "contentType": "image/jpeg"
          }
        }
      ]
    }
  }
}
```

The `experiment` object contains useful metadata about the experiment. To render the front end and include the experiment, use the `meta` and `variationsCollection` objects. Amplitude Experiment delivers the variant identifier and matches it to an option in the `meta` object. After you set the variant, you can:

- Iterate through items in the `variationsCollection` object to show the variation with the matching ID.
- Make a direct call to Contentful with the variant ID to avoid searching the array.

For more information, review the following React and TypeScript example:

```typescript
import React, { useEffect, useState } from "react";
import { Experiment } from "@amplitude/experiment-js-client";
import sdk from "contentful-sdk";

// --- Types (adjust to your schema) ---
type Hero = {
  __typename: "Hero";
  sys: { id: string };
  // ...other fields you render
};

type HeroBanner = {
  experimentId?: string; // key you use in Amplitude Experiment
  meta?: Record<string, string>; // maps variant key -> variation id
  variationsCollection?: { items: Array<Hero | null | undefined> };
};

// --- Experiment init ---
const CLIENT_KEY =
  process.env.NEXT_PUBLIC_AMPLITUDE_EXPERIMENT_CLIENT_KEY ?? "";
export const experiment = Experiment.initialize(CLIENT_KEY, {
  // Only enable verbose logging in dev if you like:
  debug: process.env.NODE_ENV !== "production",
});

// --- Component hook snippet ---
export function useHeroVariant(userId: string | undefined) {
  const [hero, setHero] = useState<Hero | null>(null);

  useEffect(() => {
    // Skip until we have a user id
    if (!userId) return;

    let isMounted = true;

    (async () => {
      try {
        // 1) Fetch variants for this user
        await experiment.fetch({ user_id: userId });

        // 2) Load the banner/experiment mapping from Contentful
        const heroBanner = await sdk.getEntry<HeroBanner>("ENTRY_ID_HERE");
        if (!heroBanner) return;

        // 3) Resolve the variant key from the experiment
        const experimentKey = heroBanner.experimentId ?? "control";
        const { value: variantKey } = experiment.variant(experimentKey);
        if (!variantKey) return;

        // 4) Map the variant key -> variation id through Contentful metadata
        const variationId = heroBanner.meta?.[variantKey];
        if (!variationId) return;

        // 5) Find the matching Hero item
        const match =
          heroBanner.variationsCollection?.items.find(
            (item): item is Hero =>
              !!item &&
              item.__typename === "Hero" &&
              item.sys?.id === variationId,
          ) ?? null;

        if (isMounted) setHero(match);
      } catch (err) {
        // Consider forwarding to your logger
        console.error("Failed to resolve hero variant", err);
        if (isMounted) setHero(null);
      }
    })();

    return () => {
      isMounted = false;
    };
  }, [userId]); // Note: don't include heroBanner (it's local) or setHero

  return hero;
}
```

Account for cases where users receive `off` as the value that `experiment.variant()` returns.

================================================================================
# Proxy requests to Experiment with AWS Cloudfront
URL: https://amplitude.com/docs/feature-experiment/advanced-techniques/proxy-requests-to-experiment-with-aws-cloudfront
================================================================================

# Proxy requests to Experiment with AWS Cloudfront

Set up a reverse proxy to circumvent domain blocking in particular regions or by certain extensions and DNS servers. Because experiment APIs are latency sensitive, Amplitude recommends an edge-hosted solution to minimize round-trip time from the proxy to Amplitude.

## Create a distribution

Follow these steps to create a new CloudFront distribution that proxies requests to Amplitude Experiment's evaluation servers. Leave any configuration field not mentioned in the steps at its default value.

1. In AWS, go to CloudFront and select **Create distribution**.
2. In the **Origin domain** field, enter `api.lab.amplitude.com` for the US data center or `api.lab.eu.amplitude.com` for the EU data center.
3. In the **Default cache behavior** section, select `GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE` for **Allowed HTTP methods** and `OPTIONS` for **Cache HTTP methods**. The **Cache HTTP methods** field appears after you select an **Allowed HTTP methods** value.
4. In the **Cache key and origin requests** section, select the `CachingDisabled` cache policy, the `AllViewExceptHostHeader` origin request policy, and the `CORS-with-preflight-and-SecurityHeadersPolicy` response headers policy. These selections require that you choose **Cache policy and origin request policy (recommended)** rather than **Legacy cache settings**.
5. In the **Web Application Firewall (WAF)** section, select **Do not enable security protections**.
6. Select **Create distribution**.

## Test the distribution

Test the new distribution with a `curl` request. To find the distribution domain name, select the new distribution from the list in CloudFront and copy the subdomain. Replace `SUBDOMAIN` in the following command with that subdomain. Replace `APIKEY` with your deployment or project API key to authorize the request.

Successful requests return a `200` response.

```bash
curl -i 'https://SUBDOMAIN.cloudfront.net/v1/vardata' -H 'Authorization: Api-Key APIKEY'
```

================================================================================
# Server-side rendering
URL: https://amplitude.com/docs/feature-experiment/advanced-techniques/server-side-rendering
================================================================================

# Server-side rendering

Use the JavaScript Server SDK and JavaScript Client SDK together to create a server-side rendered experience.

> **Example:**
>
> For a complete example, go to the [experiment-node-server demo app](https://github.com/amplitude/experiment-node-server/tree/main/packages/ssr-demo) on GitHub.

## Installation

Install both the JavaScript Server SDK and JavaScript Client SDK.

#### npm

```bash
npm install --save @amplitude/experiment-js-client @amplitude/experiment-node-server
```

#### yarn

```bash
yarn add @amplitude/skylab-js-client @amplitude/skylab-js-server
```

## Initialize the Server SDK

On server startup, initialize the Server SDK. To distinguish it from the Client SDK `Experiment` object, this example aliases the Server SDK `Experiment` object as `ExperimentServer`.

```js
let ExperimentServer;
if (typeof window === 'undefined') {
  console.debug('Initializing Server Experiment');
  ExperimentServer = require('@amplitude/experiment-node-server').Experiment.initialize(
    'server-IAxMYws9vVQESrrK88aTcToyqMxiiJoR',
    { debug: true },
  );
}

export { ExperimentServer };
```

## Fetch variants on request

On each request, fetch variants using the server-side SDK. The result is a plain JavaScript object that maps feature keys to variant values. Store the result where your rendering code can access it in both server-side and client-side contexts.

```js
const allFlags = await experimentServer.fetchV2({
  id: 'userId',
});

// store the result where the rendering code can access it
global.appProps = { flags: allFlags };
```

## Initialize the Client SDK on render

At the start of your server-side render, initialize the Client SDK with the fetched variants. Instantiate an `ExperimentClient` that the render scope can access (for example, through a React ContextProvider). In the server-side context, create a new `ExperimentClient` every time. In the client-side context, create a new `ExperimentClient` only if one doesn't already exist.

```js
import { ExperimentClient } from '@amplitude/experiment-js-client';

let experimentClient;

const render = (appProps) => {
const isServerSide = typeof window === 'undefined';
  if (isServerSide) {
    console.debug('Initializing Client Experiment');
    // on the server, create a new ExperimentClient every time
    experimentClient = new ExperimentClient(
      'client-IAxMYws9vVQESrrK88aTcToyqMxiiJoR',
      {
        initialVariants: appProps['features'],
      },
    );
  } else if (!experiment) {
    // in the client, only create the ExperimentClient once
    experimentClient = Experiment.initialize(
      'client-IAxMYws9vVQESrrK88aTcToyqMxiiJoR',
      {
        initialVariants: appProps['features'],
      },
    );
  }
}

// use a provider or store the ExperimentClient so the render scope can access it
```

## Get variants on render

After you initialize the Client SDK, fetch the flag status in any component.

```js
// experimentClient should be the same ExperimentClient instance from the previous step
experimentClient.variant('flag-key');
```

================================================================================
# Finalize your experiment's advanced settings
URL: https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences
================================================================================

# Finalize your experiment's advanced settings

The final step in creating your experiment is to specify advanced settings. These settings include:

- **Exposure settings**: Settings for the exposure event that triggers before your audience receives your experiment.
- **Stats Preferences**: Statistical settings for experiment analysis.
- **Bucketing options**: Settings for bucketing and targeting your audience.

##### To set advanced settings

1. In your experiment, scroll down to the Advanced section and click the **edit** icon.
2. Set your preferences using the definitions below.
3. Click **Save and Close**.

After you save your settings, [test your experiment](https://amplitude.com/docs/feature-experiment/workflow/experiment-test).

## Exposure settings

Exposure settings are the configuration rules that define when and how Amplitude marks a user as exposed to an experiment or feature. These settings determine the logic that triggers an exposure event: whether a user counts as exposed the first time they qualify for an experiment, the first time they interact with a feature, or under custom criteria.

In your Experiment Design options, click **Advanced** and then click **Exposure Settings** to specify the settings you want. You can modify any of the following:

### Exposure event

An exposure event is the moment when a user becomes eligible for an experiment variant or feature. Amplitude shows users the experiment variant regardless of whether they interact with it. This event serves as the anchor point for experiment analysis and ensures that Amplitude attributes downstream behaviors and outcomes to the correct variant. By logging exposure events, Experiment prevents biases such as double counting or misattribution. Exposure events also establish a consistent link between user actions and the experiment they were exposed to.

You can specify:

- **Exposure Event**: Choose which exposure event triggers the experiment. The default is the Amplitude Exposure event. Amplitude recommends leaving this setting as is, but you can specify a custom exposure event.
- **Proxy Exposure Event**: For Feature Experiments, a proxy exposure event is a placeholder used to estimate the duration of the experiment based on historical data of that event. The default is Any Active Event. You can specify any recorded event as the proxy.
- **Custom Exposure Settings**: Choose whether to further customize your exposure settings with:

  - **Attribution**: Choose whether the exposure event activates only on the first instance of the user triggering it, or at any instance.
  - **Window**: Choose whether the experiment triggers within a specific time period of the event.

## Stats Preferences

Statistical preferences are the configurable settings that determine how Amplitude analyzes and displays experiment results. These preferences let teams choose parameters such as:

- [_CUPED_](#cuped) toggled off
- [_Bonferroni Correction_](#bonferroni-correction) toggled on
- [_Custom Exposure Settings_](#custom-exposure-settings) toggled off
- [_Test Type_](#test-type) set to Sequential
- [_Confidence Level_](#confidence-level) set to 95%

You can modify the Stats Preferences at any step of an experiment. They're most useful for the final analysis after the experiment ends.

> **Note:**
>
> This article continues directly from the [Help Center article on learning from your experiment](https://amplitude.com/docs/feature-experiment/workflow/experiment-learnings). If you haven't read that article, do so before continuing here.

### CUPED

Controlled-experiment using pre-existing data, also known as CUPED, is an optional statistical technique that reduces variance in Amplitude Experiment. When you toggle CUPED on, Amplitude Experiment accounts for possible varying treatment effects across user segments. CUPED isn't the best choice for every experiment. For example, avoid CUPED when targeting only new users.

The random bucketing process can deliver unbalanced groups of users to each variant. This unbalance is pre-exposure bias, and CUPED addresses it. Without CUPED, pre-exposure bias persists in your experiment. This is why you may notice differences in the mean-per-variant when running the same experiment with and without CUPED.

For a more technical explanation, refer to this [detailed blog post](https://bytepawn.com/reducing-variance-in-ab-testing-with-cuped.html).

For more on how CUPED affects experiment results, refer to this [blog](https://amplitude.com/blog/amplitude-experiment-cuped).

### Bonferroni Correction

Amplitude Experiment uses the Bonferroni correction to address potential problems with [multiple hypothesis testing](https://amplitude.com/docs/feature-experiment/advanced-techniques/bonferroni-correction). Although a trusted statistical method, you may not want to use it in every case. One example is when you want to compare results with those generated by an internal system that doesn't support the Bonferroni method. In this case, and if you accept higher false positive rates, toggle the **Bonferroni Correction** off.

### Statistical Method

Select which statistical method you want to use:

- **Sequential testing**: A statistical method that analyzes results continuously as data comes in, instead of only at a fixed sample size. This approach lets teams review experiment results continuously without inflating false positive risk. Because the method corrects for repeated looks at the data, it's useful for making faster decisions when effects are strong. It requires careful setup to avoid bias. Refer to [Sequential Testing](https://amplitude.com/docs/feature-experiment/under-the-hood/experiment-sequential-testing) for more information.
- **T-Testing**: A traditional statistical test that compares the means of two groups, such as the control and treatment groups, to determine if differences are statistically significant. It assumes normally distributed data and fixed sample sizes. A t-test is simple and widely understood, but it's less flexible if you want to check results continuously or handle more complex outcome distributions. Refer to [T-testing](https://amplitude.com/docs/feature-experiment/experiment-theory/analyze-with-t-test) for more information.
- **Bayesian**: A statistical method that compares groups by calculating the probability that one variant outperforms another. Unlike traditional methods that rely on p-values and fixed hypothesis testing, Bayesian statistics provides direct probability estimates that align with how teams make decisions. Bayesian methods excel when you want continuous insight into experiment performance. They're valuable when you need to incorporate prior knowledge, make decisions with smaller sample sizes, or require probability statements that directly answer business questions like "How likely is this variant to succeed?" Refer to [Bayesian Statistics](https://amplitude.com/docs/feature-experiment/experiment-theory/bayesian-statistics) for more information.
- **Thompson Sampling**: A [Bayesian](https://www.andrew.cmu.edu/course/18-847F/lectures/18687Nov182019.pdf) bandit approach that dynamically allocates more traffic to variants that appear to perform better. Instead of waiting until an experiment ends, Thompson Sampling balances exploration and exploitation in real time. This approach improves user experience by gradually sending more users to promising variants. It doesn't provide a classic p-value, but relies on posterior probabilities, making it a useful choice when you need adaptive decision-making.

### Confidence Level

The confidence level measures how confident Experiment is that it generates the same results for the experiment across repeated rollouts. The default confidence level of 95% means that 5% of the time, you might interpret the results as statistically significant when they're not. Lowering your experiment's confidence level makes it more likely that your experiment reaches statistical significance, but the likelihood of a false positive goes up. Don't go below 80%, because the experiment's results may no longer be reliable.

## Bucketing options

Specify how bucketing works in your experiment. You can specify:

- **Evaluation Mode**: Select whether Amplitude evaluates the experiment remotely on Amplitude servers or locally on your own machine. By default, Amplitude evaluates experiments remotely. Refer to [Performance and Caching](https://amplitude.com/docs/feature-experiment/under-the-hood/performance-and-caching) for more information.
- **Sticky Bucketing**: Specify whether to serve users the same variant after allocation, even if the rollout or targeting criteria change. When sticky bucketing is on, Amplitude doesn't re-bucket users when the targeting criteria change. The default is off. Refer to [Sticky Bucketing](https://amplitude.com/docs/feature-experiment/advanced-techniques/sticky-bucketing#how-sticky-bucketing-works) for more information.
- **Bucketing Salt**: A string value used as part of the hashing process. The bucketing salt assigns users deterministically into experiment variants. By combining the bucketing salt with identifiers such as the user ID and experiment key, Experiment generates a random-looking but repeatable hash that places each user into the same variant across sessions. Changing the bucketing salt reshuffles assignments and re-randomizes users for that experiment.

================================================================================
# Flag Prerequisites
URL: https://amplitude.com/docs/feature-experiment/advanced-techniques/flag-prerequisites
================================================================================

# Flag Prerequisites

When you run new experiments or roll out new feature flags, some features only apply to users if another feature is also enabled. Evaluate those dependencies first, then use the results in the evaluation of your flag or experiment.

Experiment lets you create dependencies for your flags and experiments on prerequisite flags or experiments.

> **Note:** Plan availability
>
> Flag prerequisites are only available to Enterprise customers. If your organization is on another plan, the Dependencies card doesn't appear in the Feature Flags configuration UI. To learn more or upgrade, contact [Sales](https://amplitude.com/sales-contact).

## Configure flag prerequisites

Configure flag prerequisites in the Dependencies card for the feature flag. Go to _Experiment > Feature Flags > your feature flag_ and scroll to the Dependencies card.

This card summarizes all dependencies for that feature flag, including its prerequisite flags, experiments, mutual exclusions, and holdout groups. The card also lists the flags and experiments that depend on it.

1. To configure new prerequisites, select the **edit** icon.
2. Select **Add Dependency** to add a new prerequisite flag or experiment.
3. Select **Select a flag or experiment**, then select the item you want.

> **Note:**
>
> A flag or experiment qualifies as a prerequisite if:
>
> - It's in the same project. - It has a compatible evaluation mode. Local evaluation mode flags and experiments can only have local evaluation mode prerequisites. Remote evaluation mode flags and experiments can have both remote and local prerequisites.
>
> You can't add a prerequisite that creates a circular dependency loop.

4. Select the variants the flag depends on. A special variant, `Off`, represents users that the prerequisite flag or experiment excludes.
5. Select **Save**.

## Workflow considerations

Before you activate a flag or start an experiment, confirm that prerequisite flags are active and that your variant assignment works as expected. You can't start the experiment until its prerequisite flags and experiments are active.

For flags and experiments with dependents, Amplitude blocks the following actions:

- Deleting a variant or changing a variant key if another flag or experiment depends on it.
- Archiving that flag or experiment.

## How evaluation works with prerequisite flags

This example shows how evaluation works when prerequisite flags exist.

For example, you want your new feature flag (Flag-B) to roll out only to users who first saw a different feature (Flag-A). In Flag-B, you add a dependency on the `On` variant of Flag-A and activate both flags.

When Amplitude evaluates users for Flag-B:

1. Amplitude checks if the user is in Flag-B's cohort.

   - If the user belongs to the cohort, Amplitude serves the configured variant.
2. Amplitude then evaluates the user for dependencies, in this case Flag-A.

   - If the user doesn't receive the `On` variant for Flag-A, Amplitude excludes them from Flag-B.
3. Amplitude then evaluates the user for Flag-B.

Targeting for Flag-B determines which variant (if any) the user receives. The flag dependency on Flag-A has no effect at this point.

## Prerequisites compared to user property targeting

You can achieve similar outcomes using either flag prerequisites or targeting rules based on `[Experiment]` user properties. The two approaches differ in important ways.

### Use `[Experiment]` user properties in targeting

When Amplitude assigns a user to a flag or experiment variant, Amplitude sets a user property in the format `[Experiment] <flag_key>` with the variant key as the value. You can reference this user property in another flag's targeting rules to target users based on their previous assignments.

Example: Target users where `[Experiment] Flag-A` equals `on`.

Limitations:

- Timing issues: Amplitude sets the `[Experiment]` user property when it ingests the assignment or exposure event. If you evaluate a dependent flag before the property syncs, the user may not match the targeting rule.
- No formal dependency tracking: The UI doesn't show the relationship between flags. You must manually track which flags depend on others.
- Potential inconsistency: Race conditions between property sync and flag evaluation can give users inconsistent experiences.

### Use flag prerequisites

Flag prerequisites formally define dependencies between flags. When you add Flag-A as a prerequisite for Flag-B, Amplitude evaluates Flag-A first and uses the result to determine Flag-B eligibility.

Advantages:

- Evaluated together: Amplitude evaluates prerequisites in sequence during a single evaluation call, which eliminates timing issues.
- Visible dependencies: The Dependencies card shows all relationships, making complex flag hierarchies easier to manage.
- Consistent bucketing: Users get consistent experiences because prerequisite evaluation happens atomically.
- Protected changes: Amplitude prevents you from archiving prerequisite flags or changing variant keys that other flags depend on.

### When to use each approach

| Use case | Recommended approach |
| --- | --- |
| Assign users to Flag-A before they see Flag-B | Flag prerequisites |
| Build release groups with sub-features | Flag prerequisites |
| Chain mutually exclusive experiments | Flag prerequisites |
| Target users exposed to a flag days or weeks ago | User property targeting |
| Analyze experiment results by previous flag exposure | User property targeting |
| Run ad-hoc segmentation for debugging | User property targeting |

## Common use cases

Flag prerequisites support many use cases. Common examples include:

### Release groups

Use flag prerequisites to build a primary feature with multiple sub-features. Sub-features require the primary feature to be `On` unless the cohort in a sub-feature individually includes the user. Amplitude applies the targeting and bucketing from the primary feature to every sub-feature that lists the primary feature as a prerequisite. Otherwise, the users receiving the primary feature and the sub-features wouldn't match.

Common use cases for release groups:

- Active development of large feature releases across many developers and teams.
- Provisioning users to primary SKUs with add-ons.
- Simplifying feature flag logic in code.

![Diagram of example release group.](https://amplitude.com/docs/images/experiment/release-group.drawio.svg)

This example contains a `primary-feature` flag and `sub-feature` flags that list the primary feature as a prerequisite.

The `primary-feature` flag targets all users where the user property `premium` is `true` with 100% allocation. Sub-features only evaluate if the user has the required user property and meets the sub-feature's criteria. The exception is when the sub-feature individually includes the user, which typically occurs during a testing phase.

- The `sub-feature-1` flag adds targeting criteria for users where the user property `beta` is `true`. To receive `sub-feature-1`, a user must have both `premium` and `beta` user properties equal to `true`.
- The `sub-feature-2` flag allocates 100% of users. Amplitude assigns every user with `premium` equal to `true` to this feature.
- The `sub-feature-3` flag allocates 0% of users. Amplitude allocates no users to `sub-feature-3`, even if `premium` is `true`.

### Chained mutual exclusion groups

Use flag prerequisites to build complex hierarchies of mutually exclusive experiments that start at different times. Dependent experiments list a prerequisite on an existing active experiment evaluating to `off`. This rule targets all users that the existing experiment didn't allocate. Continue this chain to add more mutually exclusive experiments when the previous experiment doesn't allocate all users.

![Diagram of example mutual exclusion group](https://amplitude.com/docs/images/experiment/advanced-mutex-group.drawio.svg)

In this example, `experiment-1` runs now, and `experiment-2`, which is mutually exclusive to `experiment-1`, runs at a later time.

- The `experiment-1` experiment allocates 20% of users 50/50 control/treatment.
- The `experiment-2` experiment lists `experiment-1` as a prerequisite and allocates 100% of users 50/50 control/treatment.

Experiment assigns 20% of users to `experiment-1` and the remaining 80% to `experiment-2`. Experiment doesn't assign any user to variants for both `experiment-1` and `experiment-2`.

================================================================================
# Multi-armed bandit experiments
URL: https://amplitude.com/docs/feature-experiment/workflow/multi-armed-bandit-experiments
================================================================================

# Multi-armed bandit experiments

In a traditional A/B test, Amplitude Experiment assesses all the variants in your experiment until it reaches a statistically significant result. From there, you can choose to roll out the winning variant, or roll all users back to the control variant. Your decision depends on why a particular variant outperformed the others.

Sometimes, that reason isn't relevant. You want to identify the best-performing variant and send as much traffic to it as possible. For example:

- [Optimizing hero images, messaging, or color changes to UI elements](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment).
- In-product layout changes, like information hierarchy or order of operations.
- Optimizing menus or navigation.
- Ad optimization for seasonal or time-sensitive promotions or events.
- Hyperparameter tuning for ML models.

Unlike a traditional A/B test, multi-armed bandits don't use statistical significance to determine success. They also don't use a control or baseline variant. Amplitude Experiment also displays results differently for the two experiment types. A later section covers those differences.

Multi-armed bandit experiments use [Thompson sampling](https://en.wikipedia.org/wiki/Thompson_sampling). Amplitude Experiment doesn't support other statistical methodologies for multi-armed bandits.

## Before you begin

- You can evaluate multi-armed bandit experiments locally or remotely.
- You can configure multi-armed bandit experiments to reallocate traffic hourly, daily, or weekly.
- Amplitude Experiment requires at least 100 exposures in each variant before it reallocates traffic.
- Multi-armed bandits respect all mutual exclusion groups and holdouts that you associate with them.
- The flag config history shows each reallocation. Amplitude makes entries under the user `ampex_data_monster`.

## Reallocation schedule

Reallocation runs at fixed times, not relative to the experiment start date. The schedule depends on the reallocation frequency you choose. You can't configure these times.

## Create a multi-armed bandit experiment

Creating a multi-armed bandit experiment is almost identical to [creating an A/B test in Amplitude Experiment](https://amplitude.com/docs/feature-experiment/overview). The next section covers the differences.

## Differences between multi-armed bandits and A/B tests

### Metrics

A multi-armed bandit experiment requires a primary metric. Amplitude Experiment uses the primary metric to optimize your experiment. You can include secondary metrics, but Amplitude Experiment uses them for reporting only.

In an A/B test, your primary metric can be a guardrail metric: a metric that you don't want your experiment to negatively affect. Clickthrough rate is a common guardrail metric. A multi-armed bandit experiment optimizes for a metric, so a guardrail metric doesn't apply. You can't optimize for a change you don't want to occur. Primary metrics for multi-armed bandit experiments must be success metrics ("will increase" or "will decrease"). Amplitude Experiment supports both binary metrics and continuous metrics.

To optimize two metrics in your multi-armed bandit experiment, create a custom metric that's a weighted average of both. If you face a tradeoff between metrics you want to optimize, run an A/B test instead.

### Traffic allocation

Allocation for a multi-armed bandit experiment always begins with a uniform distribution, because the model can't know which variant is most effective before it collects data. The allocation changes after data starts arriving.

A multi-armed bandit adjusts the allocation between the variants only. It doesn't adjust the percentage rollout.

### Confidence level

The confidence level in a multi-armed bandit experiment has a different role than in an A/B test. The confidence level can accelerate traffic to the winning variant. For example, if your experiment's confidence level is 95%, and the multi-armed bandit has already allocated at least 95% of the experiment's traffic to the winning variant, Amplitude Experiment assumes confidence and allocates 100% of traffic to that variant from that point on.

### Duration estimate and MDE

In multi-armed bandit experiments, the minimum detectable effect (MDE) helps calculate the duration estimate. These experiments are automated and optimize for a metric, so the MDE doesn't affect the experiment after it starts running.

When calculating the duration estimate before the experiment starts, Amplitude Experiment simulates what happens when all variants except one share the same baseline mean (computed from historical data). When measuring an increase, the exception variant has `mean * (1+MDE)`. When measuring a decrease, the exception variant has `mean * (1-MDE)`. Amplitude Experiment then calculates how long the multi-armed bandit might take to assign all traffic to one variant. The duration estimate caps at 31 days.

### Displayed results

Amplitude Experiment doesn't display variant jumping while a multi-armed bandit runs, because variant jumping is expected behavior in these experiments.

Amplitude Experiment doesn't display the data quality card for multi-armed bandit experiments. Most checks for this display don't apply to this experiment type. You can't make changes to the experiment that affect traffic allocation while the experiment runs.

The Bandits card resembles the non-cumulative exposure chart in the Monitor card, but normalizes to 100%. The Bandits card lets you visualize the percentage of traffic each variant receives on a given day.

## Notifications

Amplitude Experiment sends notifications to experiment editors when a multi-armed bandit allocates 70%, 80%, 90%, or 100% of traffic to a variant. Amplitude Experiment also sends a notification if the bandit takes a long time to complete or if the experiment's end date arrives. Notifications can go through Slack or email. For more information, go to [Integrate Slack with Amplitude](https://amplitude.com/docs/analytics/integrate-slack).

To configure your notifications, go to _Settings > Personal settings > Notifications_.

================================================================================
# Experiment Evaluation Proxy
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-evaluation-proxy
================================================================================

# Experiment Evaluation Proxy

The Evaluation Proxy is a Service to enable, enhance, and optimize [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation) running within your infrastructure.

![Architecture diagram of the Evaluation Proxy with Amplitude Experiment](https://amplitude.com/docs/images/experiment/evaluation-proxy.drawio.svg)

- **Enable local evaluation on unsupported platforms**: Use remote [Evaluation APIs](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api) and [SDKs](https://amplitude.com/docs/sdks/experiment-sdks) to run local evaluation in your infrastructure.
- **Automatically track exposure events for local evaluations**: The proxy deduplicates identical exposure events for 24 hours.
- **Enhance local evaluation with large cohort targeting**: Targeted cohorts sync hourly to the Evaluation Proxy, and the proxy adds cohorts to the user before evaluation.

## Configuration

The evaluation proxy is either configured using a `yaml` file (recommended, more configuration options), or using environment variables.

The default location for the configuration yaml file is `/etc/evaluation-proxy-config.yaml`. You may also configure the file location using the `PROXY_CONFIG_FILE_PATH` environment variable.

- [`projects`](#projects) (required)
- [`configuration`](#configuration-1) (optional).

> **Tip:** Recommended configuration
>
> Replace the fields in the configuration with values specific to your account/infrastructure.
>
> ```yaml
> projects:
>   - apiKey: "YOUR API KEY"
>     secretKey: "YOUR SECRET KEY"
>     managementKey: "YOUR MANAGEMENT API KEY"
>
> configuration:
>   redis:
>     uri: "YOUR REDIS URI" # e.g. "redis://localhost:6379"
> ```

Environment configuration supports only a single project. The proxy uses environment variable configuration only if the configuration file isn't found.

- **`AMPLITUDE_API_KEY`**

  - **Description**: The project's [API key](https://amplitude.com/docs/apis/keys-and-tokens#api-key).
- **`AMPLITUDE_SECRET_KEY`**

  - **Description**: The project's [secret key](https://amplitude.com/docs/apis/keys-and-tokens#secret-key).
- **`AMPLITUDE_EXPERIMENT_MANAGEMENT_API_KEY`**

  - **Description**: The [Experiment management API key](https://amplitude.com/docs/apis/keys-and-tokens#management-api-key). Must be created for the same project as the configured API and secret key. Used to automatically access and update deployments used for the project.
- **`AMPLITUDE_REDIS_URI`**

  - **Description**: Optional. The entire URI to connect to Redis. Include the protocol, host, port, and optional username, password, and path (for example `redis://localhost:6379`).
- **`AMPLITUDE_REDIS_PREFIX`**

  - **Description**: Optional. The prefix to connect
- **`AMPLITUDE_REDIS_USE_CLUSTER`**

  - **Description**: Optional. If `AMPLITUDE_REDIS_URI` is a cluster URL, pass this as `true`. The setting defaults to `false`.
- **`AMPLITUDE_REDIS_READ_FROM`**

  - **Description**: Optional. Read routing strategy for Redis Cluster. Options: `REPLICA_PREFERRED` (default, prefer replicas) or `ANY` (read from any node in the cluster).
- **`AMPLITUDE_SERVER_URL`**

  - **Description**: Optional. The server URL, including protocol and host, to fetch flags from.
- **`AMPLITUDE_COHORT_SERVER_URL`**

  - **Description**: Optional. The server URL, including protocol and host, to download cohorts from.

| Field | Type | Description |
| --- | --- | --- |
| `projects` | array | Required. Refer to [`projects`](#projects). |
| `configuration` | object | Optional. Refer to [`configuration`](#configuration-1) |

### projects

A required array of objects with the following fields, all which are required.

- **`id`**

  - **Description**: The project's ID. Found in the project settings.
- **`apiKey`**

  - **Description**: The project's [API key](https://amplitude.com/docs/apis/keys-and-tokens#api-key).
- **`secretKey`**

  - **Description**: The project's [secret key](https://amplitude.com/docs/apis/keys-and-tokens#secret-key).
- **`managementKey`**

  - **Description**: The [Experiment management API key](https://amplitude.com/docs/apis/keys-and-tokens#management-api-key). Must be created for the same project as the configured API and secret key. Used to automatically access and update deployments used for the project.

### configuration

An optional object of extra configuration.

- **`redis`**

  - **Description**: Optional (Recommended). Refer to [`redis`](#redis). Configure the proxy to use redis as persistent storage.
- **`flagSyncIntervalMillis`**

  - **Description**: Optional. The polling interval to update flag configurations (default `10000`).
- **`maxCohortSize`**

  - **Description**: Optional. The maximum size of targeted cohorts that the proxy can download (default `2147483647`).
- **`serverUrl`**

  - **Description**: Optional. The server URL, including protocol and host, to fetch flags from. (default `https://api.lab.amplitude.com`)
- **`cohortServerUrl`**

  - **Description**: Optional. The server URL, including protocol and host, to download cohorts from. (default `https://cohort.lab.amplitude.com`)

> **Info:** EU data residency
>
> To use the evaluation proxy with the EU data center, set the [`serverUrl`](#configuration-1) and [`cohortServerUrl`](#configuration-1) configurations to hit the EU data center endpoints:
>
> ```yaml
> configuration:
>   # Other configurations...
>   serverUrl: "https://api.lab.eu.amplitude.com"
>   cohortServerUrl: "https://cohort.lab.eu.amplitude.com"
> ```

#### redis

Configure the evaluation proxy to use Redis as a persistent storage. Highly recommended to enable the evaluation proxy to run efficiently.

- **`uri`**

  - **Description**: Required. The full URI to connect to Redis with. Include the protocol, host, port, and optional username, password, and path.
- **`readOnlyUri`**

  - **Description**: Optional. Optional URI to connect to read only replicas for high scaling high volume reads to Redis read replicas.
- **`useCluster`**

  - **Description**: Optional. If `uri` is a cluster URL, set this to `true`. Defaults to `false`.
- **`readFrom`**

  - **Description**: Optional. Read routing strategy for **cluster mode only**: `REPLICA_PREFERRED` (default, prefer replicas) or `ANY` (read from any node in the cluster).
- **`prefix`**

  - **Description**: Optional. A prefix for all keys saved by the evaluation proxy. Defaults to `amplitude`.

## Deployment

The evaluation proxy is stateless. Deploy multiple instances behind a load balancer for high availability and scalability.

For example, a kubernetes deployment with greater than one replica.

### Kubernetes

Use the evaluation proxy [Helm chart](https://github.com/amplitude/evaluation-proxy-helm) to install the proxy service on kubernetes or generate the files needed to deploy the service manually. The repository also contains an [example of running the evaluation proxy on kubernetes](https://github.com/amplitude/evaluation-proxy-helm/tree/main/example) locally using `minikube`.

#### Helm

##### Add helm repo

```bash
helm repo add \
    evaluation-proxy-helm https://amplitude.github.io/evaluation-proxy-helm
```

##### Configure `values.yaml`

Configure the chart values. The recommended approach to configuring and installing the helm chart is using a values.yaml configuration file.

The chart's `evaluationProxy` value contents exactly match the evaluation proxy's configuration file fields.

```yaml
evaluationProxy:
  # At least one project is required.
  projects:
    - apiKey: "YOUR API KEY"
      secretKey: "YOUR SECRET KEY"
      managementKey: "YOUR MANAGEMENT API KEY"
  configuration: {}
#    redis:
#      uri: "redis://redis-master.default.svc.cluster.local:6379"
```

##### Install helm chart

```bash
helm install -f values.yaml \
    evaluation-proxy evaluation-proxy-helm/evaluation-proxy
```

### Docker

You may run [the docker image](https://hub.docker.com/r/amplitudeinc/evaluation-proxy) directly. First, create a [configuration](#configuration) file, then run the docker image mounting the file as a volume to the expected directory in the container.

```bash
docker run \
    -v CONFIG_FILE_PATH:/etc/evaluation-proxy-config.yaml \
    amplitudeinc/evaluation-proxy
```

> **Tip:** Docker compose example
>
> The [evaluation-proxy GitHub repository](https://github.com/amplitude/evaluation-proxy) also contains an example using `docker compose` to run the proxy alongside a local Redis image.

## Evaluation

The Evaluation Proxy exposes remote [Evaluation API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api) and [SDK](https://amplitude.com/docs/sdks/experiment-sdks) endpoints to run local evaluation within your cluster. This is useful to enable platforms and languages which aren't supported by local evaluation SDKs. As an added benefit, fetch requests made to the evaluation proxy can target cohorts of users, and have assignment events tracked automatically to Amplitude.

**You must send requests to the service using `http` on port `3546`.**

> **Example:** Kubernetes
>
> A Kubernetes deployed Evaluation Proxy service (named `evaluation-proxy`) running within a kubernetes namespace `main` is from within the cluster at: `http://evaluation-proxy.main.svc.cluster.local:3546`

## Best practices

### Production deployment

#### Resource requirements

Configure each pod with 4 CPU cores and 9 GiB RAM for a capacity of approximately 5,000 requests each second.

- **Minimum replicas**: Deploy at least two replicas for high availability.
- **Horizontal scaling**: Add pods to increase capacity. For example, four pods provide approximately 20,000 requests each second.

#### JVM heap for large cohorts

If you use local evaluation and download large cohorts greater than 5M users, set a 6 GiB max JVM heap size.

```yaml
- name: JAVA_TOOL_OPTIONS
  value: "-Xms128m -Xmx6144m"
```

#### Redis configuration

##### Option 1: Standalone with replicas

Use for small deployments.

```yaml
configuration:
  redis:
    uri: "rediss://primary:6379"
    readOnlyUri: "rediss://replica:6379"  # Optional, for high-volume read scaling.
```

**Recommended specs**: 12+ GiB memory, cache.m7g.xlarge or equivalent.

##### Option 2: Redis Cluster (recommended for high scale)

Use Redis Cluster when you have 10M+ users or many large cohorts.

```yaml
configuration:
  redis:
    uri: "rediss://cluster:6379"
    useCluster: true
```

**Recommended specs**: 2-3 shards, 1-2 replicas per shard, 12+ GiB per node.

> **Tip:** Cluster-based approach
>
> Prefer a cluster-based approach as cohort size and count increase. Test with your data to finalize the configuration.

### Performance characteristics

#### Latency

- **Normal**: 1-5ms.
- **During background cohort refresh**: p95 latency can reach up to 50ms. This typically occurs when you add or remove a significant number of users from a cohort, or attach new large cohorts to a flag.

#### Cold start

Initial startup time scales with the number and size of targeted cohorts.

Set the readiness probe `initialDelaySeconds: 600` to prevent pod restarts during startup.

### Monitoring and alerts

The proxy exposes metrics at `http://proxy:9090/metrics`.

Configure these critical alerts:

```promql
# Error rate > 1%
rate(http_requests_total{status=~"5.."}[5m]) > 0.01

# P95 latency > 100ms sustained (ignore during cohort refresh spikes)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.1

# Memory > 85%
container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85

# Redis errors
rate(redis_errors_total[5m]) > 0
```

### Troubleshooting

| Issue | Solution |
| --- | --- |
| High latency (sustained >100ms) | Check Redis latency: `redis-cli --latency -h <host>` |
| Cohorts not loading | Verify `managementKey`, check logs for sync errors. |
| Proxy won't start | Verify Redis connectivity, check all API keys. |
| Cold start taking too long | Normal for large cohorts (5-10 min), increase readiness `initialDelaySeconds`. |

### Capacity planning

| Load | Pods | Redis |
| --- | --- | --- |
| <10k req/s | 2 | Standalone + replica |
| 10-20k req/s | 3-4 | Standalone or 2-shard cluster |
| >20k req/s | 5+ | 2-3 shard cluster |

================================================================================
# Troubleshoot your experiment
URL: https://amplitude.com/docs/feature-experiment/workflow/experiment-troubleshoot
================================================================================

# Troubleshoot your experiment

Unexpected issues sometimes surface when you create and roll out an experiment or flag. This page describes common issues and their workarounds.

This article assumes you know how to set up and run an experiment in Amplitude Experiment. If you need a refresher, review the Help Center articles on [an overview of Amplitude Experiment](https://amplitude.com/docs/feature-experiment/overview) and [configuring your experiment](https://amplitude.com/docs/feature-experiment/workflow/configure) before proceeding.

> **Tip:** Use an A/A test to validate your setup
>
> If you suspect a systemic instrumentation or assignment issue, [run an A/A test](https://amplitude.com/docs/feature-experiment/aa-testing) to confirm your experimentation platform works correctly before debugging individual experiments.

## Experiment troubleshooting checklist

Use these questions as a quick checklist for experiment discrepancies. If you can answer "yes" to a question, that question is unlikely to be the cause of your issue.

**Do the device ID and user ID align?**

The device ID and user ID you use for experiments and flags must match the device ID and user ID in Amplitude Analytics.

**Is your flag enabled?**

A flag must be enabled in a deployment to appear in that deployment.

**Does your variant use a name other than "off"?**

Don't name a variant "Off." Amplitude reserves this value for users in the "OFF/FALLBACK" bucket.

A fallback is the default variant (usually "control") that Amplitude serves when Experiment can't assign a user to a treatment group.

**Do your assignment event and exposure event point to different events?**

The assignment event and the exposure event can use the same event, but they represent different concepts. Each plays a different role in your experiment:

- The assignment event immediately precedes a user's assignment to a variant.
- The exposure event is the event the user must trigger to receive the variant.

Amplitude assigns users at the beginning of their session, and users might not trigger the exposure event until later in the flow.

Confirm the correct events map to each.

Go to [Feature Assignment Events](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#assignment-events) for more information.

**Are your old properties synced to new ones?**

If you run an experiment on a single platform (for example, iOS), users with a different platform value (for example, Android) might still appear. This happens when users access your product on multiple platforms and their most recent event comes from a platform other than the one your experiment targets. The assignment event fires automatically and doesn't contain non-Experiment properties such as `platform`.

## Other experiment discrepancies

Some experiment issues fall outside your direct control. This section highlights those scenarios.

### Targeting on user properties is delayed

Amplitude Experiment updates cohort targeting each hour. Targeting on properties sent directly to Experiment applies in real time. Targeting on user properties that Amplitude stores can lag by up to one hour because of the [CDN](https://amplitude.com/docs/feature-experiment/under-the-hood/experiment-performance-scaling).

### A user didn't receive their assigned variant

Amplitude Experiment generates data on the server side and stores assignments as user properties. A user might not receive their assigned variant because of timeouts, network errors, or ad blockers.

A user can also receive one variant one day and a different variant the next. Because [bucketing in Amplitude Experiment relies on user ID and device ID](https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping), users can land in a second bucket when you rely on device ID alone and the user's device ID changes. This happens when the user opens incognito mode or clears their cache and cookies after their last visit.

### A user doesn't log any events past the assignment event

A user can trigger the assignment event without logging any other active events:

- The user's ad blockers or analytics blockers interfere with analytics application events. Feature Experiment requests come from backend servers and usually pass through, but blockers can affect Web Experiments.
- A user opens the app, which sends a request to Amplitude's backend servers and triggers the assignment event. The user then leaves the app before performing any other actions that trigger an analytics event.
- Amplitude's servers receive assignment requests directly from your backend instead of from the end user. Check if you instrumented experiments in offline campaigns such as group push notifications or marketing emails.

================================================================================
# Exposures Without Assignments
URL: https://amplitude.com/docs/feature-experiment/troubleshooting/exposures-without-assignments
================================================================================

# Exposures Without Assignments

The Exposures without Assignments chart appears in the Diagnostics card. The chart shows the cumulative number of unique users who triggered an exposure event without a corresponding assignment event each day.

If a large number or percentage of users appear in the chart, interpret your experiment results carefully. Investigate what happens if someone gets exposed to the experiment when they shouldn't:

- Is the experience bad?
- Can the user view the experience?
- What does it mean if a user can view both experiments when they're mutually exclusive?

Exposure without assignment can also affect future experiments, so investigate and fix the issue.

> **Note:**
>
> This chart doesn't appear if you selected the assignment event as the exposure event, or if you're using [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation).

> **Note:**
>
> Sometimes Amplitude delays the exposure event and sends it on a different day than the assignment event. For example, Amplitude sends the assignment event today and the exposure event tomorrow. An issue exists if, between the assignment and the exposure events, the user's properties change in a way that affects whether they should be targeted. Otherwise, ignore this warning.

## Causes

A significant number of users in the Exposures without Assignments chart can come from a few scenarios:

- Identity mismatch between assignment and exposure.

  - The user ID and device ID are incorrect, switched, or [missing](https://amplitude.com/docs/apis/analytics/http-v2#device-ids-and-user-ids-minimum-length) on either assignment or exposure. For example, sending the device ID as the user ID or vice versa.
- Account switching on the same device.

  - If a real user has multiple accounts on the same device, and you don't call fetch on login or logout, the value that `variant()` accesses triggers an exposure for the new user without an assignment event. Re-call `fetch()` whenever the user identity changes.
- Exposures for fallback variants.

  - If you manually track exposure events, don't track exposure events for fallback or default variant values. For example, if a user doesn't get assigned a variant for an experiment and you show the user the control, don't track an exposure event with the variant value `control` for that user.

## Debugging

To debug exposure without assignment, open the chart in analytics and view user streams. Common problems include:

1. Users who only have assignment or exposure events. This issue likely comes from an identity mismatch between assignment (`fetch()`) and the exposure tracked through analytics.
2. User login followed by exposure events without an assignment event. This issue likely comes from account switching on the same device. Use user lookup with the device ID to determine whether multiple logged-in users share the same device. If a user has multiple accounts on the same device, calls `fetch()`, signs out, signs in to another account, and then calls `variant()`, this pattern occurs. Re-call `fetch()` whenever the user identity changes.

## Problems with your experiment

- Users can get exposed to the experiment when they don't meet the rule-based targeting criteria. This happens because Amplitude checks the rule-based targeting on the `fetch()` call.
- Users can get exposed to both experiments even though the experiments are mutually exclusive.

================================================================================
# Sample Ratio Mismatch
URL: https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch
================================================================================

# Sample Ratio Mismatch

In Amplitude Experiment, a sample ratio mismatch (SRM) occurs when the observed variant allocation significantly differs from the specified allocation. For example, you allocated 50% of your Experiment traffic to control and 50% to treatment, but you observe a ratio of 55% control to 45% treatment.

An SRM points to biases in the data and can lead to unexpected results when unresolved. Treat the results of any experiment with an SRM as suspect.

Amplitude uses a [sequential version of a chi-squared test](https://arxiv.org/abs/2011.03567) with alpha = .01 to detect an SRM.

This guide describes the causes of an SRM, ordered from most to least common, and how to debug each one. It assumes you use the end-to-end Experiment product, but you can apply many of these steps to experiments set up in Experiment Results.

> **Info:** More information
>
> This list doesn't cover every possible cause of an SRM. Go to [SRM Checker](https://www.lukasvermeer.nl/srm/docs/faq/#what-can-we-do-about-sample-ratio-mismatch) for more debugging strategies.
>
> If you still have issues with an SRM after using this guide, contact support.

## Amplitude exposure events versus custom exposure events

Amplitude recommends the [Amplitude exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events). On client-side SDKs, Amplitude tracks the exposure event automatically when the variant loads from the cache, not when Amplitude fetches the variants. Local evaluation doesn't produce an assignment event, so you must use a custom exposure event.

If you use a custom exposure event, send it when the user experiences the variant. The custom exposure event can happen before assignment. When this happens, the user property isn't yet set, and Amplitude doesn't count the initial custom exposure event as an exposure in analysis. Amplitude exposure events don't cost extra.

## Variant distribution weights changed during the experiment

As a best practice, don't change a running experiment in a way that causes users to jump between variants. Changes like this can cause an SRM.

For example, changing 50% treatment / 50% control to 60% treatment / 40% control can cause users to jump between variants while the experiment runs. The SRM test assumes that traffic allocation doesn't change while the experiment runs. Go to [Interpret the Cumulative Exposures Graph in Amplitude Experiment](https://amplitude.com/docs/feature-experiment/advanced-techniques/cumulative-exposure-change-slope) for more context about why you shouldn't change traffic allocation mid-experiment.

## Experiment exposures started before analysis window begins

If one variant causes users to return to the product more, that variant can show more users than expected when the exposure window and analysis window don't align.

> **Example:**
>
> The experiment ran from January 1 to January 30, and analysis ran from January 15 to January 30. Every two weeks, 100 users enter control and 100 enter treatment. The treatment is so good that all treatment users return to the product every day. The control is so bad that control users never come back.
>
> During the analysis window, you have 100 exposed users in control and 200 exposed users in treatment. The comparison between control and treatment isn't fair.

If the analysis window and the traffic window differ, change the analysis window to cover the full traffic window and check whether the SRM remains.

## Variant jumping

[Variant jumping](https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping) describes when a user moves from one variant to another, sometimes multiple times. Variant jumping makes it difficult to attribute the metric to a specific variant. Amplitude Experiment's built-in diagnostics include charts that track the percentage of users who jump between variants.

If users jump between variants, check whether the cause is anonymous users (people who log in and out frequently) or changing device IDs. Review this in the [User Stream](https://amplitude.com/docs/analytics/user-data-lookup).

As a best practice, don't change a running experiment in a way that causes users to jump between variants. Changes like this can cause an SRM.

## Significantly more users converted from assignment to exposure for one variant

You can find the assignment-to-exposure funnel chart in the Diagnostics card of an experiment. Enter your conversion rates and sample size into [this calculator](https://www.socscistatistics.com/tests/ztest/default2.aspx) to check whether your metrics are statistically significant at the 95% level. The conversion rates between variants should be similar, within randomness, because the assignment event splits users into two randomly assigned, equal cohorts. Amplitude sends exposure events immediately before the user experiences a variant, so users have the same experience between the assignment and exposure events.

## Variant added or removed in the experiment

Adding or removing a variant while an experiment runs can cause an SRM. As a best practice, don't change a running experiment in a way that causes users to jump between variants.

## Sticky bucketing enabled or disabled during experiment

Enabling or disabling sticky bucketing on a running experiment can cause a mismatch between the variant you expect and the variant a user receives. After you enable sticky bucketing on an experiment, Amplitude always evaluates each user to the same previously bucketed variant, regardless of the current targeting, even if you later disable sticky bucketing.

## Changes that affected a segment of users

Check whether the SRM affects only a certain segment of users. For example, filter by [country, OS version, app version, or platform](https://amplitude.com/docs/get-started/user-property-definitions).

To troubleshoot this kind of problem, find out which specific users the change affected. Ask questions like:

- Did someone push an instrumentation bug to an app version that a subset of users have?
- Did someone push a change that affects only users in a certain country?

## Individual allocation of many users

Check whether you [individually allocated](https://amplitude.com/docs/feature-experiment/implementation#individual-inclusions) a large number of users. Individual allocation isn't random and can skew your ratio, which can cause an SRM.

Adding a few users to the experiment individually is fine, but avoid adding many users.

## First experiment

If an SRM appears in your first experiment, the cause might be an instrumentation bug. If you have multiple SRMs, the SRM is unlikely to result from a false positive, and you should find the root cause.

## Users often logging out

Frequent logouts can appear in the variant jumping chart. Check whether the experiment causes users to log out more often. This pattern is common in financial institutions, where the application logs users out after 15 minutes of inactivity.

1. On logout, the application regenerates the device ID.
2. Amplitude treats the session as a new user.
3. Amplitude buckets the user into a different variant.
4. The user ID later resolves.

This pattern can also relate to anonymous users from sign-up flow experiments.

## Missing exposures

Check whether two users have the same sequence of events but one is missing an exposure event. You might not send the exposure event in some cases, or you might send it when you shouldn't.

## Incorrectly sending exposures for fallback variant

If you don't use the Experiment SDK to track exposures automatically, check whether you incorrectly send exposures for a fallback (default) variant.

Check for cases where the flag defaults to control and Amplitude counts that as an exposure when the flag doesn't return a response in time. When this happens, control has more exposures than treatment.

> **Note:**
>
> Client-side SDKs don't send exposure events for fallback variants.

================================================================================
# Variant jumping
URL: https://amplitude.com/docs/feature-experiment/troubleshooting/variant-jumping
================================================================================

# Variant jumping

Variant jumping occurs when a user receives two or more variants for a single flag or experiment. Variant jumping above a certain threshold can raise concerns about the trustworthiness of an analysis.

Some types of variant jumping are normal: common and usually explainable. Other types are harder to track down.

## Debug variant jumping

The best way to debug variant jumping is to identify a user who jumped variants and analyze their user timeline. If you use remote evaluation, check the `Assignment` event to identify assignment or exposure discrepancies.

To find a user who jumped variants, follow these steps:

1. Navigate to the diagnostics card of your experiment.
2. Find the variant jumping charts, and click **Open in Chart**.
3. Click the bar for users who jumped variants and click **View User Streams**.

Show only assignment and exposure events in the user timeline for easier analysis.

When you debug a user timeline, note the following:

- Did you introduce any [targeting changes](#targeting-changes) while your flag or experiment was active? Could the timing of the change have affected the variant assigned to this user? Click the version number to check the version history.
- Which bucketing key did the flag or experiment use? Does the value for this property change between assignments or exposures for this user?
- Does the user have missing exposures or assignment events? If so, the missing event might have been sent for a different user.
- Is an assignment missing a user property it should have? If so, double-check the server upload timestamp of the assignment event and compare it to surrounding active events. Events sent from the client can arrive at Amplitude after the assignment event, even if the client order is different.

## Normal variant jumping

Normal variant jumping can occur due to:

- [Targeting changes](#targeting-changes): Someone changed targeting rules while your experiment was running.
- [Anonymous identity merging](#anonymous-identity-merging): Anonymous users, bucketed by Amplitude ID, can receive different variants until Amplitude resolves them through a matching user ID.

### Targeting changes

The following actions can cause a user to jump variants:

- Adding or removing a variant.
- Changing variant distribution weights.
- Targeting dynamic cohorts.
- Changing the bucketing key.
- Updating mutual exclusion.

> **Tip:** Avoid variant jumping by enabling sticky bucketing
>
> Enable sticky bucketing before you make targeting changes to prevent variant jumping. Sticky bucketing can cause a [sample ratio mismatch (SRM)](https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch).

### Anonymous identity merging

The way Amplitude handles anonymous users can lead to variant jumping. Amplitude merges anonymous IDs with the correct existing user IDs (if one exists) as soon as Amplitude has enough information to do so. This merging can happen if a user uses your app on different devices without logging in, or if the device ID regenerates upon logout.

[Learn more about Amplitude's identity resolution and merging users.](https://amplitude.com/docs/data/sources/instrument-track-unique-users)

To identify this type of variant jumping, find the assignment event where the user jumped between variants. Then compare the Amplitude ID for both events. If the Amplitude ID differs between the events, anonymous identity merging is the likely cause.

To avoid this type of variant jumping, bucket by user ID if you target only logged-in users who have user IDs. Alternately, bucket by device ID if you target only anonymous users (for example, a sign-up experiment).

### Inclusion list

Imagine you have some user IDs in the inclusion list. You call `fetch()` and pass the user ID into the call. The call returns the control experience, per the inclusion list. The next time you call `fetch()`, you don't pass in the user ID. The user no longer meets the inclusion list criteria, so Amplitude hashes the bucketing key, and the user can receive a different variant instead. The same outcome can happen if you include device IDs.

In the following example, the user has a user ID. The user matches the inclusion list and receives the `signin-up-new_design` experience.

![A user who has a user ID matches the inclusion list and receives the `signin-up-new_design` experience.](https://amplitude.com/docs/images/experiment-troubleshoot/inclusion-list-variant-jumping-1.png)

In this example, no user ID exists. The user doesn't match the inclusion list, falls into the "all other users" segment, and receives the `signin-up-original-view` experience instead.

![A user with no user ID doesn't match the inclusion list, falls into the "all other users" segment, and receives the `signin-up-original-view` experience.](https://amplitude.com/docs/images/experiment-troubleshoot/inclusion-list-variant-jumping-2.png)

## Abnormal variant jumping

Instances of abnormal variant jumping don't fit any of the [normal explanations](#normal-variant-jumping). Abnormal variant jumping can be difficult to track down. The most frequent cause is identity mismatch: the user identity differs between assignment and exposure tracking. Identity mismatch almost always comes from an implementation inconsistency.

The following examples aren't exhaustive, but they illustrate common identity issues in Amplitude Experiment.

### Multiple logged in accounts on a single device

Consider this timeline for a person with multiple user accounts (U1 and U2), for your app on a single device.

1. Open the app as user U1 and fetch variants. U1 receives `treatment` for `experiment-1`.
2. Expose U1 to `experiment-1` variant `treatment`.
3. Log out of U1 and into U2, fetching variants asynchronously on login.
4. Before the fetch for U2 resolves, U2 sees the exposure to `experiment-1` variant `treatment`.
5. The fetch for U2 resolves. U2 receives `control` for `experiment-1`.
6. Expose U2 to `experiment-1` variant `control`.

In this case, user U2 jumped variants, from `treatment` to `control`, after seeing U1's stored variant. To avoid this case, either wait for the fetch to resolve before you render the user experience, or call the SDK's `clear()` method on logout to clear all stored variants from the SDK. Clearing variants wipes the SDK's variant storage and prevents the user from seeing cached variants. Clearing variants doesn't protect the user from viewing a fallback experience before the fetch request resolves.

If you keep a consistent device ID across logins, you can check for this type of variant jumping. Search for different users that share the same device ID.

### Inconsistent identity input between assignment and exposure

In Amplitude, the user ID and device ID properties identify your user and [resolve their Amplitude ID](https://amplitude.com/docs/data/sources/instrument-track-unique-users). If the device ID or user ID used to fetch and evaluate assignments differs from the device ID and user ID used to track the exposure event, you can see variant jumping, SRMs, and inconsistent or unexpected bucketing behavior.

For example, you might send events through a proxy or CDP that masks IDs before sending to Amplitude. In this case, the identity used to fetch variants differs from the identity included in the exposure events.

Another common cause is a simple implementation error. For example, the following cases have caused variant jumping:

- Additional characters in the ID. Note the extra quotes around the actual identity:

  - `15a4f7e9-db4e-4c57-82c7-e57a2995803a`
  - `"15a4f7e9-db4e-4c57-82c7-e57a2995803a"`
- Inconsistent capitalization, especially with UUIDs:

  - `15a4f7e9-db4e-4c57-82c7-e57a2995803a`
  - `15A4F7E9-DB4E-4C57-82C7-E57A2995803A`

## Remove users who variant jumped from experiment analysis

As you analyze results, be careful when you remove data, because removing data can introduce bias. It's better to understand the cause of variant jumping and fix any implementation bugs to prevent recurrence in future experiments. If removing users who jumped variants is the best course of action, use the Filter card on the Experiment Analysis tab. If the `All exposed users` segment is enabled by default, click it and select _Experiment Segments > Exclude users who variant jumped_.

================================================================================
# Debug metric spikes and dips
URL: https://amplitude.com/docs/feature-experiment/troubleshooting/debug-metric-spikes-dips
================================================================================

# Debug metric spikes and dips

When you notice unexpected spikes or dips in experiment metrics, identify what changed and determine if the data is trustworthy. This guide provides systematic approaches to debug these anomalies.

## Check cumulative exposures for traffic changes

Examine the cumulative exposures chart in your experiment dashboard. The chart shows exposure patterns over time and highlights data quality issues.

Orange dots on the cumulative exposures chart mark detected anomalies in exposure traffic. Orange dots can signal:

- Sudden increases or decreases in exposure counts.
- Changes to experiment configuration.
- Instrumentation or tracking issues.

Data quality check failures appear when traffic to the experiment drops significantly. Review these warnings to determine if the spike or dip relates to exposure problems rather than metric changes.

For more details, refer to [Interpret the cumulative exposures graph](https://amplitude.com/docs/feature-experiment/advanced-techniques/cumulative-exposure-inflection-points).

## Review experiment configuration changes

Check if the experiment's percentage rollout or traffic allocation increased during the period when you noticed the metric spike or dip.

To review recent changes:

1. Go to the experiments or flags table on your Amplitude homepage.
2. Sort by _Last Modified_ to see which experiments changed recently.
3. Check the experiment's configuration history to identify what changed and when.

Use the [Experiment Management API](https://amplitude.com/docs/apis/experiment/experiment-management-api-version-endpoints) or [Slack Notifications](https://amplitude.com/docs/feature-experiment/notifications) to retrieve versions of flag configurations during a specific time period. Compare configurations to identify changes that might explain the metric spike or dip.

## Check for releases and deployments

Review whether any code releases, app version updates, or feature deployments occurred during the time window of the metric change.

Releases and annotations in your dashboard help identify temporal correlations between deployments and metric changes. If you haven't added annotations yet, add them after you identify the cause to document the incident.

## Segment by platform and version

Metric spikes or dips often affect specific platforms or app versions rather than all users. Group your analysis by these default user properties:

- App version: New app releases can introduce bugs or behavioral changes.
- OS version: Operating system updates can affect app performance.
- Platform: Issues might only affect iOS, Android, or web users.
- Device type: Specific device models or screen sizes might behave differently.

Amplitude SDKs track these properties automatically, making them reliable dimensions for debugging. Refer to [user properties](https://amplitude.com/docs/data/user-properties-and-events#user-properties) for more information.

To analyze by these dimensions:

1. In your experiment results, add a group-by clause for the relevant property.
2. Look for spikes or dips that affect only specific segments.
3. Investigate what's unique about the affected segment.

## Identify and filter outliers

Outlier users or data quality issues cause some metric spikes:

### Bot traffic

Automated bot traffic can skew metrics significantly. Amplitude provides several ways to handle bots:

- Use [bot traffic blocking](https://amplitude.com/docs/data/block-bot-traffic) to filter known bots automatically.
- Configure [block filters and drop filters](https://amplitude.com/docs/data/troubleshooting/instrumentation-issues#block-and-filter-internal-users) to exclude specific traffic patterns.

### Instrumentation issues

Rapid-fire events from a single user can indicate instrumentation bugs:

- Components that re-render frequently might send events on each render.
- Events might need debouncing to prevent duplicate tracking.
- Event handlers might fire multiple times unintentionally.

Use the [frequency chart](https://amplitude.com/docs/feature-experiment/advanced-techniques/winsorization-in-experiment#filtering-out-users) to identify users with unusually high event counts. Investigate these outliers, which can distort your metrics.

To filter outliers from your analysis:

1. Create a frequency distribution of your metric.
2. Identify users with extreme values.
3. Investigate whether these users represent genuine behavior or data quality issues.
4. Apply filters to exclude confirmed outliers from your experiment results.

## Analyze exposure events by flag key

Create an event segmentation chart to identify spikes in exposure events for specific flag keys:

1. Go to _Analytics > Event Segmentation_.
2. Select the `[Experiment] Exposure` event.
3. Group by the `flag_key` property.
4. Sort by _Row Change_ to see which flags had the largest exposure changes.

This analysis can reveal if a specific experiment suddenly received more traffic, which might explain metric changes.

## Use Root Cause Analysis

Root Cause Analysis identifies which user segments contributed most to a metric change. You can't filter exclusively by experiment properties, but you can:

1. Go to the metric that showed a spike or dip.
2. Use Root Cause Analysis to identify contributing factors.
3. Look for patterns in user properties, behaviors, or segments.

## Resolve and prevent recurrence

After identifying the cause of a metric spike or dip:

1. Document your findings with annotations in your dashboard.
2. If a bug caused the issue, fix the instrumentation.
3. If bots or outliers caused the issue, configure filters to prevent future occurrences.
4. If a configuration change caused the issue, restart the experiment with stable settings.
5. Review [sample ratio mismatch troubleshooting](https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch) if the spike or dip correlates with exposure distribution changes.

================================================================================
# Web Experiment Out-of-the-box Widgets
URL: https://amplitude.com/docs/web-experiment/out-of-the-box-widgets
================================================================================

# Web Experiment Out-of-the-box Widgets

Out-of-the-box (OOTB) widgets let you test new website elements like modals and banners without designing or coding them yourself. OOTB widgets are pre-built, configurable components that you customize directly in the visual editor to match your brand. OOTB widgets remove engineering setup like SDK and UI framework integration, so you can validate ideas, gather learnings, and scale your experimentation program. Saved design presets and reusable styles keep your brand consistent across experiments. For more information on the Visual Editor and setting up a Web experiment, go to [The Visual Editor](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor).

OOTB widgets use native Amplitude AI so you can design your Web experiment through conversational prompts instead of supplying precise brand style details.

The OOTB widgets library includes:

- **Buttons**: a call to action (CTA) button. When a customer clicks a CTA button, the button triggers an action, like sending the customer to a shopping cart or to your support agents.
- **Banner**: a notification or messaging bar. Message banners announce important information to your customer, like expected service disruptions, sales, or other communications.
- **Banner with a button**: a notification or messaging bar that includes a CTA button. Banners with buttons announce a specific action you want customers to perform, like applying a promo code to a shopping cart.
- **Modal pop-up**: a pop-up modal that appears on your page after a set amount of time. Modals include CTA buttons and messaging. A pop-up modal appears after a few seconds and encourages customers to take an action, like signing up for an account.

You can also save customized widgets to the library for future use. For example, if you create a notification banner that announces a quarterly sale, you can save that customized message and reuse it later. Everyone in your organization with permission to create experiments can access all saved widgets.

## Add an OOTB widget to a web experiment

You can add an OOTB widget to any Web experiment. After you create your [Web experiment](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment), click **Open Visual Editor**. Click **Insert** from the top menu bar. The OOTB widgets modal opens, and you can select the widget you want. Drag and drop the widget onto your experiment. The **Insert** panel also lets you [upload an image](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#image-upload) directly.

### Edit a widget

By default, you can edit the following aspects of any widget:

- Text.
- Button label.
- Background color.
- Font.

When you select a widget, the element toolbar lets you delete the widget or save the widget to the library.

The Visual Editor displays all changes in real time.

### Customize widget placement

You can freely move banners and buttons around your website:

- Banners can appear at the top or bottom of a page.
- Buttons can appear almost anywhere on the page. After you place a button, it snaps to a grid layout.

You can't customize the location of the pop-up modal, which appears in the center of your page.

For more information on moving banners and buttons, go to [Moving elements](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#move).

### AI Stylizer

AI Stylizer is an AI assistant in the Visual Editor that helps you refine copy, apply brand-aware styling, and clean up spacing, alignment, and hierarchy in your OOTB widgets. You don't need precise brand style guide details, design experience, or coding skills. Use natural-language prompts to align widgets or elements to your brand's style.

When you select a widget in the Visual Editor, the **AI Stylizer** control appears in the element toolbar as a sparkle icon. Select the control to open the AI Stylizer popover. You can describe what you want in plain language.

For example, enter "Make this headline more urgent", "Create a promotion for summer that contains a 15% discount promocode", or "Make the CTA more action-oriented." AI Stylizer updates the selected element based on your request.

You can use AI Stylizer in the following ways.

When you use AI Stylizer on a pre-built widget (modal, CTA, banner, or promo block), AI Stylizer adapts the element to your brand. AI Stylizer updates typography, colors, and spacing to match your site and can create or refine headline and CTA text so the component looks like part of your product.

- **Headlines:** make headlines more compelling or urgent, simplify and clarify the topic, or tailor headlines for new visitors.
- **Buttons (CTAs):** make CTAs more action-oriented, increase contrast and visibility, and customize CTAs for returning customers.
- **Banners:** make banners more prominent, simplify the messaging, and customize banners for new visitors.
- **Text blocks:** simplify and clarify the content, shift from features to benefits, and customize text for mobile users.

You can choose a preset prompt or enter your own custom prompt. AI Stylizer can generate multiple variants in a single interaction so you can pick the one you want. You can also revert to a previous AI-generated version from the AI Stylizer version history. You can continue to edit AI-created styles and copy after you apply the styles to your experiment. For example, you select a hero headline and ask AI Stylizer to make the headline more urgent or clearer. You like the first design and apply the design to your experiment. After you receive feedback from colleagues, you return and update the headline with their suggestions.

> **Note:** Using AI Stylizer on embedded OOTB widgets
>
> You can't use AI Stylizer on an element that contains an embedded OOTB widget. If you try to apply AI Stylizer to such an element, Amplitude displays an error message.

================================================================================
# Web Experiment actions
URL: https://amplitude.com/docs/web-experiment/actions
================================================================================

# Web Experiment actions

Actions define how variants modify your site. Actions relate to variants rather than a specific page, and apply to specific [Pages](https://amplitude.com/docs/web-experiment/pages) to control where they take effect.

Experiment applies variant actions during evaluation. Evaluation runs on the initial page load and any time state pushes to or pops from the session history. On a history state change, the SDK first reverts all applied element change and custom code actions, then reevaluates and reapplies actions for the updated page.

## Element changes

Element changes modify existing elements on your site. Web Experiment applies these changes by editing the inner text of an element or appending style to the element based on the change you make in the visual editor.

The visual editor supports the following element changes:

- _[Display](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor)_: Show or remove the element from the DOM.
- _[Visibility](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor)_: Show or hide the element.
- _[Text](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor)_: Update an element's inner text, color, and size.
- _[Background](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor)_: Update a background image or color.
- _[Move](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#move)_: Move the position of an element.

For image and background changes, you can [upload an image](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#image-upload) directly in the Visual Editor instead of pasting a URL. Amplitude hosts the file and applies its CDN URL to the element.

## URL redirect

URL redirects load a new URL when a targeted user lands on a targeted page in your experiment. URL redirects happen on the client, and aren't the same as a server redirect with a `3xx` response. Use URL redirect when your variants are different URLs or pages. For example, use it to test landing pages or different versions of a page built in a CMS. URL redirect works with standard A/B tests and [multi-armed bandits](https://amplitude.com/docs/feature-experiment/workflow/multi-armed-bandit-experiments).

URL redirects retain any query parameters on the original page URL. For example, you create a variant to redirect users from `https://example.com` to `https://example.com/get-started`. If a user clicks a link `https://example.com?utm_source=facebook`, Web Experiment redirects that user to `https://example.com/get-started?utm_source=facebook`.

### Set up a URL redirect

Set up URL redirect through the [Visual Editor](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment#the-visual-editor) as a variant action. To add a URL redirect to a treatment variant, follow these steps:

1. [Create a Web Experiment](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment) and open the Visual Editor.
2. Click the Treatment three-dot menu, select **Edit**, then under **Action** select **URL Redirect**.
3. In the URL Redirect panel, add each URL you want to test as a separate variant and click **Apply**.

### Configuration limits

Visual experimentation and Amplitude's low-code implementation apply the following limits when you use URL redirect:

| Setting | Limit | Reason |
| --- | --- | --- |
| Evaluation mode | `local` | Optimizes test performance and minimizes latency impact to end users. |
| Bucketing unit | `User` | Evaluation mode is `local`. |
| Keys | `deviceID` | Evaluation mode is `local`. |
| Audience | `all users` | URL redirect logic uses local evaluation mode. |
| Deployment | Project API key | Simplifies setup requirements. |

> **Note:**
>
> It's possible for the URL redirect test to have a [Sample Ratio Mismatch (SRM)](https://amplitude.com/docs/feature-experiment/troubleshooting/sample-ratio-mismatch). Redirect tests work by loading the redirected page, ideally as fast as possible. The sequence for this is:
>
> Control flow:
>
> - Load control page HTML.
> - Browser parses and loads dependencies (including the experiment script).
> - Experiment script initializes, evaluates the user, and logs an impression if the user is in control.
>
> Treatment flow:
>
> - Load control page HTML.
> - Browser parses and loads dependencies (including the experiment script).
> - Experiment script evaluates the user, assigns the user to treatment, and triggers a redirect.
> - Load treatment page HTML.
> - Browser parses and loads dependencies (including the experiment script again).
> - Experiment script initializes and logs the impression for treatment.
>
> Because the treatment flow involves more steps before logging the impression, users who bounce quickly (for example, after clicking an ad by mistake) are more likely to count in control than in treatment. This imbalance can appear as Sample Ratio Mismatch (SRM).
>
> Researchers report similar effects: if treatment slows performance, more users leave before impressions log, which reduces recorded impressions. Faster performance has the opposite effect, producing more recorded users in treatment than in control.
>
> To resolve SRM, follow these guidelines:
>
> - Reduce latency in evaluation. The longer the delay before logging impressions, the more pronounced the SRM effect.
> - Use local evaluation where possible. For example, target only on browser properties instead of slower remote attributes like Country. Local evaluation reduces the chance that users drop off before logging. The following configuration example shows local evaluation in Amplitude Experiment.
>
> Further reading:
>
> - [Causes of SRM](https://www.lukasvermeer.nl/srm/docs/causes/)
> - [Pitfalls in Metric Interpretation (KDD 2017)](https://exp-platform.com/Documents/2017-08%20KDDMetricInterpretationPitfalls.pdf)
> - [Diagnosing SRM in Online Experiments (KDD 2019)](https://exp-platform.com/Documents/2019_KDDFabijanGupchupFuptaOmhoverVermeerDmitriev.pdf)

### SEO best practices for redirects

Client-side redirects in experiments can affect how search engines index and rank your pages. Follow these best practices to minimize SEO impact.

#### Use temporary redirects during tests

Web Experiment uses client-side redirects that simulate a `302` temporary redirect through `window.location.replace`. The `302` signal tells search engines to keep the original URL indexed during the experiment. Don't use `noindex` tags on your control page, because `noindex` removes the page from search results.

After you pick a winner, implement a permanent server-side `301` or `308` redirect to consolidate link equity to the winning URL.

#### Add canonical tags on variant pages

Add a canonical tag on your experiment variant page that points back to the original (control) URL. The canonical tag prevents search engines from indexing the variant as a separate page.

```html
<!-- On the variant/redirect destination page -->
<link rel="canonical" href="https://example.com/original-page" />
```

#### Prefer server-side redirects for SEO-critical pages

For pages where SEO is critical, consider using server-side assignment and redirects at the edge (CDN) or server level. Server-side redirects execute before the page renders, which:

- Ensures search engine crawlers interpret redirects reliably.
- Minimizes page flicker for users.
- Provides proper HTTP status codes to crawlers.

If you must use client-side redirects, make sure the redirect logic runs in the `<head>` to reduce layout shift.

#### Keep tests short and avoid cloaking

Don't show different content or URLs to users compared to what Googlebot sees. Search engines consider this practice cloaking and may penalize your site. End experiments promptly and remove test logic after concluding to avoid SEO drift.

#### Ensure goals cover both URLs

Define conversion goals that include both control and variant URLs. Web Experiment [preserves query parameters](#url-redirect) through redirects, so attribution and session continuity stay intact.

#### Use Search Console if issues arise

If a search engine indexes your experiment variant page unexpectedly, use [Google Search Console's URL removal tool](https://search.google.com/search-console/removals) as a temporary fix while you address the underlying issue.

## Custom code

> **Note:**
>
> Custom code requires a Growth or Enterprise plan.

Web Experiment applies custom code actions as an optional part of the element changes action. Use the custom code action to write JavaScript, CSS, and HTML that adds elements or customizes your site in ways the visual editor doesn't support.

Apply custom code to specific [Pages](https://amplitude.com/docs/web-experiment/pages) using the **Apply to** dropdown. The dropdown lets you run different code depending on which Page is active.

> **Tip:**
>
> You can use custom code together with the [element changes](#element-changes). For example, an engineer can build a custom code component with placeholder text. A non-technical user can then use the visual editor to edit the placeholder text without touching the custom code.

Web Experiment applies custom code to your site in the following order:

1. Adds CSS in a `<style>` tag in the page's `<head>`.
2. Parses HTML into a DOM element.
3. Wraps JavaScript in a function, and adds it to a `<script>` tag in the page's `<head>`.
4. Calls the wrapped JavaScript function and passes parsed HTML and utilities as arguments.

### JavaScript

Web Experiment wraps any custom JavaScript in a function, and calls the function when the variant action applies. The function has two parameters you can use with your custom JavaScript code.

- `html`: The custom HTML code parsed as a DOM element object.
- `utils`: An object that contains utility functions you can use in your custom code.

#### Utilities

Web Experiment provides the following utilities:

- `waitForElement(selector: string): Promise<Element>`: Returns a promise that resolves when `waitForElement` finds an element matching the selector in the DOM. Uses `MutationObserver` to listen for elements.
- `remove: (()=> void) | undefined`: A function that you can set inside the JavaScript you inject. Web Experiment calls the `remove` function on page change, when Amplitude reevaluates experiments and reapplies variants.

  The `remove` function is useful for cleaning up changes to the page in single page apps, where the page doesn't fully reload. For example, if you inject an HTML element on a specific page, set the `remove` function to remove that element when the page changes.

### HTML

Web Experiment parses custom HTML as a DOM element, and passes the element to the custom JavaScript code for insertion. The custom HTML can use existing CSS styles and classes, or new CSS that you define.

### CSS

Use custom CSS styles to manipulate existing CSS classes and styles, or add new styles for elements you add with custom HTML. Web Experiment adds custom CSS to a `<style>` tag in the page's `<head>` element.

### Examples

> **Tip:**
>
> Generative AI like ChatGPT or equivalents can create HTML and CSS for simple elements. ChatGPT generated the initial modal and banner examples that follow, and Amplitude then modified them.

#### Insert an element

To insert an element onto your page, follow this pattern:

1. Write the HTML and CSS for the element you want to add to the page.
2. Identify the selector of the parent element you want to insert your new element into. The parent is often the `body`.
3. Paste the following JavaScript code, and update `PARENT_SELECTOR` with the parent element selector from step 2.

   ```js
   utils.waitForElement("PARENT_SELECTOR").then(function (e) {
     e.appendChild(html);
     utils.remove = function () {
       html.remove();
     };
   });
   ```

If you want to insert your element into the parent element at a specific position, use `insertBefore()` instead of `appendChild()`.

#### Add a banner

This example adds a discount code banner to the top of the page.

#### JS

```js
utils.waitForElement("body").then(function (e) {
  e.insertBefore(html, e.firstChild);
  utils.remove = function () {
    html.remove();
  };
});
```

#### CSS

```css
.announcement-banner {
  background-color: #fafafa;
  color: #333;
  padding: 10px;
  text-align: center;
  font-family: Arial, sans-serif;
  border-bottom: solid #e5e5e5;
  border-bottom-width: 1px;
}

.announcement-banner p {
  margin: 0;
  font-size: 16px;
}
```

#### HTML

```html
<div class="announcement-banner">
  <p>🎉 Big Sale: Get 25% off on all items! Use code <strong>SAVE25</strong></p>
</div>
```

#### Add a modal

This example adds a modal to the page after a 1 second delay.

#### JS

```js
var modal = html;
utils.waitForElement("body").then(function (body) {
  // Append the modal element to the body.
  body.appendChild(modal);

  // Get the close button element
  var closeBtn = document.getElementsByClassName("close")[0];

  // When the user clicks on the close button (x), close the modal
  closeBtn.onclick = function () {
    modal.style.display = "none";
  };

  // When the user clicks anywhere outside of the modal, close it
  window.onclick = function (event) {
    if (event.target == modal) {
      modal.style.display = "none";
    }
  };

  // Show the modal after a 1 second delay.
  window.setTimeout(function () {
    modal.style.display = "block";
  }, 1000);

  // Remove the modal on teardown.
  utils.remove = function () {
    modal.remove();
  };
});
```

#### CSS

```css
/* TODO: Style the action button */
.cta-btn {
  color: white;
  background-color: #000;
  border: #000 solid 1px;
}
.cta-btn:hover {
  color: black;
  background-color: #fff;
  border: #000 solid 1px;
}

/*
 * Modal Boilerplate
 */

/* Modal container */
.modal {
  display: none; /* Hidden by default */
  position: fixed;
  z-index: 1; /* Stay on top */
  left: 0;
  top: 0;
  width: 100%; /* Full width */
  height: 100%; /* Full height */
  background-color: rgba(0, 0, 0, 0.5); /* Black background with opacity */
}

/* Modal content box */
.modal-content {
  background-color: #fff;
  margin: 15% auto; /* Center the modal */
  padding: 20px;
  border-radius: 4px;
  border: 1px solid #888;
  width: 40%; /* Width of the modal */
  max-width: 800px;
  position: relative;
}

/* Close button */
.close {
  color: #999;
  float: right;
  font-size: 28px;
  font-weight: bold;
}

.close:hover,
.close:focus {
  color: #000;
  text-decoration: none;
  cursor: pointer;
}

/* Modal header */
.modal-header {
  margin: 0;
  padding: 0 0 15px 0;
  font-size: 24px;
  font-weight: bold;
}

/* Modal body */
.modal-body {
  margin: 20px 0;
  font-size: 16px;
}

/* Call to Action container */
.cta-container {
  text-align: right;
}

/* Call to Action button */
.cta-btn-base {
  padding: 10px 20px;
  font-size: 16px;
  cursor: pointer;
  bottom: 20px;
  right: 20px;
}
```

#### HTML

```html
<div class="modal">
  <div class="modal-content">
    <span class="close">&times;</span>
    <h2 class="modal-header">
      <!-- TODO: Update modal header text -->
      Join the mailing list!
    </h2>
    <p class="modal-body">
      <!-- TODO: Update modal body text -->
      To get updates on new posts to the blog join the exclusive mailing list
      today!
    </p>
    <div class="cta-container">
      <!-- TODO: Update button link -->
      <a href="https://example.com">
        <button class="cta-btn cta-btn-base">
          <!-- TODO: Update CTA button text -->
          Subscribe
        </button>
      </a>
    </div>
  </div>
</div>
```

================================================================================
# Web Experiment Pages
URL: https://amplitude.com/docs/web-experiment/pages
================================================================================

# Web Experiment Pages

In a Web Experiment, Pages scope variants to specific URLs so you can run tests on targeted pages without affecting unrelated parts of your site.

A Page defines the conditions under which a web experiment applies to your site, and includes:

- A unique name.
- URL targeting conditions.
- A page trigger.
- A Visual Editor URL to help preview the experiment.

## Create a page

When you create a new Web Experiment, specify a page by:

- **Manual URL input**: Enter a specific URL to define the page.
- **Import a saved page**: Select a page from a previous experiment.

After you add the page, continue with experiment setup, or go directly to the Visual Editor.

## Update a page or create another page

You can update or create another page for any Web experiment.

1. Go to _Experiments_ and select the Web experiment you want.
2. Go to _Settings > Pages_ for the Web experiment.
3. Rename the page, update its Visual Editor URL, configure the page trigger, or update the page targeting rules.
4. Select **Select a page** to add another page:

   1. Select **Custom** to open the Page details information for the second page.
   2. Name the page and enter the Visual Editor URL, Trigger Type, and Triggering Rules.
5. Select **Save Changes**.

### Page targeting rules

- **URL Matches**

  - **Description**: Match the page URL, ignore query parameters or hash fragments.
  - **Examples**:

    - `https://example.com/pricing`
    - ✅ https://example.com/pricing#details
    - ❌https://example.com/pricing/enterprise
- **URL Matches Exactly**

  - **Description**: Match the full page URL exactly.
  - **Examples**:

    - `https://example.com/pricing?utm_source=facebook`
    - ❌https://example.com/pricing
    - ❌ https://example.com/pricing?utm\_source=tiktok
- **URL Matches Pattern**

  - **Description**: Match the full page URL, including wildcards (`*`).
  - **Examples**:

    - `https://example.com/blog/*`
    - ✅ https://example.com/blog/my-first-post
    - ✅ https://example.com/blog/my-second-post#get-started
- **URL Contains**

  - **Description**: Match the full page URL, where the URL contains a specific substring.
  - **Examples**:

    - `/blog/my-first`
    - ✅ https://example.com/blog/my-first-post
    - ❌ https://example.com/blog/my-second-post
- **URL Starts With**

  - **Description**: Match the full page URL, where the URL starts with an exact substring.
  - **Examples**:

    - `https://example.com/blog`
    - ✅ https://example.com/blog/my-first-post
    - ❌ https://example.com/pricing
- **URL Ends With**

  - **Description**: Match the full page URL, where the URL ends with an exact substring.
  - **Examples**:

    - `/blog/my-first-post`
    - ✅ https://example.com/blog/my-first-post
    - ❌ https://example.com/blog/my-first-post#get-started
- **URL Matches Regex**

  - **Description**: Match the full page URL with a regular expression you define.
  - **Examples**:

    - [Learn Regex](https://www.regular-expressions.info/quickstart.html)
    - [Test Regex](https://regex101.com/)

### Page triggers

This feature is in Early Access and may continue to evolve. This documentation may not reflect the latest updates.

Page triggers define when an experiment evaluates a Page's conditions to determine whether the experiment activates on a webpage. Page targeting rules determine where an experiment runs; page triggers determine when variant actions apply.

When you create a new experiment, the default trigger type is **Immediately**. Immediate triggers evaluate the page conditions whenever the URL changes, including initial page load. Configure other trigger types to activate the experiment based on user behavior, page events, or custom conditions.

#### Trigger types

- **Immediately**

  - **Description**: Evaluates when the URL or route changes.
  - **Parameters**: None
- **On Event Tracked**

  - **Description**:

    - Evaluates when a specific Amplitude analytics event occurs.
    - If you don't load Analytics and Web Experiment together through the unified script tag, call `window.amplitude.add(window.webExperiment.plugin())` to ensure the web experiment client receives tracked event notifications.
  - **Parameters**:

    - Event name.
    - Optional property filters to match specific event property values.
  - **Limitation**: Property filters only match properties present in the raw event payload at the time the SDK tracks the event. Properties that Amplitude computes or enriches server-side aren't available in the client-side event object and can't satisfy a filter condition. This includes derived properties, lookup properties, and array-expanded properties (for example, `products.label`). Use only properties your instrumentation explicitly sets when calling the tracking method.
- **When Element Appears**

  - **Description**: Evaluates when an element matching a CSS selector appears in the DOM.
  - **Parameters**: CSS selector of the element.
- **When Element Becomes Visible**

  - **Description**: Evaluates when an element becomes visible in the viewport.
  - **Parameters**:

    - CSS selector of the element.
    - Optional visibility ratio (0-100) that defines how much of the element must be visible.
- **After Time on Page**

  - **Description**: Evaluates after the user spends a minimum amount of continuous time with the page in focus.
  - **Parameters**: Duration in seconds.
- **When User Exits**

  - **Description**: Evaluates when the cursor moves upward toward the top of the viewport.
  - **Parameters**: Optional minimum time (in seconds) the user must spend on the page before exit intent can trigger.
- **When Scrolled To**

  - **Description**: Evaluates when the user scrolls to a specific element or scroll percentage.
  - **Parameters**:

    - Element mode: CSS selector and pixel offset from the bottom of the viewport. Positive values trigger before the element enters view, negative values trigger after the element is partially visible.
    - Percent mode: Percentage of page scrolled (0-100).
- **When User Interacts**

  - **Description**: Evaluates on click, hover, or focus of a specified element.
  - **Parameters**:

    - CSS selector of the element.
    - Interaction type (click, hover, or focus).
    - Optional time threshold in milliseconds. For hover and focus, the user must maintain the interaction for this duration before the trigger fires. For click, the trigger fires after this delay.
- **Manual**

  - **Description**: Evaluates when a developer calls `window.webExperiment.toggleManualPageObject(name, active)`. Pass the name identifier and `true` to activate the page object, or `false` to deactivate.
  - **Parameters**: Name identifier that matches the manual trigger key configured in the page settings.

> **Note:** Trigger evaluation
>
> When a trigger fires, the experiment evaluates the page's targeting conditions. If the conditions match, the page becomes active. Triggers don't bypass conditions: the experiment activates only when both the trigger and the targeting rules match.

## Manage page scope for variants

Scope each variant to a specific page so the variant's changes apply only where you intend. Page scoping applies to all variant types.

### Visual editor

When you use the Visual Editor to make changes, such as text edits or style updates, those changes associate with the page you select during the preview session. For each change, specify the page or pages it applies to.

Page scoping in the Visual Editor lets you:

- Assign updates or changes to a specific page.
- Avoid applying the same change across all views.
- Maintain better isolation and clarity across your experiment setup.

> **Tip:** Validate the page scope
>
> Check the page scope for each change to avoid cross-page conflicts or unintended edits.

### Custom code

When you add custom code or URL redirects as variants, explicitly define which page or pages the variant applies to.

Page scoping lets you create a single experiment that includes custom code with different behaviors, depending on the active page.

### URL redirect

Select the page in the URL redirect variant's settings. Scoping the redirect to a specific page or set of pages ensures that Amplitude redirects users only when the specified page is active.

## Technical details

- The experiment evaluates pages after bucketing, so a page's variant affects only users who qualify for the experiment.
- Amplitude deactivates pages that conflict with other pages in your experiment.

## Known limitations

- `On Event Tracked` property filters can only match raw event properties sent by the SDK at tracking time. Server-side properties, including derived properties, lookup properties, and array-expanded properties, aren't available at trigger evaluation time, so you can't use them as filter conditions.
- Pages aren't available in:

  - Feature Flags.
  - Experiment Templates.
  - Management API.
  - Experiments that you convert to templates or flags.
- You can't delete or archive a page.
- Amplitude doesn't include pages in universal search or experiment table views.
- Pages don't appear in alerts or notifications.

================================================================================
# Web Experiment targeting
URL: https://amplitude.com/docs/web-experiment/targeting
================================================================================

# Web Experiment targeting

Web Experiments target both pages and audiences. Amplitude evaluates page targeting first, then audience targeting. Both targeting methods evaluate locally in the browser when the page first loads.

Web Experiments use [Pages](https://amplitude.com/docs/web-experiment/pages) to control where experiment variants apply on your website. Pages define when a web experiment applies, including targeting conditions that match specific URLs and visual editor URLs for previewing experiments.

## Audience targeting

By default, a new Web Experiment targets all users. Audience targeting lets you target specific users for your experiment. Users outside the target audience receive the default experience and don't count towards analysis.

If any segment matches, Amplitude buckets that user into a variant based on the rollout and variant distribution. For a segment to match, the user must meet all conditions you set.

### Browser properties (local)

Browser properties are available client-side and don't require network requests, so Amplitude evaluates them with low latency.

| Parameter | Description |
| --- | --- |
| New Users | A new user that Amplitude first sees after a specified date. When the Web Experiment SDK first sees a user, the SDK stores the date in local storage and never updates the date. This rule may not function as intended shortly after the initial script installation, when all users are new. The New Users rule isn't the same as new user queries in charts and cohorts. |
| Returning Users | A returning user that Amplitude first sees before a certain date. The same local considerations apply as with New Users. |
| Referring URL | Matches users who land on your site from a specific referrer. For more information, refer to [document.referrer](https://developer.mozilla.org/en-US/docs/Web/API/Document/referrer) on MDN. |
| Landing Page URL | The SDK sets the landing page URL in session storage when the SDK loads. |
| URL Query Parameters (Not persisted) | The current query parameters on the page at the time of evaluation. Use this rule for UTM parameter targeting. |
| URL Query Parameters (Persisted) | The query parameters that Amplitude stores locally to maintain variant assignments across navigations or reloads. Use this rule when you need consistent UTM-based targeting beyond the initial page visit. |
| Device Category | Target users by their device type. `Desktop`, `Mobile`, or `Tablet`. |
| User Agent | Target based on the user agent. Useful for targeting bots. For example, exclude all users where user agent contains `Googlebot`. |
| Cookies | The cookies in the window at the time of evaluation. |
| Language | The language in the user's browser. |
| Browser | The user's browser: `Safari`, `Chrome`, `Firefox`, `Edge`, `Opera`. |
| Freeform properties | Custom user properties that the [`IntegrationPlugin`](https://amplitude.com/docs/web-experiment/implementation#integrate-with-a-third-party-customer-data-platform) sets. |

### Real-time behavioral targeting (local)

Real-time behavioral targeting is in Early Access. Behavior and APIs may change before general availability.

Real-time behavioral targeting lets you target users based on actions they perform during their current and previous sessions. Amplitude evaluates these conditions in real time as users perform Amplitude events, without network requests.

When users become eligible: the moment a behavioral condition becomes true (for example, after the 5th article view), users become eligible and Amplitude buckets them into the experiment.

How it works: the Web Experiment SDK listens to Amplitude events as they fire in the browser, stores behavioral signals in localStorage, and evaluates targeting conditions in real time.

The SDK persists events for real-time behavioral targeting only when the events appear in your targeting conditions and the experiment is on. There's no historical look-back before you turn on the experiment, so the effective evaluation window is whichever is shorter: the time window you set, or how long the experiment has been running.

If your site gates the script on cookie consent, behavioral signals stay in memory until the visitor grants consent. Refer to [Cookies and consent management](https://amplitude.com/docs/web-experiment/cookies-and-consent-management).

> **Note:**
>
> If you don't load Analytics and Web Experiment together through the unified script tag, call `window.amplitude.add(window.webExperiment.plugin())` so the web experiment client receives tracked event notifications.

#### Supported conditions

Real-time behavioral targeting supports event-based conditions:

- Event counts: target users who performed an event a specific number of times.
- Time windows: evaluate behavior within the current session or over a specific time period (last X hours or days).
- Property filtering: combine multiple property conditions with AND logic (for example, "where `category = 'electronics'` AND `item_value > 200`").
- Cross-page tracking: behavioral signals persist across page navigations and browser sessions.

#### Example targeting rules

| Use case | Targeting rule |
| --- | --- |
| Engaged readers | User performed `page_view` ≥ 5 times this session. |
| Article readers | User performed `page_view` where `page_type = 'article'` ≥ 5 times this session. |
| Frequent visitors | User performed `page_view` ≥ 10 times in last 30 days. |
| Premium electronics shoppers | User performed `add_to_cart` where `category = 'electronics'` AND `item_value > 200` ≥ 1 time this session. |

#### Real-time behavioral targeting vs. cohorts

| Feature | Real-time behavioral targeting | Cohort targeting |
| --- | --- | --- |
| Where it runs | Client-side (browser) | Amplitude servers |
| When users become eligible | Instantly, as the event fires | Approx 1 min or 1 hour depending on type of cohort ([Syncs](https://amplitude.com/docs/data/destinations/syncs), [Cohort Targeting](https://amplitude.com/docs/feature-experiment/cohort-targeting)). |
| Network request needed | No | Yes |
| Can target mid-session behavior | Yes | No |
| Time window | Shorter. Supports current session, or rolling window of last X hours or days. | Longer. Supports specific intervals and rolling window up to last 12 months. |
| Best for | In-session intent signals (cart adds, page views, engagement depth) | Historical behavior, complex queries, long lookback windows |

Use real-time behavioral targeting when:

- You react to what a user does in this session, not only what they did days or weeks ago. For example, you show a discount after someone views three product pages this session.
- You target behavior that isn't in a cohort yet, including anonymous users who aren't synced to a cohort but show intent during the current session.
- You run experiments tied to high-intent moments: cart abandonment, repeat page views, scroll depth, where the targeting condition and the experiment need to resolve in the same session.
- You need eligibility to update the instant the user crosses the threshold, without waiting on cohort refreshes or remote membership checks.

Use cohorts instead when:

- You target stable attributes or longer historical patterns, such as users who purchased three or more times in the last 90 days or users in a curated churn-risk cohort.
- You target behavior from before the experiment started. Real-time behavioral targeting only persists events after you turn on the experiment.
- You need complex server-side queries across historical data beyond a few weeks.

### User properties (remote)

You can perform advanced targeting based on [Amplitude ID resolution](https://amplitude.com/docs/feature-experiment/remote-evaluation#amplitude-id-resolution), [IP geolocation](https://amplitude.com/docs/feature-experiment/remote-evaluation#geolocation), [property canonicalization](https://amplitude.com/docs/feature-experiment/remote-evaluation#canonicalization), [behavioral cohorts](https://amplitude.com/docs/feature-experiment/remote-evaluation#cohort-membership), and historical [user properties](https://amplitude.com/docs/feature-experiment/remote-evaluation#user-properties). Targeting user properties may increase page display latency because Amplitude needs to make network requests.

| Parameter | Description |
| --- | --- |
| Enriched User Properties | Properties that [user enrichment](https://amplitude.com/docs/feature-experiment/remote-evaluation#user-enrichment) resolves. |
| Amplitude User Properties | Amplitude Analytics' historical user data. |
| Experiment User Properties | The variant that other experiments assigned to the user. |
| User Cohorts | A set of [users](https://amplitude.com/docs/feature-experiment/cohort-targeting) that you define in Amplitude. |

#### Page display delay

When active web experiments target a page, Amplitude injects an anti-flicker overlay if at least one experiment targets the page and has "Anti-flicker" on. The overlay is a blank element that covers your page while it loads. Amplitude removes the overlay after it evaluates the remote properties, or after a 1-second timeout.

## Bucketing

Bucketing refers to the variant a user sees, based on the rollout and distribution. Rollout is the percentage of users in the experiment. Distribution defines which variant a user experiences if the user is in the rollout. Web Experiment distributes variants evenly by default. Amplitude recommends this distribution.

Bucketing stays consistent when the user has the same ID. Because most experiments bucket by Device ID, Web Experiment may place users in a different bucket if they visit on a new device, browser, or use private browsing.

Increasing rollout doesn't re-bucket users who're already in the rollout. For example, if your experiment has rolled out to 10% of users, and you increase the rollout to 50%, the original 10% of users don't change variants. If you change the distribution from evenly distributed to `20% -> control, 80% -> treatment`, users who started in the control jump to the treatment.

### Cross-subdomain identity

Experiments created on or after July 13, 2026 bucket users with a cross-subdomain identity, so a user keeps the same variant across subdomains of the same domain (`www.example.com`, `shop.example.com`). Amplitude fixes this at experiment creation; editing or restarting an experiment doesn't change it.

## Mutual exclusion, holdout groups, and dependencies

Mutual exclusion groups, holdout groups, and flag dependencies for Web Experiment are in Early Access. Behavior and APIs may change before general availability.

Web experiments support [mutual exclusion groups](https://amplitude.com/docs/feature-experiment/advanced-techniques/mutually-exclusive-experiments), [holdout groups](https://amplitude.com/docs/feature-experiment/advanced-techniques/holdout-groups-exclude-users), and [flag dependencies](https://amplitude.com/docs/feature-experiment/under-the-hood/flag-dependencies).

- **Mutual exclusion**: Add an experiment to a mutual exclusion group to prevent users from seeing it alongside another experiment. Mutual exclusion groups can contain a mix of web and feature experiments.
- **Holdout groups**: Add an experiment to a holdout group to measure long-term impact by withholding a percentage of users. Holdout groups can contain a mix of web and feature experiments.
- **Flag dependencies**: Flag dependencies let you gate one experiment on the result of another — for example, only evaluate experiment B if the user is in the treatment variant of experiment A. Web experiments support flag dependencies, but a web experiment can only depend on other web experiments — not on feature flags or feature experiments.

================================================================================
# Web Experiment performance
URL: https://amplitude.com/docs/web-experiment/performance
================================================================================

# Web Experiment performance

Web Experiment intentionally minimizes its impact on page performance.

## Script size

The Web Experiment script is dynamic and includes all your experiment configurations to avoid multiple synchronous downloads. Script size starts at a base size and scales with each experiment.

|  | Uncompressed | Compressed |
| --- | --- | --- |
| Base script | 79KB | 20KB |
| Per-flag size | ~1KB | ~100B |

To avoid constantly increasing script sizes, deactivate or archive experiments when they're complete.

> **Note:** Custom code impact on flag size
>
> Custom code increases a flag's code size by the size of the custom code itself.

## Caching

Web Experiment uses two caching layers: CDN and browser. The two layers make script delivery to your site more reliable.

### CDN cache

Amplitude caches the Web Experiment script on a CDN. When a user requests the script, the browser loads the script from the CDN if another user loaded it in the same geographic area. The CDN cache has a max age of one minute and serves stale content for up to one hour while the script reloads. If the origin returns an error, the CDN serves a stale response for the maximum time possible.

The cache control response header that configures CDN caching is:

`max-age=60,stale-while-revalidate=3600,stale-if-error=31536000`

### Browser cache

The browser cache serves the Web Experiment script without a network request for 60 seconds, or for the maximum time if the server returns an error. The browser cache serves the script from memory (0ms latency) when a user loads multiple pages on your site, or reloads the same page within a one-minute window.

The cache control response header that configures browser caching is:

`max-age=60,stale-while-revalidate=3600`

## Evaluation

Web Experiment evaluation runs locally using information available synchronously in the browser. Evaluation is CPU bound and usually takes less than 1ms to apply variant actions.

## Ad blockers

Amplitude can't identify users with ad blocking software enabled. Those users don't log assignment events or impressions, and don't experience the experiment.

================================================================================
# Feature and Web Experiment Use Cases
URL: https://amplitude.com/docs/feature-experiment/feature-vs-web-experimentation
================================================================================

# Feature and Web Experiment Use Cases

You can create experiments using either Feature Experiment or Web Experiment. Select Feature or Web Experiment based on your needs for the experiment and the results you want to generate.

## Feature Experiment use cases

Feature Experiment is most useful for Product experts, Data Engineering and Operations, and Data Analysts. These roles focus on deep knowledge of specific products or data repositories. Their goals are to:

- Launch new products.
- Reduce risk.
- Analyze and report on data.
- Adhere to security and compliance requirements.
- Monitor performance.
- Ship code.
- Run complex models.

Feature experimentation uses feature flags to create the variants you want. Flags are switches that let you modify your product's experience without changing code. Use flags to set up experiments in your product or to stage and roll out new features to your users. Your code uses the [Amplitude Experiment SDK](https://amplitude.com/docs/sdks/experiment-sdks) or [APIs](https://amplitude.com/docs/apis) to communicate with Experiment. Feature flags require knowledge of your code to use them for experimentation. For more information, go to [Feature Flags](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts).

## Web Experiment use cases

Web Experiment is most useful for digital marketers and growth marketing. These roles focus on strategic changes to the existing functionality and design of your website. Their goals are to:

- Optimize the website or the lifecycle of the product.
- Drive adoption.
- Personalize experiences.
- Test hypotheses.
- Track key product insights (KPIs).
- Upsell or cross-sell products.

Web experimentation uses a visual editor to create variations of your website. With the visual editor, you can select and alter content or element properties. Web Experiment lets less technical users, or users with fewer permissions in your system, create experiments without engineering resources. Web experiments use pages to control where your experiment's variants apply on your website. Pages let you scope experiments to specific URLs without affecting unrelated parts of your site. For more information about creating experiments with the web editor, go to [Setting up a Web Experiment](https://amplitude.com/docs/web-experiment/set-up-a-web-experiment).

For a full description of the functional differences between Feature and Web Experiment, go to [Feature and Web Experiment functional comparison](https://amplitude.com/docs/feature-experiment/feature-and-web-experiment-functional-comparison).

> **Note:**
>
> The [Website Conversion Agent](https://amplitude.com/docs/amplitude-ai/website-conversion-agent) can help you identify high-impact pages and generate experiment strategies to increase conversion, whether you use Feature or Web Experiment.

================================================================================
# Feature and Web Experiment Functional Comparison
URL: https://amplitude.com/docs/feature-experiment/feature-and-web-experiment-functional-comparison
================================================================================

# Feature and Web Experiment Functional Comparison

The following tables show which Experiment functionality applies to Feature experimentation, Web experimentation, or both. To compare differences and use cases, go to [Feature and Web Experiment use cases](https://amplitude.com/docs/feature-experiment/feature-vs-web-experimentation).

#### Planning

| Functionality | Feature | Web |
| --- | --- | --- |
| Client-side or server-side implementation | ✅ | ❌ |
| Local or remote evaluation | ✅ | ✅ |
| **Stats method** |  |  |
| Sequential | ✅ | ✅ |
| T-test | ✅ | ✅ |
| Thompson Sampling (MAB) | ✅ | ✅ |
| Bayesian | ✅ | ✅ |
| **Reduce chance of error** |  |  |
| Bonferroni correction | ✅ | ✅ |
| **Group experiments** |  |  |
| Mutual exclusion | ✅ | ❌ |
| Holdouts | ✅ | ❌ |
| % of audience to rollout | ✅ | ❌ |
| **Exposure event type** |  |  |
| Exposure event | ✅ | ✅ |
| Custom exposure | ✅ | ❌ |
| **Bucketing salt** |  |  |
| Bucketing salt | ✅ | ✅ |

#### Creating experiments

| Functionality | Feature | Web |
| --- | --- | --- |
| Name | ✅ | ✅ |
| Project | ✅ | ✅ |
| Template | ✅ | ❌ |
| Recommended settings | ✅ | ✅ |
| Link | ✅ | ✅ |
| Tag | ✅ | ✅ |
| Variants | ✅ | ✅ |
| Payload | ✅ | ❌ |
| **When to run** |  |  |
| When to start | ✅ | ✅ |
| When to end | ✅ | ✅ |
| Traffic estimate, control mean estimator, power duration estimator for each day | ✅ | ✅ |
| OOTB web editor widgets | ❌ | ✅ |
| Project-level user permissions | ✅ | ✅ |
| Notifications through Slack channel, webhook, or Microsoft Teams | ✅ | ✅ |

#### Experiment types

| Functionality | Feature | Web |
| --- | --- | --- |
| A/B | ✅ | ✅ |
| Multi-armed bandit | ✅ | ✅ |

#### Metrics

| Functionality | Feature | Web |
| --- | --- | --- |
| Primary metric | ✅ | ✅ |
| Secondary metric | ✅ | ✅ |
| Guardrail vs success | ✅ | ✅ |
| Direction | ✅ | ✅ |
| Minimum detectable effect | ✅ | ✅ |
| Winsorization | ✅ | ✅ |
| CUPED | ✅ | ✅ |
| Attribution | ✅ | ✅ |
| Window | ✅ | ✅ |

#### Targets

| Functionality | Feature | Web |
| --- | --- | --- |
| User or behavioral cohort | ✅ | ✅ |
| Stratified sampling | ✅ | ✅ |
| Rollout | ✅ | ✅ |
| Sticky bucketing | ✅ | ❌ |

#### QA

| Functionality | Feature | Web |
| --- | --- | --- |
| Deployment | ✅ | ✅ |
| Dependencies | ✅ | ❌ |
| Testers | ✅ | ❌ |

#### Analysis

| Functionality | Feature | Web |
| --- | --- | --- |
| Assignment and exposure charts | ✅ | ❌ |
| Variant jumping | ✅ | ✅ |
| Anonymous exposures | ✅ | ✅ |
| Exposures without assignments | ✅ | ❌ |
| Rollout options | ✅ | ✅ |

================================================================================
# Experiment templates
URL: https://amplitude.com/docs/feature-experiment/workflow/experiment-templates
================================================================================

# Experiment templates

Experiment templates capture the configuration of an [experiment](https://amplitude.com/docs/feature-experiment/workflow/create). Templates include features like goals, deployments, and audience targeting that you can reuse for future experiments. Templates help when you:

- Run similar experiments across different features or time periods.
- Enforce consistent experiment standards across your team.
- Launch new experiments quickly with proven configurations.
- Test variations of the same feature with different audiences.

## Permissions

To create, edit, or delete experiment templates, you need the Manage Experiments permission. For more information, refer to [User roles and permissions](https://amplitude.com/docs/admin/account-management/user-roles-permissions).

### Template limitations

- Templates are only available on Feature Experiments.
- Templates are project-specific.
- You can't apply templates to pre-existing experiments.
- You can't convert a template back into an active experiment.
- Template changes don't sync to experiments created from that template.

## Creating templates

### Create a template from an experiment

To create a template from an existing feature experiment:

1. Navigate to the experiment you want to save as a template.
2. Click **More options** (three dots) and select **Use as a template**.
3. Enter a template name and optional description.
4. Select which components to include in the template:

   - [Deployment rules](https://amplitude.com/docs/feature-experiment/workflow/configure-delivery).
   - [Goals and metrics](https://amplitude.com/docs/feature-experiment/workflow/define-goals).
   - User targeting rules.
   - Evaluation mode and [bucketing](https://amplitude.com/docs/feature-experiment/workflow/define-audience#set-the-target-audience-for-your-experiment) settings.
5. Click **Use as a template**.

The template appears in the Templates library.

### Create a template from the Templates library

1. Navigate to _Experiments > Templates_.
2. Click **Create Template**.
3. Enter a name for the template.
4. Select the project where you want to apply the template. Templates apply to a single project.
5. Optionally, enter a description of the template.
6. Click **Create**.

## Create an experiment from a template

To use a template when creating a new experiment:

1. From the Experiments page, click **Create Experiment > Feature Experiment**. For more information, go to [Create a new experiment](https://amplitude.com/docs/feature-experiment/workflow/create).
2. Enter information about your experiment.
3. In the Apply a template section, select a template from the drop-down menu.
4. Click **Create**.
5. Make any adjustments to the experiment.
6. When ready, click **Start Experiment**.

The new experiment inherits the template configuration but operates independently. Changes to the template don't affect experiments created from it.

## Manage experiment templates

### View all templates

Navigate to _Experiment > Templates_ to view your organization's template library. The Templates page shows:

- Template name and description.
- Number of experiments created from this template.
- Last modified date.
- Goals.
- Evaluation mode.
- Number of segments in the template.
- Bucketing unit.

### Edit a template

1. Navigate to _Experiment > Templates_.
2. Click the name of the template you want to edit.
3. Modify the configuration as needed.
4. Click **Save**.

Changes to a template only affect future experiments created from it. Amplitude doesn't update existing experiments created from the template.

### Archive a template

1. Navigate to _Experiment > Templates_.
2. Open the template you want to archive.
3. Click the **three-dot** menu icon and then click **Archive**.
4. Confirm the archive.

Archiving a template doesn't affect any experiments created from the template.

## What templates include

Templates can include the following experiment configurations:

### Experiment settings

- Evaluation mode ([local](https://amplitude.com/docs/feature-experiment/local-evaluation) or [remote](https://amplitude.com/docs/feature-experiment/remote-evaluation)).
- [Bucketing unit](https://amplitude.com/docs/feature-experiment/workflow/define-audience#set-the-target-audience-for-your-experiment) (user or group).

### Goals and metrics

For more information about defining experiment goals, go to [Define your experiment's goals](https://amplitude.com/docs/feature-experiment/workflow/define-goals).

- Primary and secondary metrics.
- Success and guardrail metrics.
- [Minimum detectable effects](https://amplitude.com/docs/feature-experiment/workflow/finalize-statistical-preferences#stats-preferences).

### Audience targeting

For more information about audience targeting, refer to [Define your experiment's audience](https://amplitude.com/docs/feature-experiment/workflow/define-audience).

- Segment definitions.
- Property-based targeting rules.
- Geographic or demographic filters.
- User ID targeting.

## Best practices

- Name templates descriptively. Use names that clearly indicate the template's purpose, like "Product Page Conversion Test" or "Mobile Onboarding Experiment."
- Add detailed descriptions. Include guidance on when to use the template and any special considerations.
- Review templates regularly. Archive or update templates that are outdated or no longer align with your experimentation standards.
- Start with successful experiments. Create templates from experiments with proven configurations and [clear learnings](https://amplitude.com/docs/feature-experiment/workflow/experiment-learnings).
- Document template variations. If you create multiple similar templates, document the differences between them.

================================================================================
# The Experiment Analysis view
URL: https://amplitude.com/docs/feature-experiment/analysis-view
================================================================================

# The Experiment Analysis view

The Experiment Analysis view shows the details of your experiment. Select your experiment and then go to _Activity > Analysis_ to view high-level statistical measurements. These measurements help you determine whether your experiment succeeded.

This article describes each column in the Analysis table and how each column relates to your experiment.

## Analysis filters and chart options

You can filter the information in the Analysis view by time, user, or any other available property.

Select **Select property** to filter the analysis view by any available property.

To turn your experiment analysis into a chart, select **Open in Chart**. Amplitude converts the experiment information into the most likely charts and opens them in a new tab. You can then modify the charts.

## Metric name, Control, and status

The Metric name column contains the names of the selected metrics. The top metric is the [recommendation metric](https://amplitude.com/docs/feature-experiment/key-terms). All other metrics are secondary metrics. Hover over a metric's name to read its definition.

The Control column contains information about the control group for your experiment.

The On column contains information about experiment activity while the experiment runs. If the experiment is complete or rolled back, the On column may show other relevant information such as treatment or property.

> **Note:**
>
> To examine segments of users, use the filter card at the top of the Analysis tab.

## Relative performance

Relative performance measures the relative difference between variant performance and control performance. This is also known as the relative lift. To cross-check this value, expand a single metric's section and divide the absolute lift for a variant by the absolute value of the control for that metric.

## Confidence interval

The confidence interval is a range of values that includes the parameter you're trying to measure. In this case, the parameter is the difference in the [means](https://en.wikipedia.org/wiki/Arithmetic_mean) between the variant and the control. The confidence interval isn't a probability. Interpret the confidence interval as follows: if you ran this experiment 100 times with the confidence level set to 95, you can expect the true parameter value to fall within this range at least 95 times.

The confidence interval reveals characteristics of what the experiment has observed so far:

- **Confidence Interval contains 0**: Not enough evidence exists to decide whether a difference exists between control and treatment.
- **Confidence Interval greater than 0**: The interval (upper and lower confidence bounds) is greater than zero. Amplitude Experiment has accumulated enough observations to reach statistical significance, and you can conclude that the variant has a positive effect compared to control. For example, when you look at lift, a variant with a confidence interval greater than zero performs better than the control.
- **Confidence Interval less than 0**: Amplitude Experiment has accumulated enough observations to reach statistical significance, and you can conclude that the variant has a negative effect compared to control. When you look at lift, a variant with a confidence interval less than zero performs worse than the control.

If you have multiple variants, select the one you want to view in the confidence interval chart from the drop-down above the chart.

To access the confidence interval, hover over the control or the experiment metric.

## Significance

Significance is the likelihood that the performance shown for each test variant differs from zero rather than reflecting random fluctuations in the data. The higher this value, the more confident you can be in your results. More formally, significance is _1 - p-value_.

## Absolute value

The specific meaning of the absolute value depends on the metric type. For unique conversions, Experiment expresses values as a percentage that indicates the percentage of users (over the total number of exposed users) who converted for each variant. The numerator [Conversions](https://amplitude.com/docs/get-started/understand-conversion-rate) and denominator [Exposures](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) appear below the percentage.

Otherwise, the value indicates the aggregate (total events, sum of property value, average of property value) for each exposed user. The denominator is the total number of exposures. For example, 10 total events / 4 exposures = on average, an exposed user had 2.5 conversion events.

To access the absolute value, hover over the control or metric name.

## Winsorization statistics

When you enable [winsorization](https://amplitude.com/docs/feature-experiment/advanced-techniques/winsorization-in-experiment) for your experiment, the Analysis view shows additional information about how winsorization affected your data:

- **Number of winsorized users**: The count of users whose values winsorization adjusted.
- **Percentage of winsorized users**: The proportion of total users that winsorization affected.

This information helps you understand the impact of outlier handling on your experiment results. If winsorization affects a high percentage of users, investigate the underlying data distribution or adjust your winsorization settings.

## Common questions

### Why is my graph displaying an error state?

A common mistake is to generate a chart using only one variant. The Experiment Results chart must include two or more variants to display comparison results. If you don't include both the control and at least one variant, your chart doesn't display anything.

### Why is reaching significance taking longer than it should?

When you use a [T-test](https://amplitude.com/docs/feature-experiment/experiment-theory/analyze-with-t-test), you must wait until your experiment reaches the specified sample size before Experiment Results runs the p-value and confidence-interval computations.

With sequential testing, even with a large MDE, reaching statistical significance can take time if your experiment's lift is small. A T-test generally requires fewer samples to detect the same lift.

### How is the Retention metric calculated?

Amplitude uses three parameters to calculate Return On for the Retention metric:

- **The starting event**: the event that occurs _after_ the exposure event. The starting event marks the beginning of the retention window.
- **The return event**: the event you want the user to perform after the starting event. The user counts as retained if they trigger the return event.
- **Return on the nth day/week/month**: the number of days, weeks, or months between the user performing the starting event and the return event. Amplitude calculates this parameter in 24-hour increments and doesn't use calendar dates.

For example, suppose a user performs an exposure event, a starting event, and a return event:

- Exposure event: `Page Viewed`
- Starting event: `Sign Up`
- Return event: `Add to cart`

With a return-on **nth day** value of seven, the user counts as retained if `Add to cart` triggers between seven days and seven days plus 24 hours after `Sign Up`.

If the return-on value is an **nth week** value of one (one week, or seven days), the user counts as retained if they perform `Add to cart` any time between days seven (week 1) and 14 (week 2) after `Sign Up`.

For a return-on **nth month** value of one (30 days), the user counts as retained if they perform `Add to cart` any time between days 30 (month 1) and 60 (month 2) after `Sign Up`.

================================================================================
# Data model
URL: https://amplitude.com/docs/feature-experiment/data-model
================================================================================

# Data model

In Amplitude, your organization is the top-most hierarchical level. Within an organization, Experiment follows the project structure that Amplitude Analytics defines. Associate all Experiment data with an Analytics project.

An Amplitude project contains [flags](#flags-and-experiments), [experiments](#flags-and-experiments), and [deployments](#deployments).

![An illustration of the hierarchical nature of Feature Experiment with the top-most level being Organization then Project, then Flag/Experiment that contains each variant and the deployments associated with the variants](https://amplitude.com/docs/images/experiment/data-model.drawio.svg)

## Projects

Experiment uses the same projects as Amplitude Analytics. As a best practice, create a project for each product and for each environment. Because [flags](#flags-and-experiments), [experiments](#flags-and-experiments), and [deployments](#deployments) only exist within a single project, duplicate these objects across projects for the same product.

> **Tip:** Copy a flag to another project
>
> When developing a new feature with an experiment, create the experiment in the dev environment project to develop and test the implementation. Then, copy the experiment into the prod project to run the experiment in prod.

A deployment serves a group of flags or experiments for use in an application. Each [project](#projects) has a deployment that uses the project API key as the deployment key, available by default. Amplitude randomly generates deployment keys. On creation, Amplitude assigns each experiment deployment an associated deployment key. Experiment uses the deployment key to identify the deployment and authorize requests to the evaluation servers.

> **Note:** Client and server deployments
>
> Deployments are either client or server deployments. Use client-side deployments to initialize client-side SDKs and server-side deployments to initialize server-side SDKs or authorize requests to the Evaluation API.

## Deployments

Deployments belong to Analytics projects, and a project can have multiple deployments. Name deployments after the platform (client-side) or service (server-side) to which Experiment serves variants (for example: `android`, `ios`, `web`). The default project API key deployment is useful for getting started. However, explicit deployments for each platform or service work better for larger organizations or teams that share the same Amplitude project across multiple platforms for the same application.

Add deployments to [Flags and Experiments](https://amplitude.com/docs/feature-experiment/workflow/feature-flag-rollouts#create-a-new-flag) in the same project. When Experiment's evaluation servers receive a request to fetch variants for a user, Experiment uses the deployment key to look up all associated flags and experiments for evaluation.

## Flags and experiments

Experiment uses feature flags and experiments to serve a variable experience to a user. The flag key identifies flags and experiments. Flags and experiments associate with `0-n` [deployments](#deployments) and contain `1-k` [variants](#variants). The evaluation mode (local or remote) determines whether the flag or experiment supports [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation), and local mode can limit the flag's targeting capabilities.

Feature flags and experiments share the same underlying data model, and you can migrate from one to the other retroactively. The most visible difference appears in the product interface: experiments guide you through an experiment lifecycle and let you define success metrics and perform analysis. Flags contain more basic functionality, and don't include special planning and analysis sections.

### Flags

Use flags for standard feature flagging without user analysis. New flags come with a default variant, `on`.

> **Example:** Flag use cases
>
> - Rolling out a feature to a subset of users (for example, beta customers).
> - Different experience for a behavioral cohort (for example, power users).

### Experiments

Use experiments for feature experimentation on users. New experiments come with two default variants, `control` and `treatment`.

> **Example:** Experiment use cases
>
> - Run an A/B test for a new feature in your application.
> - Experiment on multiple recommendation algorithms on your server.

## Variants

A variant is the part of the experiment that you change for the customer. For example, if you experiment with the text on a call to action (CTA) button, each version of the text is a variant. Variants can exist within a flag or an experiment. A single variant can associate with a single flag, or multiple variants can associate with a larger experiment. An experiment requires variants so that you can test the experiment's hypothesis.

> **Note:** SDK use
>
> Only the `value` and `payload` are available when accessing a variant from an SDK or the [Evaluation API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api).

| Property | Requirement | Description |
| --- | --- | --- |
| `value` | Required | A string that identifies the variant in the instrumentation. Amplitude checks the value string for equality when an SDK or [Evaluation API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api) accesses the variant. Format must be lowercase, kebab-case, or snake\_case. |
| `payload` | Optional | Dynamic JSON payload for sending arbitrary data with the variant. For example, you can send a hex code to change the color of a component in your application. |
| `name` | Optional | Name for the variant. The name is like `value`, but doesn't have formatting limitations, and you can change it without breaking the instrumentation in your code base. |
| `description` | Optional | A more detailed description of the variant. Use the description to explain what the user experiences when viewing the variable experience in more detail. |

## Users

Experiment users map to a user within Amplitude Analytics. Alongside flag configurations, users are an input to [evaluation](https://amplitude.com/docs/feature-experiment/implementation). Flag and experiment targeting rules access user properties.

Pass users to evaluation through `fetch` requests for [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), or directly to the `evaluate` function for [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation).

> **Warning:**
>
> Include either a user ID or device ID in the user object for evaluation to succeed. For example, remote evaluation returns a 400 error if both the User ID and Device ID are null, empty, or missing.

| Property | Type | Description |
| --- | --- | --- |
| `user_id` | `string` | The [User ID](https://amplitude.com/docs/get-started/identify-users) is the primary identifier for the user. This value is typically the user's ID within your system. Experiment uses the User ID when resolving the Amplitude ID on enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), where the Amplitude ID is the default bucketing key. |
| `device_id` | `string` | The Device ID is the secondary identifier for the user. An analytics SDK usually generates the Device ID randomly on the client side, or sets it in a cookie on the server side. Experiment also uses the Device ID when resolving the Amplitude ID on enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), where the Amplitude ID is the default bucketing key. |
| `user_properties` | `object` | Optional object of custom properties used when evaluating the user during local or remote evaluation. |
| `groups` | `object` | Beta. Optional object that lists groups associated with this user. Format is an object where the key is the group type, and the value is an array of group value strings (for example, `{"org name":["Amplitude"]}`). |
| `group_properties` | `object` | Beta. Optional object that lists group properties associated with this user. Format is a nested object where the key is the group type, and the value is an object where the key is the group value, and the value is an object of properties (for example, `{"org name":{"Amplitude":{"plan":"enterprise"}}}`). |

This feature is in Early Access. This feature may continue to evolve. This documentation may not yet reflect the latest updates.

> **Note:**
>
> If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting), you can perform bucketing and analysis on groups rather than users. Contact your representative to gain access.
>
> Include groups with the user when sent with the fetch request (recommended). Or, identify groups with the user through a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or with [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).
>
> All Experiment SDKs support groups. The following table lists minimum versions:
>
> | SDK | Minimum version |
> | --- | --- |
> | [Android](https://amplitude.com/docs/sdks/experiment-sdks/experiment-android) | 1.9.0 |
> | [iOS](https://amplitude.com/docs/sdks/experiment-sdks/experiment-ios) | 1.10.0 |
> | [React Native](https://amplitude.com/docs/sdks/experiment-sdks/experiment-react-native) | 1.1.0 |
> | [JavaScript](https://amplitude.com/docs/sdks/experiment-sdks/experiment-javascript) | 1.5.6 |
> | [Ruby](https://amplitude.com/docs/sdks/experiment-sdks/experiment-ruby) | 1.4.0 |
> | [Go](https://amplitude.com/docs/sdks/experiment-sdks/experiment-go) | 1.7.0 |
> | [Python](https://amplitude.com/docs/sdks/experiment-sdks/experiment-python) | 1.3.0 |
> | [JVM](https://amplitude.com/docs/sdks/experiment-sdks/experiment-jvm) | 1.3.0 |
> | [Node](https://amplitude.com/docs/sdks/experiment-sdks/experiment-node-js) | 1.5.0 |
> | [PHP](https://amplitude.com/docs/sdks/experiment-sdks/experiment-php) | 1.0.0 |

### Full user definition

#### Full user definition

```json
{
    "user_id": string,
    "device_id": string,
    "country": string,
    "region": string,
    "city": string,
    "dma": string,
    "language": string,
    "platform": string,
    "version": string,
    "os": string,
    "device_manufacturer": string,
    "device_brand": string,
    "device_model": string,
    "carrier": string,
    "library": string,
    "user_properties": object,
    "groups": object,
    "group_properties": object
}
```

================================================================================
# Performance and caching
URL: https://amplitude.com/docs/feature-experiment/under-the-hood/performance-and-caching
================================================================================

# Performance and caching

Amplitude Experiment [evaluation](https://amplitude.com/docs/feature-experiment/implementation) supports two modes, [local](https://amplitude.com/docs/feature-experiment/local-evaluation) and [remote](https://amplitude.com/docs/feature-experiment/remote-evaluation). Each mode has different performance characteristics and tradeoffs.

## Performance

Evaluation performance depends on the type of evaluation you use, and on where the request originates.

Amplitude hosts data centers in the US (`us-west-2`) and EU (`eu-central-1`). Requests to a data center geographically distant from the data center that hosts your Amplitude data experience higher latency.

### Remote evaluation

[Remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation) uses [Fastly](https://fastly.com) to [cache](#cdn-caching) evaluation results for a user. Cache hits serve variants from the edge, which improves performance.

The following results come from synthetic remote evaluation test requests to Amplitude's US data center, collected over the last six months. Latency includes Domain Name System (DNS) resolution, transport layer security (TLS) connection, and the remote evaluation request response round trip.

#### Global

| Cache | Average |
| --- | --- |
| HIT | 35.9ms |
| MISS | 194.21ms |

#### North America

| Cache | Average |
| --- | --- |
| HIT | 48.21ms |
| MISS | 100.56ms |

#### South America

| Cache | Average |
| --- | --- |
| HIT | 18.18ms |
| MISS | 262.62ms |

#### Europe

| Cache | Average |
| --- | --- |
| HIT | 31.96ms |
| MISS | 214.3ms |

#### Asia

| Cache | Average |
| --- | --- |
| HIT | 28.84ms |
| MISS | 239.09ms |

### Local evaluation

[Local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation) pre-fetches flag configurations and uses them to evaluate all users. This avoids a network request and speeds up evaluation compared to remote evaluation.

The following results measure a single flag evaluation, collected over 10 executions of 10,000 iterations with randomized user inputs evaluated against one flag configuration. Amplitude selected these results at random from three possible flag configurations.

| SDK | Average | Median | Cold Start |
| --- | --- | --- | --- |
| [Node.js](https://amplitude.com/docs/sdks/experiment-sdks/experiment-node-js) | 0.025ms | 0.018ms | 3ms |
| [Go](https://amplitude.com/docs/sdks/experiment-sdks/experiment-go) | 0.098ms | 0.071ms | 0.7ms |
| [JVM](https://amplitude.com/docs/sdks/experiment-sdks/experiment-jvm) | 0.007ms | 0.005ms | 6ms |

## CDN caching

> **Note:** Content delivery network
>
> - A content delivery network (CDN) is a geographically distributed group of servers that work together to deliver internet content quickly.
> - The CDN caches variant responses for [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation) and flag configurations for [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation).

After Experiment computes and retrieves a response for a request, Experiment caches that request for reuse to make future requests faster. Experiment uses a CDN to cache the experiments and feature flags for a user, which provides low-latency access on subsequent requests.

### Cache time-to-live (TTL)

Experiment caches requests to the server on the CDN. The cache uses a time-to-live (TTL) policy and expires after 60 minutes regardless of whether the key is accessed within that window. The 60-minute cache window starts from the first page load.

### Cache key

The CDN caches the exact request received, including user information. Any change in user info misses the CDN cache unless Experiment cached that exact same request before.

### Cache invalidation

To prevent stale results when underlying flags change, Experiment invalidates (deletes) cached results for an entire deployment when you update any flag or experiment associated with that deployment. The SDKs retrieve results for all experiments and feature flags for a given deployment for a user, so Experiment invalidates all results for a deployment whenever a single flag changes. Experiment also invalidates all cached requests for a deployment when you add or remove the deployment from a flag.

A [scheduled start](https://amplitude.com/docs/feature-experiment/workflow/experiment-test#schedule-your-experiment) is an exception: reaching the scheduled start time doesn't invalidate the cache. Requests made in the hour before the start time keep their cached pre-start response until the 60-minute TTL expires, while requests made at or after the start time receive a variant right away.

### Dynamic targeting cache considerations

Experiment lets you target feature flags and experiments based on dynamic properties, such as user properties and behavioral cohorts synced from Analytics. These properties aren't included in the fetch request, so you may receive cached experiment results for up to one hour (the TTL) until the cache misses and Experiment re-evaluates the user with the most recent dynamic properties.

#### Amplitude user properties

Experiment's remote evaluation servers can target based on user properties that Amplitude previously identified for the user. The CDN caches responses based only on user properties passed explicitly in the request, so the caller may receive stale results for up to one hour, even after Analytics updates the user properties to values that would assign the user to a different variant.

#### Behavioral cohorts

You may want to use behavioral cohorts defined in Amplitude Analytics in your flag and experiment targeting. Experiment computes cohorts hourly and the CDN cache TTL is also hourly, so targeting a user to a variant may take up to two hours.

> **Tip:**
>
> Amplitude Experiment recommends dynamic cohort targeting only for flags and experiments where inclusion in a variant isn't time sensitive.

================================================================================
# JSON payloads
URL: https://amplitude.com/docs/feature-experiment/json-payloads
================================================================================

# JSON payloads

JSON payloads attach dynamic configuration data to experiment variants. Use payloads to change your application's behavior or appearance remotely, without redeploying code.

Strongly typed payloads let you define a schema so Amplitude validates each payload before it reaches your application.

## How JSON payloads work

When you create a variant, you can attach a JSON payload that contains any variables you want to pass to your application.

When your application evaluates an experiment:

- The Experiment SDK fetches the variant for the user.
- The SDK returns the variant along with its payload.
- Your application reads the payload and configures the experience dynamically.

This pattern works for simple flags and for complex experiments where the payload defines layouts, feature options, copy, or configuration values.

### Workflow overview

A typical workflow follows these steps:

1. **Create a variant with a payload**: Add a JSON object to a variant that defines the configuration for that experience.
2. **Activate the flag or experiment**: Deploy the configuration to your users.
3. **Evaluate in your application**: Use an Experiment SDK (web, mobile, or backend) to fetch the variant for each user.
4. **Use the payload**: Read `variant.payload` and apply the configuration to your UI or business logic.

## Create variants with payloads

Add JSON payloads to variants through the Amplitude UI or programmatically through the Management API.

### Through the UI

To add a payload when creating or editing a variant:

1. Navigate to _Experiment > Feature Flags_ or _Experiments_.
2. Select the flag or experiment you want to configure.
3. In the **Variants** section, create or edit a variant.
4. Enter the variant name, value, and description.
5. In the **Payload** field, add your JSON configuration.
6. Select **Apply**, then save your changes.

> **Example:** Example payload – config for a blog layout
>
> ```json
> {
>   "layout": "cards",
>   "titlePosition": "above",
>   "gradient": false,
>   "showDescription": true,
>   "cardCount": 3
> }
> ```

For more details on creating variants in the UI, go to [Create variations](https://amplitude.com/docs/feature-experiment/experiment-quick-start#create-variations).

### Through the Management API

Create variants with payloads using the Experiment Management API. Include the `payload` field in your request body.

> **Example:** API request example
>
> ```bash
> curl --request POST \
>   --url 'https://experiment.amplitude.com/api/1/flags/{id}/variants' \
>   --header 'Authorization: Bearer <management-api-key>' \
>   --header 'Content-Type: application/json' \
>   --data '{
>     "key": "cards-layout",
>     "name": "Cards Layout",
>     "description": "Blog posts displayed in card format",
>     "payload": {
>       "layout": "cards",
>       "titlePosition": "above",
>       "gradient": false,
>       "showDescription": true,
>       "cardCount": 3
>     },
>     "rolloutWeight": 1
>   }'
> ```

Refer to the full API docs for:

- [Create variant for flags](https://amplitude.com/docs/apis/experiment/experiment-management-api-flags#create-variant).
- [Create variant for experiments](https://amplitude.com/docs/apis/experiment/experiment-management-api-experiments#create-variant).

## Access payloads in your application

After you create variants with payloads, your application must:

1. Initialize the Experiment SDK.
2. Fetch the variants for the user.
3. Access `variant.payload`.
4. Apply the configuration with sensible defaults.

The exact syntax varies by SDK, but the pattern stays consistent.

### JavaScript / TypeScript (browser)

```javascript
import { Experiment } from '@amplitude/experiment-js-client';

// Initialize the Experiment client
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  // Optional configuration
});

// Fetch variants for the current user
await experiment.fetch({
  user_id: 'user123',
  device_id: 'device456',
});

// Get the variant for a specific flag
const variant = experiment.variant('blog-layout-flag');

// Access the payload with defaults
if (variant && variant.payload) {
  const layout = variant.payload.layout || 'list';
  const titlePosition = variant.payload.titlePosition || 'below';
  const showDescription = variant.payload.showDescription !== false;
  const cardCount = variant.payload.cardCount || 5;

  configureBlogLayout({
    layout,
    titlePosition,
    showDescription,
    cardCount,
  });
}
```

### React example

```javascript
import { useEffect, useState } from 'react';
import { Experiment } from '@amplitude/experiment-js-client';

function BlogComponent() {
  const [layoutConfig, setLayoutConfig] = useState({
    layout: 'list',        // defaults
    titlePosition: 'below',
    showDescription: true,
    cardCount: 5,
  });

  useEffect(() => {
    async function fetchExperiment() {
      const experiment = Experiment.initialize('<DEPLOYMENT_KEY>');
      await experiment.fetch({ user_id: 'user123' });

      const variant = experiment.variant('blog-layout-flag');
      if (variant && variant.payload) {
        setLayoutConfig({
          layout: variant.payload.layout || 'list',
          titlePosition: variant.payload.titlePosition || 'below',
          showDescription: variant.payload.showDescription !== false,
          cardCount: variant.payload.cardCount || 5,
        });
      }
    }

    fetchExperiment();
  }, []);

  return (
    <BlogLayout
      layout={layoutConfig.layout}
      titlePosition={layoutConfig.titlePosition}
      showDescription={layoutConfig.showDescription}
      cardCount={layoutConfig.cardCount}
    />
  );
}
```

### Server-side example (Node.js)

```javascript
import { Experiment } from '@amplitude/experiment-node-server';

const experiment = Experiment.initialize('<DEPLOYMENT_KEY>');

async function getExperimentConfig(userId: string) {
  const user = { user_id: userId };
  const variants = await experiment.fetch(user);
  const variant = variants['blog-layout-flag'];

  if (variant && variant.payload) {
    return {
      layout: variant.payload.layout || 'list',
      titlePosition: variant.payload.titlePosition || 'below',
      showDescription: variant.payload.showDescription !== false,
      cardCount: variant.payload.cardCount || 5,
    };
  }

  // Fallback config
  return {
    layout: 'list',
    titlePosition: 'below',
    showDescription: true,
    cardCount: 5,
  };
}

// Example usage in a route
app.get('/api/blog-config', async (req, res) => {
  const config = await getExperimentConfig(req.user.id);
  res.json(config);
});
```

## Strongly typed JSON payloads (advanced)

JSON payloads are flexible by default: attach any valid JSON to a variant and read it from `variant.payload`. For more control, define the expected type for variant payloads so Amplitude validates each payload before it reaches your application.

Strongly typed payloads let you:

- Set the expected type once per flag or experiment.
- Validate each variant payload against that type in Amplitude.
- Catch configuration issues, such as wrong types or missing required fields, before they reach production.

> **Note:** JSON payload model
>
> Strongly typed payloads build on the JSON payload model and SDKs this page covers. How you access `variant.payload` in code doesn't change. You gain stronger guarantees about the payload's shape.

### Strongly typed payload use cases

Strong typing is useful when:

- Multiple teams depend on the same configuration, such as product, engineering, and solutions.
- Payloads are complex, with nested objects, arrays, and multiple keys.
- You use flags as remote config or as layers that many experiments share.
- You want to reduce runtime errors and make configuration safer.

### Define the expected type for variant payloads

In the UI, select **Set payload type** to choose a payload type for the flag or experiment. The type applies to all variants.

**Payload type options:**

- **None**: No type enforcement. Use any valid JSON as a payload.
- **String**: `variant.payload` is a string.
- **Number**: `variant.payload` is a number.
- **Object**: `variant.payload` is an object with no fixed keys or types.
- **Array**: `variant.payload` is an array.
- **Custom Schema**: Define an object shape with specific keys and types. Choose the type of each field, such as string, number, boolean, or array, and mark any field as required. Amplitude represents the shape as a JSON schema and validates every variant payload against it.

If a variant payload doesn't match the type you select:

- The UI shows an error on the variant and indicates the problem, such as a missing required field or an incorrect type.
- You can't save the change until the payload is valid.
- Amplitude prevents you from deploying broken configuration.

Type enforcement complements validation in your application code instead of replacing it:

- Amplitude enforces the payload type when you edit payloads.
- Your application can validate or narrow types at runtime if needed.

### Example: Custom Schema

Suppose you want a payload with `headerText` required:

```json
{
  "buttonColor": "#4A90D9",
  "headerText": "Welcome back!",
  "maxRetries": 3,
  "features": ["dark_mode", "analytics"],
  "showBanner": true
}
```

When you choose **Custom Schema** and define these keys and types, Amplitude uses a JSON schema equivalent to:

```json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "buttonColor": {
      "type": "string"
    },
    "headerText": {
      "type": "string"
    },
    "maxRetries": {
      "type": "integer"
    },
    "features": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "showBanner": {
      "type": "boolean"
    }
  },
  "required": [
    "features",
    "buttonColor",
    "headerText",
    "maxRetries",
    "showBanner"
  ]
}
```

Mark any field as required in the Custom Schema. Payloads that omit a required field or use the wrong type for a key fail validation and block saving until you fix them.

### Using strongly typed payloads in your code

When you enable strong typing, the SDK interaction stays the same: read payloads from `variant.payload` and apply your own types in code.

In TypeScript, you can define a matching type:

```typescript
type BlogLayoutPayload = {
  layout: 'list' | 'cards' | 'grid';
  titlePosition: 'above' | 'below';
  gradient?: boolean;
  showDescription: boolean;
  cardCount: number;
};
```

Then cast and use:

```typescript
const variant = experiment.variant('blog-layout-flag');

if (variant && variant.payload) {
  const payload = variant.payload as BlogLayoutPayload;

  configureBlogLayout({
    layout: payload.layout,
    titlePosition: payload.titlePosition,
    showDescription: payload.showDescription,
    cardCount: payload.cardCount,
  });
}
```

The JSON schema in Amplitude enforces this contract when you create or edit payloads. Your application enforces the same contract through types or runtime checks.

## Best practices

Follow these best practices to keep your payloads robust and maintainable.

### 1. Always provide default values

Provide sensible defaults when accessing payload properties. Defaults ensure your application behaves correctly when:

- The user doesn't receive a variant.
- The payload is missing expected properties.
- You turn off the experiment.

Examples:

```javascript
const layout = variant?.payload?.layout || 'list';
const cardCount = variant?.payload?.cardCount || 5;
const showDescription = variant?.payload?.showDescription !== false;
```

### 2. Validate payload structure

Even with strongly typed payloads, validate payloads in your application:

```javascript
function validateLayoutPayload(payload: any): boolean {
  const validLayouts = ['list', 'cards', 'grid'];

  if (!validLayouts.includes(payload.layout)) {
    console.error('Invalid layout in payload:', payload.layout);
    return false;
  }

  if (typeof payload.cardCount !== 'number' || payload.cardCount < 1) {
    console.error('Invalid cardCount in payload:', payload.cardCount);
    return false;
  }

  return true;
}

const variant = experiment.variant('blog-layout-flag');
if (variant?.payload && validateLayoutPayload(variant.payload)) {
  applyLayoutConfig(variant.payload);
}
```

When using a JSON schema in Amplitude, keep your application-side validation aligned with that schema.

### 3. Keep payloads simple

Keep payload structures focused on configuration:

- Avoid large payloads. Aim for less than 10 KB.
- Avoid deeply nested or highly coupled structures.
- Avoid sensitive data. Payloads are visible in network traffic and logs.

### 4. Document your payload schema

Document the expected structure of your payloads, especially when multiple teams touch the same flag or experiment.

Example JSDoc-style documentation:

```javascript
/**
 * Blog Layout Flag Payload Schema
 * @typedef {Object} BlogLayoutPayload
 * @property {('list'|'cards'|'grid')} layout         - The layout style
 * @property {('above'|'below')}       titlePosition  - Title position relative to content
 * @property {boolean}                 gradient       - Whether to show gradient backgrounds
 * @property {boolean}                 showDescription- Whether to show post descriptions
 * @property {number}                  cardCount      - Number of cards to display (1-10)
 */
```

If you're using strongly typed payloads in Amplitude, keep your JSON schema and code documentation or types in sync. Everyone then shares the same contract.

## Common use cases

JSON payloads, with or without strong typing, support a wide range of use cases.

### Remote configuration

Use payloads to remotely configure features without deploying code:

```json
{
  "apiEndpoint": "https://api.v2.example.com",
  "timeout": 5000,
  "retries": 3,
  "enableCache": true
}
```

### UI customization

Configure UI elements like colors, layouts, and text:

```json
{
  "primaryColor": "#007AFF",
  "buttonText": "Get Started",
  "showBanner": true,
  "bannerMessage": "Limited time offer!"
}
```

### Feature rollout levels

Gradually expose functionality through configuration flags:

```json
{
  "enableAdvancedSearch": true,
  "enableFilters": true,
  "maxResults": 50,
  "showRecommendations": true
}
```

### Content variations

Test different content approaches:

```json
{
  "headline": "Transform Your Workflow",
  "subheadline": "Get started in minutes, not hours",
  "ctaText": "Start Free Trial",
  "showTestimonials": true
}
```

## Payload availability

When you access a variant from the SDK or Evaluation API, you can use the `value` and `payload` properties:

- `value`: the variant's value, such as "on", "off", "control", or "treatment".
- `payload`: the JSON configuration you attach.

The Management API and Amplitude UI provide other variant properties like `name` and `description`.

For more information about the variant data model, go to [Variants](https://amplitude.com/docs/feature-experiment/data-model#variants).

================================================================================
# Flag Dependencies
URL: https://amplitude.com/docs/feature-experiment/under-the-hood/flag-dependencies
================================================================================

# Flag Dependencies

Flag dependencies define relationships between flags to enforce evaluation order. Amplitude passes the result of each flag's evaluation to all subsequent evaluations to decide whether dependent flags [evaluate](https://amplitude.com/docs/feature-experiment/implementation#flag-dependencies) based on the result of the dependency.

Flag dependencies help implement:

- [Flag prerequisites](https://amplitude.com/docs/feature-experiment/advanced-techniques/flag-prerequisites).
- [Mutual exclusion groups](https://amplitude.com/docs/feature-experiment/advanced-techniques/mutually-exclusive-experiments).
- [Holdout groups](https://amplitude.com/docs/feature-experiment/advanced-techniques/holdout-groups-exclude-users).

## Flag prerequisites

![Flag prerequisites](https://amplitude.com/docs/images/experiment/release-group.drawio.svg)

_Available for flags and experiments._

> **Note:** Plan availability
>
> Flag prerequisites are only available to Enterprise customers. To learn more or upgrade, contact [Sales](https://amplitude.com/sales-contact).

Flag prerequisites is a generic implementation of flag dependencies that lets any flag or experiment depend on any other flag or experiment. Web experiments can only depend on other web experiments. They don't support dependencies on feature flags or feature experiments. Prerequisite evaluation can check specific variants or target users that the prerequisite flag or experiment didn't include.

Use flag prerequisites to:

- Actively develop large feature releases with many developers and teams.
- Build provisioning for users to primary SKUs with add-ons.
- Simplify complex feature flag logic in code.
- Build complex hierarchies of mutually exclusive experiments that start at different times.

For more information, go to [Flag Prerequisites](https://amplitude.com/docs/feature-experiment/advanced-techniques/flag-prerequisites).

## Mutual exclusion groups

![Mutual exclusion group](https://amplitude.com/docs/images/experiment/mutex-group.drawio.svg)

_Available for experiments only._

A mutual exclusion group ensures that your users only join a single experiment. In Experiment, a mutual exclusion group defines multiple slots, each with a percentage of traffic allocated to that slot. The mutual exclusion group is a flag with a variant for each slot. Experiments in the group add a dependency on one or more variants of the mutual exclusion group flag.

Amplitude doesn't return the variant result of a mutual exclusion group's evaluation and doesn't assign it as a user property.

For more information, go to [Set up and run mutually exclusive experiments](https://amplitude.com/docs/feature-experiment/advanced-techniques/mutually-exclusive-experiments).

## Holdout groups

![Holdout groups](https://amplitude.com/docs/images/experiment/holdout-group.drawio.svg)

_Available for experiments only._

A holdout group withholds a percentage of traffic from a group of experiments to measure the long-term and combined impact of multiple experiments. Experiment implements a holdout group with a flag that has two variants: `holdout` and `on`. Amplitude allocates the holdout percentage defined at group creation to the `holdout` variant. Experiments in the group depend on the holdout group's `on` variant.

Amplitude doesn't return the variant result of a holdout group's evaluation, but assigns it as a user property to enable holdout analysis.

For more information, go to [Holdout Groups](https://amplitude.com/docs/feature-experiment/advanced-techniques/holdout-groups-exclude-users).

## Local evaluation SDK support

Amplitude [SDKs](https://amplitude.com/docs/sdks) support flag dependencies (mutual exclusion and holdout groups) starting at the versions listed below.

> **Warning:** Older local evaluation SDK versions
>
> Earlier local evaluation SDK versions don't consider mutual exclusion or holdout groups. Two experiments in a mutual exclusion group that an old local evaluation SDK version evaluates aren't mutually exclusive.

| SDK | Local evaluation flag dependencies support |
| --- | --- |
| [Node.js](https://amplitude.com/docs/sdks/experiment-sdks/experiment-node-js) | `1.4.0+` |
| [Ruby](https://amplitude.com/docs/sdks/experiment-sdks/experiment-ruby) | `1.1.0+` |
| [JVM](https://amplitude.com/docs/sdks/experiment-sdks/experiment-jvm) | `1.1.0+` |
| [Go](https://amplitude.com/docs/sdks/experiment-sdks/experiment-go) | `1.1.0+` |
| [Python](https://amplitude.com/docs/sdks/experiment-sdks/experiment-python) | `1.1.0+` |
| [PHP](https://amplitude.com/docs/sdks/experiment-sdks/experiment-php) | `1.0.0+` |

================================================================================
# Experiment Node.js SDK
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-node-js
================================================================================

# Experiment Node.js SDK

- **Package:** `@amplitude/experiment-node-server` ([npm](https://www.npmjs.com/package/@amplitude/experiment-node-server))
- **Latest version:** 1.13.6

Official documentation for Amplitude Experiment's server-side Node.js SDK implementation.

This documentation has separate sections for [remote](https://amplitude.com/docs/feature-experiment/remote-evaluation) and [local](https://amplitude.com/docs/feature-experiment/local-evaluation) evaluation:

## Remote evaluation

Implements fetching variants for a user using [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation).

### Install

> **Note:** Node version compatibility
>
> The Node Server SDK works with Node 10+.

#### npm

```bash
npm install --save @amplitude/experiment-node-server
```

#### yarn

```bash
yarn add @amplitude/experiment-node-server
```

> **Tip:** Quick start
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants for the user](#fetch)
> 3. [Access a flag's variant](#fetch)
>
> ```js
> import { Experiment } from '@amplitude/experiment-node-server';
>
> // (1) Initialize the experiment client
> const experiment = Experiment.initializeRemote('<DEPLOYMENT_KEY>', {
>     fetchTimeoutMillis: 500,
>     fetchRetries: 1,
>     fetchRetryBackoffMinMillis: 0,
>     fetchRetryTimeoutMillis: 500,
> });
>
> // (2) Fetch variants for a user
> const user = {
>     user_id: 'user@company.com',
>     device_id: 'abcdefg',
>     user_properties: {
>         'premium': true,
>     },
> };
> const variants = experiment.fetchV2(user);
>
> // (3) Access a flag's variant
> const variant = variants['YOUR-FLAG-KEY'];
> if (variant?.value === 'on') {
>     // Flag is on
> } else {
>     // Flag is off
> }
> ```

### Initialize remote

Initialize the SDK client in your server on startup. The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) argument you pass to the `apiKey` parameter must live within the same project that you are sending analytics events to.

```js
initializeRemote(apiKey: string, config?: RemoteEvaluationConfig): RemoteEvaluationClient
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `apiKey` | required | The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) that authorizes fetch requests and determines which flags to evaluate for the user. |
| `config` | optional | The client [configuration](#configuration) used to customize SDK client behavior. |

> **Warning:** Timeout and retry configuration
>
> **The default timeout and retry configuration options are too high for most server environments**. Configure the timeout and retry options to best fit your performance requirements. If [remote evaluation performance](https://amplitude.com/docs/feature-experiment/under-the-hood/performance-and-caching#remote-evaluation) is too slow, consider using [local evaluation](#local-evaluation).

```js
import { Experiment } from '@amplitude/experiment-node-server';

const experiment = Experiment.initializeRemote('<DEPLOYMENT_KEY>', config: {
    fetchTimeoutMillis: 500,
    fetchRetries: 1,
    fetchRetryBackoffMinMillis: 0,
    fetchRetryTimeoutMillis: 500,
});
```

#### Configuration

Configure the SDK client during initialization.

> **Info:** EU data center
>
> If you're using Amplitude's EU data center, configure the `serverZone` option on initialization.

| Name | Description | Default Value |
| --- | --- | --- |
| `debug` | Enable additional debug logging. | `false` |
| `logLevel` | The minimum log level to output. Options: `Verbose`, `Debug`, `Info`, `Warn`, `Error`, `Disable`. Go to [custom logging](#custom-logging). | `Error` |
| `loggerProvider` | Custom logger implementation. Implement the `LoggerProvider` interface to integrate with your logging solution. Go to [custom logging](#custom-logging). | `ConsoleLogger` |
| `serverZone` | The Amplitude data center to use. Either `"us"` or `"eu"` | `"us"` |
| `serverUrl` | The host to fetch variants from. | `https://api.lab.amplitude.com` |
| `fetchTimeoutMillis` | The timeout for fetching variants in milliseconds. This timeout only applies to the initial request, not subsequent retries | `10000` |
| `fetchRetries` | The number of retries to attempt if a request to fetch variants fails. | `8` |
| `fetchRetryBackoffMinMillis` | The minimum (initial) backoff after a request to fetch variants fails. The SDK scales this delay by the `fetchRetryBackoffScalar` | `500` |
| `fetchRetryBackoffMaxMillis` | The maximum backoff between retries. If the scaled backoff becomes greater than the max, the SDK uses the max for all subsequent requests | `10000` |
| `fetchRetryBackoffScalar` | Scales the minimum backoff exponentially. | `1.5` |
| `fetchRetryTimeoutMillis` | The request timeout for retrying variant fetches. | `10000` |

### Fetch

Fetches variants for a [user](https://amplitude.com/docs/feature-experiment/data-model#users) and returns the results. The function [remote evaluates](https://amplitude.com/docs/feature-experiment/remote-evaluation) the user for flags associated with the deployment used to initialize the SDK client.

```js
fetchV2(user: ExperimentUser, fetchOptions?: FetchOptions): Promise<Variants>
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | required | The [user](https://amplitude.com/docs/feature-experiment/data-model#users) for whom to fetch variants. |
| `fetchOptions` | optional | The [options](#fetch-options) for the fetch request. |

**FetchOptions**

| Name | Description | Default Value |
| --- | --- | --- |
| `flagKeys` | Specific flags or experiments to evaluate. If undefined, null, or empty, the SDK evaluates all flags and experiments. | `undefined` |
| `tracksExposure` | To track or not track an exposure event for this fetch request. If `undefined`, uses the server's default behavior (doesn't track exposure). | `undefined` |
| `tracksAssignment` | To track or not track an assignment event for this fetch request. If `undefined`, uses the server's default behavior (does track assignment). | `undefined` |

```js
const user = {
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
};
const variants = await experiment.fetchV2(user);
```

After fetching variants for a user, you may to access the variant for a specific flag.

```js
const variant = variants['YOUR-FLAG-KEY'];
if (variant?.value === 'on') {
    // Flag is on
} else {
    // Flag is off
}
```

## Local evaluation

Implements evaluating variants for a user using [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation). If you plan to use local evaluation, [understand the tradeoffs](https://amplitude.com/docs/feature-experiment/local-evaluation#targeting-capabilities).

### Install

Install the Node.js Server SDK with `npm` or `yarn`.

#### npm

```bash
npm install --save @amplitude/experiment-node-server
```

#### yarn

```bash
yarn add @amplitude/experiment-node-server
```

> **Tip:** Quick start
>
> 1. [Initialize the local evaluation client.](#initialize-local)
> 2. [Start the local evaluation client.](#start)
> 3. [Evaluate a user.](#evaluate)
>
> ```js
> import { Experiment } from '@amplitude/experiment-node-server';
>
> // (1) Initialize the local evaluation client with a server deployment key.
> const experiment = Experiment.initializeLocal('<DEPLOYMENT_KEY>', {
>   // (Recommended) Enable local evaluation cohort targeting.
>   cohortSyncConfig: {
>     apiKey: '<API_KEY>',
>     secretKey: '<SECRET_KEY>'
>   }
> });
>
> // (2) Start the local evaluation client.
> await experiment.start();
>
> // (2) Evaluate a user.
> const user = { device_id: 'abcdefg' };
> const variants = experiment.evaluateV2(user);
> ```

#### Account-level bucketing and analysis (v1.5.0+)

If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users. Reach out to your representative to gain access to this beta feature.

Include groups in the user sent with the fetch request (recommended), or identify groups with the user through a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or through [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).

```js
await fetch({
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
    groups: {'org name': ['Amplitude']}
});
```

To pass freeform group properties, refer to this example:

```js
await fetch({
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
    group_properties: {'org name': ['Amplitude']}
});
```

### Initialize Local

Initializes a [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation) client.

> **Warning:** Server deployment key
>
> You must [initialize](#initialize-local) the local evaluation client with a server [deployment](https://amplitude.com/docs/feature-experiment/data-model#deployments) key to get access to local evaluation flag configs.

```js
initializeLocal(apiKey: string, config?: LocalEvaluationConfig): LocalEvaluationClient
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `apiKey` | required | The server [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) that authorizes fetch requests and determines which flags to evaluate for the user. |
| `config` | optional | The client [configuration](#configuration) used to customize SDK client behavior. |

> **Tip:** Flag streaming
>
> Use the `streamUpdates` [configuration](#configuration) to get flag config updates pushed to SDK (default false), instead of polling every `flagConfigPollingIntervalMillis` milliseconds. The time for SDK to receive the update after saving is generally less than one second. The SDK falls back to polling if streaming fails. Configure `flagConfigPollingIntervalMillis` [configuration](#configuration) as well for fallback.

#### Configuration

You can configure the SDK client on initialization.

> **Info:** EU data center
>
> If you're using Amplitude's EU data center, configure the `serverZone` option on initialization.

**LocalEvaluationConfig**

| Name | Description | Default Value |
| --- | --- | --- |
| `debug` | Set to `true` to enable debug logging. | `false` |
| `logLevel` | The minimum log level to output. Options: `Verbose`, `Debug`, `Info`, `Warn`, `Error`, `Disable`. Go to [custom logging](#custom-logging). | `Error` |
| `loggerProvider` | Custom logger implementation. Implement the `LoggerProvider` interface to integrate with your logging solution. Go to [custom logging](#custom-logging). | `ConsoleLogger` |
| `serverZone` | The Amplitude data center to use. Either `"us"` or `"eu"` | `"us"` |
| `serverUrl` | The host to fetch flag configurations from. | `https://api.lab.amplitude.com` |
| `bootstrap` | Bootstrap the client with a map of flag key to flag configuration | `{}` |
| `flagConfigPollingIntervalMillis` | The interval (in milliseconds) to poll for updated flag configs after calling `start()` | `30000` |
| `assignmentConfig` | **Deprecated.** Configuration for automatically tracking assignment events after an evaluation. | `null` |
| `exposureConfig` | Configuration for tracking exposure events after an evaluation. | `null` |
| `streamUpdates` | Enable streaming to replace polling for receiving flag config updates. Instead of polling every second, Amplitude servers push updates to SDK generally within a second. If the stream fails for any reason, the SDK falls back to polling automatically and retries streaming after some interval. | `false` |
| `streamServerUrl` | The stream server url to stream from. | `https://stream.lab.amplitude.com` |
| `streamFlagConnTimeoutMillis` | The timeout for establishing a valid flag config stream. This includes time for a connection to be established to stream server and time for receiving initial flag configs. | `1500` |
| `cohortSyncConfig` | Configuration to enable cohort downloading for [local evaluation cohort targeting](#local-evaluation-cohort-targeting). | `undefined` |

**AssignmentConfig**

| Name | Description | Default Value |
| --- | --- | --- |
| `apiKey` | The analytics API key and NOT the experiment deployment key | _required_ |
| `cacheCapacity` | The maximum number of assignments stored in the assignment cache | `65536` |
| [Analytics SDK Options](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#configure-the-sdk) | Options to configure the underlying Amplitude Analytics SDK used to track assignment events |  |

**ExposureConfig**

| Name | Description | Default Value |
| --- | --- | --- |
| `apiKey` | The analytics API key and NOT the experiment deployment key | _required_ |
| `cacheCapacity` | The maximum number of exposures stored in the exposure cache | `65536` |
| [Analytics SDK Options](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#configure-the-sdk) | Options to configure the underlying Amplitude Analytics SDK used to track exposure events |  |

**CohortSyncConfig**

| Name | Description | Default Value |
| --- | --- | --- |
| `apiKey` | The analytics API key and NOT the experiment deployment key | _required_ |
| `secretKey` | The analytics secret key | _required_ |
| `maxCohortSize` | The maximum size of cohort that the SDK downloads. The SDK doesn't download cohorts larger than this size. | `2147483647` |
| `cohortPollingIntervalMillis` | The interval, in milliseconds, to poll Amplitude for cohort updates (60000 minimum). | `60000` |
| `cohortServerUrl` | The cohort server endpoint from which to fetch cohort data. For hitting the EU data center, set `serverZone` to `eu`. Setting this value overrides `serverZone` defaults. | `https://cohort-v2.lab.amplitude.com` |

### Start

Start the local evaluation client, pre-fetching local evaluation mode flag configs for [evaluation](#evaluate) and starting the flag config poller at the [configured](#configuration) interval.

```js
start(): Promise<void>
```

You should await the result of `start()` to ensure that flag configs are ready to be used before calling [`evaluateV2()`](#evaluate)

```js
await experiment.start();
```

### Evaluate

Executes the [evaluation logic](https://amplitude.com/docs/feature-experiment/implementation) using the flags pre-fetched on [`start()`](#start). You must give `evaluate` a user object argument. You can optionally pass an array of flag keys if you require only a specific subset of flag variants.

> **Tip:** Exposure tracking
>
> Set [`exposureConfig`](#configuration) to enable exposure tracking. Then, set `tracksExposure` to `true` in `EvaluateOptions` when calling `evaluateV2()`.

```js
evaluateV2(user: ExperimentUser, flagKeys?: string[], options?: EvaluateOptions): Record<string, Variant>
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | required | The [user](https://amplitude.com/docs/feature-experiment/data-model#users) to evaluate. |
| `flagKeys` | optional | Specific flags or experiments to evaluate. If undefined, null, or empty, the SDK evaluates all flags and experiments. |
| `options` | optional | The [options](#evaluate-options) for the evaluation request. |

```js
// The user to evaluate
const user = { device_id: 'abcdefg' };

// Evaluate all flag variants
const allVariants = experiment.evaluateV2(user);

// Evaluate a specific subset of flag variants
const specificVariants = experiment.evaluateV2(user, [
  'my-local-flag-1',
  'my-local-flag-2',
]);
```

**EvaluateOptions**

| Name | Description | Default Value |
| --- | --- | --- |
| `tracksExposure` | If `true`, the SDK tracks an exposure event for the evaluated variants. | `false` |

### Local evaluation cohort targeting

Since version `1.13.4`, the local evaluation SDK client supports downloading cohorts for local evaluation targeting. You must configure the `cohortSyncConfig` option with the analytics `apiKey` and `secretKey` on initialization to enable this support.

```js
const experiment = Experiment.initializeLocal('<DEPLOYMENT_KEY>', {
  // (Recommended) Enable local evaluation cohort targeting.
  cohortSyncConfig: {
    apiKey: '<API_KEY>',
    secretKey: '<SECRET_KEY>'
  }
});
```

Consider configuring the `maxCohortSize` to avoid downloading large cohorts, which may cause your service to run out of memory. The SDK doesn't download cohorts that are too large.

## Custom logging

Control log verbosity with the `logLevel` configuration, or implement the `LoggerProvider` interface to integrate your own logger.

### Log levels

- `Verbose`: Detailed debugging logs
- `Debug`: Development and troubleshooting logs
- `Info`: General information
- `Warn`: Warnings
- `Error`: Errors (default)
- `Disable`: No logs

### Custom logger

Implement the `LoggerProvider` interface to use your own logging solution:

```ts
import { Experiment, LogLevel, LoggerProvider } from '@amplitude/experiment-node-server';

class MyCustomLogger implements LoggerProvider {
  verbose(message, ...optionalParams) {
    // Implement verbose logging
  }

  debug(message, ...optionalParams) {
    // Implement debug logging
  }

  info(message, ...optionalParams) {
    // Implement info logging
  }

  warn(message, ...optionalParams) {
    // Implement warn logging
  }

  error(message, ...optionalParams) {
    // Implement error logging
  }
}

// Initialize with custom logger
const experiment = Experiment.initializeLocal(
  '<DEPLOYMENT_KEY>',
  {
    logLevel: LogLevel.Debug,
    loggerProvider: new MyCustomLogger()
  }
);
```

> **Note:** Backward compatibility
>
> The `debug` configuration field is still supported. When set to `true`, it overrides `logLevel` to `Debug`.

## Access Amplitude cookies

If you use the Amplitude Analytics SDK on the client-side, the Node.js server SDK provides an `AmplitudeCookie` class with convenience functions for parsing and interacting with the Amplitude identity cookie. Use the class for server-side rendering (SSR) use cases where you need to:

- Maintain consistent Device IDs between server-rendered and client-side code.
- Access the user's Amplitude identity before the page renders.
- Generate and set cookies on the server when they don't exist yet.

The class ensures that the Device ID on the server matches the Device ID set on the client, even if the client didn't generate a Device ID yet.

```js
import { AmplitudeCookie } from '@amplitude/experiment-node-server';
import { v4 as uuidv4 } from 'uuid';
// Get the cookie name for the Amplitude API key
// For Browser SDK 2.0 cookies, pass true as second parameter:
// const ampCookieName = AmplitudeCookie.cookieName('amplitude-api-key', true);
const ampCookieName = AmplitudeCookie.cookieName('amplitude-api-key');
let deviceId = null;
// Try to get device ID from existing cookie
if (req.cookies[ampCookieName]) {
  deviceId = AmplitudeCookie.parse(req.cookies[ampCookieName]).device_id;
  // For Browser SDK 2.0: AmplitudeCookie.parse(req.cookies[ampCookieName], true).device_id;
}
// If no device ID found, generate a new one and set the cookie
if (!deviceId) {
  deviceId = uuidv4();
  const ampCookieValue = AmplitudeCookie.generate(deviceId);
  // For Browser SDK 2.0: AmplitudeCookie.generate(deviceId, true);
  res.cookie(ampCookieName, ampCookieValue, {
    domain: '.your-domain.com', // this should be the same domain used by the Amplitude JS SDK
    httpOnly: false,
    secure: false
  });
}
```

================================================================================
# Experiment Android SDK
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-android
================================================================================

# Experiment Android SDK

- **Package:** `com.amplitude:experiment-android-client` ([maven](https://search.maven.org/artifact/com.amplitude/experiment-android-client))
- **Latest version:** 1.16.1

Official documentation for Amplitude Experiment's Client-side Android SDK implementation.

## Install

Add to the dependencies in your Android project's `build.gradle` file.

```groovy
dependencies {
  implementation 'com.amplitude:experiment-android-client:<VERSION>'
}
```

> **Tip:** Quick start
>
> The right way to initialize the Experiment SDK depends on whether you use an Amplitude SDK for analytics or a third party (for example, Segment).
>
> #### Amplitude
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```kotlin
> class MyApplication : Application() {
>
>     override fun onCreate() {
>         super.onCreate()
>
>         // (1) Initialize the experiment client
>         val client = Experiment.initializeWithAmplitudeAnalytics(
>             this, "DEPLOYMENT_KEY", ExperimentConfig()
>         )
>
>         // (2) Fetch variants
>         try {
>             // NOTE: The future returned resolves after a network call. Do not
>             //       wait for this future on the main application thread in
>             //       production applications to avoid ANR if the user has a poor
>             //       network connection.
>             client.fetch().get()
>         } catch (e: Exception) {
>             e.printStackTrace()
>         }
>
>         // (3) Lookup a flag's variant
>         val variant = client.variant("<FLAG_KEY>")
>         if (variant.value == "on") {
>             // Flag is on
>         } else {
>             // Flag is off
>         }
>     }
> }
> ```
>
> #### Third party
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants for a user](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```kotlin
> class MyApplication : Application() {
>
>    override fun onCreate() {
>        super.onCreate()
>
>        // (1) Initialize the experiment client
>        val client = Experiment.initialize(this, "DEPLOYMENT_KEY", ExperimentConfig.builder()
>            .exposureTrackingProvider(object : ExposureTrackingProvider {
>                override fun track(exposure: Exposure) {
>                    // TODO: Implement exposure tracking
>                    // Analytics.with(context).track(
>                    //     "\$exposure",
>                    //     Properties()
>                    //         .putValue("flag_key", exposure.flagKey)
>                    //         .putValue("variant", exposure.variant)
>                    //         .putValue("experiment_key", exposure.experimentKey));
>                    // )
>                }
>            }).build()
>        )
>
>        // (2) Fetch variants for a user
>        val user = ExperimentUser.builder()
>            .userId("user@company.com")
>            .deviceId("abcdefg")
>            .userProperty("premium", true)
>            .build()
>        try {
>             // NOTE: The future returned resolves after a network call. Do not
>             //       wait for this future on the main application thread in
>             //       production applications to avoid ANR if the user has a poor
>             //       network connection.
>            client.fetch(user).get()
>        } catch (e: Exception) {
>            e.printStackTrace()
>        }
>
>        // (3) Lookup a flag's variant
>        val variant = client.variant("<FLAG_KEY>")
>        if (variant.value == "on") {
>            // Flag is on
>        } else {
>            // Flag is off
>        }
>    }
> }
> ```

## Initialize

Initialize the SDK client in your application on startup. The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) argument you pass to the `apiKey` parameter must live within the same project that you are sending analytics events to.

#### Amplitude

```kotlin
fun initializeWithAmplitudeAnalytics(
    application: Application, apiKey: String, config: ExperimentConfig
)
```

#### Third party

```kotlin
fun initialize(application: Application, apiKey: String, config: ExperimentConfig)
```

- **`application`**

  - **Requirement**: required
  - **Description**: The Android `Application` context. Used to persist variants across sessions.
- **`apiKey`**

  - **Requirement**: required
  - **Description**: The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) which authorizes fetch requests and determines which flags the SDK evaluates for the user.
- **`config`**

  - **Requirement**: optional
  - **Description**: The client [configuration](#configuration) used to customize SDK client behavior.

The initializer returns a singleton instance, so subsequent initializations for the same instance name return the initial instance. To create multiple instances, use the `instanceName` [configuration](#configuration).

#### Amplitude

```kotlin
val experiment = Experiment.initializeWithAmplitudeAnalytics(
    context,
    "DEPLOYMENT_KEY", 
    ExperimentConfig().apply {
        // must match the name you used for your Amplitude Analytics instance
        instanceName = "myCustomInstance"
    }
)
```

#### Third party

```kotlin
val experiment = Experiment.initialize(
    context,
    "DEPLOYMENT_KEY",
    ExperimentConfig().apply {
        // must match the name you used for your Amplitude Analytics instance
        instanceName = "myCustomInstance"
    }
)
```

### Configuration

SDK client configuration occurs during initialization.

| Name | Description | Default Value |
| --- | --- | --- |
| `debug` | **Deprecated.** When `true`, sets `logLevel` to `Debug`. Use `logLevel` instead. | `false` |
| `logLevel` | The minimum log level to output. The SDK ignores messages below this level. Options: `LogLevel.DISABLE`, `LogLevel.ERROR`, `LogLevel.WARN`, `LogLevel.INFO`, `LogLevel.DEBUG`, `LogLevel.VERBOSE`. Go to [Custom logging](#custom-logging). | `LogLevel.ERROR` |
| `loggerProvider` | Custom logger implementation. Must implement the `LoggerProvider` interface. Go to [Custom logging](#custom-logging). | `AndroidLoggerProvider()` |
| `fallbackVariant` | The default variant to fall back if a variant for the provided key doesn't exist. | `{}` |
| `initialVariants` | An initial set of variants to access. This field helps bootstrap the client SDK with values rendered by the server using server-side rendering (SSR). | `{}` |
| `source` | The primary source of variants. Set the value to `Source.INITIAL_VARIANTS` and configure `initialVariants` to bootstrap the SDK for SSR or testing purposes. | `Source.LOCAL_STORAGE` |
| `serverZone` | Select the Amplitude data center to get flags and variants from | `ServerZone.US` |
| `serverUrl` | The host to fetch remote evaluation variants from. For hitting the EU data center, use `serverZone`. | `https://api.lab.amplitude.com` |
| `flagsServerUrl` | The host to fetch local evaluation flags from. For hitting the EU data center, use `serverZone`. | `https://flag.lab.amplitude.com` |
| `fetchTimeoutMillis` | The timeout for fetching variants in milliseconds. | `10000` |
| `retryFetchOnFailure` | Whether to retry variant fetches in the background if the request doesn't succeed. | `true` |
| `automaticExposureTracking` | If true, calling [`variant()`](#variant) tracks an exposure event through the configured `exposureTrackingProvider`. If no exposure tracking provider is set, this configuration option does nothing. | `true` |
| `fetchOnStart` | If true or null, always [fetch](#fetch) remote evaluation variants on [start](#start). If false, never fetch on start. | `true` |
| `pollOnStart` | Poll for local evaluation flag configuration updates every minute on [start](#start). | `true` |
| `automaticFetchOnAmplitudeIdentityChange` | Only matters if you use the `initializeWithAmplitudeAnalytics` initialization function to integrate with the Amplitude Analytics SDK. If `true` any change to the user ID, device ID or user properties from analytics triggers the experiment SDK to fetch variants and update its cache. | `false` |
| `userProvider` | An interface used to provide the user object to `fetch()` when called. | `null` |
| `exposureTrackingProvider` | Implement and configure this interface to track exposure events through the experiment SDK, either automatically or explicitly. | `null` |
| `instanceName` | Custom instance name for experiment SDK instance. **The value of this field is case-sensitive.** | `null` |
| `initialFlags` | A JSON string representing an initial set of flag configurations to use for local evaluation. | `undefined` |

> **Note:** EU Data Center
>
> If you use Amplitude's EU data center, configure the `serverZone` option on initialization to `ServerZone.EU`.

### Integrations

If you use either Amplitude or Segment Analytics SDKs to track events into Amplitude, set up an integration on initialization. Integrations automatically implement [provider](#providers) interfaces to enable a more streamlined developer experience by making it easier to **manage user identity** and **track exposures events**.

#### Amplitude integration

The Amplitude Experiment SDK is set up to integrate with the Amplitude Analytics SDK.

#### Java

```java
Amplitude.getInstance().init("API_KEY");
ExperimentClient experiment = Experiment.initializeWithAmplitudeAnalytics( 
    context, "DEPLOYMENT_KEY", new ExperimentConfig());
```

#### Kotlin

```kotlin
Amplitude.getInstance().init("API_KEY")
val experiment = Experiment.initializeWithAmplitudeAnalytics( 
    context,
    "DEPLOYMENT_KEY",
    ExperimentConfig(),
)
```

If you use a custom instance name for analytics, set the same value in the `instanceName` configuration option in the Experiment SDK.

Using the integration initializer configures implementations of the [user provider](#user-provider) and [exposure tracking provider](#exposure-tracking-provider) interfaces to pull user data from the Amplitude Analytics SDK and track exposure events.

**Supported Versions**

| Analytics Kotlin SDK Version | Analytics Legacy SDK Version | Experiment SDK Version |
| --- | --- | --- |
| `1.0.0+` | `2.36.0+` | `1.5.1+` |

#### Segment integration

Configure Experiment's integration with Segment Analytics manually. Then configure the Experiment SDK on initialization with an instance of the exposure tracking provider. Make sure this happens _after_ the analytics SDK loads and initializes.

#### Java

```java
class SegmentExposureTrackingProvider implements ExposureTrackingProvider {
    private Analytics analytics;
    public SegmentExposureTrackingProvider(Analytics analytics) {
        this.analytics = analytics;
    }
    @Override
    public void track(Exposure exposure) {
        analytics.track(
                "$exposure",
                new Properties()
                    .putValue("flag_key", exposure.flagKey)
                    .putValue("variant", exposure.variant)
                    .putValue("experiment_key", exposure.experimentKey));
    }
}
```

#### Kotlin

```kotlin
class SegmentExposureTrackingProvider(
    private val analytics: Analytics
): ExposureTrackingProvider {

    override fun track(exposure: Exposure) {
        analytics.track(
            "\$exposure",
            Properties()
                .putValue("flag_key", exposure.flagKey)
                .putValue("variant", exposure.variant)
                .putValue("experiment_key", exposure.experimentKey)
        )
    }
}
```

Configure the Experiment SDK on initialization with an instance of the exposure tracking provider.

#### Java

```java
Analytics analytics = // Initialize segment analytics
ExperimentConfig config = ExperimentConfig.builder()
    .exposureTrackingProvider(new SegmentExposureTrackingProvider(analytics))
    .build();
ExperimentClient experiment = Experiment.initialize(context, "DEPLOYMENT_KEY", config);
```

#### Kotlin

```kotlin
val analytics = // Initialize segment analytics
val config = ExperimentConfig.builder()
    .exposureTrackingProvider(SegmentExposureTrackingProvider(analytics))
    .build()
val experiment = Experiment.initialize(context, "DEPLOYMENT_KEY", config)
```

When [fetching variants](#fetch), pass the segment anonymous ID and user ID for the device ID and user ID, respectively.

#### Java

```java
String userId = analytics.getAnalyticsContext().traits().userId();
String deviceId = analytics.getAnalyticsContext().traits().anonymousId();
try {
    ExperimentUser user = ExperimentUser.builder()
        .userId(userId)
        .deviceId(deviceId)
        .build();
    experiment.fetch(user).get();
} catch (Exception e) {
    e.printStackTrace();
}
```

#### Kotlin

```kotlin
val userId = analytics.analyticsContext.traits().userId()
val deviceId = analytics.analyticsContext.traits().anonymousId()
try {
    val user = ExperimentUser.builder()
        .userId(userId)
        .deviceId(deviceId)
        .build()
    experiment.fetch(user).get()
} catch (e: Exception) {
    e.printStackTrace()
}
```

## Fetch

Fetches variants for a [user](https://amplitude.com/docs/feature-experiment/data-model#users) and stores the results in the client for fast access. The function [remote evaluates](https://amplitude.com/docs/feature-experiment/remote-evaluation) the user for flags associated with the deployment used to initialize the SDK client.

```kotlin
fun fetch(user: ExperimentUser? = null, options: FetchOptions? = null): Future<ExperimentClient>
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to fetch. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. |
| `options` | optional | Explicit flag keys to fetch. |

Amplitude Experiment recommends calling `fetch()` during application start up so that the user gets the most up-to-date variants for the application session. Furthermore, wait for the fetch request to return a result before rendering the user experience to avoid the interface "flickering".

#### Java

```java
try {
    ExperimentUser user = ExperimentUser.builder()
        .userId("user@company.com")
        .userProperty("premium", true)
        .build();
    experiment.fetch(user).get();
} catch (Exception e) {
    e.printStackTrace();
}
```

#### Kotlin

```kotlin
try {
    val user = ExperimentUser.builder()
        .userId("user@company.com")
        .userProperty("premium", true)
        .build()
    experiment.fetch(user).get()
} catch (e: Exception) {
    e.printStackTrace()
}
```

If you're using an [integration](#integrations) or a custom [user provider](#user-provider) then you can fetch without inputting the user.

#### Java

```java
experiment.fetch(null);
```

#### Kotlin

```kotlin
experiment.fetch()
```

> **Tip:** Fetch when user identity changes
>
> If you want the most up-to-date variants for the user, it's recommended that you call `fetch()` whenever the user state changes in a meaningful way. For example, if the user logs in and receives a user ID, or has a user property set which may affect flag or experiment targeting rules.
>
> For **user properties**, Amplitude recommends passing new user properties explicitly to `fetch()` instead of relying on user enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation). Remote user-property sync through a separate system has no timing guarantees for `fetch()`, which can create a race condition.

If `fetch()` times out (default 10 seconds) or fails for any reason, the SDK client returns and retries in the background with back-off. You may configure the timeout or disable retries in the [configuration options](#configuration) during SDK client initialization.

#### Account-level bucketing and analysis (v1.9.0+)

If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users. Reach out to your representative to gain access to this beta feature.

Include groups in the user sent with the fetch request (recommended), or identify them with the user using a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or using [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).

#### Java

```java
try {
    ExperimentUser user = ExperimentUser.builder()
        .userId("user@company.com")
        .userProperty("premium", true)
        .group("org name", ["Amplitude"])
        .build();
    experiment.fetch(user).get();
} catch (Exception e) {
    e.printStackTrace();
}
```

#### Kotlin

```kotlin
try {
    val user = ExperimentUser.builder()
        .userId("user@company.com")
        .userProperty("premium", true)
        .group("org name", ["Amplitude"])
        .build()
    experiment.fetch(user).get()
} catch (e: Exception) {
    e.printStackTrace()
}
```

To pass freeform group properties, refer to this example:

#### Java

```java
try {
    ExperimentUser user = ExperimentUser.builder()
        .userId("user@company.com")
        .userProperty("premium", true)
        .groupProperty("org name", ["Amplitude"])
        .build();
    experiment.fetch(user).get();
} catch (Exception e) {
    e.printStackTrace();
}
```

#### Kotlin

```kotlin
try {
    val user = ExperimentUser.builder()
        .userId("user@company.com")
        .userProperty("premium", true)
        .groupProperty("org name", ["Amplitude"])
        .build()
    experiment.fetch(user).get()
} catch (e: Exception) {
    e.printStackTrace()
}
```

## Start

> **Info:** Fetch vs start
>
> Use `start` if you're using client-side [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation). If you're only using [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), call [fetch](#fetch) instead of `start`.

Start the Experiment SDK to get flag configurations from the server and fetch remote evaluation variants for the user. The SDK is ready when the returned future resolves.

```kotlin
fun start(user: ExperimentUser? = null): Future<ExperimentClient>
```

| Parameter | Requirement | Description                                                                                                                                                                                                                                                                                                                                                                                    | | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | | `user`    | optional    | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to fetch variants. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. Also sets the user in the SDK for reuse. | `null` |

Call `start()` when your application is initializing, after user information is available to fetch variants. The future resolves after loading local evaluation flag configurations and fetching remote evaluation variants.

Configure the behavior of `start()` by setting `fetchOnStart` in the SDK configuration on initialization to improve performance based on the needs of your application.

- If your application never relies on remote evaluation, set `fetchOnStart` to `false` to avoid increased startup latency caused by remote evaluation.
- If your application relies on remote evaluation, but not right at startup, you may set `fetchOnStart` to `false` and call `fetch()` and await the future separately.

#### Amplitude

```kotlin
try {
    experiment.start().get();
} catch (e: Exception) {
    e.printStackTrace();
}
```

#### Third-party

```kotlin
ExperimentUser user = ExperimentUser.builder()
    .userId("user@company.com")
    .userProperty("premium", true)
    .build();
try {
    experiment.start(user).get();
} catch (e: Exception) {
    e.printStackTrace();
}
```

## Variant

Access a [variant](https://amplitude.com/docs/feature-experiment/data-model#variants) for a [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) from the SDK client's local store.

> **Info:** Automatic exposure tracking
>
> When you use an [integration](#integrations) or set a custom [exposure tracking provider](#exposure-tracking-provider), `variant()` tracks an exposure event through the tracking provider. To disable this functionality, [configure](#configuration) `automaticExposureTracking` to be `false`, and track exposures manually using [`exposure()`](#exposure).

```kotlin
fun variant(key: String, fallback: Variant? = null): Variant
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) to access the variant for. |
| `fallback` | optional | The value to return if the SDK found no variant for the given `flagKey`. |

When determining which variant a user has been bucketed into, compare the variant `value` to a well-known string.

#### Java

```java
Variant variant = client.variant("<FLAG_KEY>");
if (variant.is("on")) {
    // Flag is on
} else {
    // Flag is off
}
```

#### Kotlin

```kotlin
Variant variant = client.variant("<FLAG_KEY>");
if (variant.value == "on") {
    // Flag is on
} else {
    // Flag is off
}
```

> **Note:** Access the variant's payload
>
> A variant may also include a dynamic [payload](https://amplitude.com/docs/feature-experiment/data-model#variants) of arbitrary data. Access the `payload` field from the variant object after checking the variant's `value`.
>
> The `payload` on Android is of type `Object` (`Any?`) meaning you must cast the payload to the expected type. Cast JSON object and array types as `org.json.JSONObject` and `org.json.JSONArray` respectively.
>
> For example, if the payload is `{"key":"value"}`:
>
> #### Java
>
> ```java
> Variant variant = experiment.variant("<FLAG_KEY>");
> if (variant.is("on") && variant.payload != null) {
>     try {
>         String value = ((JSONObject) variant.payload).getString("key");
>     } catch (Exception e) {
>         e.printStackTrace();
>     }
> }
> ```
>
> #### Kotlin
>
> ```kotlin
> val variant = experiment.variant("<FLAG_KEY>")
> if (variant.value == "on") {
>     try {
>         val value = (variant.payload as JSONObject).getString("key")
>     } catch (e: Exception) {
>         e.printStackTrace()
>     }
> }
> ```

A `null` variant `value` means that the user hasn't been bucketed into a variant. You may use the built in **fallback** parameter to provide a variant to return if the store doesn't contain a variant for the given flag key.

#### Java

```java
Variant variant = experiment.variant("<FLAG_KEY>", new Variant("control"));
if (variant.is("control")) {
    // Control
} else if (variant.is("treatment")) {
    // Treatment
}
```

#### Kotlin

```kotlin
val variant = experiment.variant("<FLAG_KEY>", Variant("control"))
if (variant.value == "control") {
    // Control
} else if (variant.value == "treatment") {
    // Treatment
}
```

## All

Access all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) stored by the SDK client.

```kotlin
fun all(): Map<String, Variant>
```

#### Java

```java
experiment.all();
```

#### Kotlin

```kotlin
experiment.all()
```

## Clear

Clear all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) in the cache and storage.

```kotlin
fun clear()
```

You can call `clear` after user logout to clear the variants in cache and storage.

#### Java

```java
experiment.clear();
```

#### Kotlin

```kotlin
experiment.clear()
```

## Exposure

Manually track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for the current variant of the given flag key through configured [integration](#integrations) or custom [exposure tracking provider](#exposure-tracking-provider). Generally used in conjunction with setting the `automaticExposureTracking` [configuration](#configuration) optional to `false`.

```kotlin
fun exposure(key: String)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) variant to track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for. |

#### Java

```java
Variant variant = experiment.variant("<FLAG_KEY>");

// Do other things...

experiment.exposure("<FLAG_KEY>");
if (variant.is("control")) {
    // Control
} else if (variant.is("treatment")) {
    // Treatment
}
```

#### Kotlin

```kotlin
val variant = experiment.variant("<FLAG_KEY>")

// Do other things...

experiment.exposure("<FLAG_KEY>")
if (variant.value == "control") {
    // Control
} else if (variant.value == "treatment") {
    // Treatment
}
```

## Providers

> **Tip:** Integrations
>
> If you use Amplitude or Segment analytics SDKs along side the Experiment Client SDK, Amplitude recommends you use an [integration](#integrations) instead of implementing custom providers.

Provider implementations enable a more streamlined developer experience by making it easier to manage user identity and track exposures events.

### User provider

The SDK client uses the user provider to access the most up-to-date user information only when needed (for example, when the SDK calls [`fetch()`](#fetch)). The user provider is optional, but helps if you have a user information store already set up in your application. With a user provider, you don't need to manage two separate user info stores in parallel. Separate stores can create divergent user state if the application updates the user store and experiment isn't (or vice versa).

```kotlin
interface ExperimentUserProvider {
    fun getUser(): ExperimentUser
}
```

To use your custom user provider, set the `userProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

#### Java

```java
ExperimentConfig config = ExperimentConfig.builder()
    .userProvider(new CustomUserProvider())
    .build();
ExperimentClient experiment = Experiment.initialize(
    context, "<DEPLOYMENT_KEY>", config);
```

#### Kotlin

```kotlin
val config = ExperimentConfig.builder()
    .userProvider(CustomUserProvider())
    .build()
val experiment = Experiment.initialize(context, "<DEPLOYMENT_KEY>", config)
```

### Exposure tracking provider

Amplitude highly recommends implementing an exposure tracking provider. [Exposure tracking](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) increases the accuracy and reliability of experiment results and improves visibility into which flags and experiments a user is exposed to.

```kotlin
interface ExposureTrackingProvider {
    fun track(exposure: Exposure)
}
```

The implementation of `track()` should track an event of type `$exposure` (a.k.a name) with two event properties, `flag_key` and `variant`, corresponding to the two fields on the `Exposure` object argument. Finally, the event tracked must eventually end up in Amplitude Analytics for the same project that the \[deployment\] used to [initialize](#initialize) the SDK client lives within, and for the same user that the SDK [fetched](#fetch) variants for.

To use your custom user provider, set the `exposureTrackingProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

#### Java

```java
ExperimentConfig config = ExperimentConfig.builder()
    .exposureTrackingProvider(new CustomExposureTrackingProvider())
    .build();
ExperimentClient experiment = Experiment.initialize(
    context, "<DEPLOYMENT_KEY>", config);
```

#### Kotlin

```kotlin
val config = ExperimentConfig.builder()
    .exposureTrackingProvider(CustomExposureTrackingProvider())
    .build()
val experiment = Experiment.initialize(context, "<DEPLOYMENT_KEY>", config)
```

## Bootstrapping

You may want to bootstrap the experiment client with an initial set of flags or variants when you get variants from an external source (for example, not from calling `fetch()` on the SDK client). Use cases include [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation) or integration testing on specific variants.

### Bootstrapping variants

To bootstrap the client with a predefined set of variants, set the flags and variants in the `initialVariants` [configuration](#configuration) object, then set the `source` to `Source.InitialVariants` so that the SDK client prefers the bootstrapped variants over any fetched & stored variants from before for the same flags.

```swift
let config = ExperimentConfigBuilder()
    .initialVariants(["<FLAG_KEY>": Variant("<VARIANT>")])
    .source(Source.InitialVariants)
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

#### Java

```java
ExperimentConfig config = ExperimentConfig.builder()
    .initialVariants(Map.of("<FLAG_KEY>", new Variant("<VARIANT>")))
    .source(Source.INITIAL_VARIANTS)
    .build();
ExperimentClient experiment = Experiment.initialize(
    context, "<DEPLOYMENT_KEY>", config);
```

#### Kotlin

```kotlin
val config = ExperimentConfig.builder()
    .initialVariants(mapOf("<FLAG_KEY>" to Variant("<VARIANT>")))
    .source(Source.INITIAL_VARIANTS)
    .build()
val experiment = Experiment.initialize(context, "<DEPLOYMENT_KEY>", config)
```

### Bootstrapping flag configurations

You may choose to bootstrap the SDK with an initial set of local evaluation flag configurations using the `initialFlags` configuration. Experiment evaluates these when you call [variant](#variant), unless you load an updated flag config or variant with [start](#start) or [fetch](#fetch).

To download initial flags, use the [evaluation flags API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api#flags-api)

```swift
let config = ExperimentConfigBuilder()
    .initialFlags("<FLAGS_JSON>")
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

#### Java

```java
ExperimentConfig config = ExperimentConfig.builder()
    .initialFlags("<FLAGS_JSON>")
    .build();
ExperimentClient experiment = Experiment.initialize(
    context, "<DEPLOYMENT_KEY>", config);
```

#### Kotlin

```kotlin
val config = ExperimentConfig.builder()
    .initialFlags("<FLAGS_JSON>")
    .build()
val experiment = Experiment.initialize(context, "<DEPLOYMENT_KEY>", config)
```

## Custom logging

Control log verbosity with the `logLevel` configuration, or implement the `LoggerProvider` interface to integrate your own logger.

### Log levels

- `LogLevel.DISABLE` - No logs.
- `LogLevel.ERROR` - Errors only (default).
- `LogLevel.WARN` - Errors and warnings.
- `LogLevel.INFO` - Errors, warnings, and info.
- `LogLevel.DEBUG` - Errors, warnings, info, and debug.
- `LogLevel.VERBOSE` - All messages including verbose details.

#### Kotlin

```kotlin
// Set log level to debug
val experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .logLevel(LogLevel.DEBUG)
        .build()
)


```

#### Java

```java
// Set log level to debug
ExperimentClient experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .logLevel(LogLevel.DEBUG)
        .build()
);

```

### Custom logger

Implement the `LoggerProvider` interface to use your own logging solution.

#### Kotlin

```kotlin
// Implement the LoggerProvider interface
class CustomLoggerProvider : LoggerProvider {
    override fun verbose(msg: String) {
        // Send verbose logs to your logging service
        myLoggingService.verbose(msg)
    }

    override fun debug(msg: String) {
        myLoggingService.debug(msg)
    }

    override fun info(msg: String) {
        myLoggingService.info(msg)
    }

    override fun warn(msg: String) {
        myLoggingService.warn(msg)
    }

    override fun error(msg: String) {
        myLoggingService.error(msg)
    }
}

// Initialize with custom logger
val experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .loggerProvider(CustomLoggerProvider())
        .logLevel(LogLevel.WARN)
        .build()
)
```

#### Java

```java
// Implement the LoggerProvider interface
class CustomLoggerProvider implements LoggerProvider {
    @Override
    public void verbose(String msg) {
        // Send verbose logs to your logging service
        myLoggingService.verbose(msg);
    }

    @Override
    public void debug(String msg) {
        myLoggingService.debug(msg);
    }

    @Override
    public void info(String msg) {
        myLoggingService.info(msg);
    }

    @Override
    public void warn(String msg) {
        myLoggingService.warn(msg);
    }

    @Override
    public void error(String msg) {
        myLoggingService.error(msg);
    }
}

// Initialize with custom logger
ExperimentClient experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .loggerProvider(new CustomLoggerProvider())
        .logLevel(LogLevel.WARN)
        .build()
);
```

### Debug flag (deprecated)

The `debug` configuration flag is deprecated. Use `logLevel` instead.

#### Kotlin

```kotlin
// Deprecated: Sets logLevel to Debug
val experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .debug(true)
        .build()
)

// Preferred: Use logLevel instead
val experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .logLevel(LogLevel.DEBUG)
        .build()
)
```

#### Java

```java
// Deprecated: Sets logLevel to Debug
ExperimentClient experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .debug(true)
        .build()
);

// Preferred: Use logLevel instead
ExperimentClient experiment = Experiment.initialize(
    context,
    "<DEPLOYMENT_KEY>",
    ExperimentConfig.builder()
        .logLevel(LogLevel.DEBUG)
        .build()
);
```

================================================================================
# Experiment iOS SDK
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-ios
================================================================================

# Experiment iOS SDK

- **Package:** `AmplitudeExperiment` ([cocoapods](https://cocoapods.org/pods/AmplitudeExperiment))
- **Latest version:** 1.20.2

Official documentation for Amplitude Experiment's Client-side iOS SDK implementation.

> **Info:** Unified SDK
>
> Install the [Unified SDK for Swift](https://amplitude.com/docs/sdks/analytics/ios/unified-sdk) to access the Experiment SDK along with other Amplitude products (Analytics, Session Replay). The Unified SDK provides a single entry point for all Amplitude features and simplifies the integration process by handling the initialization and configuration of all components.

## Install

> **Warning:** Import statement
>
> CocoaPods and Swift Package Manager/Carthage have different import statements.
>
> - CocoaPods: `import AmplitudeExperiment`
> - SPM/Carthage: `import Experiment`

#### CocoaPods

```ruby
pod 'AmplitudeExperiment', '~> <VERSION>'
```

#### Swift Package Manager

- Package URL: `https://github.com/amplitude/experiment-ios-client`

> **Tip:** Quick start
>
> The right way to initialize the Experiment SDK depends on whether you use an Amplitude SDK for analytics or a third party (for example, Segment).
>
> #### Amplitude
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```swift
> // (1) Initialize the experiment client with Amplitude Analytics
> let experiment = Experiment.initializeWithAmplitudeAnalytics(
>     apiKey: "DEPLOYMENT_KEY",
>     config: ExperimentConfigBuilder().build()
> )
>
> // (2) Fetch variants
> experiment.fetch(user: nil) { _, error in
>
>     // (3) Lookup a flag's variant
>     let variant = experiment.variant("FLAG_KEY")
>     if variant.value == "on" {
>         // Flag is on
>     } else {
>         // Flag is off
>     }
> }
> ```
>
> #### Third party
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants for a user](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```swift
>
> // (1) Initialize the experiment client and implement a
> //     custom exposure tracking provider.
> class ExposureTracker: ExposureTrackingProvider {
>    func track(exposure: Exposure) {
>        // TODO: Implement exposure tracking
>        // analytics.track(name: "$exposure", properties: [
>        //     "flag_key": exposure.flagKey,
>        //     "variant": exposure.variant,
>        //     "experiment_key": exposure.experimentKey
>        // ])
>    }
> }
> let experiment = Experiment.initialize(
>    apiKey: "DEPLOYMENT_KEY",
>    config: ExperimentConfigBuilder()
>        .exposureTrackingProvider(ExposureTracker())
>        .build()
> )
> // (2) Fetch variants with the user.
> let user = ExperimentUserBuilder()
>    .userId("user@company.com")
>    .deviceId("abcdefg")
>    .userProperty("premium", value: true)
>    .build()
> experiment.fetch(user: user) { _, error in
>
>    // (3) Lookup a flag's variant
>    let variant = experiment.variant("FLAG_KEY")
>    if variant.value == "on" {
>        // Flag is on
>    } else {
>        // Flag is off
>    }
> }
> ```

## Initialize

Initialize the SDK client in your application on startup. The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) argument you pass to the `apiKey` parameter must live within the same project that you are sending analytics events to.

#### Amplitude

```swift
func initializeWithAmplitudeAnalytics(
    apiKey: String,
    config: ExperimentConfig
) -> ExperimentClient
```

#### Third party

```swift
func initialize(apiKey: String, config: ExperimentConfig) -> ExperimentClient
```

- **`apiKey`**

  - **Requirement**: required
  - **Description**: The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) that authorizes fetch requests and determines which flags to evaluate for the user.
- **`config`**

  - **Requirement**: optional
  - **Description**: The client [configuration](#configuration) used to customize SDK client behavior.

The initializer returns a singleton instance, so subsequent initializations for the same instance name always return the initial instance. To create multiple instances, use the `instanceName` [configuration](#configuration).

#### Amplitude

```swift
let experiment = Experiment.initializeWithAmplitudeAnalytics(
    apiKey: "DEPLOYMENT_KEY",
    config: ExperimentConfigBuilder()
    .instanceName("myCustomInstance")  // case-sensitive name matching your Analytics instance
    .build()
)
```

#### Third party

```swift
// (1) Initialize the experiment client and implement a
//     custom exposure tracking provider.
class ExposureTracker: ExposureTrackingProvider {
    func track(exposure: Exposure) {
        // TODO: Implement exposure tracking
        // analytics.track(name: "$exposure", properties: [
        //     "flag_key": exposure.flagKey,
        //     "variant": exposure.variant,
        //     "experiment_key": exposure.experimentKey
        // ])
    }
}
let experiment = Experiment.initialize(
    apiKey: "DEPLOYMENT_KEY",
    config: ExperimentConfigBuilder()
        .exposureTrackingProvider(ExposureTracker())
        .instanceName("myCustomInstance")  // case-sensitive name matching your Analytics instance
        .build()
)
```

### Configuration

Configure the SDK client during initialization.

#### Configuration options

| Name | Description | Default Value |
| --- | --- | --- |
| `debug` | **Deprecated.** When `true`, sets `logLevel` to `.debug`. Use `logLevel` instead. | `false` |
| `logLevel` | The minimum log level to output. Options: `.off` (no logging), `.error` (errors only), `.warn` (errors and warnings), `.log` (errors, warnings, and logs), `.debug` (all messages including debug). Go to [Custom logging](#custom-logging). | `.warn` |
| `loggerProvider` | Custom logger implementation. Must implement the `CoreLogger` protocol. Go to [Custom logging](#custom-logging). | `DefaultLogger()` |
| `fallbackVariant` | The default variant to fall back if a variant for the provided key doesn't exist. | `{}` |
| `initialVariants` | An initial set of variants to access. This field helps bootstrap the client SDK with values rendered by the server using server-side rendering (SSR). | `{}` |
| `source` | Select whether `initialVariants` or variants stored on the device take precedence. Use `.InitialVariants` when bootstrapping variants. | `.LocalStorage` |
| `serverZone` | Select the Amplitude data center to get flags and variants from, `.US` or `.EU` | `.US` |
| `serverUrl` | The host to fetch variants from. | `https://api.lab.amplitude.com` |
| `flagsServerUrl` | The host to fetch local evaluation flags from. For hitting the EU data center, use `serverZone`. | `https://flag.lab.amplitude.com` |
| `fetchTimeoutMillis` | The timeout for fetching variants in milliseconds. | `10000` |
| `retryFetchOnFailure` | Whether to retry variant fetches in the background if the request doesn't succeed. | `true` |
| `automaticExposureTracking` | If true, calling [`variant()`](#variant) tracks an exposure event through the configured `exposureTrackingProvider`. Without an exposure tracking provider, this configuration option does nothing. | `true` |
| `fetchOnStart` | If true or nil, always [fetch](#fetch) remote evaluation variants on [start](#start). If false, never fetch on start. | `true` |
| `pollOnStart` | Poll for local evaluation flag configuration updates on [start](#start). Set `flagConfigPollingIntervalMillis` to control the interval. | `true` |
| `flagConfigPollingIntervalMillis` | Interval, in milliseconds, between local evaluation flag configuration polls. The SDK enforces a minimum of `60000` milliseconds. | `300000` |
| `automaticFetchOnAmplitudeIdentityChange` | Only matters if you use the `initializeWithAmplitudeAnalytics` initialization function to integrate with the Amplitude Analytics SDK. If `true` any change to the user ID, device ID or user properties from analytics triggers the experiment SDK to fetch variants and update its cache. | `false` |
| `userProvider` | An interface used to provide the user object to `fetch()` when called. | `null` |
| `exposureTrackingProvider` | Implement and configure this interface to track exposure events through the experiment SDK, either automatically or explicitly. | `null` |
| `customRequestHeaders` | A closure that returns headers to attach to variant-fetch and flag-configuration requests. | `{ [:] }` |
| `instanceName` | Custom instance name for experiment SDK instance. **The value of this field is case-sensitive.** | `"$default_instance"` |
| `initialFlags` | A JSON string representing an initial set of flag configurations to use for local evaluation. | `nil` |

> **Info:** Eu data center
>
> If you're using Amplitude's EU data center, configure the `serverZone` option on initialization to `.EU`.

### Integrations

If you use either Amplitude or Segment Analytics SDKs to track events into Amplitude, you'll want to set up an integration on initialization. Integrations automatically implement [provider](#providers) interfaces to enable a more streamlined developer experience by making it easier to **manage user identity** and **track exposures events**.

#### Amplitude integration

The Amplitude Experiment SDK is set up to integrate with the Amplitude Analytics SDK.

```swift
Amplitude.instance().initializeApiKey("API_KEY")
let experiment = Experiment.initializeWithAmplitudeAnalytics( 
    apiKey: "DEPLOYMENT_KEY",
    config: ExperimentConfigBuilder().build()
)
```

If you are using a custom instance name for analytics, set the same value in the `instanceName` [configuration option](#configuration) in the experiment SDK.

The integration initializer configures implementations of the [user provider](#user-provider) and [exposure tracking provider](#exposure-tracking-provider) interfaces to pull user data from the Amplitude Analytics SDK and track exposure events.

**Supported Versions**

All generally available versions of the next-generation [Amplitude Analytics Swift](https://amplitude.com/docs/sdks/analytics/ios/ios-swift-sdk) SDK support this integration.

| Analytics SDK Version | Experiment SDK Version |
| --- | --- |
| `8.8.0+` | `1.6.0+` |

#### Segment integration

Experiment's integration with Segment Analytics must be configured manually. The Experiment SDK must then be configured on initialization with an instance of the exposure tracking provider. Make sure this happens _after_ the analytics SDK has been loaded an initialized.

```swift
class SegmentExposureTrackingProvider : ExposureTrackingProvider {
    private let analytics: Analytics
    init(analytics: Analytics) {
        self.analytics = analytics
    }
    func track(exposure: Exposure) {
        analytics.track("$exposure", properties: [
            "flag_key": exposure.flagKey,
            "variant": exposure.variant,
            "experiment_key": exposure.experimentKey
        ])
    }
}
```

The Experiment SDK must then be configured on initialization with an instance of the exposure tracking provider.

```swift
let analytics: Analytics = /* Initialize Segment Analytics */
let config = ExperimentConfigBuilder()
    .exposureTrackingProvider(SegmentExposureTrackingProvider(analytics))
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

When [fetching variants](#fetch), pass the segment anonymous ID and user ID for the device ID and user ID, respectively.

```swift
let userId = SEGState.sharedInstance().userInfo.userId
let deviceId = SEGState.sharedInstance().userInfo.anonymousId

let user = ExperimentUserBuilder()
    .userId(userId)
    .deviceId(deviceId)
    .build()
experiment.fetch(user: user, completion: nil)
```

## Fetch

Fetches variants for a [user](https://amplitude.com/docs/feature-experiment/data-model#users) and stores the results in the client for fast access. The function [remote evaluates](https://amplitude.com/docs/feature-experiment/remote-evaluation) the user for flags associated with the deployment used to initialize the SDK client.

```swift
func fetch(user: ExperimentUser?, options: FetchOptions?, completion: ((ExperimentClient, Error?) -> Void)?)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to evaluate. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. |
| `options` | optional | Explicit flag keys to fetch. |
| `completion` | optional | Callback when the variant fetch (success or failure). If the fetch request fails, the error is returned in the second parameter of this callback. |

Amplitude Experiment recommends calling `fetch()` during application start up so that the user gets the most up-to-date variants for the application session. Furthermore, you'll need to wait for the fetch request to return a result before rendering the user experience to avoid the interface "flickering".

```swift
let user = ExperimentUserBuilder()
    .userId("user@company.com")
    .userProperty("premium", value: true)
    .build()
experiment.fetch(user: user) { experiment, error in
    // Do something...
}
```

If you're using an [integration](#integrations) or a custom [user provider](#user-provider) then you can fetch without inputting the user.

```swift
experiment.fetch(user: nil, completion: nil)
```

> **Tip:** Fetch when user identity changes
>
> If you want the most up-to-date variants for the user, it's recommended that you call `fetch()` whenever the user state changes in a meaningful way. For example, if the user logs in and receives a user ID, or has a user property set which may affect flag or experiment targeting rules.
>
> In the case of **user properties**, Amplitude recommends passing new user properties explicitly to `fetch()` instead of relying on user enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation). Remote user-property sync through a separate system has no timing guarantees for `fetch()`, which can create a race condition.

If `fetch()` times out (default 10 seconds) or fails for any reason, the SDK client returns and retries in the background with back-off. You may configure the timeout or disable retries in the [configuration options](#configuration) during SDK client initialization.

#### Account-level bucketing and analysis (v1.10.0+)

If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users. Reach out to your representative to gain access to this beta feature.

Groups must either be included in the user sent with the fetch request (recommended), or identified with the user through a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or through [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).

```swift
let user = ExperimentUserBuilder()
    .userId(userId)
    .deviceId(deviceId)
    .group("org name", "Amplitude")
    .build()
experiment.fetch(user: user, completion: nil)
```

To pass freeform group properties, refer to this example:

```swift
let user = ExperimentUserBuilder()
    .userId(userId)
    .deviceId(deviceId)
    .groupProperty("org name", "Amplitude", "plan", "enterprise")
    .build()
experiment.fetch(user: user, completion: nil)
```

## Start

> **Info:** Fetch vs start
>
> Use `start` if you're using client-side [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation). If you're only using [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), call [fetch](#fetch) instead of `start`.

Start the SDK by getting flag configurations from the server and fetching remote evaluation variants for the user. The SDK is ready when the completion callback runs.

```swift
func start(_ user: ExperimentUser? = nil, completion: ((Error?) -> Void)? = nil)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to fetch variants. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. Also sets the user in the SDK for reuse. |
| `completion` | optional | The completion block, called when the SDK has finished starting. If fetch is called on start, the completion block is called after the fetch response is received. |

Call `start()` when your application initializes, after user information is available to evaluate or [fetch](#fetch) variants. The completion block runs after the SDK loads local evaluation flag configurations and fetches remote evaluation variants.

Configure the behavior of `start()` by setting `fetchOnStart` in the SDK configuration on initialization to improve performance based on the needs of your application.

- If your application never relies on remote evaluation, set `fetchOnStart` to `false` to avoid increased startup latency caused by remote evaluation.
- If your application relies on remote evaluation, but not right at startup, you may set `fetchOnStart` to `false` and call `fetch()` separately.

#### Amplitude

```swift
experiment.start() { error in
    // SDK Started
}
```

#### Third party

```swift
let user = ExperimentUserBuilder()
    .userId("user@company.com")
    .deviceId("abcdefg")
    .userProperty("premium", value: true)
    .build()
experiment.start(user) { error in
    // SDK Started
}
```

## Variant

Access a [variant](https://amplitude.com/docs/feature-experiment/data-model#variants) for a [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) from the SDK client's local store.

> **Info:** Automatic exposure tracking
>
> When you use an [integration](#integrations) or set a custom [exposure tracking provider](#exposure-tracking-provider), `variant()` automatically tracks an exposure event through the tracking provider. To disable this functionality, [configure](#configuration) `automaticExposureTracking` to be `false`, and track exposures manually using [`exposure()`](#exposure).

```swift
func variant(_ key: String, fallback: Variant? = nil) -> Variant
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) to access the variant for. |
| `fallback` | optional | The value to return if no variant was found for the given `flagKey`. |

When determining which variant a user has been bucketed into, you'll want to compare the variant `value` to a well-known string.

```swift
let variant = experiment.variant("<FLAG_KEY>")
if variant.value == "on" {
    // Flag is on
} else {
    // Flag is off
}
```

> **Info:** Access the variant's payload
>
> A variant may also be configured with a dynamic [payload](https://amplitude.com/docs/feature-experiment/data-model#variants) of arbitrary data. Access the `payload` field from the variant object after checking the variant's `value`.
>
> The `payload` in iOS is of type `Any?`, so cast the payload to the expected type to retrieve the value. For example, if the payload is `{"key":"value"}`:
>
> ```swift
> let variant = client.variant("<FLAG_KEY>")
> if variant.value == "on" {
>     if let payload = variant.payload as? [String:String] {
>         let value = payload["key"]
>     }
> }
> ```

A `null` variant `value` means that the user hasn't been bucketed into a variant. You may use the built in **fallback** parameter to provide a variant to return if the store doesn't contain a variant for the given flag key.

```swift
let variant = experiment.variant("<FLAG_KEY>", fallback: Variant("control"))
if variant.value == "control" {
    // Control
} else if variant.value == "treatment" {
    // Treatment
}
```

## All

Access all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) stored by the SDK client.

```swift
func all() -> [String:Variant]
```

## Clear

Clear all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) in the cache and storage.

```swift
func clear()
```

You can call `clear` after user logout to clear the variants in cache and storage.

```swift
experiment.clear()
```

## Current user

Set the user that the client uses for later `fetch()` and `start()` calls, or read the client's current user.

```swift
experiment.setUser(ExperimentUserBuilder()
    .userId("user@company.com")
    .deviceId("abcdefg")
    .build()
)

let currentUser = experiment.getUser()
```

Passing a non-empty user directly to `fetch()` also sets that user on the client.

## Stop

Stop local-evaluation flag polling started by [`start`](#start). This doesn't clear variants already stored by the client.

```swift
experiment.stop()
```

## Exposure

Manually track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for the current variant of the given flag key through configured [integration](#integrations) or custom [exposure tracking provider](#exposure-tracking-provider). Generally used in conjunction with setting the `automaticExposureTracking` [configuration](#configuration) optional to `false`.

```swift
func exposure(key: String)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) variant to track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for. |

```swift
let variant = experiment.variant("<FLAG_KEY>")

// Do other things...

experiment.exposure(key: "<FLAG_KEY>")
if variant.value == "control" {
    // Control
} else if variant.value == "treatment" {
    // Treatment
}
```

## Assignment tracking

Control whether subsequent remote-evaluation requests are marked for assignment tracking. This setting is stored for the configured Experiment instance.

```swift
experiment.setTracksAssignment(false) // Send no-track on later fetch requests.
```

## Providers

> **Tip:** Integrations
>
> If you use Amplitude or Segment analytics SDKs along side the Experiment Client SDK, Amplitude recommends you use an [integration](#integrations) instead of implementing custom providers.

Provider implementations enable a more streamlined developer experience by making it easier to manage user identity and track exposures events.

### User provider

The SDK client uses the user provider to access the most up-to-date user information only when needed (for example, when [`fetch()`](#fetch) is called). The user provider is optional, but helps if you have a user information store already set up in your application. With a user provider, you don't need to manage two separate user info stores in parallel. Separate stores can create divergent user state if the application user store is updated and experiment isn't (or vice versa).

```swift
protocol ExperimentUserProvider {
    func getUser() -> ExperimentUser
}
```

To use your custom user provider, set the `userProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```swift
let config = ExperimentConfigBuilder()
    .userProvider(CustomUserProvider())
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

### Exposure tracking provider

Amplitude highly recommends implementing an exposure tracking provider. [Exposure tracking](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) increases the accuracy and reliability of experiment results and improves visibility into which flags and experiments a user is exposed to.

```swift
protocol ExposureTrackingProvider {
    func track(exposure: Exposure)
}
```

The implementation of `track()` should track an event of type `$exposure` (a.k.a name) with two event properties, `flag_key` and `variant`, corresponding to the two fields on the `Exposure` object argument. Finally, the event tracked must eventually end up in Amplitude Analytics for the same project that the \[deployment\] used to [initialize](#initialize) the SDK client lives within, and for the same user that variants were [fetched](#fetch) for.

To use your custom user provider, set the `exposureTrackingProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```swift
ExperimentConfig config = ExperimentConfigBuilder()
    .exposureTrackingProvider(CustomExposureTrackingProvider(analytics))
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

## Bootstrapping

You may want to bootstrap the experiment client with an initial set of flags or variants when variants come from an external source (for example, not from calling `fetch()` on the SDK client). Use cases include [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation) or integration testing on specific variants.

### Bootstrapping variants

To bootstrap the client with a predefined set of variants, set the flags and variants in the `initialVariants` [configuration](#configuration) object, then set the `source` to `Source.InitialVariants` so that the SDK client prefers the bootstrapped variants over any previously fetched & stored variants for the same flags.

```swift
let config = ExperimentConfigBuilder()
    .initialVariants(["<FLAG_KEY>": Variant("<VARIANT>")])
    .source(Source.InitialVariants)
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

### Bootstrapping flag configurations

You may choose to bootstrap the SDK with an initial set of local evaluation flag configurations using the `initialFlags` configuration. The SDK evaluates these flag configurations when [variant](#variant) is called, unless an updated flag config or variant is loaded with [start](#start) or [fetch](#fetch).

To download initial flags, use the [evaluation flags API](https://amplitude.com/docs/apis/experiment/experiment-evaluation-api#flags-api)

```swift
let config = ExperimentConfigBuilder()
    .initialFlags("<FLAGS_JSON>")
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

## Custom logging

Control log verbosity with the `logLevel` configuration, or implement the `CoreLogger` protocol to integrate your own logging solution.

### Log levels

- `.off` - No logging
- `.error` - Errors only
- `.warn` - Errors and warnings (default)
- `.log` - Errors, warnings, and logs
- `.debug` - All messages including debug

```swift
// Only log errors
let config = ExperimentConfigBuilder()
    .logLevel(.error)
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

### Custom logger

Implement the `CoreLogger` protocol to use your own logging solution:

```swift
// Implement the CoreLogger protocol
class CustomLogger: CoreLogger {
    func error(message: String) {
        // Send error logs to your logging service
        myLoggingService.error(message)
    }

    func warn(message: String) {
        myLoggingService.warn(message)
    }

    func log(message: String) {
        myLoggingService.log(message)
    }

    func debug(message: String) {
        myLoggingService.debug(message)
    }
}

// Initialize with custom logger
let config = ExperimentConfigBuilder()
    .loggerProvider(CustomLogger())
    .logLevel(.warn)
    .build()
let experiment = Experiment.initialize(apiKey: "<DEPLOYMENT_KEY>", config: config)
```

================================================================================
# Experiment React Native SDK
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-react-native
================================================================================

# Experiment React Native SDK

- **Package:** `@amplitude/experiment-react-native-client` ([npm](https://www.npmjs.com/package/@amplitude/experiment-react-native-client))
- **Latest version:** 1.8.0

Official documentation for Amplitude Experiment's Client-side React Native SDK.

## Install

Install the Experiment JavaScript Client SDK. This library depends on `@react-native-async-storage/async-storage` which you also need to install.

> **Info:** Web and Expo support
>
> Use this SDK for react-native apps built for web or built using [Expo](https://expo.dev/) (Expo Go not yet supported).

#### npm

```bash
npm install @amplitude/experiment-react-native-client
npm install @react-native-async-storage/async-storage
```

#### yarn

```bash
yarn add @amplitude/experiment-react-native-client
yarn add @react-native-async-storage/async-storage
```

#### expo

```bash
expo install @amplitude/experiment-react-native-client
expo install @react-native-async-storage/async-storage
```

You need to install the native modules to run the SDK on iOS.

```bash
cd ios
pod install
```

> **Tip:** Quick start
>
> The right way to initialize the Experiment SDK depends on whether you use an Amplitude SDK for analytics or a third party (for example, Segment).
>
> #### Amplitude
>
> If you use `@amplitude/analytics-react-native`, go to [Amplitude Analytics plugin](#amplitude-analytics-plugin) for a plugin-based setup instead.
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```js
> import { Experiment } from '@amplitude/experiment-react-native-client';
>
> // (1) Initialize the experiment client with Amplitude Analytics.
> const experiment = Experiment.initializeWithAmplitudeAnalytics(
>     'DEPLOYMENT_KEY'
> );
>
> // (2) Fetch variants and await the promise result.
> await experiment.fetch();
>
> // (3) Lookup a flag's variant.
> const variant = experiment.variant('FLAG_KEY');
> if (variant.value === 'on') {
>     // Flag is on
> } else {
>     // Flag is off
> }
> ```
>
> #### Third party
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants for a user](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```js
> import { Experiment } from '@amplitude/experiment-react-native-client';
>
> // (1) Initialize the experiment client with Amplitude Analytics and
> // implement an exposure tracking provider.
> const experiment = Experiment.initialize(
>    'DEPLOYMENT_KEY',
>    {
>        exposureTrackingProvider: {
>            track: (exposure) => {
>                // TODO: Implement exposure tracking
>                // analytics.track('$exposure', exposure)
>            }
>        }
>    }
> );
>
> // (2) Fetch variants with the user and await the promise result.
> const user = {
>    user_id: 'user@company.com',
>    device_id: 'abcdefg',
>    user_properties: {
>        premium: true,
>    },
> }
> await experiment.start(user);
>
> // (3) Lookup a flag's variant.
> const variant = experiment.variant('FLAG_KEY');
> if (variant.value === 'on') {
>    // Flag is on
> } else {
>    // Flag is off
> }
> ```

## Amplitude Analytics plugin

If you use `@amplitude/analytics-react-native`, install `@amplitude/plugin-experiment-react-native` to integrate Experiment as a plugin. The plugin reads the current Analytics identity and sends exposure events through Analytics.

The Experiment React Native plugin is in early access and may change before general availability. Install the exact beta version. Don't use a caret range:

#### npm

```bash
npm install @amplitude/plugin-experiment-react-native@1.0.0-beta.1
```

#### yarn

```bash
yarn add @amplitude/plugin-experiment-react-native@1.0.0-beta.1
```

#### expo

```bash
expo install @amplitude/plugin-experiment-react-native@1.0.0-beta.1
```

Add the plugin, initialize Analytics, then start Experiment and access variants:

```ts
import { add, init } from '@amplitude/analytics-react-native';
import { experimentPlugin } from '@amplitude/plugin-experiment-react-native';

const experiment = experimentPlugin({
  // Required unless Experiment and Analytics share the same project.
  deploymentKey: 'DEPLOYMENT_KEY',
});

await add(experiment).promise;
await init('AMPLITUDE_API_KEY').promise;

await experiment.experiment?.start();
const variant = experiment.experiment?.variant('FLAG_KEY');
```

The plugin works as follows:

- `experimentPlugin()` calls `Experiment.initializeWithAmplitudeAnalytics()` inside its `setup()` function. Don't call `Experiment.initializeWithAmplitudeAnalytics()` separately.
- Set `deploymentKey` to the Experiment deployment key. If Experiment and Analytics share the same Amplitude project, the plugin uses the Analytics API key by default.
- You can register the Experiment plugin before or after Analytics `init()`. This example registers it before `init()` to match the package quick start.
- `start()` fetches variants by default unless `fetchOnStart` is `false`. Before `start()` completes, `variant()` can return a cached assignment, an initial variant, or a fallback. Await `start()` when your app requires the startup fetch to complete.
- Set `automaticFetchOnAmplitudeIdentityChange` to `true` to fetch after Analytics identity changes, including login or logout. The default is `false`.

## Initialize

Initialize the SDK client in your application on startup. The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) argument you pass to the `apiKey` parameter must live within the same project that you are sending analytics events to.

#### Amplitude

```js
initializeWithAmplitudeAnalytics(apiKey: string, config?: ExperimentConfig): ExperimentClient
```

#### Third party

```js
initialize(apiKey: string, config?: ExperimentConfig): ExperimentClient
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `apiKey` | required | The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) that authorizes fetch requests and determines which flags to evaluate for the user. |
| `config` | optional | The client [configuration](#configuration) used to customize SDK client behavior. |

The initializer returns a singleton instance, so subsequent initializations for the same instance name always return the initial instance. To create multiple instances, use the `instanceName` [configuration](#configuration).

#### Amplitude

```js
import { Experiment } from '@amplitude/experiment-js-client';

const experiment = initializeWithAmplitudeAnalytics('DEPLOYMENT_KEY');
```

> **Note:** Instance name
>
> If you're using a custom instance name for analytics, you need to set the same value in the `instanceName` [configuration option](#configuration) in the experiment SDK, or visa versa.

#### Third party

```js
import { Experiment } from '@amplitude/experiment-js-client';

const experiment = Experiment.initialize(
    'DEPLOYMENT_KEY',
    {
        exposureTrackingProvider: {
            track: (exposure) => {
                // TODO: Implement exposure tracking
                // analytics.track('$exposure', exposure)
            }
        }
    }
);
```

### Configuration

SDK client configuration occurs during initialization.

| Name | Description | Default Value |
| --- | --- | --- |
| `debug` | **Deprecated.** When `true`, sets `logLevel` to `Debug`. Use `logLevel` instead. | `false` |
| `logLevel` | The minimum log level to output. The SDK ignores messages below this level. Options: `Disable`, `Error`, `Warn`, `Info`, `Debug`, `Verbose`. Go to [Custom logging](#custom-logging) for details. | `LogLevel.Error` |
| `loggerProvider` | Custom logger implementation. Must implement the `Logger` interface. Go to [Custom logging](#custom-logging) for details. | `null` (uses default ConsoleLogger) |
| `fallbackVariant` | The default variant to fall back if a variant for the provided key doesn't exist. | `{}` |
| `initialVariants` | An initial set of variants to access. This field helps bootstrap the client SDK with values rendered by the server using server-side rendering (SSR). | `{}` |
| `source` | The primary source of variants. Set the value to `Source.InitialVariants` and configured `initialVariants` to bootstrap the SDK for SSR or testing purposes. | `Source.LocalStorage` |
| `serverZone` | Select the Amplitude data center to get flags and variants from, `us` or `eu`. | `us` |
| `serverUrl` | The host to fetch remote evaluation variants from. For hitting the EU data center, use `serverZone`. | `https://api.lab.amplitude.com` |
| `flagsServerUrl` | The host to fetch local evaluation flags from. For hitting the EU data center, use `serverZone`. | `https://flag.lab.amplitude.com` |
| `fetchTimeoutMillis` | The timeout for fetching variants in milliseconds. | `10000` |
| `retryFetchOnFailure` | Whether to retry variant fetches in the background if the request doesn't succeed. | `true` |
| `automaticExposureTracking` | If true, calling [`variant()`](#variant) tracks an exposure event through the configured `exposureTrackingProvider`. If no exposure tracking provider is set, this configuration option does nothing. | `true` |
| `fetchOnStart` | If true or undefined, always [fetch](#fetch) remote evaluation variants on [start](#start). If false, never fetch on start. | `true` |
| `pollOnStart` | Poll for local evaluation flag configuration updates every minute on [start](#start). | `true` |
| `automaticFetchOnAmplitudeIdentityChange` | Only matters if you use the `initializeWithAmplitudeAnalytics` initialization function to integrate with the Amplitude Analytics SDK. If `true` any change to the user ID, device ID or user properties from analytics triggers the experiment SDK to fetch variants and update its cache. | `false` |
| `userProvider` | An interface used to provide the user object to `fetch()` when called. | `null` |
| `exposureTrackingProvider` | Implement and configure this interface to track exposure events through the experiment SDK, either automatically or explicitly. | `null` |
| `instanceName` | Custom instance name for experiment SDK instance. **The value of this field is case-sensitive.** | `null` |
| `initialFlags` | A JSON string representing an initial set of flag configurations to use for local evaluation. | `undefined` |
| `httpClient` | (Advanced) Use your own HTTP client implementation to handle network requests made by the SDK. | Default HTTP client |

> **Info:** EU data center
>
> If you're using Amplitude's EU data center, configure the `serverZone` option on initialization to `eu`.

### Integrations

If you use either Amplitude or Segment Analytics SDKs to track events into Amplitude, Amplitude recommends that you set up an integration on initialization. Integrations implement [provider](#providers) interfaces to enable a more streamlined developer experience by making it easier to **manage user identity** and **track exposures events**.

#### Amplitude integration

The Amplitude Experiment SDK is set up to integrate with the Amplitude Analytics SDK.

```js
import * as amplitude from '@amplitude/analytics-browser';
import { Experiment } from '@amplitude/experiment-js-client';

amplitude.init('API_KEY');
const experiment = Experiment.initializeWithAmplitudeAnalytics('DEPLOYMENT_KEY'); 
```

When you use the integration initializer, it configures implementations of the [user provider](#user-provider) and [exposure tracking provider](#exposure-tracking-provider) interfaces to pull user data from the Amplitude Analytics SDK and track exposure events.

#### Segment integration

When you use Segment as your analytics provider, configure the integration with Experiment manually on initialization with an instance of the exposure tracking provider. Ensure this happens after the analytics SDK loads and initializes.

```js
analytics.ready(() => {
    const experiment =  Experiment.initialize('DEPLOYMENT_KEY', {
        exposureTrackingProvider: {
            track: (exposure) => {
                analytics.track('$exposure', exposure)
            }
        }
    });
});
```

When [fetching variants](#fetch), pass the segment anonymous ID and user ID for the device ID and user ID, respectively.

```js
await experiment.fetch({
    user_id: analytics.user().id(),
    device_id: analytics.user().analyticsId(),
});
```

## Fetch

Fetches variants for a [user](https://amplitude.com/docs/feature-experiment/data-model#users) and stores the results in the client for fast access. The function [remote evaluates](https://amplitude.com/docs/feature-experiment/remote-evaluation) the user for flags associated with the deployment used to initialize the SDK client.

```js
fetch(user?: ExperimentUser): Promise<Client>
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to evaluate. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. |

Amplitude Experiment recommends calling `fetch()` during application start up so that the user gets the most up-to-date variants for the application session. Wait for the fetch request to return a result before rendering the user experience to avoid the interface "flickering".

```js
const user = {
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
};
await experiment.fetch(user);
```

If you're using an [integration](#integrations) or a custom [user provider](#user-provider) then you can fetch without inputting the user.

```js
await experiment.fetch();
```

> **Tip:** Fetch when user identity changes
>
> If you want the most up-to-date variants for the user, it's recommended that you call `fetch()` whenever the user state changes in a meaningful way. For example, if the user logs in and receives a user ID, or has a user property set which may affect flag or experiment targeting rules.
>
> In the case of **user properties**, Amplitude recommends passing new user properties explicitly to `fetch()` instead of relying on user enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation). Remote user-property sync through a separate system has no timing guarantees for `fetch()`, which can create a race condition.

If `fetch()` times out (default 10 seconds) or fails for any reason, the SDK client returns and retries in the background with back-off. You may configure the timeout or disable retries in the [configuration options](#configuration) during SDK client initialization.

#### Account-level bucketing and analysis (v1.1.0+)

If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users. Reach out to your representative to gain access to this beta feature.

Include groups in the user sent with the fetch request (recommended), or identify groups with the user through a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or through [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).

```js
await fetch({
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
    groups: {'org name': ['Amplitude']}
});
```

To pass freeform group properties, refer to this example:

```js
await fetch({
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        'premium': true,
    },
    group_properties: {'org name': ['Amplitude']}
});
```

## Start

> **Info:** Fetch vs start
>
> Use `start` if you're using client-side [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation). If you're only using [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), call [fetch](#fetch) instead of `start`.

Start the Experiment SDK to get flag configurations from the server and fetch remote evaluation variants for the user. The SDK is ready when the returned promise resolves.

```js
start(user?: ExperimentUser): Promise<void>
```

| Parameter | Requirement | Description                                                                                                                                                                                                                                                                                                                                                                        | | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | `user`    | optional    | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to fetch variants. This user information merges with user information from any [integrations](#integrations) through the [user provider](#user-provider), and prefers properties passed explicitly to `fetch()` over provided properties. Also sets the user in the SDK for reuse. | `undefined` |

Call `start()` when your application is initializing, after user information is available to evaluate or [fetch](#fetch) variants. The promise resolves after loading local evaluation flag configurations and fetching remote evaluation variants.

Set `fetchOnStart` in the SDK configuration to set the behavior of `start()` to improve the performance of your application.

- If your application never relies on remote evaluation, set `fetchOnStart` to `false` to avoid increased startup latency caused by remote evaluation.
- If your application relies on remote evaluation, but not right at startup, you may set `fetchOnStart` to `false` and call `fetch()` and await the promise separately.

#### Amplitude

```js
await experiment.start();
```

#### Third party

```js
const user = {
    user_id: 'user@company.com',
    device_id: 'abcdefg',
    user_properties: {
        premium: true
    }
};
await experiment.start(user);
```

If the client is bootstrapped with `initialVariants` or `initialFlags` and doesn't need to await the results of `start()`, call and await `cacheReady()` to ensure async storage has loaded variants and flags.

## Variant

Access a [variant](https://amplitude.com/docs/feature-experiment/data-model#variants) for a [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) from the SDK client's local store.

> **Info:** Automatic exposure tracking
>
> When you use an [integration](#integrations) or set a custom [exposure tracking provider](#exposure-tracking-provider), `variant()` automatically tracks an exposure event through the tracking provider. To disable this functionality, [configure](#configuration) `automaticExposureTracking` to be `false`, and track exposures manually using [`exposure()`](#exposure).

```js
variant(key: string, fallback?: string | Variant): Variant
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) to access the variant for. |
| `fallback` | optional | The value to return if no variant was found for the given `flagKey`. |

When determining which variant a user has been bucketed into, you'll want to compare the variant `value` to a well-known string.

```js
const variant = experiment.variant('<FLAG_KEY>');
if (variant.value === 'on') {
    // Flag is on
} else {
    // Flag is off
}
```

> **Info:** Access a variant's payload
>
> A variant may also be configured with a dynamic [payload](https://amplitude.com/docs/feature-experiment/data-model#variants) of arbitrary data. Access the `payload` field from the variant object after checking the variant's `value`.
>
> ```js
> const variant = experiment.variant('<FLAG_KEY>');
> if (variant.value === 'on') {
>     const payload = variant.payload;
> }
> ```

A `null` variant `value` means that the user hasn't been bucketed into a variant. You may use the built in **fallback** parameter to provide a variant to return if the store doesn't contain a variant for the given flag key.

```js
const variant = experiment.variant('<FLAG_KEY>', { value: 'control' });
if (variant.value === 'control') {
    // Control
} else if (variant.value === 'treatment') {
    // Treatment
}
```

## All

Access all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) stored by the SDK client.

```js
all(): Variants
```

## Clear

Clear all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) in the cache and storage.

```js
clear(): void
```

You can call `clear` after user logout to clear the variants in cache and storage.

```js
experiment.clear();
```

## Exposure

Manually track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for the current variant of the given flag key through configured [integration](#integrations) or custom [exposure tracking provider](#exposure-tracking-provider). Generally used in conjunction with setting the `automaticExposureTracking` [configuration](#configuration) optional to `false`.

```js
exposure(key: string): void
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `key` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) variant to track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for. |

```js
const variant = experiment.variant('<FLAG_KEY>');

// Do other things...

experiment.exposure('<FLAG_KEY>');
if (variant === 'control') {
    // Control
} else if (variant === 'treatment') {
    // Treatment
}
```

## Providers

> **Tip:** Integrations
>
> If you use Amplitude or Segment analytics SDKs along side the Experiment Client SDK, Amplitude recommends you use an [integration](#integrations) instead of implementing custom providers.

Provider implementations enable a more streamlined developer experience by making it easier to manage user identity and track exposures events.

### User provider

The SDK client uses the user provider to access the most up-to-date user information only when needed (for example, when [`fetch()`](#fetch) is called). The user provider is optional, but helps if you have a user information store already set up in your application. With a user provider, you don't need to manage two separate user info stores in parallel. Separate stores can create divergent user state if the application user store is updated and experiment isn't (or vice versa).

```js
interface ExperimentUserProvider {
  getUser(): ExperimentUser;
}
```

To use your custom user provider, set the `userProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```js
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    userProvider: new CustomUserProvider(),
});
```

### Exposure tracking provider

Amplitude highly recommends implementing an exposure tracking provider. [Exposure tracking](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) increases the accuracy and reliability of experiment results and improves visibility into which flags and experiments a user is exposed to.

```js
export interface ExposureTrackingProvider {
  track(exposure: Exposure): void;
}
```

The implementation of `track()` should track an event of type `$exposure` (a.k.a name) with two event properties, `flag_key` and `variant`, corresponding to the two fields on the `Exposure` object argument. Finally, the event tracked must eventually end up in Amplitude Analytics for the same project that the \[deployment\] used to [initialize](#initialize) the SDK client lives within, and for the same user that variants were [fetched](#fetch) for.

To use your custom user provider, set the `exposureTrackingProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```js
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    exposureTrackingProvider: new CustomExposureTrackingProvider(),
});
```

## Bootstrapping

You may want to bootstrap the experiment client with an initial set of flags and variants when variants come from an external source (for example, not from calling `fetch()` on the SDK client). Use cases include [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation), [server-side rendering](https://amplitude.com/docs/feature-experiment/advanced-techniques/server-side-rendering), or integration testing on specific variants.

To bootstrap the client, set the flags and variants in the `initialVariants` [configuration](#configuration) object, then set the `source` to `Source.InitialVariants` so that the SDK client prefers the bootstrapped variants over any previously fetched & stored variants for the same flags.

```js
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    initialVariants: { /* Flags and variants */ },
    source: Source.InitialVariants,
});
```

## HTTP client

You can provide a custom HTTP client implementation to handle network requests made by the SDK. This is useful for environments with specific networking requirements or when you need to customize request handling.

```js
export interface SimpleResponse {
  status: number;
  body: string;
}

export interface HttpClient {
  request(
    requestUrl: string,
    method: string,
    headers: Record<string, string>,
    data: string,
    timeoutMillis?: number,
  ): Promise<SimpleResponse>;
}
```

To use your custom HTTP client, set the `httpClient` [configuration](#configuration) option with an instance of your implementation on SDK initialization.

```js
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
    httpClient: new CustomHttpClient(),
});
```

## Custom logging

Control log verbosity with the `logLevel` configuration or implement the `Logger` interface to use your own logging solution.

### Log levels

- `LogLevel.Disable` - No logging.
- `LogLevel.Error` - Errors only (default).
- `LogLevel.Warn` - Errors and warnings.
- `LogLevel.Info` - Errors, warnings, and informational messages.
- `LogLevel.Debug` - Errors, warnings, info, and debug messages.
- `LogLevel.Verbose` - All messages including verbose details.

```js
import { Experiment, LogLevel } from '@amplitude/experiment-react-native-client';

// Only log errors
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Error
});

// Log errors and warnings
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Warn
});

// Log everything (verbose)
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Verbose
});
```

### Custom logger

Implement the `Logger` interface to use your own logging solution.

```js
import { Experiment, Logger, LogLevel } from '@amplitude/experiment-react-native-client';

// Implement the Logger interface
class CustomLogger implements Logger {
  error(message, ...optionalParams) {
    // Send errors to your logging service
    myLoggingService.error(message, ...optionalParams);
  }

  warn(message, ...optionalParams) {
    myLoggingService.warn(message, ...optionalParams);
  }

  info(message, ...optionalParams) {
    myLoggingService.info(message, ...optionalParams);
  }

  debug(message, ...optionalParams) {
    myLoggingService.debug(message, ...optionalParams);
  }

  verbose(message, ...optionalParams) {
    myLoggingService.verbose(message, ...optionalParams);
  }
}

// Initialize with custom logger
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  loggerProvider: new CustomLogger(),
  logLevel: LogLevel.Warn
});
```

### Debug flag (deprecated)

The `debug` configuration flag is deprecated. Use `logLevel` instead.

```js
// Deprecated: Sets logLevel to Debug
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  debug: true
});

// Preferred: Use logLevel instead
const experiment = Experiment.initialize('<DEPLOYMENT_KEY>', {
  logLevel: LogLevel.Debug
});
```

================================================================================
# Experiment Flutter SDK
URL: https://amplitude.com/docs/sdks/experiment-sdks/experiment-flutter
================================================================================

# Experiment Flutter SDK

- **Package:** `amplitude_experiment` ([pubdev](https://pub.dev/packages/amplitude_experiment))
- **Latest version:** 1.0.0

Official documentation for Amplitude Experiment's Client-side Flutter SDK.

## Install

Add the `amplitude_experiment` package to your `pubspec.yaml`:

```yaml
dependencies:
  amplitude_experiment: ^1.0.0
```

Then run:

```bash
flutter pub get
```

### Web installation (optional)

The Flutter Experiment SDK uses Dart's JavaScript interoperability to enable the [Experiment JavaScript SDK](https://amplitude.com/docs/sdks/experiment-sdks/experiment-javascript) for Flutter Web. This requires that you make the SDK available within the global JavaScript scope. Add the following script tag to `web/index.html` in your Flutter project:

```html
<script src="https://unpkg.com/@amplitude/experiment-js-client@1.21.3/dist/experiment.umd.js"></script>
```

> **Info:** Platform support
>
> The SDK supports Android, iOS, and Web platforms.

> **Tip:** Quick start
>
> The right way to initialize the Experiment SDK depends on whether you use an Amplitude SDK for analytics or a third party (for example, Segment).
>
> #### Amplitude
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```dart
> import 'package:amplitude_flutter/amplitude.dart';
> import 'package:amplitude_flutter/configuration.dart';
> import 'package:amplitude_experiment/amplitude_experiment.dart';
>
> // (1) Initialize the Amplitude Analytics SDK.
> final amplitude = Amplitude(Configuration(apiKey: 'API_KEY'));
> await amplitude.isBuilt;
>
> // (2) Initialize the experiment client with Amplitude Analytics.
> final experiment = await Experiment.initializeWithAmplitude(
>   'DEPLOYMENT_KEY',
>   ExperimentConfig(),
> );
>
> // (3) Fetch variants and await the result.
> await experiment.fetch();
>
> // (4) Look up a flag's variant.
> final variant = await experiment.variant('FLAG_KEY');
> if (variant.value == 'on') {
>   // Flag is on
> } else {
>   // Flag is off
> }
> ```
>
> #### Third party
>
> 1. [Initialize the experiment client](#initialize)
> 2. [Fetch variants for a user](#fetch)
> 3. [Access a flag's variant](#variant)
>
> ```dart
> import 'package:amplitude_experiment/amplitude_experiment.dart';
>
> // (1) Initialize the experiment client and implement an
> // exposure tracking provider.
> final experiment = await Experiment.initialize(
>   'DEPLOYMENT_KEY',
>   ExperimentConfig(
>     trackingProvider: MyExposureTracker(),
>   ),
> );
>
> // (2) Fetch variants with the user and await the result.
> final user = ExperimentUser(
>   userId: 'user@company.com',
>   deviceId: 'abcdefg',
>   userProperties: {'premium': true},
> );
> await experiment.fetch(user);
>
> // (3) Look up a flag's variant.
> final variant = await experiment.variant('FLAG_KEY');
> if (variant.value == 'on') {
>   // Flag is on
> } else {
>   // Flag is off
> }
> ```

## Initialize

Initialize the SDK client in your application on startup. The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) argument you pass to the `apiKey` parameter must live within the same project that you are sending analytics events to.

#### Amplitude

```dart
static Future<ExperimentClient> initializeWithAmplitude(String apiKey, ExperimentConfig config)
```

#### Third party

```dart
static Future<ExperimentClient> initialize(String apiKey, ExperimentConfig config)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `apiKey` | required | The [deployment key](https://amplitude.com/docs/feature-experiment/data-model#deployments) that authorizes fetch requests and determines which flags to evaluate for the user. |
| `config` | required | The client [configuration](#configuration) used to customize SDK client behavior. |

The initializer returns a singleton instance, so subsequent initializations for the same instance name always return the initial instance. To create multiple instances, use the `instanceName` [configuration](#configuration).

#### Amplitude

```dart
import 'package:amplitude_flutter/amplitude.dart';
import 'package:amplitude_flutter/configuration.dart';
import 'package:amplitude_experiment/amplitude_experiment.dart';

final amplitude = Amplitude(Configuration(apiKey: 'API_KEY'));
await amplitude.isBuilt;

final experiment = await Experiment.initializeWithAmplitude(
  'DEPLOYMENT_KEY',
  ExperimentConfig(),
);
```

> **Note:** Instance name
>
> If you're using a custom instance name for analytics, you need to set the same value in the `instanceName` [configuration option](#configuration) in the experiment SDK, or vice versa. Both SDKs default to `$default_instance`.

#### Third party

```dart
import 'package:amplitude_experiment/amplitude_experiment.dart';

final experiment = await Experiment.initialize(
  'DEPLOYMENT_KEY',
  ExperimentConfig(
    trackingProvider: MyExposureTracker(),
  ),
);
```

### Configuration

SDK client configuration occurs during initialization.

- **`logLevel`**

  - **Description**: The minimum log level to output. Options: `LogLevel.none`, `LogLevel.error`, `LogLevel.warn`, `LogLevel.info`, `LogLevel.debug`, `LogLevel.verbose`.
  - **Default Value**: `LogLevel.warn`
- **`fallbackVariant`**

  - **Description**: The default variant to fall back to if a variant for the provided key doesn't exist.
  - **Default Value**: `null`
- **`initialVariants`**

  - **Description**: An initial set of variants to access. This field helps bootstrap the client SDK with values rendered by the server using server-side rendering (SSR).
  - **Default Value**: `{}`
- **`source`**

  - **Description**: The primary source of variants. Set the value to `Source.initialVariants` and configure `initialVariants` to bootstrap the SDK for SSR or testing purposes.
  - **Default Value**: `Source.localStorage`
- **`serverZone`**

  - **Description**: Select the Amplitude data center to get flags and variants from, `us` or `eu`.
  - **Default Value**: `ServerZone.us`
- **`serverUrl`**

  - **Description**: The host to fetch remote evaluation variants from. For hitting the EU data center, use `serverZone`.
  - **Default Value**: `https://api.lab.amplitude.com`
- **`flagsServerUrl`**

  - **Description**: The host to fetch local evaluation flags from. For hitting the EU data center, use `serverZone`.
  - **Default Value**: `https://flag.lab.amplitude.com`
- **`fetchTimeoutMillis`**

  - **Description**: The timeout for fetching variants in milliseconds.
  - **Default Value**: `10000`
- **`retryFetchOnFailure`**

  - **Description**: Whether to retry variant fetches in the background if the request doesn't succeed.
  - **Default Value**: `true`
- **`automaticExposureTracking`**

  - **Description**: If true, calling [`variant()`](#variant) tracks an exposure event through the configured `trackingProvider`. If no tracking provider is set, this configuration option does nothing.
  - **Default Value**: `true`
- **`fetchOnStart`**

  - **Description**: If true, always [fetch](#fetch) remote evaluation variants on [start](#start). If false, never fetch on start.
  - **Default Value**: `true`
- **`pollOnStart`**

  - **Description**: Poll for local evaluation flag configuration updates on [start](#start). Set the polling interval with `flagConfigPollingIntervalMillis`.
  - **Default Value**: `true`
- **`flagConfigPollingIntervalMillis`**

  - **Description**: The interval in milliseconds between flag configuration polls when you enable `pollOnStart`. The SDK raises any value below 60000 to 60000.
  - **Default Value**: `300000`
- **`automaticFetchOnAmplitudeIdentityChange`**

  - **Description**: Only matters if you use `initializeWithAmplitude`. If `true`, any change to the user ID, device ID, or user properties from analytics triggers the experiment SDK to fetch variants and update its cache.
  - **Default Value**: `false`
- **`userProvider`**

  - **Description**: An interface that provides the user object to `fetch()` when called. Go to [User provider](#user-provider).
  - **Default Value**: `null`
- **`trackingProvider`**

  - **Description**: Implement and configure this interface to track exposure events through the experiment SDK, either automatically or explicitly. Go to [Exposure tracking provider](#exposure-tracking-provider).
  - **Default Value**: `null`
- **`instanceName`**

  - **Description**: Custom instance name for experiment SDK instance. **The value of this field is case-sensitive.**
  - **Default Value**: `$default_instance`
- **`initialFlags`**

  - **Description**: A JSON string representing an initial set of flag configurations to use for local evaluation.
  - **Default Value**: `null`

> **Info:** EU data center
>
> If you're using Amplitude's EU data center, configure the `serverZone` option on initialization to `ServerZone.eu`.

### Integrations

If you use Amplitude Analytics SDK to track events into Amplitude, Amplitude recommends that you set up an integration on initialization. Integrations implement [provider](#providers) interfaces to enable a more streamlined developer experience by making it easier to **manage user identity** and **track exposure events**.

#### Amplitude integration

The Amplitude Experiment SDK is set up to integrate with the Amplitude Analytics SDK.

```dart
import 'package:amplitude_flutter/amplitude.dart';
import 'package:amplitude_flutter/configuration.dart';
import 'package:amplitude_experiment/amplitude_experiment.dart';

final amplitude = Amplitude(Configuration(apiKey: 'API_KEY'));
await amplitude.isBuilt;

final experiment = await Experiment.initializeWithAmplitude(
  'DEPLOYMENT_KEY',
  ExperimentConfig(),
);
```

When you use the integration initializer, it configures implementations of the [user provider](#user-provider) and [exposure tracking provider](#exposure-tracking-provider) interfaces to pull user data from the Amplitude Analytics SDK and track exposure events.

## Fetch

Fetches variants for a [user](https://amplitude.com/docs/feature-experiment/data-model#users) and stores the results in the client for fast access. The function [remote evaluates](https://amplitude.com/docs/feature-experiment/remote-evaluation) the user for flags associated with the deployment used to initialize the SDK client.

```dart
Future<void> fetch([ExperimentUser? user, FetchOptions? options])
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to evaluate. The SDK merges this user information with user information provided from [integrations](#integrations) through the [user provider](#user-provider), preferring properties passed explicitly to `fetch()` over provided properties. |
| `options` | optional | Fetch options such as a list of specific flag keys to fetch. |

Amplitude Experiment recommends calling `fetch()` during application start up so that the user gets the most up-to-date variants for the application session. Wait for the fetch request to return a result before rendering the user experience to avoid the interface "flickering".

```dart
final user = ExperimentUser(
  userId: 'user@company.com',
  deviceId: 'abcdefg',
  userProperties: {'premium': true},
);
await experiment.fetch(user);
```

If you're using an [integration](#integrations) or a custom [user provider](#user-provider) then you can fetch without inputting the user.

```dart
await experiment.fetch();
```

> **Tip:** Fetch when user identity changes
>
> If you want the most up-to-date variants for the user, it's recommended that you call `fetch()` whenever the user state changes in a meaningful way. For example, if the user logs in and receives a user ID, or has a user property set which may affect flag or experiment targeting rules.
>
> In the case of **user properties**, Amplitude recommends passing new user properties explicitly to `fetch()` instead of relying on user enrichment before [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation). Remote user-property sync through a separate system has no timing guarantees for `fetch()`, which can create a race condition.

If `fetch()` times out (default 10 seconds) or fails for any reason, the SDK client returns and retries in the background with back-off. You may configure the timeout or disable retries in the [configuration options](#configuration) during SDK client initialization.

#### Account-level bucketing and analysis

If your organization has purchased the [Accounts add-on](https://amplitude.com/docs/analytics/account-level-reporting) you may perform bucketing and analysis on groups rather than users. Reach out to your representative to gain access to this beta feature.

Include groups in the user sent with the fetch request (recommended), or identify groups with the user through a group identify call from the [Group Identify API](https://amplitude.com/docs/apis/analytics/group-identify) or through [`setGroup()` from an analytics SDK](https://amplitude.com/docs/sdks/analytics/browser/browser-sdk-2#user-groups).

```dart
await experiment.fetch(
  ExperimentUser(
    userId: 'user@company.com',
    deviceId: 'abcdefg',
    userProperties: {'premium': true},
    groups: {'org name': ['Amplitude']},
  ),
);
```

## Start

> **Info:** Fetch vs start
>
> Use `start` if you're using client-side [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation). If you're only using [remote evaluation](https://amplitude.com/docs/feature-experiment/remote-evaluation), call [fetch](#fetch) instead of `start`.

Start the Experiment SDK to get flag configurations from the server and fetch remote evaluation variants for the user. The SDK is ready when the returned future completes.

```dart
Future<void> start(ExperimentUser? user)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `user` | optional | Explicit [user](https://amplitude.com/docs/feature-experiment/data-model#users) information to pass with the request to fetch variants. This user information merges with user information from any [integrations](#integrations) through the [user provider](#user-provider), and prefers properties passed explicitly to `fetch()` over provided properties. Also sets the user in the SDK for reuse. |

Call `start()` when your application is initializing, after user information is available to evaluate or [fetch](#fetch) variants. The future completes after loading local evaluation flag configurations and fetching remote evaluation variants.

Set `fetchOnStart` in the SDK configuration to control the behavior of `start()` to improve the performance of your application.

- If your application never relies on remote evaluation, set `fetchOnStart` to `false` to avoid increased startup latency caused by remote evaluation.
- If your application relies on remote evaluation, but not right at startup, you may set `fetchOnStart` to `false` and call `fetch()` and await the future separately.

#### Amplitude

```dart
await experiment.start(null);
```

#### Third party

```dart
final user = ExperimentUser(
  userId: 'user@company.com',
  deviceId: 'abcdefg',
  userProperties: {'premium': true},
);
await experiment.start(user);
```

## Variant

Access a [variant](https://amplitude.com/docs/feature-experiment/data-model#variants) for a [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) from the SDK client's local store.

> **Info:** Automatic exposure tracking
>
> When you use an [integration](#integrations) or set a custom [exposure tracking provider](#exposure-tracking-provider), `variant()` automatically tracks an exposure event through the tracking provider. To disable this functionality, [configure](#configuration) `automaticExposureTracking` to be `false`, and track exposures manually using [`exposure()`](#exposure).

```dart
Future<Variant> variant(String flagKey, [Variant? fallbackVariant])
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `flagKey` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) to access the variant for. |
| `fallbackVariant` | optional | The value to return if no variant was found for the given `flagKey`. |

When determining which variant a user has been bucketed into, compare the variant `value` to a well-known string.

```dart
final variant = await experiment.variant('FLAG_KEY');
if (variant.value == 'on') {
  // Flag is on
} else {
  // Flag is off
}
```

> **Info:** Access a variant's payload
>
> A variant may also be configured with a dynamic [payload](https://amplitude.com/docs/feature-experiment/data-model#variants) of arbitrary data. Access the `payload` field from the variant object after checking the variant's `value`.
>
> ```dart
> final variant = await experiment.variant('FLAG_KEY');
> if (variant.value == 'on') {
>   final payload = variant.payload;
> }
> ```

A `null` variant `value` means that the user hasn't been bucketed into a variant. You may use the built in **fallback** parameter to provide a variant to return if the store doesn't contain a variant for the given flag key.

```dart
final variant = await experiment.variant(
  'FLAG_KEY',
  Variant(value: 'control'),
);
if (variant.value == 'control') {
  // Control
} else if (variant.value == 'treatment') {
  // Treatment
}
```

## All

Access all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) stored by the SDK client.

```dart
Future<Map<String, Variant>> all()
```

## Clear

Clear all [variants](https://amplitude.com/docs/feature-experiment/data-model#variants) in the cache and storage.

```dart
Future<void> clear()
```

You can call `clear` after user logout to clear the variants in cache and storage.

```dart
await experiment.clear();
```

## Exposure

Manually track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for the current variant of the given flag key through configured [integration](#integrations) or custom [exposure tracking provider](#exposure-tracking-provider). Generally used in conjunction with setting the `automaticExposureTracking` [configuration](#configuration) option to `false`.

```dart
Future<void> exposure(String flagKey)
```

| Parameter | Requirement | Description |
| --- | --- | --- |
| `flagKey` | required | The **flag key** to identify the [flag or experiment](https://amplitude.com/docs/feature-experiment/data-model#flags-and-experiments) variant to track an [exposure event](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) for. |

```dart
final variant = await experiment.variant('FLAG_KEY');

// Do other things...

await experiment.exposure('FLAG_KEY');
if (variant.value == 'control') {
  // Control
} else if (variant.value == 'treatment') {
  // Treatment
}
```

## Providers

> **Tip:** Integrations
>
> If you use the Amplitude Analytics SDK alongside the Experiment Client SDK, Amplitude recommends you use an [integration](#integrations) instead of implementing custom providers.

Provider implementations enable a more streamlined developer experience by making it easier to manage user identity and track exposure events.

### User provider

The SDK client uses the user provider to access the most up-to-date user information only when needed (for example, when [`fetch()`](#fetch) is called). The user provider is optional, but helps if you have a user information store already set up in your application. With a user provider, you don't need to manage two separate user info stores in parallel. Separate stores can create divergent user state if the application user store is updated and experiment isn't (or vice versa).

```dart
abstract interface class UserProvider {
  ExperimentUser getUser();
}
```

To use your custom user provider, set the `userProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```dart
class CustomUserProvider implements UserProvider {
  @override
  ExperimentUser getUser() {
    // Return the current user from your app's user store
    return ExperimentUser(userId: 'user@company.com');
  }
}

final experiment = await Experiment.initialize(
  'DEPLOYMENT_KEY',
  ExperimentConfig(
    userProvider: CustomUserProvider(),
  ),
);
```

### Exposure tracking provider

Amplitude highly recommends implementing an exposure tracking provider. [Exposure tracking](https://amplitude.com/docs/feature-experiment/under-the-hood/event-tracking#exposure-events) increases the accuracy and reliability of experiment results and improves visibility into which flags and experiments a user is exposed to.

```dart
abstract interface class ExposureTrackingProvider {
  void track(Exposure exposure);
}
```

The implementation of `track()` should track an event of type `$exposure` (a.k.a name) with two event properties, `flag_key` and `variant`, corresponding to the two fields on the `Exposure` object argument. Finally, the event tracked must eventually end up in Amplitude Analytics for the same project that the deployment used to [initialize](#initialize) the SDK client lives within, and for the same user that variants were [fetched](#fetch) for.

To use your custom exposure tracking provider, set the `trackingProvider` [configuration](#configuration) option with an instance of your custom implementation on SDK initialization.

```dart
class CustomExposureTracker implements ExposureTrackingProvider {
  @override
  void track(Exposure exposure) {
    // Track the exposure event to your analytics provider
    analytics.track('\$exposure', {
      'flag_key': exposure.flagKey,
      'variant': exposure.variant,
    });
  }
}

final experiment = await Experiment.initialize(
  'DEPLOYMENT_KEY',
  ExperimentConfig(
    trackingProvider: CustomExposureTracker(),
  ),
);
```

## Bootstrapping

You may want to bootstrap the experiment client with an initial set of flags and variants when variants come from an external source (for example, not from calling `fetch()` on the SDK client). Use cases include [local evaluation](https://amplitude.com/docs/feature-experiment/local-evaluation), [server-side rendering](https://amplitude.com/docs/feature-experiment/advanced-techniques/server-side-rendering), or integration testing on specific variants.

### Initial variants

To bootstrap the client, set the flags and variants in the `initialVariants` [configuration](#configuration) option, then set the `source` to `Source.initialVariants` so that the SDK client prefers the bootstrapped variants over any previously fetched and stored variants for the same flags.

```dart
final experiment = await Experiment.initialize(
  'DEPLOYMENT_KEY',
  ExperimentConfig(
    initialVariants: {
      'flag-key-1': Variant(value: 'on'),
      'flag-key-2': Variant(value: 'treatment'),
    },
    source: Source.initialVariants,
  ),
);
```

### Initial flags

To bootstrap the client with a set of flag configurations for local evaluation, set the `initialFlags` [configuration](#configuration) option with a JSON string of flag configurations.

```dart
final experiment = await Experiment.initialize(
  'DEPLOYMENT_KEY',
  ExperimentConfig(
    initialFlags: '{"flag-key-1": ...}',
  ),
);
```