Troubleshoot your experiment

Unexpected issues sometimes surface when you create and roll out an experiment or flag. This page describes common issues and their workarounds.

This article assumes you know how to set up and run an experiment in Amplitude Experiment. If you need a refresher, review the Help Center articles on an overview of Amplitude Experiment and configuring your experiment before proceeding.

Experiment troubleshooting checklist

Use these questions as a quick checklist for experiment discrepancies. If you can answer "yes" to a question, that question is unlikely to be the cause of your issue.

Do the device ID and user ID align?

The device ID and user ID you use for experiments and flags must match the device ID and user ID in Amplitude Analytics.

Is your flag enabled?

A flag must be enabled in a deployment to appear in that deployment.

Does your variant use a name other than "off"?

Don't name a variant "Off." Amplitude reserves this value for users in the "OFF/FALLBACK" bucket.

A fallback is the default variant (usually "control") that Amplitude serves when Experiment can't assign a user to a treatment group.

Do your assignment event and exposure event point to different events?

The assignment event and the exposure event can use the same event, but they represent different concepts. Each plays a different role in your experiment:

• The assignment event immediately precedes a user's assignment to a variant.
• The exposure event is the event the user must trigger to receive the variant.

Amplitude assigns users at the beginning of their session, and users might not trigger the exposure event until later in the flow.

Confirm the correct events map to each.

Go to Feature Assignment Events for more information.

Are your old properties synced to new ones?

If you run an experiment on a single platform (for example, iOS), users with a different platform value (for example, Android) might still appear. This happens when users access your product on multiple platforms and their most recent event comes from a platform other than the one your experiment targets. The assignment event fires automatically and doesn't contain non-Experiment properties such as platform.

Other experiment discrepancies

Some experiment issues fall outside your direct control. This section highlights those scenarios.

Targeting on user properties is delayed

Amplitude Experiment updates cohort targeting each hour. Targeting on properties sent directly to Experiment applies in real time. Targeting on user properties that Amplitude stores can lag by up to one hour because of the CDN.

A user didn't receive their assigned variant

Amplitude Experiment generates data on the server side and stores assignments as user properties. A user might not receive their assigned variant because of timeouts, network errors, or ad blockers.

A user can also receive one variant one day and a different variant the next. Because bucketing in Amplitude Experiment relies on user ID and device ID, users can land in a second bucket when you rely on device ID alone and the user's device ID changes. This happens when the user opens incognito mode or clears their cache and cookies after their last visit.

A user doesn't log any events past the assignment event

A user can trigger the assignment event without logging any other active events:

• The user's ad blockers or analytics blockers interfere with analytics application events. Feature Experiment requests come from backend servers and usually pass through, but blockers can affect Web Experiments.
• A user opens the app, which sends a request to Amplitude's backend servers and triggers the assignment event. The user then leaves the app before performing any other actions that trigger an analytics event.
• Amplitude's servers receive assignment requests directly from your backend instead of from the end user. Check if you instrumented experiments in offline campaigns such as group push notifications or marketing emails.