Variant jumping

Variant jumping occurs when a user receives two or more variants for a single flag or experiment. Variant jumping above a certain threshold can raise concerns about the trustworthiness of an analysis.

Some types of variant jumping are normal: common and usually explainable. Other types are harder to track down.

Debug variant jumping

The best way to debug variant jumping is to identify a user who jumped variants and analyze their user timeline. If you use remote evaluation, check the Assignment event to identify assignment or exposure discrepancies.

To find a user who jumped variants, follow these steps:

1. Navigate to the diagnostics card of your experiment.
2. Find the variant jumping charts, and click Open in Chart.
3. Click the bar for users who jumped variants and click View User Streams.

Show only assignment and exposure events in the user timeline for easier analysis.

When you debug a user timeline, note the following:

• Did you introduce any targeting changes while your flag or experiment was active? Could the timing of the change have affected the variant assigned to this user? Click the version number to check the version history.
• Which bucketing key did the flag or experiment use? Does the value for this property change between assignments or exposures for this user?
• Does the user have missing exposures or assignment events? If so, the missing event might have been sent for a different user.
• Is an assignment missing a user property it should have? If so, double-check the server upload timestamp of the assignment event and compare it to surrounding active events. Events sent from the client can arrive at Amplitude after the assignment event, even if the client order is different.

Normal variant jumping

Normal variant jumping can occur due to:

• Targeting changes: Someone changed targeting rules while your experiment was running.
• Anonymous identity merging: Anonymous users, bucketed by Amplitude ID, can receive different variants until Amplitude resolves them through a matching user ID.

Targeting changes

The following actions can cause a user to jump variants:

• Adding or removing a variant.
• Changing variant distribution weights.
• Targeting dynamic cohorts.
• Changing the bucketing key.
• Updating mutual exclusion.

Anonymous identity merging

The way Amplitude handles anonymous users can lead to variant jumping. Amplitude merges anonymous IDs with the correct existing user IDs (if one exists) as soon as Amplitude has enough information to do so. This merging can happen if a user uses your app on different devices without logging in, or if the device ID regenerates upon logout.

Learn more about Amplitude's identity resolution and merging users.

To identify this type of variant jumping, find the assignment event where the user jumped between variants. Then compare the Amplitude ID for both events. If the Amplitude ID differs between the events, anonymous identity merging is the likely cause.

To avoid this type of variant jumping, bucket by user ID if you target only logged-in users who have user IDs. Alternately, bucket by device ID if you target only anonymous users (for example, a sign-up experiment).

Inclusion list

Imagine you have some user IDs in the inclusion list. You call fetch() and pass the user ID into the call. The call returns the control experience, per the inclusion list. The next time you call fetch(), you don't pass in the user ID. The user no longer meets the inclusion list criteria, so Amplitude hashes the bucketing key, and the user can receive a different variant instead. The same outcome can happen if you include device IDs.

In the following example, the user has a user ID. The user matches the inclusion list and receives the signin-up-new_design experience.

In this example, no user ID exists. The user doesn't match the inclusion list, falls into the "all other users" segment, and receives the signin-up-original-view experience instead.

Abnormal variant jumping

Instances of abnormal variant jumping don't fit any of the normal explanations. Abnormal variant jumping can be difficult to track down. The most frequent cause is identity mismatch: the user identity differs between assignment and exposure tracking. Identity mismatch almost always comes from an implementation inconsistency.

The following examples aren't exhaustive, but they illustrate common identity issues in Amplitude Experiment.

Multiple logged in accounts on a single device

Consider this timeline for a person with multiple user accounts (U1 and U2), for your app on a single device.

1. Open the app as user U1 and fetch variants. U1 receives treatment for experiment-1.
2. Expose U1 to experiment-1 variant treatment.
3. Log out of U1 and into U2, fetching variants asynchronously on login.
4. Before the fetch for U2 resolves, U2 sees the exposure to experiment-1 variant treatment.
5. The fetch for U2 resolves. U2 receives control for experiment-1.
6. Expose U2 to experiment-1 variant control.

In this case, user U2 jumped variants, from treatment to control, after seeing U1's stored variant. To avoid this case, either wait for the fetch to resolve before you render the user experience, or call the SDK's clear() method on logout to clear all stored variants from the SDK. Clearing variants wipes the SDK's variant storage and prevents the user from seeing cached variants. Clearing variants doesn't protect the user from viewing a fallback experience before the fetch request resolves.

If you keep a consistent device ID across logins, you can check for this type of variant jumping. Search for different users that share the same device ID.

Inconsistent identity input between assignment and exposure

In Amplitude, the user ID and device ID properties identify your user and resolve their Amplitude ID. If the device ID or user ID used to fetch and evaluate assignments differs from the device ID and user ID used to track the exposure event, you can see variant jumping, SRMs, and inconsistent or unexpected bucketing behavior.

For example, you might send events through a proxy or CDP that masks IDs before sending to Amplitude. In this case, the identity used to fetch variants differs from the identity included in the exposure events.

Another common cause is a simple implementation error. For example, the following cases have caused variant jumping:

• Additional characters in the ID. Note the extra quotes around the actual identity:
  • 15a4f7e9-db4e-4c57-82c7-e57a2995803a
  • "15a4f7e9-db4e-4c57-82c7-e57a2995803a"
• Inconsistent capitalization, especially with UUIDs:
  • 15a4f7e9-db4e-4c57-82c7-e57a2995803a
  • 15A4F7E9-DB4E-4C57-82C7-E57A2995803A

Remove users who variant jumped from experiment analysis

As you analyze results, be careful when you remove data, because removing data can introduce bias. It's better to understand the cause of variant jumping and fix any implementation bugs to prevent recurrence in future experiments. If removing users who jumped variants is the best course of action, use the Filter card on the Experiment Analysis tab. If the All exposed users segment is enabled by default, click it and select Experiment Segments > Exclude users who variant jumped.