Set up and run mutually exclusive experiments

When you run several experiments at the same time, you may want to keep users who receive one experiment from seeing a second, related experiment. These experiments might solve the same problem in different ways, and you don't want to confuse users who see both. The interaction effect can also distort your results.

Experiment lets you set two or more experiments as mutually exclusive. Mutually exclusive experiments don't share users: users who receive experiment A don't receive experiment B, and the reverse also holds.

To learn more about the underlying implementation, refer to flag dependencies.

When to use mutually exclusive experiments

Use mutually exclusive experiments in these situations:

• Simultaneous experiments that occur in the same area of your product and share the same goal.
• Simultaneous experiments that occur in the same funnel and share the same goal.

You can also run these experiments one after the other instead of simultaneously.

Create a mutual exclusion group

Note these guidelines when you use mutual exclusion groups:

• Distribute traffic evenly between your slots. A slot is the percentage of the group that you want to receive an experiment. You can add multiple slots to your group to distribute traffic across the group.
• Don't add a running experiment to a mutual exclusion group. Adding a running experiment can severely compromise data integrity by removing users from the active experiments. Add experiments to a mutual exclusion group before they start running.
• Don't remove a running experiment from a mutual exclusion group. Removing a running experiment can compromise data integrity by exposing users to the other experiments in the group.
• Don't delete a mutual exclusion group with running experiments, for the same reason. Delete the mutual exclusion group after all experiments in the group conclude.

To create a mutual exclusion group

1. In the Experiment feature, go to Experiments > MutEx and Holdouts tab.
2. If your project has no groups, click Add a new mutual exclusion group to create a new group. If you already have groups, click Create Group, then select Mutual Exclusion Group.
3. Enter a name and description for your group, then enter delivery settings such as evaluation mode, bucketing, and key.
4. Create the number of slots you want and assign experiments to each slot. Experiments in different slots are mutually exclusive. You can add a maximum of 20 slots to a mutual exclusion group.

5. (Optional) Specify individuals or cohorts to add to your mutual exclusion group from the Individuals or Cohorts tabs. This step is helpful when you want to test an experiment in a mutual exclusion group to confirm that a specific user gets assigned to a specific experiment. Don't add the same users or cohorts to more than one slot, because the first slot determines inclusion.
6. Set the traffic allocation percentages for each experiment. By default, Amplitude distributes traffic evenly between them, but you can edit these percentages manually.
7. Click Add Group.

Advanced use cases

• To increase traffic allocation to an experiment, change the slot percentage when you create the group. If the group already exists, assign a single experiment to multiple slots within the same group.

• Adding an experiment to multiple mutual exclusion groups further limits the experiment's traffic, because Amplitude evaluates each user for each mutual exclusion group the user belongs to.

  For example, consider the following two mutual exclusion groups:

  • Mutual exclusion group 1 has experiment A in slot 1 and experiment B in slot 2, where each experiment receives 50% of the traffic.
  • Mutual exclusion group 2 has experiment A in slot 1 and experiment C in slot 2, where each experiment receives 50% of the traffic.

  In this case, experiment A receives 0.5 * 0.5 = 0.25, or 25% of the total traffic.

  Instead of adding an experiment to multiple mutual exclusion groups, create one group that includes all the relevant experiments. In this example, that group contains experiments A, B, and C.

• Adding an experiment to both a holdout group and a mutual exclusion group further limits the experiment's traffic, because Amplitude evaluates each user for both groups. For example, consider the following holdout group and mutual exclusion group:

  • The holdout group contains experiment A, with a holdout percentage of 5%.
  • The mutual exclusion group contains experiment A in slot 1 and experiment B in slot 2, where each experiment receives 50% of the traffic.

  In this case, experiment A receives 0.95 * 0.5 = 0.475, or 47.5% of the total traffic.

To learn more, refer to working with holdout groups in Amplitude Experiment.

Common questions

Is it best to create a mutual-exclusion group before launching an experiment?

Yes. Avoid configuration changes that affect targeting after the experiment starts. If you add an active experiment to a mutual-exclusion group, Amplitude Experiment may reassign users.

How do the slot % allocations affect user targeting?

When you create mutual-exclusion groups, you set an allocation percentage: the probability that Amplitude assigns a user to an experiment. For example, an allocation percentage of 25% means users have a 25% chance of joining the experiment. Allocation percentages can total 100% or less. If less than 100%, unused traffic receives the "off" treatment of the experiment.

When does Amplitude Experiment apply mutual exclusion?

Amplitude Experiment follows a specific order of operations when assigning users to experiments:

individual user qualification → mutual exclusion → sticky bucketing → target segment

Because Amplitude Experiment evaluates individual user qualification before mutual-exclusion groups, users targeted under Individual Users can see multiple experiments even when those experiments share a mutual-exclusion group. The same rule applies to individual inclusions for cohorts: Amplitude Experiment also applies them before mutual-exclusion groups.

Why does a user see more than one experiment when mutual exclusion is configured?

The same user can see more than one experiment, even when the experiments use mutual exclusion. Possible reasons:

• The order of operations described above (individual user qualification runs before mutual exclusion).
• How Amplitude tracks unique users. For example, a user might anonymously use more than one device before logging in. Until Amplitude identifies and merges the user into the existing ID, Amplitude treats the session as a different user available for assignment.
• Variant jumping, where a user sees two or more variants for a single flag or experiment.