Holdout groups: Advanced use cases

This article covers advanced use cases for holdout groups in Amplitude Experiment.

Case 1: Streamline multiple experiments and holdout groups

Adding an experiment to multiple holdout groups can limit the experiment's traffic. Amplitude evaluates each user against every holdout group the user belongs to.

For example, consider two holdout groups:

• Holdout group 1 contains experiment A and experiment B, with a holdout percentage of 5%.
• Holdout group 2 contains experiment A and experiment C, also with a holdout percentage of 5%.

Because experiment A belongs to both holdout groups, it receives the majority of the total traffic:

0.95 * 0.95 = 0.9025 (90.25%)

Instead of adding an experiment to multiple holdout groups, create a single group that contains all relevant experiments. A single group distributes traffic more evenly across experiments.

In the example above, create one holdout group that contains all three experiments (A, B, and C).

Case 2: Handle experiments with holdout groups and mutual exclusion

Adding an experiment to both a holdout group and a mutual exclusion group can further limit the experiment's traffic. Amplitude evaluates each user against both groups.

For example, consider the following holdout group and mutual exclusion group:

• The holdout group has a holdout percentage of 5% and contains experiment A.
• The mutual exclusion group directs half the traffic to experiment A in slot 1 and the other half to experiment B in slot 2.

In this scenario, experiment A receives about half of the total traffic:

0.95 * 0.5 = 0.475 (47.5%)

You can use holdout groups with mutual exclusion, but plan for the potential traffic limits as you roll out your experiments.

For more information, refer to mutually exclusive experiments.