The cumulative exposures graph: Divergent lines

This article reviews divergent lines with similar slopes versus divergent lines with varying slopes. Divergent lines start from a common point but slowly spread apart from each other.

Divergent lines with similar slopes

Sometimes your cumulative exposure graph shows divergent lines with similar slopes. This can happen when your experiment starts before all variants are ready.

In this example, the two variants began receiving traffic on two separate days, February 23 and February 28, resulting in a pair of staggered lines on the graph.

Ideally, you shouldn't begin an experiment until all variants are ready to receive traffic. Adding a new variant after the experiment is underway can present a misleading picture of the results, since all variants weren't subject to the same conditions for the same length of time.

The novelty effect

Another potential issue is the novelty effect: where the newness (novelty) of a treatment can sway experiment results. In the example above, users exposed to the green variant may have had more time to adjust to the new experience. Finally, users exposed to that green variant have had more opportunities to trigger the primary metric, especially if it’s an unbounded time metric, thus making the comparison between variants unequal.

A central requirement of experimentation is to make sure that the only difference between treatment and control is the feature you are testing. This way, you know any differences you find are the result of causation, and not just correlation.

Divergent lines with different slopes

Your cumulative exposures graph can show divergent lines with different slopes for a number of reasons. If you’re using a custom exposure event, users may get old, cached variants of your experiment if they keep triggering the exposure event without triggering the assignment event.

For example, Amplitude could assign a user to the control variant without triggering the exposure event. If you later set the traffic allocation for the control variant to 0%, that user could return and trigger the exposure event without triggering a new assignment event. Amplitude counts that user as a control exposure.

This reasoning also holds for experiments with sticky bucketing on.

In this example, on March 15, this user rolled out their experiment to 100% for the control variant. Because this experiment uses sticky bucketing, the graph still shows the number of “on” users increasing even after this user set the traffic allocation to 0%. This happens because Amplitude allocates variants when the SDK or API requests them, so the variant can stick to the user even if the user never receives it.

Sticky bucketing and traffic allocation

When you select sticky bucketing and change the traffic allocation, you won't get the target traffic allocation. Instead, you get a weighted average between the two allocations, since users who already have buckets stay in their buckets. You have to wait a bit to get close to the target traffic allocation.

If your experiment has sticky bucketing turned on, and you’re planning to roll out a variant after it ends, you should delete the appropriate branch in the code and remove the feature flag. If you don't want to make a code deployment, you can also turn off sticky bucketing.