Sample Ratio Mismatch

In Amplitude Experiment, a sample ratio mismatch (SRM) occurs when the observed variant allocation significantly differs from the specified allocation. For example, you allocated 50% of your Experiment traffic to control and 50% to treatment, but you observe a ratio of 55% control to 45% treatment.

An SRM points to biases in the data and can lead to unexpected results when unresolved. Treat the results of any experiment with an SRM as suspect.

Amplitude uses a sequential version of a chi-squared test with alpha = .01 to detect an SRM.

This guide describes the causes of an SRM, ordered from most to least common, and how to debug each one. It assumes you use the end-to-end Experiment product, but you can apply many of these steps to experiments set up in Experiment Results.

Amplitude exposure events versus custom exposure events

Amplitude recommends the Amplitude exposure event. On client-side SDKs, Amplitude tracks the exposure event automatically when the variant loads from the cache, not when Amplitude fetches the variants. Local evaluation doesn't produce an assignment event, so you must use a custom exposure event.

If you use a custom exposure event, send it when the user experiences the variant. The custom exposure event can happen before assignment. When this happens, the user property isn't yet set, and Amplitude doesn't count the initial custom exposure event as an exposure in analysis. Amplitude exposure events don't cost extra.

Variant distribution weights changed during the experiment

As a best practice, don't change a running experiment in a way that causes users to jump between variants. Changes like this can cause an SRM.

For example, changing 50% treatment / 50% control to 60% treatment / 40% control can cause users to jump between variants while the experiment runs. The SRM test assumes that traffic allocation doesn't change while the experiment runs. Go to Interpret the Cumulative Exposures Graph in Amplitude Experiment for more context about why you shouldn't change traffic allocation mid-experiment.

Experiment exposures started before analysis window begins

If one variant causes users to return to the product more, that variant can show more users than expected when the exposure window and analysis window don't align.

If the analysis window and the traffic window differ, change the analysis window to cover the full traffic window and check whether the SRM remains.

Variant jumping

Variant jumping describes when a user moves from one variant to another, sometimes multiple times. Variant jumping makes it difficult to attribute the metric to a specific variant. Amplitude Experiment's built-in diagnostics include charts that track the percentage of users who jump between variants.

If users jump between variants, check whether the cause is anonymous users (people who log in and out frequently) or changing device IDs. Review this in the User Stream.

As a best practice, don't change a running experiment in a way that causes users to jump between variants. Changes like this can cause an SRM.

Significantly more users converted from assignment to exposure for one variant

You can find the assignment-to-exposure funnel chart in the Diagnostics card of an experiment. Enter your conversion rates and sample size into this calculator to check whether your metrics are statistically significant at the 95% level. The conversion rates between variants should be similar, within randomness, because the assignment event splits users into two randomly assigned, equal cohorts. Amplitude sends exposure events immediately before the user experiences a variant, so users have the same experience between the assignment and exposure events.

Variant added or removed in the experiment

Adding or removing a variant while an experiment runs can cause an SRM. As a best practice, don't change a running experiment in a way that causes users to jump between variants.

Sticky bucketing enabled or disabled during experiment

Enabling or disabling sticky bucketing on a running experiment can cause a mismatch between the variant you expect and the variant a user receives. After you enable sticky bucketing on an experiment, Amplitude always evaluates each user to the same previously bucketed variant, regardless of the current targeting, even if you later disable sticky bucketing.

Changes that affected a segment of users

Check whether the SRM affects only a certain segment of users. For example, filter by country, OS version, app version, or platform.

To troubleshoot this kind of problem, find out which specific users the change affected. Ask questions like:

• Did someone push an instrumentation bug to an app version that a subset of users have?
• Did someone push a change that affects only users in a certain country?

Individual allocation of many users

Check whether you individually allocated a large number of users. Individual allocation isn't random and can skew your ratio, which can cause an SRM.

Adding a few users to the experiment individually is fine, but avoid adding many users.

First experiment

If an SRM appears in your first experiment, the cause might be an instrumentation bug. If you have multiple SRMs, the SRM is unlikely to result from a false positive, and you should find the root cause.

Users often logging out

Frequent logouts can appear in the variant jumping chart. Check whether the experiment causes users to log out more often. This pattern is common in financial institutions, where the application logs users out after 15 minutes of inactivity.

1. On logout, the application regenerates the device ID.
2. Amplitude treats the session as a new user.
3. Amplitude buckets the user into a different variant.
4. The user ID later resolves.

This pattern can also relate to anonymous users from sign-up flow experiments.

Missing exposures

Check whether two users have the same sequence of events but one is missing an exposure event. You might not send the exposure event in some cases, or you might send it when you shouldn't.

Incorrectly sending exposures for fallback variant

If you don't use the Experiment SDK to track exposures automatically, check whether you incorrectly send exposures for a fallback (default) variant.

Check for cases where the flag defaults to control and Amplitude counts that as an exposure when the flag doesn't return a response in time. When this happens, control has more exposures than treatment.