Sample ratio mismatches: Debug issues with experiment allocations

In Amplitude Experiment, a sample ratio mismatch (SRM) occurs when the observed allocation for variants significantly differs from the specified allocation.

For example, if you set your experiment's traffic allocation to split equally between the control and treatment variants, but the control receives 55% of the traffic, that's an SRM.

SRMs point to biases in the data and can lead to unexpected results if unresolved. Treat the results of any experiment affected by an SRM with caution.

Potential causes of SRMs include:

• Instrumentation errors.
• Changing traffic allocation during the experiment.
• Adding or removing a variant during the experiment.
• Turning sticky bucketing on or off during the experiment.

The cumulative assignment or exposure charts help you identify the cause of an SRM. Look for timestamps where the control and treatment time series diverge. The cause often appears at that point.

In some cases, variant jumping causes SRMs. Variant jumping happens when the same user sees two or more variants. Variant jumping often occurs with authentication patterns that make it difficult to determine if a user already has a variant assignment. Examples include:

• Applications with short-lived sessions.
• Applications with large numbers of anonymous users.

You might receive an SRM warning when analyzing a time frame shorter than the actual duration of the experiment. You can ignore these warnings if your analysis of the full experiment window doesn't trigger a similar warning.

To check for SRM issues, go to the data quality guide on the Analyze tab. Select Implementation & Instrumentation to view a count of any detected SRM issues.

Learn more about debugging sample ratio mismatches in Amplitude Experiment.