Finalize your experiment's statistical preferences

This article helps you:

Understand the default statistical preferences in your experiment's results
Understand when to modify the default settings

Amplitude Experiment uses default statistical settings for experiment analyses:

CUPED toggled off
Bonferroni Correction toggled on
Custom Exposure Settings toggled off
Test Type set to Sequential
Confidence Level set to 95%

As you review your results, your experiment may benefit from modifying one or more of the default settings. To modify the default statistical preferences, follow the steps below:

From the Settings tab, click the Edit icon in the Analysis Settings card.
In the fly-out that appears, go to Stats Preferences.
Make the desired edits to the default settings, and then click Save.

The Stats Preferences can be modified at any step of an experiment, but they are most beneficial for the final analysis after the experiment has ended.

Note

This article continues directly from the article in our Help Center on learning from your experiment. If you haven’t read that, do so before continuing here.

CUPED

Controlled-experiment using pre-existing data, also known as CUPED, is an optional statistical technique meant to reduce variance in Amplitude Experiment. Toggling CUPED on means that Amplitude Experiment will account for possible varying treatment effects for different user segments. There are situations where CUPED would not be the best choice for your experiment, such as targeting only new users in your test.

The random bucketing process can sometimes deliver unbalanced groups of users to each variant. This is known as pre-exposure bias, and it’s one of the things CUPED is meant to address. If you don’t use CUPED for your experiment, this bias will persist. This is why you may notice differences in the mean-per-variant when running the same experiment with and without CUPED.

For a more technical explanation, see this detailed blog post.

Read more about CUPED and how it can affect your experiment results in this blog.

Bonferroni Correction

Amplitude Experiment uses the Bonferroni correction to address potential problems with multiple hypothesis testing. Although a trusted statistical method, there are situations where you may not want to use it when analyzing your experiment results. One might be if you want to compare results with those generated by an internal system that does not support the Bonferroni method. In this case, and if you're willing to accept higher false positive rates, toggle the Bonferroni Correction off.

Custom Exposure Settings

You can allocate a timeframe, or window, to the metric event for it to be considered a true conversion event. The window can be set in seconds, minutes, hours, or days.

Test Type

You can choose between a sequential test and a T-test when analyzing your experiment. Sequential is usually—but not always—the better choice. For example, you may have too small of a sample size for sequential testing.

Read more about how to utilize a T-test and sequential testing in Amplitude's Help Center.

Confidence Level

The confidence level measures how confident Amplitude Experiment is that it would generate the same results for the experiment if you were to roll it out again and again. The default confidence level of 95% means that 5% of the time, you might interpret the results as statistically significant when they're not. Lowering your experiment’s confidence level will make it more likely that your experiment reaches statistical significance, but the likelihood of a false positive goes up. You should not go below 80%, as the experiment's results may no longer be reliable at this point.