A/B testing in a Funnel Analysis chart

In Amplitude, A/B testing lets you compare the funnel conversion performance of two or more user segments against each other. View results as improvement, which describes the performance of a segment compared to the baseline, or as statistical significance, which shows the probability of observing a difference as extreme as what you saw, assuming the control and treatment have the same mean.

By default, Amplitude uses the first segment added to the funnel analysis as the baseline. Change this in the Baseline segment drop-down menu.

A/B Test - Improvement

This chart displays the conversion rate for each segment across all steps in your funnel. You can have more than one variant in an A/B test, but you can only have one baseline.

A/B Test - Significance

On this chart, a high value for a variant suggests it converts better than the baseline. A low value suggests it doesn't.

Amplitude considers the results statistically significant when the results meet these conditions:

• A sample size above 30 for both variants.
• Sample size * conversion rate >= 5 and sample size * (1-conversion rate) >= 5 for both variants.
• A significance of 95% or greater.

For more details, refer to how Amplitude calculates statistical significance.

Understand the breakdown table

The data table below the chart breaks down the data. As with all data tables in Amplitude, you can export the data as a CSV file. The columns include:

• Count: The number of users or groups that entered the funnel.
• Converted: The number of users or groups that completed all the steps in the funnel with all conditions met.
• % Conversion: The number of converted users or groups, divided by the number of users or groups that entered the funnel.
• % Improvement over Baseline: Amplitude calculates this with the equation (% conversion for that variant - % conversion for the baseline) / (% conversion for the baseline). The percentage in the data table is green when the value is a positive number.
• Significance: The likelihood that the performance displayed for each test variant is actually different from zero, and not due to random fluctuations in the data. The higher this value is, the more confident you can be in your results. More formally, this is 1 - p-value.