What is statistical significance?

Statistical significance measures the likelihood that your A/B test results are not due to chance. In other words, it helps you determine if differences in your test results are real or just random fluctuations.

How to calculate statistical significance?

To calculate statistical significance, start by defining your null hypothesis, which states there is no difference between the groups. Collect your data, including sample size, mean, and standard deviation for each group. Calculate your test statistic—such as a t-test for means or a chi-square test for proportions—to measure the difference between the groups. Determine your p-value, which shows the probability that the observed difference happened by chance. If the p-value is below a predetermined threshold (commonly 0.05), you can reject the null hypothesis and conclude that the difference is statistically significant.