How to Use Power Analysis

Power analysis is a calculation that helps you figure out the minimum sample size needed to detect an effect in your experiment, assuming there is one. This tool is crucial during the planning phase of most experiments, helping teams avoid tests that are too small to recognize meaningful differences or unnecessarily large, wasting resources.

Think of power analysis as your experiment’s “crystal ball.” With this calculation, you can answer such questions as:

• How many people should I include in my A/B test to trust the outcomes?

• If I change something on my website, how likely will I notice a real difference in ?

• What’s the smallest effect size I can spot with my number of users?

• How confident can I be that the effect I see isn’t just due to random chance?

• Should I run the experiment for longer, or do I already have enough data?

To truly understand power analysis, you need to be familiar with four key concepts.

Effect size

Effect size is the magnitude of the difference or relationship you’re trying to see in your experiment.

In A/B testing, the effect size could be the difference in between two web page versions. Other experiments might note a 5% increase in (CTRs) or a two-second decrease in page load time. The effect size changes depending on your improvement and your target outcome.

The smaller the effect, the harder it is to spot. You may need a larger sample size to identify anything meaningful or worth acting on.

Sample size

Sample size refers to the number of data points or observations you need in your experiment. The appropriate sample size ensures your experiment has enough information to provide trustworthy conclusions.

Web tests often use the number of users or sessions as the sample size. A small sample count increases the risk of missing a genuine effect. Larger samples yield more reliable results but can complicate data collection and increase effort.

the ideal sample size for your experiment.

Significance level

The significance level is the threshold to determine whether your observed effect is statistically significant—like a false positive “alarm bell.” This value represents the chance you’re willing to take when concluding there’s an effect when there isn’t one.

Commonly set at 0.05 (or 5%), this probability threshold means that there’s a 5% chance of concluding that an effect exists when it doesn’t.

A lower significance level reduces the danger of false positives (making your test more accurate and reliable) but needs a larger sample size to maintain power.

the statistical significance between the control and variant in your experiments. .

Power

Power is the main element of the “power analysis” calculator. This percentage is the probability of noticing a real effect when it exists.

Most teams aim for a power of 0.8 (or 80%)—this means there’s an 80% chance of correctly identifying an effect if there is one. In other words, you’ll catch true differences four out of five times.

Power is the opposite of a , which is when you fail to spot an effect that’s there. A higher power means you’re less likely to encounter these faults, but you’ll need more data and a larger participant count.

Power analysis typically occurs before an experiment starts, so the process is sometimes called “pre-study” power analysis.

Using the calculation before the study begins helps you:

• Determine the sample size you need
• Estimate how long your experiment should run
• Assess if you have the resources to uncover your desired effect
• Avoid conducting underpowered studies that waste time and resources

Suppose you’re developing an A/B test for a new checkout process. A pre-study power analysis might tell you that you need 10,000 visitors per variant to detect a 2% increase in conversion rate reliably.

You can also evaluate the power after your experiment ends—a “post-hoc” analysis.

Post-hoc power analysis enables you to:

• Decipher non-significant results
• Understand the sensitivity of your completed experiment
• Prepare better for future similar experiments

Let’s say your A/B test findings came back inconclusive. A post-hoc analysis may reveal that your test only had 40% power to notice that 2% increase in conversion. This insight suggests your experiment was underpowered and might have missed a noteworthy effect.

However, most statisticians argue against using post-hoc analysis to interpret current data.

The calculation can lead to circular reasoning and misinterpretation—using the same data to estimate the impact level and calculate power can make non-significant results always appear underpowered.

You’ll go around in circles and spend more time accounting for mistakes than if you properly planned and prepared in the first place. Because of this, pre-study power analysis is a more reliable approach, and post-hoc analysis is best used as a learning tool to improve future studies.

Applying power analysis to A/B testing helps you avoid the pitfalls of stopping tests too early or for an extended period.

The calculation takes the guesswork out of deciding how many users to include in your experiment and ensures your tests are reliable and actionable. Product judgments will be based on accurate, credible data, not gut feeling or noise.

Putting power analysis to work in your A/B tests involves following this process:

1. Define your smallest meaningful effect: What is the smallest change that would make a difference to your business? Is it a 1% increase in conversion rate? A 0.05-second decrease in load time? This value is your minimum detectable effect (MDE).
2. Set your significance level: Typically, this level is 5% (0.05), meaning you would accept a 5% chance that the observed effect is due to random variation rather than a genuine difference.
3. Choose your desired power: Aim for a power of at least 0.8. This level means you’ll catch genuine changes 80% of the time.
4. Calculate sample size: Input these variables into an online calculator or statistical software tool. The result will tell you how many users you need in each group or variant (A and B) to identify the effect confidently.
5. Estimate test time: Based on your site’s traffic, calculate how long it will take to execute the test and reach the required sample count.

Case study

Imagine you’re testing a call-to-action (CTA) button. You want to uncover a 2% absolute increase in CTRs. Your current rate is 10%, and you get 20,000 weekly visitors. In this case, your MDE is 2% (12% vs. 10%), the significance level is 5%, and the desired power is 80%.

A power calculation might show that you need roughly 4,700 visitors per variant. With your traffic, this means you’d need to run the test for about 33 days. (4,700 * 2 variants / 20,000 weekly visitors * 7 days).

Sticking to the sample size and testing period determined through power analysis ensures valid and reliable outcomes.

For example, if you end the test too early (e.g., before 33 days), you risk not having enough data to detect the change in CTR, leading to inconclusive conclusions. You won’t know whether the new CTA button is worth implementing.

Power analysis also helps you make sense of your results after the A/B test. If you hit that 2% increase you were aiming for, you can feel confident it’s not just down to luck. Your experiment was designed to catch this change so you can count on what you see.

If the opposite occurs (you don’t see a significant difference), power analysis lets you discern whether it’s because there’s no effect or if your test wasn’t strong enough to find one.

Maybe the button change makes a difference, but it is smaller than you expected. This insight can guide your next steps—you might need a bigger test or perhaps try a different approach.

Balancing power and practicality

Power analysis isn’t just about hitting a certain number. You need to balance statistical rigor with practical constraints. Sometimes, you might choose to run a lower-powered test if the potential impact of your change is huge or the cost of a false positive is low.

For instance, if you’re testing a major redesign that could dramatically boost conversion, you might accept a lower power to get results faster.

Similarly, if you’re testing something easily reversible (such as an email subject line), the likelihood of a false positive is less severe.

In these scenarios, the potential benefits of acting outweigh the need for high confidence in your results.

research often involves smaller sample counts and more qualitative data than large-scale A/B tests. Statistical significance is also often less important—a single user struggling with your interface can provide more valuable revelations than a large dataset with no apparent issues.

Despite this, power analysis still plays a vital role in UX studies, adding rigor to your process and increasing certainty in your findings.

Qualitative and quantitative studies

As in A/B tests, power analysis helps UX researchers determine the number of participants needed, even in studies. Use the method to determine how many users need to be tested to reliably spot usability issues or differences in user preferences.

For quantitative metrics such as task completion time, error rates, or satisfaction scores, power analysis works similarly to A/B testing. Apply the calculation to ensure sufficient sample sizes to spot meaningful differences in these .

Most UX studies use mixed methods (combining qualitative and quantitative techniques). Since power analysis works for both, you can use it to strategize the quantitative parts of your experiment, ensuring they’re strong enough to complement your qualitative discoveries.

Agile approach

Whatever your study or data type, power analysis helps balance your in-depth observations with broader patterns. Researchers can use the calculation to decide when they’ve collected enough evidence to draw accurate conclusions and when they need to broaden their sample.

This tactic is particularly valuable in , where you might conduct multiple rounds of small tests. Here, you’ll want to know when you’ve gathered enough data across each iteration to make confident decisions. Using power analysis helps you save time and resources in the long run.

The calculation can also be useful when explaining your research tactics to stakeholders or publishing UX findings, as it allows you to back up your judgments with evidence. You have a scientific basis for your sample size, making it easier to secure buy-in and build trust.

Even with the best intentions, it’s easy to stumble into challenges when using power analysis. Awareness of these pitfalls is the first step to dodging them, leading to more effective, insightful, and valuable experiments.

Overestimating effect sizes

It’s tempting to be optimistic when determining a change measurement, but assuming an effect is too large can result in underpowered studies. Be realistic and conservative in your estimates, using what you know about your business, product, and past studies to guide you.

Ignoring variability

Different metrics have different levels of noise. Metrics like conversion rates and are often more variable than expected. Use historical data to inform and support your power calculations.

Neglecting multiple comparisons

You need to adjust your significance level if you’re testing multiple variants or metrics. Otherwise, you put the experiment in danger of false positives.

Stopping tests early

It’s exciting to see early results but resist the urge to stop a test before reaching your predetermined number of observations. Stick to your strategy to ensure valid conclusions.

Power analysis is a method to steer your experiments, not a strict formula. Use the calculation to influence your decisions, but consider other factors such as opportunity costs, potential hazards, and business priorities.

Experimentation aims to conduct tests that provide actionable learnings, not just statistically relevant data. Following these best practices will help you achieve this.

• Use appropriate tools, like statistical software or reputable online calculators, designed for power analysis.

• Consider the practical significance of your effect size. How meaningful is the change for your business?

• If your ideal observation count is out of reach, perform a series of smaller tests instead of one big one.

• Keep track of your assumptions and calculations to help you learn from each study and refine your approach.

• When defining meaningful impact sizes, involve business leaders to ensure your experiments align with business goals and expectations.

While power analysis is vital for experiment planning and ensuring dependable results, it’s not the only tool in the shed. The calculation complements other statistical methods, and using it alongside them can help you get the most out of your data.

P-values

P-values and power analysis go hand in hand. A p-value helps tell you if a result has statistical importance (real or due to chance), while power analysis asks if you can reliably recognize the effect if it exists.

Power analysis gives you more control over planning and experiment design, but p-values help .

Confidence intervals

Confidence intervals show the range where the actual effect likely lies. This range complements power analysis: you use power analysis to help organize your sample size and confidence intervals to indicate the precision of your findings.

Using both methods gives you a fuller picture of your experiment’s validity and outcomes.

Bayesian methods

use prior knowledge and enable you to update your beliefs about an experiment's outcomes as the data comes in.

Compared to power analysis, which requires upfront planning and a fixed sample size, Bayesian methods allow for flexible stopping and incorporating what you already know into your insights.

Bayesian analysis can be more flexible than power analysis but often uses more complicated calculations.

Sequential analysis

lets you look at a study's results as data accumulates. This method differs from traditional power analysis, where a fixed sample count means you do one analysis at the end.

means you can check in several times during an experiment and potentially stop it early—for example, if you’ve reached your desired result or threshold. This strategy can be more efficient but requires careful planning to ensure statistical reliability.

From A/B testing to UX research, power analysis helps you design experiments right: solid enough to highlight real effects but not wastefully large.

By using the calculation, you’re setting yourself up for:

• More consistent results
• Efficient use of resources
• Confident decision-making
• Improved experimental design over time

You’ll never have the “perfect experiment” (which doesn’t exist), but you will be able to prepare better ones. Each time you use power analysis, you learn and refine your techniques. The tool is a must for anyone serious about making data-driven choices.

If you want to put these principles into practice, can help. The makes it easy to carry out statistically sound tests.

Adopt Amplitude to create experiments that are perfect for your needs. Reduce the risk of inconclusive tests, maximize each experiment’s insights, accelerate your cycle, and build a culture of experimentation in your organization.

Don’t let uncertainty hold back your product. Get started with Amplitude today.