---
title: "Feature flags vs. A/B testing: when to use each | Amplitude"
description: "Compare feature flags vs. A/B testing, learn when a rollout becomes an experiment, and see how to use both to release and measure product changes."
canonical: "https://amplitude.com/explore/experiment/feature-flags-vs-ab-testing"
---

# Feature flags vs. A/B testing: when to use each | Amplitude

Compare feature flags vs. A/B testing, learn when a rollout becomes an experiment, and see how to use both to release and measure product changes.

Source: https://amplitude.com/explore/experiment/feature-flags-vs-ab-testing

---

###### Feature flags and experimentation

# Feature flags vs. A/B testing: when to use each

Compare feature flags vs. A/B testing, learn when a rollout becomes an experiment, and see how to use both to release and measure product changes.

Table of Contents

-

Feature flags let you choose who gets a new feature. A/B testing helps you find out whether that feature makes your product better. Teams often use both: release a change to a limited audience, compare it with the existing experience, and decide whether to roll it out further.

For example, a flag can give some customers access to a shorter signup form. An A/B test randomly assigns customers to the old or new form and measures how many finish signing up.

Amplitude offers both through [Statsig, its experimentation platform](https://amplitude.com/statsig-upgrade).

Browse this guide

- [How feature flags and A/B testing differ](#flags-vs-testing)
- [When to use feature flags](#when-to-use-flags)
- [When to run an A/B test](#when-to-run-tests)
- [How to use feature flags and A/B testing together](#flags-and-tests-together)
- [Mistakes that weaken experiment results](#experiment-mistakes)
- [Experiment with Amplitude through Statsig](#experiment-with-statsig)
- [Frequently asked questions](#toc-chapter-faqs_142)

## How feature flags and A/B testing differ

Feature flags answer "Who gets this feature?" A/B tests answer "Does this version work better?" A flag can put customers into different experiences, but you still need a fair comparison and reliable measurement to learn which experience performs better.

| Dimension          | Feature flags                                                              | A/B testing                                                         |
| ------------------ | -------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| Main question      | Who gets this feature, and when?                                           | What effect does this change have on a chosen outcome?              |
| Typical action     | Enable a feature for employees, a beta group, or a percentage of customers | Randomly assign eligible users to control and treatment experiences |
| Evidence collected | Delivery, usage, and operational signals for the release                   | Outcome metrics linked to experimental assignment and exposure      |
| Main decision      | Expand access, pause delivery, or disable the feature                      | Ship, revise, or reject the change based on its measured effect     |

Suppose your team adds a shorter onboarding flow. A flag lets employees try it before customers receive access. An A/B test compares the shorter flow with the current flow among eligible new customers to estimate its effect on activation.

You can manage flags and run tests in the same product. For example, Statsig includes feature flags and analysis for simple A/B tests.

## When to use feature flags

Use feature flags to release a feature gradually, give beta testers early access, or turn off a feature that causes problems. Once the code is deployed, you can change who gets the feature without deploying a new version each time.

A team replacing a report export service might enable the new version for employees first, then a small customer group. If exports fail for large files, the team can disable the flagged path while investigating. Returning to the previous service still depends on keeping that path functional; a flag doesn't reverse database changes or recover lost data.

Other uses include giving beta testers access to an unfinished feature and enabling functionality only for supported app versions. These releases can generate useful feedback even when a controlled comparison isn't the goal.

Give temporary flags an owner and a removal condition. After the export service has fully replaced the old version, remove the obsolete branch and flag when it's safe to do so. Keep permanent access controls separate from temporary release decisions. The [feature flag best practices guide](https://amplitude.com/explore/experiment/feature-flags-best-practices) covers additional release scenarios.

## When to run an A/B test

Run an A/B test when you want to know whether a change improves a result you care about, such as completed purchases or new customer activation. Randomly assign eligible customers to the existing experience or the new version, then compare their results.

For a shorter onboarding flow, your hypothesis might be: removing an optional setup step increases the share of new customers who complete their first project within seven days. Keep the existing flow as the control and show the shorter flow to the treatment group. Track setup-related support requests as a guardrail against introducing confusion.

Choose the smallest effect worth detecting before launch. A tiny improvement may take more traffic to detect than your product can supply within a useful decision window. [Power analysis](https://amplitude.com/explore/experiment/power-analysis) helps connect the effect you want to detect with sample size and statistical power.

An A/B test is less useful when the decision is already fixed. For a mandatory legal text update, staged delivery and operational monitoring may answer the release question. For an optional onboarding redesign, a randomized test can inform whether the redesign deserves a wider launch.

## How to use feature flags and A/B testing together

Use a flag to release the change, then run an A/B test to decide whether more customers should get it. Check that the feature works before starting the test, and keep the test conditions consistent while you measure results.

For an onboarding experiment, the workflow can look like this:

1. Define activation as completing a first project within seven days, and write the hypothesis before launch.
2. Deploy both onboarding paths behind a flag. Verify the treatment with internal testers.
3. Choose eligible new customers and a stable randomization unit, such as an account when teammates share an onboarding experience.
4. Assign the experimental audience to control and treatment. Record exposure when the assigned experience can affect behavior.
5. Evaluate activation and the support-request guardrail using the planned statistical method and observation window.
6. Expand delivery if the evidence supports the change. Otherwise, keep the current flow or revise the treatment.

If the shorter flow improves activation but also creates unacceptable support demand, the team can revise the removed step before a wider release. If results are inconclusive, record the uncertainty instead of labeling the variants equivalent.

A feature can work without errors and still fail to help customers. Measure the outcome before deciding that a successful release deserves a wider rollout.

## Mistakes that weaken experiment results

An A/B test can give misleading results if the groups aren't comparable, tracking is incomplete, or you stop too early. Before declaring a winner, check how customers entered the test, what you measured, and whether the evidence supports the decision.

Watch for these problems:

- Comparing volunteers with other customers. Beta testers may differ in motivation or experience. Randomize within the eligible population when you want a controlled comparison.
- Treating a percentage rollout as proof. Giving 10% of users access doesn't by itself define a trustworthy experiment. Assignment, exposures, metrics, and analysis still matter.
- Stopping at the first favorable result. If you planned to analyze a test after collecting a set sample, repeatedly checking and stopping as soon as it looks positive increases the risk of a false positive. Follow your analysis method's stopping rules.
- Measuring an intermediate action as the whole outcome. More clicks on an onboarding button may accompany fewer completed projects. Match the primary metric to the decision.

For example, a button variant may increase clicks while seven-day activation stays uncertain. That evidence supports a claim about clicks, while the activation decision remains open. Also check that the observed group sizes match the intended allocation; an unexpected imbalance can signal an assignment or logging problem.

## Experiment with Amplitude through Statsig

Amplitude offers feature flags and A/B testing through Statsig, its experimentation platform. You can choose who gets a new feature, test different versions, and measure which one performs better.

Suppose [Amplitude Analytics](https://amplitude.com/amplitude-analytics) shows that customers leave during account setup. Your team thinks a shorter setup flow will help more people create their first project.

Run the test in Statsig. Show one randomly assigned group the current flow and another the shorter version. Measure how many customers create a project within seven days, and check whether the change increases support requests. Use the results to decide whether to release the shorter flow or revise it.

Related content:

- [Feature flag best practices](https://amplitude.com/explore/experiment/feature-flags-best-practices)
- [How to use power analysis](https://amplitude.com/explore/experiment/power-analysis)
- [Experimentation platform guide](https://amplitude.com/explore/experiment/experimentation-platform-guide)

Amplitude is the leading [all-in-one analytics platform](https://amplitude.com) for building better products. Product and engineering teams use Amplitude to connect [product analytics](https://amplitude.com/amplitude-analytics) and [web analytics](https://amplitude.com/web-analytics) with [session replay](https://amplitude.com/session-replay), [heatmaps](https://amplitude.com/heatmaps), [experimentation](https://amplitude.com/amplitude-experiment), [feature flags](https://amplitude.com/feature-management), and [guides and surveys](https://amplitude.com/guides-and-surveys) in one unified workflow, with [AI agents](https://amplitude.com/ai-agents) working across all of it. Learn how Amplitude is building [self-improving products](https://amplitude.com/self-improving-products) with [Wave](https://amplitude.com/wave), helping teams turn product data into PRs.

## Frequently asked questions

##### Are feature flags the same as A/B testing?

Feature flags and A/B testing serve different purposes. Flags control access to product functionality; A/B tests estimate the effect of a change through randomized comparisons. A flag can deliver an experiment's variants, but the test also depends on exposure tracking, outcome metrics, and an appropriate analysis method.

##### Can you run an A/B test without feature flags?

You can run an A/B test without a feature flag system. For example, a website can randomly route visitors to two different landing page URLs. The experiment still depends on consistent assignment and reliable outcome measurement. Feature flags are one way to deliver variants inside a product.

##### Is a percentage rollout an experiment?

A percentage rollout can be part of an experiment when it uses suitable random assignment and includes controlled outcome measurement. Simply enabling a feature for 10% of users doesn't establish that the feature caused a metric change. Define eligibility, exposures, success metrics, and the analysis plan before interpreting lift.

##### How is experimentation different from A/B testing?

Experimentation is the broader practice of developing hypotheses, designing tests, interpreting evidence, and making decisions. A/B testing is a specific method that compares two experiences using randomly assigned groups. For example, a team might run several A/B tests while investigating how to help new customers reach activation.

##### Does Amplitude offer feature flags and A/B testing?

Yes. Amplitude offers feature flags and A/B testing through Statsig, its experimentation platform. Use Statsig to choose who gets a feature and test whether a change improves results. For example, you can compare two signup flows and measure which one helps more customers finish creating an account.
