Test and launch your experiment
Before any users view your experiment, confirm that the variants look and function as intended. Because Experiment lets you assign specific variants by user ID, device ID, or cohort, you can confirm that Amplitude serves test devices the correct variants when they enter your experiment.
To exclude users from an experiment or flag, add them to the OFF variant.
On the Overview page for your experiment, review the Overview, Delivery, Variants, and Targeting sections. Confirm each section matches your plan.
Click Test Instrumentation to send the experiment's variants to the testers you designated when you configured the experiment's audience.
Test Instrumentation and targeting
When you test your instrumentation, Amplitude ignores the target segments you configured in the experiment. Test instrumentation sends variants only to the Testers.
Launch your experiment
When you're satisfied that your experiment works as intended, click Start Experiment to start delivering variants immediately. To start later, stop later, or both, expand Start Experiment and click Schedule.
You can set an analysis end date or accept the default Experiment analysis range. The analysis end date controls which dates appear in results. It doesn't stop variant delivery. To stop delivery at a set time, schedule a stop.
After your experiment runs, you can make a decision on your experiment when it reaches statistical significance or its analysis end date.
This scheduling flow also applies to Web Experiments.
Schedule your experiment
Schedule a start, a stop, or both so Amplitude begins and ends variant delivery at the times you choose.
Use a scheduled stop when you want Amplitude to stop assigning new users to variants at a future time. Use the analysis end date when you want to bound the results window. Those settings are independent.
- Expand Start Experiment and click Schedule.
- Set a start time, an end time, or both. Schedule a start at least one hour in advance. The end time must be after the start time, and both times must be in the future.
- Click Schedule.
If the experiment is already scheduled or running, click Edit Schedule to change or remove the start or stop times.
When you schedule a stop and the analysis end date is empty, Amplitude fills the analysis end date with the stop time. If you already set an analysis end date, Amplitude leaves it unchanged. You can still edit the analysis date range after you schedule a stop, including dates after the stop time.
After the stop time, Amplitude stops assigning new users to variants. The experiment shows Completed (Pending decision) until you make a decision.
For a feature flag, expand Activate flag and click Schedule. Amplitude turns the flag on at the start time and off at the end time.
A scheduled stop doesn't rebalance mutual exclusion or holdout groups. After the experiment auto-stops, its mutual exclusion slot stays unused, and a holdout carve-out continues to hold users back from the experiment.
Scheduled times and variant delivery
Amplitude exposes users to variants as soon as the scheduled time arrives. Users who requested a variant in the hour before the start time may continue to experience the pre-start response until their cached response expires, up to one hour later. A scheduled stop has the same cache window: users who received a variant in the hour before the stop time may continue to experience that variant until the cache expires.
Unlike a manual flag or experiment update, a scheduled start or stop doesn't trigger CDN cache invalidation, so the existing 60-minute cache TTL applies to those earlier requests. For details, refer to Cache invalidation.
For example, for an experiment that starts at 2:00 PM, a user who visited at 1:30 PM continues to experience the pre-start response until 2:30 PM, when their cached response expires. A user who arrives at 2:00 PM experiences a variant right away.
QA after rollout
After rollout, you can track how many users Amplitude exposed to each variant on a daily basis.
Go to Experiments > your experiment > Activity tab > Diagnostics to view how many users Amplitude exposed to each variant.
This view is a useful way to QA the assignment process. If one variant enrolls significantly more or significantly fewer users than expected, the difference may indicate an issue to investigate.
If you spot outliers or anomalies that concern you, click into the chart or information to investigate the potential causes. To learn more about understanding anomalies, refer to this article on Root Cause Analysis.
For a deeper validation of your experiment's instrumentation and assignment logic, run an A/A test before launching a full A/B test.
Was this helpful?