Key terms
Glossary of key experimentation terms
| Term | Definition |
|---|---|
| Assignment event | Another name for enrollment event. |
| Audience | A group of users targeted for the experiment. Amplitude usually splits this audience evenly into "control" and "variant" groups. |
| Baseline conversion rate | The current rate of your primary success metric before this experiment. |
| Bonferroni correction | A statistical technique that counteracts the multiple comparisons problem (also known as multiplicity or the look-elsewhere effect). |
| Confidence interval | A range of plausible values that contains the parameter of interest. The parameter to estimate is the difference in means between the treatment and control. For example, if the confidence level is 95 and the same experiment runs 100 times, the confidence interval in each run contains the true parameter at least 95 times. |
| Confidence / significance level | The probability of a false positive. For example, at a 95% confidence level, there's a 5% chance of detecting a change to your success metric when no actual change occurred. |
| Enrollment event | The event that converts users or customers into registered participants or members. |
| Exposure event | The event that indicates when a user has seen a change based on an experiment. |
| Guardrail metric | A metric you want to protect from regression while you increase your success metrics. For example, if you drive users to a free trial of your business product, trials of your consumer product can act as a counter metric. If business trials go up, consumer trials go down. Confirm the net effect is positive. |
| CUPED | Controlled-experiment using pre-existing data (CUPED) is an optional statistical technique that reduces variance in experimentation. |
| Hypothesis | An assumption about the methods that can solve or ease the problem statement, and the reasoning behind those methods. |
| p-value | The probability of observing data as extreme as what you saw or more, assuming no difference between treatment and control. |
| Payload | Variables attached to a variant. Use payloads to change flags and experiments remotely without a code change. |
| Primary success metric | The main metric you want to move by running this experiment. Ideally drives both customer and business success. |
| Problem statement | An explanation of the internal business or user problem you're trying to solve. |
| Primary metric | A quantitative measure that evaluates your experiment against your goals. The primary metric determines whether your hypothesis is accepted or rejected and whether your experiment has succeeded or failed. |
| Run time | The duration your experiment takes to run, based on the sample size needed per variant and your traffic levels. |
| Rollout percentage | The percentage or number of targeted users that receive this variant. |
| Sample size | The number of users or amount of traffic you need in each experimental variant to reliably detect statistical significance. |
| Secondary success metric | An additional metric you expect to move with this experiment. |
| Sequential testing | A statistical analysis where the sample size isn't fixed in advance. Sequential testing lets you conduct an A/B test, review your results, and conclude the test without inflating false positives. |
| Statistical power | The probability that you detect a change to your success metric when a change exists to detect. |
| T-test | A statistical analysis that compares means between two populations of data to decide if the difference is statistically significant. |
| Target lift / minimum detectable effect (MDE) | The percentage change you expect to drive on your primary success metric as a result of this experiment. |
| Type 1 error | Incorrectly classifying that a statistically significant difference exists between treatment and control when none exists. |
| Type 2 error | Incorrectly classifying that no difference exists between treatment and control when a difference exists. |
Was this helpful?