What is CUPED (Controlled Experiment Using Pre-Experiment Data)?

CUPED is an experimental method using data from before the experiment (pre-experiment data) to make results more accurate.

Instead of relying on the post-experiment data (data collected during the experiment), CUPED uses pre-experiment data to reduce the variance.

Reducing variance increases the experiment’s sensitivity, making it easier to detect more minor changes accurately. This makes the findings more reliable, allowing researchers to draw more confident conclusions about what to change or implement.

Here’s how CUPED works:

• Understand individual differences: Before the experiment starts, individual-level covariates (such as user demographics, past behavior, etc.) are collected. You use this pre-experiment data to build regression models to predict individual outcomes. These estimate what the results would be if there were no changes.
• Adjust results: During the experiment, measure outcomes for both the control and treatment groups. You then remove the predicted outcomes (based on pre-experiment data) from the actual results. This step removes existing differences between users in the two groups.
• Get reliable results: The difference in these adjusted outcomes between the groups shows the experiment’s real impact, with much of the pre-existing variance removed. This makes the results more trustworthy and easier to understand.

CUPED solves the problem of unclear results in experiments. It uses earlier data to ensure differences observed in the experiment are because of the changes made instead of other factors (confounding variables).

Confounding variables can make it harder to attribute any outcome differences to the experiment. Using pre-experiment data helps account for individual differences in the treatment and control groups, minimizing their influence.

CUPED experimentation helps to:

• Reduce bias and increase accuracy: Using pre-experiment data for individual-level differences between groups helps reduce experiment bias. It ensures fair comparisons, so we can confidently accredit the differences to experimental changes rather than anything else.
• Enhance sensitivity: CUPED’s adjustment process increases the experiment’s sensitivity, making it easier to detect more minor effects. It reduces data variability, enabling researchers to spot genuine, even subtle, experimental impacts.
• Improve reliability and validity: CUPED provides more accurate insights into the causal relationship between the changes and the outcomes, making the findings more dependable and actionable.

Using CUPED is particularly useful when you want to accurately measure the impact of an intervention, such as a website change, new product feature, or marketing strategy.

It’s beneficial when you have any of these:

• High participant variability: When there’s a lot of diversity among participants, such as demographics, behavior patterns, or preferences, CUPED experimentation helps you control these differences to get more accurate results.
• Potential confounding factors: If external factors could affect your experiment’s outcomes, like seasonal changes or user-specific characteristics, you can use CUPED to mitigate their impact and isolate the intervention’s actual effect.
  
• Small or subtle effects: When the expected outcome changes are minor or subtle, CUPED can increase the experiment’s sensitivity, so it’s easier to find these minor effects and determine their significance.
• Historical data: You can apply CUPED when you can access relevant historical participant data, as you can use it to create predictive models for outcome variables.
• Resource-intensive interventions: For experiments with lots of resources or high costs, it’s crucial to get accurate results. CUPED ensures the resources you spend on the intervention are justified, as it provides a reliable and precise estimate of its impact.

Many researchers use CUPED in their A/B tests to make them more accurate and definitive.

In a typical A/B test, researchers randomly assign users to a control group (A) and a treatment group (B). They show the treatment group the change (like a new webpage layout) and the control group the original version, so it serves as a comparative baseline. The goal is to make data-driven decisions about the most effective option by measuring the differences between the groups.

CUPED uses data collected before the experiment to account for inherent differences and external factors in the control and treatment groups. It adjusts the findings based on these considerations.

The adjusted results give researchers a clear picture of the impact of the change they’re testing, enabling them to determine which version performs better.

By reducing noise and isolating the actual effect of the changes, CUPED experimentation ensures conclusions drawn from the A/B test are more accurate and meaningful.

Reliability is the aim of the game at Amplitude, which is why we’ve made using CUPED with our experimentation platform a straightforward process.

Toggling CUPED experimentation is an excellent option if you want more definitive test results. In Amplitude Experiment, this means using 14 days of pre-experiment data to reduce variance and reach statistical significance faster. You can then analyze your adjusted data to conclude your experiment’s impact, using visualization and analysis tools to interpret the results effectively.

Integrating CUPED with Amplitude ensures your analyses are based on raw data and adjusted for potential biases and confounding factors. This approach leads to more accurate insights, enabling you to maximize your product, marketing strategies, or user experiences.

It can revolutionize how you translate experimental outcomes and make informed decisions that propel your business forward.