Dig deeper into experimentation data with Experiment Results

Experiment Results lets you delve into the data your experiments collect. Experiment Results can incorporate data and information from non-Amplitude feature-flagging platforms and use that external data within Amplitude's native planning, tracking, and analysis tools. Amplitude incorporates this data into the A/B tracking data that Experiment generates.

Before you begin

Before using Experiment Results, instrument the metric events relevant to your experiment. Without metric events, you can't create the success metrics and goals that Experiment Results needs to compare each variant in its analysis.

Also instrument the necessary exposure events, which represent the delivery of a variant to a user participating in the experiment.

For more information, refer to the Experiment Results FAQ.

Analyze an A/B test using Experiment Results

To create an A/B test and view the results

1. Go to Create > Chart > Experiment Results.
2. In the Metrics module, click Add Metric or Define single-use metric to define your primary metric.
3. If adding a single-use metric, use the drop-down menu to specify the metric type. Choose one of:
   • Unique conversions
   • Event totals
   • Sum of property value
   • Average of property value
   • Funnel conversion
   • Formula

The first four are available for individual event metric analyses. Funnel conversion lets you define a multi-step journey that users must complete for the conversion to count. The Formula metric lets you define a formula centered around a selected event or events.

4. Specify the event to use for this metric. You can also filter the event using a Where clause.

5. When you're done, click Done.

   Optionally, click Add Metric or Define single-use metric in the Secondary Metrics module to add a second, subordinate metric to the analysis. Add multiple secondary metrics as needed.

6. Click Add Event in the Exposure module to define your experiment's exposure event. The exposure event is the event users must trigger to join the experiment.

7. In the Variants performed by module, click Add Experiment Variant to add your variants. All experiments require at least one variant, known as the control.

Choose the properties and values that define your variant and click Apply.

8. Click Add Experiment Variant to add more variants that reflect the experiment setup in your feature flagging system.

Amplitude calculates your statistical results as they become available and displays them in the Results area. The results let you modify your experiment's statistical settings, such as switching from the default Sequential test to a T-test.

Interpret your results

The specifics vary based on the metric types you use. Four charts depict your results:

• Confidence interval of absolute performance over time: This chart applies to sequential testing only. It helps you identify when the experiment reaches statistical significance, which occurs when the confidence interval no longer includes zero.
• Cumulative exposure: This chart shows the number of users who receive your experiment over time. The x-axis shows the first date of a user's exposure, and the y-axis shows a cumulative, running total of users exposed to the experiment.
• Performance by variant: The title of this chart is the metric you focus on. The chart shows the number of users who completed each step of a funnel, or the means of each variant if the metric isn't a funnel.
• Mean over time (cumulative or non-cumulative): The x-axis shows the date the user first saw the experiment. The y-axis shows the mean of the selected metric. Click the dropdown under the metric table to select a metric. Amplitude selects the recommendation metric by default for each variant. This chart works like the conversion over time chart, except it also handles non-conversion metrics. Use this chart to identify seasonality, novelty effects, or trends over time. Usually, the mean for days near the start of the experiment exceeds the mean for days near the end, because users at the start have had more time to perform the metric event. This concern lessens when you use the exposure attribution window. View the cumulative or non-cumulative version of this chart. The cumulative view smooths out daily noise and makes the chart easier to interpret.

These charts also help you learn from your end-to-end experiment.

Group By

For more resources on group by in experiments, refer to the group by reference and the experiment learnings guide.