Dig deeper into experimentation data with Experiment Results

This article helps you:

  • Extend the analytic power of A/B tests you create in Amplitude Experiment

With Experiment Results, Amplitude Analytics customers who have invested in a non-Amplitude feature flagging platform, whether third party or homegrown, can now take advantage of Amplitude’s planning, tracking, and analysis tools for Experiment—while still using the A/B tracking data generated by their own feature flagging platform.

Before you begin

Prior to using Experiment Results, you’ll need to ensure you’ve instrumented the metric events that are relevant to your experiment. Without them, you’ll be unable to create the success metrics and goals that Experiment Results needs to compare each variant in its analysis.

Make sure you’ve instrumented the necessary exposure events, which represent the delivery of a variant to a user participating in the experiment.

There is also an Experiment Results FAQ article that could provide guidance as you use this chart for the first time. 

Analyze an A/B test using Experiment Results

To create an A/B test and see the results, follow these steps:

  1. Navigate to Create > Chart > More > Experiment Results.

  2. In the Primary Metric module, click + Add Metric or + Define single-use metric to begin setting up your primary metric. A third option, Import, allows you to import your experiment settings as JSON.

  3. If adding a single-use metric, use the drop-down menu to specify the metric type in the Define Metric fly out panel:

    • Unique conversions
    • Event totals
    • Sum of property value
    • Average of property value
    • Funnel conversion
    • Formula
    • Retention

    The first four are available for individual event metric analyses, while funnel conversion allows you to define a multi-step journey that users must complete for the conversion to count. The Formula metric allows you to define a formula centered around a selected event or events.

    The last option, Retention, allows you to measure the percentage of users who return to perform the selected event on a specific day (Return on nth day) after being exposed to the experiment. By default, the Retention metric doesn't support CUPED, exposure attribution settings, nor calendar day windows. Instead, the metric calculates exposure attribution settings using any exposure and the nth day value based on 24-hour window increments.

    Note

    Any of the above metrics can be used as a custom metric during the design phase in Amplitude Experiment

  4. Next, specify the event to use for this metric. You can also filter the event using a + where clause. When you’re finished, click Done

    where_filter.png

    Optionally, click + Add Metric or + Define single-use metric in the Secondary Metrics module to add a second, subordinate metric to the analysis. You can add multiple secondary metrics as necessary.

  5. Click + Add Event in the Exposure module to define your experiment’s exposure event. The exposure event is the event users must trigger to become part of the experiment.

  6. In the Variants performed by module, add your variants. All experiments require at least one variant, known as the control. Add a variant by clicking + Add Experiment Variant.

add_variant.png

Choose the properties and values that defines your variant and click Apply.

  1. Click + Add Experiment Variant to add more variants as necessary to reflect the experiment setup in your feature flagging system.

Amplitude calculates your statistical results on the fly and display them in the results. The results also allow you to modify your experiment's statistical settings, such as from the default Sequential test to a T-test. 

Interpret your results

While the specifics may vary depending on the metric types you’re using, you’ll see four charts depicting your results:

  • Confidence interval of absolute performance over time: This chart is for sequential testing only. It can help you identify when the experiment reaches statistical significance; which occurs when the confidence interval no longer includes zero.
  • Cumulative exposure: This chart details the number of users who are exposed to your experiment over time. The x-axis displays the first date of a user's exposure, and the y-axis displays a cumulative, running total of users exposed to the experiment.
  • Performance by variant: The title of this chart is the metric you're focused on. The chart shows the number of users who did each step of a funnel, or the means of each variant if the metric isn't a funnel.
  • Mean over time (cumulative or non-cumulative): This chart is like the Conversion Over Time graph in a funnel analysis except that it works for non-conversion metrics. The x-axis shows the date the user was first exposed, and the y-axis shows the mean of the selected metric for each variant. The chart gives the option for either a cumulative or non-cumulative view. The cumulative view can help smooth out noise and make interpretation easier.

These charts are also helpful when learning from your end-to-end experiment

Note

By default, Amplitude selects the primary metric in experiment results. You can choose a different metric in the Analysis module. Click on the metric's name in the metric table to see its results. 

Was this page helpful?

Thanks for your feedback!

July 9th, 2024

Need help? Contact Support

Visit Amplitude.com

Have a look at the Amplitude Blog

Learn more at Amplitude Academy

© 2024 Amplitude, Inc. All rights reserved. Amplitude is a registered trademark of Amplitude, Inc.