Dig deeper into experimentation data with Experiment Results

With Experiment Results, Amplitude Analytics customers who have invested in a non-Amplitude feature flagging platform, whether third party or homegrown, can now take advantage of Amplitude’s planning, tracking, and analysis tools for Experiment—while still using the A/B tracking data generated by their own feature flagging platform.

Before you begin

Prior to using Experiment Results, you’ll need to ensure you’ve instrumented the metric events that are relevant to your experiment. Without them, you’ll be unable to create the success metrics and goals that Experiment Results needs to compare each variant in its analysis.

Make sure you’ve instrumented the necessary exposure events, which represent the delivery of a variant to a user participating in the experiment.

There is also an Experiment Results FAQ article that could provide guidance as you use this chart for the first time. 

Analyze an A/B test using Experiment Results

To create an A/B test and see the results, follow these steps:

1. Click Create New > Analysis > Experiment Results.

2. In the Primary Metric module, click + Add Metric or + Define single-use metric to begin setting up your primary metric. A third option, Import, allows you to import your experiment settings as JSON.

3. If adding a single-use metric, use the drop-down menu to specify the metric type in the Define Metric fly out panel:

   • Unique conversions
   • Event totals
   • Sum of property value
   • Average of property value
   • Funnel conversion
   • Formula
   • Retention

   The first four are available for individual event metric analyses, while funnel conversion allows you to define a multi-step journey that users must complete for the conversion to count. The Formula metric allows you to define a formula centered around a selected event or events.

   The last option, Retention, allows you to measure the percentage of users who return to perform the selected event on a specific day (Return on nth day) after being exposed to the experiment. By default, the Retention metric doesn't support CUPED, exposure attribution settings, nor calendar day windows. Instead, the metric calculates exposure attribution settings using any exposure and the nth day value based on 24-hour window increments.

   Note

   Any of the above metrics can be used as a custom metric during the design phase in Amplitude Experiment. 

4. Next, specify the event to use for this metric. You can also filter the event using a + where clause. When you’re finished, click Done. 

   

   Optionally, click + Add Metric or + Define single-use metric in the Secondary Metrics module to add a second, subordinate metric to the analysis. You can add multiple secondary metrics as necessary.

5. Click + Add Event in the Exposure module to define your experiment’s exposure event. The exposure event is the event users must trigger to become part of the experiment.

6. In the Variants performed by module, add your variants. All experiments require at least one variant, known as the control. Add a variant by clicking + Add Experiment Variant.

Choose the properties and values that defines your variant and click Apply.

7. Click + Add Experiment Variant to add more variants as necessary to reflect the experiment setup in your feature flagging system.

Amplitude calculates your statistical results on the fly and display them in the results. The results also allow you to modify your experiment's statistical settings, such as from the default Sequential test to a T-test. 

Interpret your results

While the specifics may vary depending on the metric types you’re using, you’ll see four charts depicting your results:

• Confidence interval of absolute performance over time: This chart is for sequential testing only. It can help you identify when the experiment reaches statistical significance; which occurs when the confidence interval no longer includes zero.
• Cumulative exposure: This chart details the number of users who are exposed to your experiment over time. The x-axis displays the first date of a user's exposure, and the y-axis displays a cumulative, running total of users exposed to the experiment.
• Performance by variant: The title of this chart is the metric you're focused on. The chart shows the number of users who did each step of a funnel, or the means of each variant if the metric isn't a funnel.
• Mean over time (cumulative or non-cumulative): This chart is like the Conversion Over Time graph in a funnel analysis except that it works for non-conversion metrics. The x-axis shows the date the user was first exposed, and the y-axis shows the mean of the selected metric for each variant. The chart gives the option for either a cumulative or non-cumulative view. The cumulative view can help smooth out noise and make interpretation easier.

These charts are also helpful when learning from your end-to-end experiment.