Advanced metric use cases

This article covers advanced use cases you may encounter when analyzing experiment results.

Case 1: Create a funnel analysis based on your experiment's metrics

Imagine a conversion funnel with five steps, where step three represents the exposure event for your experiment. To reduce noise and increase the likelihood of reaching statistical significance, Amplitude Experiment counts metric events only after the exposure event. If the exposure event is step three of the funnel, and you include the whole funnel as a metric, the funnel conversion count is zero. To measure the actual conversion rate of your funnel, make steps three through five a standalone metric in your experiment.

You may need further analysis of your experiment's conversion rates in a funnel analysis.

To use your experiment's metrics in a Funnel Analysis chart, follow these steps:

1. Add the events for your funnel analysis in the Events module.
2. In the Measured as module, choose the Conversion time window, then specify the counting method (unique users or totals).
3. Select your analysis unit or group type (for example, Any Users) in the Segment By module.
4. Create a user segment for each variant of your experiment.
5. Select Performed to add filters with your experiment's flag key and variant. If you restarted your experiment, add an experiment key filter.
6. Set the date range for any time since to match the start date of your experiment.

The Funnel Analysis chart results may differ slightly from your experiment results. Funnel analyses and experiments handle users who variant jump differently.

For example, a funnel analysis includes all users who meet the filter requirements. The funnel analysis then computes the conversion rate of the funnel based on those filtered users. The funnel analysis may include a user even if the user reached the exposure event after completing the funnel.

Analyze your experiment data using other Amplitude Analytics metrics

Amplitude Analytics offers metrics that Amplitude Experiment doesn't. Use the steps in the previous section to analyze time to convert or return on or after retention.

Refer to this Help Center article on funnel analysis FAQs for more details.

Case 2: Analyze your experiment's results based on a subset of users

Imagine your experiment targets all users, but you want to examine the experiment's effect on a subset of users, such as exposed users in the United States only. Adding a filter on the country property doesn't generate the results you expect.

When you create a metric, Amplitude computes that metric on all exposed users. If you add a filter for users in the United States to the metric event, the numerator includes the filter but the denominator doesn't.

To filter a subset of users in your experiment results, follow these steps:

1. In your experiment, go to Activity > Analysis > external link icon.
2. In the Variants Performed By section, select Filter by to add a filter for the Country property.

Avoid analyzing your experiment's results based on just one subset. You may encounter a false positive when looking for true statistically significant results.

When you run a multiple hypothesis test in this situation, you run a separate hypothesis test for each segment. You may find a positive lift with one subset and a negative decline with another subset. The decision to roll out or roll back in these situations isn't always clear. One option is to roll out only to the group that shows positive lift.

Case 3: Threshold metrics

Sometimes, you want to define success as a user doing an event multiple times. For example, your users must make a purchase three (3) consecutive times to count as a conversion. To achieve this, create a funnel counting by uniques with three (3) purchase events.