Advanced metric use cases

This article reviews advanced use cases that you may face while analyzing your experiment's results. 

Case 1: Create a funnel analysis based on your experiment's metrics

Imagine a conversion funnel with five steps, where step three represents the exposure event for your experiment. To reduce noise and increase the likelihood of reaching statistical significance, Amplitude Experiment only counts metric events after the exposure event. If the exposure event is step three of the funnel, and you include the whole funnel as a metric, the number of conversions for the funnel will be zero. Making steps three through five a standalone metric in your experiment would be the best way to measure the actual conversion rate of your funnel.

Sometimes, you may need further analysis of your experiment's conversion rates in a funnel analysis. 

Follow these steps to use your experiment's metrics in a Funnel Analysis chart:

1. Add the events for your funnel analysis in the Events module.
2. In the Measured as module, choose the Conversion time window, then specify the counting method (unique users or totals).
3. Select your analysis unit or group type (i.e., Any Users) in the Segment By module.
4. Create a user segment for each variant of your experiment.

5. Click + Performed to add filters with your experiment's flag key and variant.
6. Set the date range for any time since to match the start date of your experiment.

The results of your Funnel Analysis chart may vary slightly from those of your experiment. This is because funnel analyses and experiments do not handle users who variant jump the same way. For example, a funnel analysis will first include all users who meet its filter requirements; then, based on those filtered users, it will compute the conversion rate of the funnel. This means that a user may be included in the funnel analysis even if they were exposed to your experiment's exposure event after they completed the funnel. 

Analyze your experiment data using other Amplitude Analytics metrics

Amplitude Analytics offers metrics that Amplitude Experiment does not. You can also use the steps in the previous section to analyze things like time to convert or retention (though these measurements will only take into account a user's first conversion). 

Read this Help Center article on funnel analysis' FAQs to learn more.

Case 2: Analyze your experiment's results based on a subset of users

Imagine your experiment targets all users, but you want to take a deeper look at the experiment's effect on a subset of users, such as exposed users in the United States only. It may be tempting to simply add a filter on the country property; however, this will not generate the results you expect. When you create a metric, that metric is computed on all exposed users. If you add a filter for users in the United States to the metric event, the numerator will include the filter but the denominator will not.

Follow these steps to filter a subset of users in your experiment results:

1. From the Analyze tab in your experiment, click Open in Chart.
2. In the Variants performed by section, click +Filter by to add a filter for the Country property.

Be cautious of analyzing your experiment's results based on just one subset. You may encounter a false positive when looking for true statistically significant results.

Remember that when you run a multiple hypothesis test.) in this situation, you are actually running a separate hypothesis test for each segment. You may see a positive lift with one subset and a negative with another subset. Your decision on whether to roll out or roll back in these situations isn't clear-cut. One option is to roll out only to the group that shows positive lift.