Value calculation in Experiment Analysis charts

Knowing how the values in your Experiment Analysis charts are calculated can help you understand what your experiments are telling you, so you can avoid making costly interpretation errors.

These values are derived from a small selection of inputs and formulas, which are described below.

Inputs

The formulas rely on the following set of inputs:

• E: The number of unique users who were exposed to the experiment.
• M: The number of unique users who triggered the metric event. T the subset of the users who have been exposed to the experiment. M is always be less than E.
• T: The total number of times the metric event was triggered. A single user can trigger the metric event more than once. Amplitude only counts metric events triggered by users included in E.
• S: The sum of all the metric events' property values.
• A: The sum of the average of all the metric events' property values, per user.
• FM: The number of unique users who triggered the events in the funnel, in the specified order.
• FT: The total number of times all the funnel events are triggered in the specified order.
• R: The number of unique users who triggered the starting event after exposure, then triggered the return event within the specified time window (nth day/week/month).

Formulas

The inputs in the previous section are then plugged into the following formulas:

• Unique conversions: (M/E) * 100
• Event totals: T/E
• Sum of property value: S/E
• Average of property value: A/E
• Funnel conversion, uniques: FM/E
• Funnel conversion, totals: FT/E
• Retention: R/E

Examples

For this example, assume the metric event has a numeric event property VALUE. This table is the chronological log of events coming into Amplitude:

In this example, the number of unique users exposed to the experiment—E in the list of notations above—is four (U1, U2, U3, U4). Of those, the number who triggered the metric event (M in the list) is two (U1 and U2). U5 doesn’t count, as they weren't exposed to the experiment.

The metric event was triggered three times, twice by U1 and once by U2. Again, U5 doesn't count.

The sum of all the metric events' property values is 30, and the sum of their average:

 ((5 + 10)/2 + (15)/1) = (7.5 + 15) = (U1 Avg + U2 Avg) = 22.5

With that information, you can plug these values into each of the formulas listed above:

• Unique conversions: (M/E) * 100 = (2/4) * 100 = 50%
• Event total: T/E = 3/4 = 0.75
• Sum of property value: S/E = 30/4 = 7.5
• Average of property value: A/E = 22.5/4 = 5.625

Funnel example

For this example, define the funnel as events ME1 and ME2, performed in that order. This table is the chronological log of the events coming into Amplitude:

Here, the number of unique users who triggered the events in the funnel in the given order—defined as FM in the list earlier—is two (U1, U2). U3, U4. and U5 didn’t qualify for the funnel: U3 didn’t trigger ME2, and U4 triggered ME2 out of order; U5 never triggered the exposure event, and thus isn’t included in the experiment at all.

The value of FT—defined as the total number of times all the events in the funnel are triggered in the specified order—here is three. U1 triggered ME1 → ME2 once, while U2 did it twice.

Knowing that, you can plug these values into each of the formulas listed above:

• Funnel conversion, uniques = FM/E = 2/4 = 0.5
• Funnel conversion, totals = FT/E = 3/4 = 0.75

Retention example

For this example, define retention as users who trigger a starting event after exposure, then trigger a return event within seven days (Day 7 retention). This table is the chronological log of events coming into Amplitude:

Here, the number of unique users exposed to the experiment (E) is four (U1, U2, U3, U4). U5 doesn't count, as they weren't exposed to the experiment.

For Day 7 retention, a user is counted as retained if they trigger the return event between seven days and seven days plus 24 hours from performing the starting event. The number of unique users who triggered the starting event after exposure and then triggered the return event within the Day 7 window—defined as R—is two (U1 and U2). U1 triggered the return event 7 days after the starting event (day 8 from exposure, which is 7 days after day 1). U2 triggered the return event 7 days after the starting event (day 9 from exposure, which is 7 days after day 2). U3 triggered the return event only 4 days after the starting event, which is outside the Day 7 window. U4 triggered the starting event but never triggered the return event. U5 never triggered the exposure event, and thus isn't included in the experiment at all.

Knowing that, you can plug these values into the retention formula:

• Retention = R/E = 2/4 = 0.5 (50%)