Analyze A/B test results in Amplitude

A/B testing runs controlled, randomized experiments to improve a website or application metric. With Amplitude's AB Test View, you can measure the impact of your experiments by comparing how each experiment group behaves in your application.

For example, you can show two different onboarding flows to different groups of new users, then use the results to identify the flow that drives more users to complete onboarding. You can also test different checkout flows to see which generates more sales.

Before you begin: Instrument your experiments

Instrument your experiments before you analyze results. Amplitude recommends using user properties to associate a user with a given experiment variation. User properties reflect traits about each individual person using your product. Use them to segment your analysis in Amplitude Analytics.

Two main methods update a user property:

1. SDKs and HTTP API: Update user properties on event action.

   • How: Send user properties with each event through Amplitude's SDKs or HTTP API.
   • Pros: User properties take effect when your app sends the event. The properties stay with the user for all subsequent events until you explicitly update the values.
   • Cons: These events count toward your monthly event volume. The events also count users as active users by default. Mark any A/B testing-related events as inactive events.

2. Identify API: Update user properties without sending an event.

   • How: Amplitude's Identify API updates a user property without sending an event.
   • Pros: Updates a user property asynchronously without sending an event. The update doesn't affect your monthly event volume count.
   • Cons: The user property doesn't take effect until the user takes an action. This isn't an issue for most experiments. The delay may affect experiments that track whether inactive users return to your application.

For example, suppose you're trying to bring users who have been inactive for more than seven days back to your app, and you're testing whether an email drives that return. If you use the Identify API to update a user property, the update only applies to users who return and trigger an event in your application. If a user remains inactive after receiving the email, the user property doesn't apply to that user. As a result, that inactive user isn't part of the experiment group that received the email, because the user property never attached. In situations like these, update user properties on an event action instead (for example, an event called Email Sent).

Learn more about how Amplitude syncs user properties.

How many user properties should you send?

Amplitude users tend to take one of two approaches when instrumenting split tests:

Use one user property per experiment

All user properties arrive as key-value pairs. This approach sets the experiment name as the key and all variations of the experiment as the potential values.

User Property Key: Experiment 1 User Property Value: variation_a

Pros: You can select experiments to segment by from the user segmentation tab. Cons: The list of user properties can grow long, depending on the number of active experiments.

Use one user property for all experiments

All user properties arrive as key-value pairs. This approach sets the key to Split Tests (or something similar) and stores the values in an array.

User Property: Split Tests User Property Value: [experiment_1_value, experiment_2_value]

You can segment on the user property Split Tests by selecting the appropriate value or test group in the chart's segmentation module.

Pros: You have only one user property related to split testing (rather than one per experiment), so your user property list stays manageable in the dashboard. Cons: Arrays can't exceed 10,000 characters when you use append or prepend. If an array exceeds this limit, Amplitude doesn't record any characters past the threshold.

Amplitude also offers a full integration with Optimizely that updates user properties for each experiment.

Viewing results in Amplitude

Review the results of your split tests after user properties update for each experiment group. Use the AB Test View to conduct this analysis.

Compare the activity between experiment groups in the segments module of the chart control panel. To do this, add your experimental groups.