Experiments

Knowing what your users respond to best is tricky. To help with this challenge, Guides and Surveys works with Amplitude Experiment. When you install the Guides and Surveys SDK, you get everything you need to run experiments on your Guides and Surveys.

Manager or Administrator role required

Running an experiment on your guide or survey requires the Manager role at a minimum. For more information about how roles impact who can use Guides and Surveys, see Getting Started | Roles and Permissions.

Run an experiment

To add experimentation to your guide or survey, click the add experiment icon.

When you add an experiment to your guide or survey, Guides and Surveys controls the experience, and Experiment controls user targeting and distribution of each variant, depending on the type of experiment you choose.

Choose an experiment type

Your first decision involves the type of experiment you want to run. Guides and Surveys offers two options.

A/B test

Choose an A/B test and create two variants of the same Guide. Amplitude decides the winner based on the data it receives. Access results just like any other experiment you run in Amplitude.

Multi-armed bandit test

Choose a Multi-armed Bandit test for a smarter, more dynamic approach to testing. With multi-armed bandit experiments, the system allocates more traffic to the higher-performing variant in real-time, helping you optimize faster.

Configure variants

After you select an experiment type, Guides and Surveys adds two variants with autogenerated keys. To rename the variant, select it and click More options. Update the name, duplicate, or delete the variant here.

Complete experiment setup

Adding variants is only the first part of experimentation in Guides and Surveys. To ensure users see variants as they should:

Make sure the experiment is running. Define a goal, review targeting, and click Start Experiment. For more information, see Manage the experiment.
If an specific user doesn't see a variant, ensure they're part of the experiments target audience.
If a user sees one variant, they should continue to see that variant. Navigate to Users > User Profiles. Search for the user and open their profile. Look at the Guide and Survey tabs to view which experiences they've seen.

Manage the experiment

Click Manage Experiment to open the experiment editor in a new tab. The experiment assumes the name of your guide or survey, and contains any variants you added.

Updating variants

Variant names stay in sync between your guide or survey and the experiment when you save the guide or survey.

For more information about working with experiments, see Feature Experiment

End the experiment

If you're ready to conclude experiment, navigate to the experiment's configuration page, click Stop Experiment. and choose one of the following:

Option	Description
Complete experiment	Declare a winner. If one of the variants is the winner, Amplitude archives the losing variant, and publishes the winning variant. If you select the control, the experiment returns to a state as if you just created it, and sets the rollout of the control variant to 100%.
Continue running experiment	The experiment remains live, enabling you to collect more data.

Insights

The Insights tab is the dashboard where you track how users engage with your guide or survey. Monitor trends in views and completions over time, and track how different variants perform relative to one another.

Time-based analysis

Track guide and survey engagement trends over predefined time periods.

Hourly
Daily
Weekly
Monthly
Quarterly

With these presets, see when users are most likely to engage with the guide or survey and if engagement changes after, for example, a new product release.

Date range selection

Select a predefined range based on the unit of time, or click the calendar icon to define your own range. Choose from:

Rolling window (Last # complete days and today)
Since date
Between dates

Use the advanced settings to:

Add a date offset to a rolling window
Exclude Today
Enable Time Range

Performance overview

The top-most chart on the Insights tab is the Performance Overview. Here, Amplitude displays high-level metrics that track how your guide or survey is performing:

Metric	Description
Guides / Surveys viewed	The number of times the guide or survey was shown to users.
Guides / Surveys completed	The number of times the guide or survey was completed by users.
Trend graph	Tracks the view or completion count over the time range specified in the date range selector.