Amplitude Experiment: A/B Testing and Feature Flagging Powered by Customer Behavior

Today we’re launching Amplitude Experiment, an experimentation and feature management solution powered by analytics and customer behavior. Experiment is the first solution to provide a complete, end-to-end workflow from hypothesis to analysis that empowers organizations to scale experimentation across teams and make bigger bets that drive growth.

It’s no secret that leading products like Stitchfix and Netflix have built extraordinary experimentation programs. They drive customer loyalty and business growth by consistently delivering the best customer experience faster than competitors. With the explosion of digital products and consumer choice, that’s more important today than ever before.

The challenge for digital teams lies in determining what the best experience is and serving it to the appropriate customer segment. A/B testing and feature flagging tools are designed for this very purpose. Yet many of these existing A/B testing tools sit disconnected from customer behavior, analytics, and teams’ workflows. Organizations that use them are forced to choose between limiting the scope of the bets they test or locking up technical resources to overcome challenges with targeting and identity.

That’s why we built Amplitude Experiment. Through the Digital Optimization System, experimentation and feature management are fully integrated with customer behavior and analytics. Now in the same system where you measure behavior and power personalization, you can run high-impact A/B tests and remotely configure experiences for any segment.

The complete experimentation workflow

When it comes to experimentation in your digital product, nothing is more important than the process. The process ensures that you have the highest chance for success—a result with a clear next steps—so that you don’t waste precious resources or steer your team down the wrong path.

In Experiment, that process is Design, Rollout, and Learn. Through those key phases, we compiled every step—from hypothesis development to targeting to measurement—into a single page. Not only is it easier to manage your experiments, it helps you ship better tests because you start and end with your customer data.

This is made possible by key functionality Experiment leverages from Amplitude’s Digital Optimization System and the Behavioral Graph:

• Behavioral & predictive data helps you target segments based on their actual or predicted behavior.
• Identity resolution serves users the same experience regardless of platform or authentication state.
• Proactive analytics uncovers the ripple effects of a test that occurs outside of statistical significance.

For auto industry disruptor SHIFT, “Amplitude Experiment is a game changer. Since it’s powered by customer behavior insights and analytics, we’re able to design better tests, learn from them faster and change the way we build and deliver experiences to our customers. Amplitude has been a great partner to Shift on product analytics, and we are excited to have been able to deepen that relationship through the new Experiment product.” — Adam Johnston, Director of Product

Design better tests

A big difference with Experiment compared to other tools is that we don’t start with the variant or feature flag. We start with experiment design.

The Experiment workflow walks you through the best practices of building an effective test. Start with the customer problem, outline supporting insights, and document your hypothesis. It’s all right there. You can even link to an existing chart or metric you use in Analytics.

We also found ways to eliminate double work. Managing and communicating experiment planning creates a lot of overhead for product managers. Through Notebooks, we created a single place to sync and share all of the details. Plus, it’s natively integrated with the test you’re running.

In the example above, we use analytics to identify an opportunity to drive increased engagement through social features. We’re able to tie our hypothesis to specific Amplitude charts and the whole Notebook is embedded in our test so anyone on the team can get up to speed quickly. That integrated experience is what makes Experiment fit seamlessly into teams’ workflows.

Roll out smarter, not harder

Even when tests are designed well, technical blockers can delay or even derail experiments. The most common ones that teams run into arise from targeting customer segments and managing user identity. With Experiment, we think you’ll be surprised with how easy we’ve made rollout. Not only does Experiment provide specific code snippets to initialize and evaluate variants, but targeting and identity management require no extra engineering work.

Through the Behavioral Graph, targeting segments by behavior doesn’t add extra work. Choose events these segments have performed, properties they have or just select a cohort. And if you have Amplitude Recommend, you can even target users based on their predicted behavior, like likelihood to purchase. In our example of testing new social functionality, we’re able to target users who take fewer than 5 classes a week and who have connected with friends—all in just a few clicks.

Experiment also leverages Amplitude’s identity resolution to natively merge users across devices and authentication states. Paired with behavioral targeting, it means engineers don’t have to run around piping user data in every time you want to run a test or insert a flag.

That’s also why we created Payloads. Payloads is a feature that passes variables into your product so you can remotely configure experiences without deploying new code. That can include everything from a different color palette to one-to-one recommendations you create in Amplitude Recommend. A great example of this would be determining the number of suggested friends to show in a social experience. Through payloads, you can configure that number remotely and even change it on the fly —no dev work required.

Finally, Experiment uses sequential testing so you can view results in real-time without compromising experiment integrity. And soon you’ll even be able to add experiments to Amplitude’s monitors so you can be notified of major changes in behavior, critical metrics or test performance. This end-to-end view of customer engagement gives you peace of mind that you’re not breaking anything and the confidence to take on bigger bets.

Learn beyond statistical significance

A lot of emphasis is placed on statistical significance in A/B tests—for good reason—but it’s never the whole story. With most A/B tests, stat sig is where analysis stops. With Amplitude Experiment, it’s where analysis begins.

You have the full power of product analytics at your fingertips. Not only can you cut results by any combination of cohorts or behaviors, but you can see the ripple effects that occur beyond the bounds of the test. Imagine knowing that a new social experience increased engagement overall, but for indoor cycling power users it had a dramatic impact on retention. By seeing the full impact of how behavior changes, it’s easier to determine your next steps or your next experiment.

This is possible without any extra technical work because Experiment automatically logs assignments for you. You get full visibility at the user level across all experiments and flags ever created and they don’t count against your data volume. Assignments instantly appear for use in analytics or future tests and they help you avoid collisions between experiences. It makes scaling experimentation across your organization manageable.

Make bigger bets that drive growth

The complete workflow in Amplitude Experiment is not just about improving your chances for success, it’s designed to help you take on bigger bets. What’s a bigger bet? A bigger bet is the difference between testing a new social experience versus testing a different location for the share button. Both may increase engagement, but if you can figure out the right experience it will unlock new possibilities, new value, and new growth. That’s what great experimentation programs enable, and that is what’s possible with Amplitude Experiment.

We’re excited to show you Amplitude Experiment and how you can use the A/B testing and feature flagging functionality to deliver the best experience to customers faster than ever before.

Explore Amplitude Experiment yourself. Request a demo today.