When building and scaling a successful experimentation program, there are many strategic decisions to make such as ideal team structure, metric definitions, statistical models, and technology solutions. Another key consideration is if your team is better suited to rely on web experimentation or if they are ready for product-led experimentation.
Before we discuss which approach is better for your company, let’s start with some history and define some key concepts.
Web experimentation as we know it today first became widely available around 2010 when companies like VWO and Optimizely launched visual editor tools that, enabled users to build A/B tests without writing any code using a “what you see is what you get,” or “WYSIWYG,” interface. While these (and similar products) have matured over time, their basic workflow remains the same:
- Use a WYSIWYG editor to preview a web page and click on elements to make visual changes. The testing tool converts your changes into JavaScript.
- Add a JavaScript tag to your site. The tag will load when your page loads and retrieve your experiment changes from the vendor’s server.
- The downloaded JavaScript executes in the browser, modifying the Document Object Model (DOM) to render the experiment changes for the visitor.
These products will always be limited to use cases in the browser. Still, not (sometimes you have to write custom JavaScript and deliver it via the experimentation tool).
Product-led experimentation, on the other hand, uses a dramatically different approach to deliver experiments. Product-led experimentation solutions do not rely on a visual editor. Instead, they deliver experiment decisions anywhere you run a test in your code and across your tech stack using APIs and SDKs.
So, how do you know which approach your team should take? Well, it really depends on what you are trying to accomplish. Here are the 5 signs that your team might be ready for product-led experimentation.
1. You want complete control over each test variant
If your team is experimenting on marketing pages and focused on optimizing webpage copy, layouts, or imagery, web experimentation may be a good approach. Often these teams build their experiences in a CMS or other site-building tool and don’t have easy access to developer resources.
However, for product and engineering teams who ship features using code, product-led experimentation will be a better fit. Using this approach, they have complete control over the experiment variants and can build each variant using the programming languages they use every day.
2. You need to roll out the experiment using software development best practices
When working with web experimentation, the process of launching a winning variant is another step in the process. Often it requires additional collaboration with your developers to scope out the feature and re-build it in code. This adds significant lag time to the process due to waiting on sprint schedules and for teams to work through their existing backlog. This is a key pain point when you have to use feature flags and experiments managed in different places.
With product-led experimentation, it is extremely simple to convert a winning variant to a feature flag. Then, it’s effortless to launch the flag using a progressive rollout which minimizes risk all while having the same monitoring and analytics capabilities as the experiment itself.
3. You need to test more complex use cases
As mentioned, web experimentation is limited to use cases that run in the browser which is why web experimentation is ideal for conversion rate optimization use cases.
But, for teams with digital products like algorithms, APIs, or server-driven experiences, product-led experimentation provides these teams the ability to test multiple variants and deliver the best experience to the user. Teams use product-led experimentation for diverse use cases like price testing, sorting algorithms, and onboarding flows. These use cases require teams to run experiments on the actual engineering or business logic, rather than just the presentation layer.
4. You need to test across platforms and devices
While web experimentation is limited to the web browser, product-led experimentation unlocks the ability to run experiments across any user touchpoint including the browser, phone, tablet, smartwatch, or API. And some solutions, like Amplitude Experiment, ensure the same user sees the same variant, even as they move between devices or log in part-way through a session.
5. You need a highly performant experience
Web experimentation will always require loading additional JavaScript in the critical path when delivering a browser-based experience to your user, which can harm page performance.
Product-led experimentation offers much lighter JavaScript tags for use cases that still require client-side testing. But, it also opens up opportunities for zero-latency server-side experimentation where network calls to a vendor can be removed from the critical path and bucketing can happen on your own servers. Product-led experimentation offers much more implementation flexibility providing better optionality when considering these key performance trade-offs.
Ultimately, the right experimentation approach for your team depends on what you are trying to optimize, what resources you have available, and which trade-offs make the most sense for your business.
If you want to learn more about how ’s approach to product-led experimentation, .