In today’s competitive landscape, product-led experimentation is more than a good idea—it’s a necessity for product and engineering teams to quantify their impact. But teams need to cultivate a culture of experimentation and infuse it into every phase of the product development process to build with confidence.
We’ve teamed up with Reforge to create a comprehensive guide to product-led experimentation, featuring insights and advice from experts in the field. The guide will help product and engineering teams establish experimentation programs and unlock the full potential of their products. In this post, we’ll give you an overview of what to expect in the guide.
Download the full guide—Build with Confidence: Your Guide to Scaling Product-Led Experimentation—to learn how to build an experimentation culture, understand the practice of experimentation, prioritize your experimentation roadmap, and more.
- Experimentation is a necessity for product teams to help them to quantify the impact of their work, clarify their product strategy, and de-risk product investments.
- Experimentation also helps product leaders tie their work to wider company goals like increasing monetization opportunities or customer retention.
- Building a culture of experimentation involves a fundamental mindset shift as well as concrete systems and processes to support your velocity of learning, not just shipping fast.
- To begin your experimentation journey, start small with one team before scaling up to the entire organization.
- Secure an executive champion who can provide resources and assist in creating channels for open communication and collaboration across departments.
- Learn more about setting up your first experimentation programs in the product-led experimentation guide.
Why experimentation matters
In today’s competitive environment, experimentation is not optional for product teams. It’s what helps teams build successful, innovative products and tie their work to the wider organization.
Saleem Malkana, Executive in Residence at Reforge, explains that experimentation helps product teams measure the impact of their work and improve product strategy. “The best product teams know that what truly matters is not just shipping those outputs, but rather what outcomes they drive,” he explains.
To measure the impact of product changes, teams need to develop hypotheses, run experiments, and evaluate the results. They can use the results of the experiments to inform their future product strategy.
Testing is also a way to de-risk product changes. Product teams can assess the impact a product decision will have before rolling it out to all users—which could lead to expensive mistakes.
Wil Pong, Head of Product for Amplitude Experiment, argues that product-led experimentation delivers faster innovation without disrupting engineering’s development cycles using more surface-level tooling, like WYSIWYG approaches.
In the past, only big companies like Netflix could offer tailored experiences and experiment with different product interfaces or recommendation engines. Now, with new applications like Amplitude Experiment, organizations of all sizes can iterate and refine their products in real time based on what users need.
Product-led experimentation enables product teams to deliver highly targeted experiments across their entire products using feature flags, which allows teams complete control to turn experiences on or off. This allows teams to “deliver highly targeted experiments and feature releases using the same processes and technology that power the rest of [their] product development,” which will ultimately drive rapid innovation without disrupting technical teams, explains Wil.
Experimentation is also crucial for product leaders, given recent shifts in accountability. Bhavik Patel, founder and Managing Director of CAUSL, explains that organizations have moved from output-based to outcome-based approaches, placing more scrutiny on product leaders. While it’s difficult to measure the ROI of product teams (organizations typically use headcount or engineering hours), tracking product impact through experimentation helps the product team tie their work to wider company goals.
Learn how to create and execute a strategic experimentation system with Reforge’sExperimentation + Testing course.
What does it mean to build a culture of experimentation
An experimentation culture should involve all parts of an organization. To establish that culture, organizations need to go through a mindset shift and build concrete systems and processes to support their velocity of learning, not just shipping fast.
Saleem explains that a culture of experimentation is not purely technical. It’s a combination of:
- People: Team members are data-driven.
- Integrated processes & communication: Experimentation is part of product development processes with healthy communication across teams.
- Technology: The organization uses a platform that enables experimentation.
Ideally, experimentation will exist at every level of the organization, with a focus on the squad level—that’s what creates the foundation for agility and adaptability. Tests should also be part of day-to-day operations, so teams get a steady stream of insights they can use to improve the product.
To create a culture of experimentation, teams should develop a mindset that embraces curiosity and “failure” since both are mechanisms for learning. To generate momentum around experimentation, it’s also important to secure an executive champion.
An executive that buys into experimentation will help provide resources like time, budget, and tools. They’ll also assist in creating channels for open communication and collaboration across departments.
How to begin your experimentation journey
Implementing experimentation programs in an organization takes time. Here are some best practices for when you’re starting.
Start small and scale up
The ultimate goal is to roll out experimentation across an organization. However, organizations should start small with low-risk tests across one team before they scale.
Saleem recommends establishing one “center of excellence:” a single team who should validate the basic experimentation stack and build up from there. Elena Verna, Growth Advisor and Interim Growth Exec, explains, “a single team needs to first learn how to test, how to learn, and how to win.”
She continues: “Once that team has developed experimentation skills and mindset, they should work on democratizing access to testing and learning. That way, the entire organization can learn how to win with experimentation.” Ultimately, team members across all departments should be able to run tests to inform their day-to-day decisions.
Bhavik suggests teams start with low-risk tests: “I wouldn’t advise starting with a complex pricing experiment, for example, but it’s a great goal to work toward.”
He also warns against getting hung up on unrealistic numbers. The majority of companies shouldn’t try to run thousands of experiments. Instead, ”the goal should be running more [tests] than they did last quarter or year,” he says.
Ideate, validate, prioritize
Running unnecessary experiments wastes resources, so teams must be selective about choosing which experiments to run.
Start your ideation with a metric tree (a diagram that shows how all elements of a product ladder up to company goals) or a sun diagram (an overview of your organization’s workstreams). Reviewing those diagrams with give you an idea of where to focus your experimentation efforts.
To test ideas, first validate the test concepts to identify which ideas are worth pursuing. To validate, you can use painted door tests, desk research, or examine results from similar experiments.
Then, consider the time to insights from each potential test. Elena emphasizes that teams should prioritize the velocity of learning and consider how long it takes to build an experiment, not just run it. Saleem also suggests breaking big challenges into smaller testable hypotheses to get a faster feedback loop.
Other factors to take into account include:
- The volume at different points in your product because you need enough traffic to run tests
- Whether the test you want to run can be measured with the analytics tools you have
Once you’ve validated your test ideas, prioritize which experiments to run based on:
- Your testing capacity
- How much impact you expect the test to have
- How confident you are that it will have the desired impact
- The effort involved in running the test
Design your experiments using best practices
Bhavik explains that well-designed experiments are rooted in the scientific method. They can be broken down into the following stages: planning before the experiment, monitoring during the experiment, and analyzing after the experiment.
Once you’ve validated your experiment ideas and chosen which tests to prioritize, you have to plan out your experiments. Document the reasons for running the experiment. Clearly articulate what is being tested and modified—that could be a design change or a new feature or experience.
Build a hypothesis and set clearly defined product metrics or KPIs for your experiment. Decide on the next steps based on the possible outcomes. Determine what happens when each variant wins or if the test doesn’t reach statistical significance.
Dive deeper into pre-experiment planning in the full guide.
During the experiment, create an analytics dashboard to continuously monitor the test. After the experiment, analyze the results and articulate the outcome. Then, the team can move forward with the predetermined next steps.
Build with confidence
For more guidance on how to run successful experimentation programs, download the product-led experimentation guide.
In the guide, you’ll also learn how to:
- Decide whether to build or buy your experimentation platform using a framework from Chad Sanderson
- Measure and evaluate your tests
- What critical capabilities you need to scale product-led experimentation
- How to use Amplitude Experiment to deliver experimentation at scale
