Use experiment briefs to design better experiments

Whether your company is just beginning its experimentation journey or has already established a culture of experimentation, using Experiment Briefs can help democratize experimentation, design better experiments, and creates transparency about the goals and next steps for each experiment.

Democratize experimentation

What do we mean by democratization? Democratization is the act of making something accessible to everyone. To democratize experimentation, organizations need to make two things accessible:

1. The ability to run experiments should be accessible to anyone in any part of the organization
2. Anyone should be able to quickly get access to the key facts and outcomes of every experiment

An excellent approach to democratize the key facts and outcomes of an experiment is to use an Experiment Brief.

Why should your team use an Experiment Brief?

Design better experiments

Running experiments is a fundamental component of the scientific method—scientific being the operative word. An experiment design is a crucial part of the process, but many teams often overlook this key step in the process. We’ll explore this in more depth in the next section.

Build transparency into your process

Without transparent documentation, it’s easy to rationalize decisions and fit them into a narrative after learning the facts, which leads to bad experimentation outcomes. By documenting clear next steps after an experiment runs, teams are much more likely to move forward based on what they learn from the experiment.

What does a good experiment look like?

Many practitioners assume that an experiment consists of starting with an idea, building a prototype, testing it, and then figuring out what to do next. Wrong.

A good experiment follows the skeleton of the scientific method. These core concepts carry over into the world of digital experiments, along with a few bespoke components. For the record, when I say “good experiment,” I am not referring to the experiment’s outcome. A “good experiment” speaks to a well-designed experiment.”

Here are the critical components of a good experiment design:

Planning (pre-experiment)

1. Document observations that clarify why we plan to run this experiment
2. Research to validate our observations. This research can be quantitative, qualitative, competitive insights, etc.
3. Build a hypothesis with clearly defined metrics or KPIs
4. Create alignment on your team’s next steps based on each possible outcome: whether each variant wins, or if the test does not reach statistical significance
5. Characteristics of the experiment set-up include the duration, audience, traffic split, critical threshold, and more
6. Clear articulation of what is being tested and modified. This could be a design change, a new feature or experience, and more
7. Confirm that the experiment can be measured

While the experiment runs

1. Analytics dashboards for ongoing monitoring

Post-experiment

1. Analyze the results
2. Articulate the outcome of the experiment and move forward with your pre-determined next steps.

The Experiment Brief

Now that we understand what good experiment design consists of, we can templatize our process to ensure that all the critical information is documented before the experiment begins, every time.

This makes it easy for anyone in your organization to learn about the experiment asynchronously which helps enable democratization.

Phase 1: Plan

Phase 2: Configure

Phase 3: Monitor

Phase 4: Analyze and Decide

Final thoughts

These Experiment Briefs can easily be categorized by date, segment, website section, and platform. They should also be accessible to everyone so that this institutional knowledge persists even if people come and go. Using Experiment Briefs can help your team supercharge your experimentation program by transparently documenting each experiment in order to democratize this information.

I’d love to hear your thoughts on this template and any modifications you would make. You can reach out to me on LinkedIn or Twitter, or check out my blog.

A link to the experiment brief template can be found by following this link → Experiment Brief Template.