Use Experiment Briefs to Design Better Experiments

Experiment briefs make it possible to design better experiments and democratize access to critical information about each experiment.

February 2, 2023
Image of Bhavik Patel
Bhavik Patel
Founder and Managing Director, CAUSL
Experiment brief

Whether your company is just beginning its experimentation journey or has already established a culture of experimentation, using Experiment Briefs can help democratize experimentation, design better experiments, and creates transparency about the goals and next steps for each experiment.

Democratize experimentation

What do we mean by democratization? Democratization is the act of making something accessible to everyone. To democratize experimentation, organizations need to make two things accessible:

  1. The ability to run experiments should be accessible to anyone in any part of the organization
  2. Anyone should be able to quickly get access to the key facts and outcomes of every experiment

An excellent approach to democratize the key facts and outcomes of an experiment is to use an Experiment Brief.

Why should your team use an Experiment Brief?

Design better experiments

Running experiments is a fundamental component of the scientific method—scientific being the operative word. An experiment design is a crucial part of the process, but many teams often overlook this key step in the process. We’ll explore this in more depth in the next section.

Build transparency into your process

Without transparent documentation, it’s easy to rationalize decisions and fit them into a narrative after learning the facts, which leads to bad experimentation outcomes. By documenting clear next steps after an experiment runs, teams are much more likely to move forward based on what they learn from the experiment.

What does a good experiment look like?

Many practitioners assume that an experiment consists of starting with an idea, building a prototype, testing it, and then figuring out what to do next. Wrong.

A good experiment follows the skeleton of the scientific method. These core concepts carry over into the world of digital experiments, along with a few bespoke components. For the record, when I say “good experiment,” I am not referring to the experiment’s outcome. A “good experiment” speaks to a well-designed experiment.”

Here are the critical components of a good experiment design:

Planning (pre-experiment)

  1. Document observations that clarify why we plan to run this experiment
  2. Research to validate our observations. This research can be quantitative, qualitative, competitive insights, etc.
  3. Build a hypothesis with clearly defined metrics or KPIs
  4. Create alignment on your team’s next steps based on each possible outcome: whether each variant wins, or if the test does not reach statistical significance
  5. Characteristics of the experiment set-up include the duration, audience, traffic split, critical threshold, and more
  6. Clear articulation of what is being tested and modified. This could be a design change, a new feature or experience, and more
  7. Confirm that the experiment can be measured

While the experiment runs

  1. Analytics dashboards for ongoing monitoring


  1. Analyze the results
  2. Articulate the outcome of the experiment and move forward with your pre-determined next steps.

The Experiment Brief

Now that we understand what good experiment design consists of, we can templatize our process to ensure that all the critical information is documented before the experiment begins, every time.

This makes it easy for anyone in your organization to learn about the experiment asynchronously which helps enable democratization.

Phase 1: Plan

Observations and Insights: What observations led to this experiment? This research can be quantitative, qualitative, competitive insights, user research, and more.
Define the type of experiment: Hypothesis test or a Do No Harm test
Hypothesis We observed: Document the qualitative, quantitative, or competitive insight  
By: Introducing or modifying the independent variable  
We expect: Define the change in behavior you expect  
Leading to: State the expected impact on your dependent variable  
For Do No Harm Tests
Metrics: When you introduce major changes, like a new experience in your product, use a Do No Harm test for both primary and guardrail metrics to detect any negative impacts on your critical metrics Primary: This could be a conversion rate  
Guardrails (optional)  
For Hypothesis Tests
Metrics Primary: This could be a conversion rate  
Secondary (optional)  
Guardrails (optional)  
Total Sample Size   
Minimum Detectable Effect Primary  
Guardrail (optional)  
Experiment Event Name  
Experiment Event Parameters  
Actions: What actions will you take based on each of the following outcomes of the test? Win  

Phase 2: Configure

Variation Designs Variation 1: Define how it will differ from the control group. You should also link to any relevant designs  
Variation 2 (optional)  
Allocation Split: This could be 50/50  
Audience cohort: Are we targeting existing customers, new users, etc.?  
Section of site: In which section of your product is the test being conducted?  
JIRA Tickets & Analysis (links) Analytics JIRA Link
Tech JIRA Link
Pre-test Analysis Link
Post-test Analysis Link

Phase 3: Monitor

Planned test date From: [MM/DD/YYYY] To: [MM/DD/YYYY]
Duration (days)
Ensure tracking in analytics is working: Ensure that: (1) The event fires on relevant platforms, devices, and user groups, (2) You have configured consistent event names for control and variant across all platforms, (3) The exposure event fires when users are exposed to the experiment 

Phase 4: Analyze and Decide

Analyze and Decide
Analysis: Document the outcome of the experiment  
Decision: Decide next steps based on your planning phase.   
Monitor ongoing performance: Continue to analyze behavior post-experiment  

Final thoughts

These Experiment Briefs can easily be categorized by date, segment, website section, and platform. They should also be accessible to everyone so that this institutional knowledge persists even if people come and go. Using Experiment Briefs can help your team supercharge your experimentation program by transparently documenting each experiment in order to democratize this information.

I’d love to hear your thoughts on this template and any modifications you would make. You can reach out to me on LinkedIn or Twitter, or check out my blog.

A link to the experiment brief template can be found by following this link → Experiment Brief Template.

Contact sales
About the Author
Image of Bhavik Patel
Bhavik Patel
Founder and Managing Director, CAUSL
Bhav is the founder of CAUSL, a Product Measurement Consultancy that helps product organizations connect the dots between their product initiatives and company goals. Bhav has previously held "Director and Head of Product Analytics" roles at companies like Gousto, MOO, PhotoBox, and most recently, Hopin. Bhav also runs London’s biggest conversion rate optimization, analytics, and product meetup called CRAP Talks.