Whether your company is just beginning its experimentation journey or has already established a culture of experimentation, using Experiment Briefs can help democratize experimentation, design better experiments, and creates transparency about the goals and next steps for each experiment.
Democratize experimentation
What do we mean by democratization? Democratization is the act of making something accessible to everyone. To democratize experimentation, organizations need to make two things accessible:
- The ability to run experiments should be accessible to anyone in any part of the organization
- Anyone should be able to quickly get access to the key facts and outcomes of every experiment
An excellent approach to democratize the key facts and outcomes of an experiment is to use an Experiment Brief.
Why should your team use an Experiment Brief?
Design better experiments
Running experiments is a fundamental component of the scientific method—scientific being the operative word. An experiment design is a crucial part of the process, but many teams often overlook this key step in the process. We’ll explore this in more depth in the next section.
Build transparency into your process
Without transparent documentation, it’s easy to rationalize decisions and fit them into a narrative after learning the facts, which leads to bad experimentation outcomes. By documenting clear next steps after an experiment runs, teams are much more likely to move forward based on what they learn from the experiment.
What does a good experiment look like?
Many practitioners assume that an experiment consists of starting with an idea, building a prototype, testing it, and then figuring out what to do next. Wrong.
A good experiment follows the skeleton of the scientific method. These core concepts carry over into the world of digital experiments, along with a few bespoke components. For the record, when I say “good experiment,” I am not referring to the experiment’s outcome. A “good experiment” speaks to a well-designed experiment.”
Here are the critical components of a good experiment design:
Planning (pre-experiment)
- Document observations that clarify why we plan to run this experiment
- Research to validate our observations. This research can be quantitative, qualitative, competitive insights, etc.
- Build a hypothesis with clearly defined metrics or KPIs
- Create alignment on your team’s next steps based on each possible outcome: whether each variant wins, or if the test does not reach statistical significance
- Characteristics of the experiment set-up include the duration, audience, traffic split, critical threshold, and more
- Clear articulation of what is being tested and modified. This could be a design change, a new feature or experience, and more
- Confirm that the experiment can be measured
While the experiment runs
- Analytics dashboards for ongoing monitoring
Post-experiment
- Analyze the results
- Articulate the outcome of the experiment and move forward with your pre-determined next steps.
The Experiment Brief
Now that we understand what good experiment design consists of, we can templatize our process to ensure that all the critical information is documented before the experiment begins, every time.
This makes it easy for anyone in your organization to learn about the experiment asynchronously which helps enable democratization.
Phase 1: Plan
| Plan |
| Observations and Insights: What observations led to this experiment? This research can be quantitative, qualitative, competitive insights, user research, and more. |
|
| Define the type of experiment: Hypothesis test or a Do No Harm test |
|
| Hypothesis |
We observed: Document the qualitative, quantitative, or competitive insight |
|
| By: Introducing or modifying the independent variable |
|
| We expect: Define the change in behavior you expect |
|
| Leading to: State the expected impact on your dependent variable |
|
For Do No Harm Tests
|
| Metrics: When you introduce major changes, like a new experience in your product, use a Do No Harm test for both primary and guardrail metrics to detect any negative impacts on your critical metrics |
Primary: This could be a conversion rate |
|
| Guardrails (optional) |
|
| For Hypothesis Tests |
| Metrics |
Primary: This could be a conversion rate |
|
| Secondary (optional) |
|
| Guardrails (optional) |
|
| Total Sample Size |
|
| Minimum Detectable Effect |
Primary |
|
| Guardrail (optional) |
|
| Experiment Event Name |
|
| Experiment Event Parameters |
|
| Actions: What actions will you take based on each of the following outcomes of the test? |
Win |
|
| Lose |
|
| Flat |
|
Phase 2: Configure
Configure
|
| Variation Designs |
Variation 1: Define how it will differ from the control group. You should also link to any relevant designs |
|
| Variation 2 (optional) |
|
| Allocation |
Split: This could be 50/50 |
|
| Audience cohort: Are we targeting existing customers, new users, etc.? |
|
| Platforms |
|
| Section of site: In which section of your product is the test being conducted? |
|
| JIRA Tickets & Analysis (links) |
Analytics JIRA |
Link |
| Tech JIRA |
Link |
| Pre-test Analysis |
Link |
| Post-test Analysis |
Link |
Phase 3: Monitor
| Monitor |
| Planned test date |
From: [MM/DD/YYYY] |
To: [MM/DD/YYYY] |
| Duration (days) |
|
| Ensure tracking in analytics is working: Ensure that: (1) The event fires on relevant platforms, devices, and user groups, (2) You have configured consistent event names for control and variant across all platforms, (3) The exposure event fires when users are exposed to the experiment |
|
Phase 4: Analyze and Decide
Analyze and Decide
|
| Analysis: Document the outcome of the experiment |
|
| Decision: Decide next steps based on your planning phase. |
|
| Monitor ongoing performance: Continue to analyze behavior post-experiment |
|
Final thoughts
These Experiment Briefs can easily be categorized by date, segment, website section, and platform. They should also be accessible to everyone so that this institutional knowledge persists even if people come and go. Using Experiment Briefs can help your team supercharge your experimentation program by transparently documenting each experiment in order to democratize this information.
I’d love to hear your thoughts on this template and any modifications you would make. You can reach out to me on LinkedIn or Twitter, or check out my blog.
A link to the experiment brief template can be found by following this link → Experiment Brief Template.
4 -minute Read,
Posted on February 2, 2023
