Discovery plans and results reports

Over time, I've come to appreciate the value of a clear documentation template as a vehicle for creating a particular culture on your team. I strive to build teams that operate with minimal process and required steps. Zero process results in chaos. Lots of process results in your team wasting time on low value activities. The key is finding the right amount of process that results in a high performing team.

Here, I'll make the case for introducing the following process for growth teams:

  1. Write a discovery plan before running an experiment or gathering data from users (e.g. running an A/B test or running a survey).
  2. Write a results report after the data has been gathered.

Honestly, I've only seen teams use documents like these inconsistently and the scope has just been limited to just A/B tests. I've regretted not making these required many times. I also have seen non-A/B test discovery activities fail to produce usable results because of an unanticipated issue that likely would have been caught had we written down the plan beforehand, so I imagine many of the benefits I've seen using these documents for A/B tests extend to other types of research/discovery activities as well.

Discovery Plans

This document is intended to be a clear articulation of why you are doing this discovery activity, what you are going to do, and what you will learn from it. It doesn't have to be longer than one page. The primary goal is to help you distill and clarify your own thinking. A close second goal for the document is to create alignment across the cross-functional growth team. Is the data scientist confident that we will be able to make the types of statements we want to make at the end given the experiment design? Is engineering confident that they can actually implement this experiment? The act of writing down the plan tends to actually improve the plan by forcing the team to think it through, even if it's not shared with anyone outside of the team!

That said, in my experience, the positive effects are amplified quite a bit when these plans are shared outside of the team as well. Instead of scheduling meetings with many stakeholders to explain the plan and gather feedback, teams can just share a document like this for asynchronous feedback. Fast forward a couple of years, team members will have come and gone. This document helps both long-tenured team members remember what was done long ago, as well as helping new members get up to speed quickly.

The key elements of this document are:

  1. Learning Objective: What are you hoping to learn? What will you do with the data you collect?
  2. Methodology: Are you running an A/B test? A survey? An unmoderated usability study? Customer interviews? Historical data analysis? What exactly are you going to do? What changes are you making to the user experience (if any)? Are there multiple variations? How do they differ? What are the exact survey questions you are going to ask? Provide details!
  3. Audience: What segments of users are you including in this A/B test or research study? Why?
  4. Data Collection and Metrics: What data are you going to collect? What are you going to measure? For an A/B test, what is your decision making metric and what are your guardrail metrics?
  5. Risks: How risky is this for the business? What could go wrong?
  6. Timing: When will you start? When will you stop?

Results Reports

This document is published after you've gathered and analyzed your data. It's primary goal is to encourage you to analyze your raw data, interpret it, and learn from it. It should be shared widely and stored in an easily accessible, searchable archive for future reference.

If you don't archive these and make them easy to find in the future, you are doomed to re-run tests that you already ran because no-one can remember the result. This first-party research is one of your most valuable assets as a business and can be a strong strategic differentiator, treat it as such.

The key elements of this document are:

  1. Summary: What is a concise summary of what you found?
  2. Methodology: How did you go about gathering your data? Did it differ at all from your discovery plan?
  3. Results: Share the raw data you gathered
  4. Interpretation: What do these observations mean in context? How does the team interpret them? What did you learn?
  5. Next Steps: What are you going to do with this information?
  6. Links: Where can readers find the raw data and any analysis that was done on it?