Test-Driven Product Management at an Early-Stage Startup

This is a guest post in our Product Innovator Series from Jennelle Nystrom, product manager at Farmstead.

Today, it’s common to hear people talk about testing early-stage products like it’s a bad thing.

“Use your gut to make decisions,” they say, as if using data and running experiments are guaranteed to do nothing more than take your business on a random walk up to your nearest local maxima.

You can’t build a better product just by A/B testing button colors, no. You can’t do it by making small incremental changes in your product’s presentation layer. To build a better product, you need to:

• Understand who your ideal customer is.
• Validate or disprove your assumptions.
• Clarify and test your company’s values.

Data can help you do all of this faster and more rigorously.

Without data, everything from the makeup of your customer personas to the copy on your homepage becomes a matter of taste, subject to just one person’s will. Every change is often a time-consuming debate.

With data, you can make better decisions faster and improve adoption and activation in your product while staying true to the values that define your business and your company.

1. Early On, Nothing Is Sacred

Having an opinion about your product is important—otherwise, you’re not going to have a product—but you shouldn’t model your product off you, or your team, or even your early customers. It makes it too easy to start building in the wrong direction for the long-term. Instead, you should be testing many different ideas of what your product is and who it’s for.

We learned this first-hand doing customer development for Farmstead, for which there are a ton of different use cases. There are people who shop:

• Once a month
• Once a week
• Once a day
• Only for produce
• Only for prepared goods
• Only for snacks
• For kids
• For parents
• For themselves
• On a paleo diet
• On a vegan diet

And the list goes on. Grocery shopping may have a uniquely high number of dimensions and preferences, but the rule is generally applicable—building a valuable product means learning first-hand what your customers want, not guessing or papering them over with your or your team’s preferences. You do this through testing.

Since we didn’t have a clear idea of our customer when we started, and we knew it, we could test as if everything was provisional and (almost) nothing was sacred.

We could roll out new copy in one day, change our landing page, create a new product grid, add a column to our sidebar—all without extended planning or conversation. Then, we could use the actual reactions of our customers to understand whether the changes we’d made were good.

These kinds of changes can occupy a lot of your time when you don’t have a test-first culture. When you account for all the communication overhead involved in debating changes and all the clashing tastes of people in different roles and responsibilities, you can end up spending weeks just clearing basic tweaks to your site. When you have a culture of experimentation, however, it’s the customers and their behavior that ultimately decide the worth of a change.

Speed is crucial here. The faster you can roll tests out and learn from them, the better you can perform as a team.

2. Test Every Day

Testing is pointless unless you’re measuring product metrics that really matter. It’s easy to optimize the presentation layer of your product, turn a few buttons different colors, and get a small lift in conversion rates.

It’s harder, but more worthwhile in the long-term, to use testing to build a better, more valuable product and improve the user experience.

We’ve mapped out every conceivable path users can take to discover Farmstead, sign up, and place their first order—from email to social—and we run tests everywhere in order to improve the product. Every single day, we make changes to see what helps and what works to get people to order from Farmstead, or come back to Farmstead if they don’t place an order the first time, or come back and purchase from us again.

A single test is often not going to tell you much, which is why it’s key that you run them every day.

While working on our onboarding experience, for instance, we knew that we wanted to get people from the Farmstead landing page to their first order as quickly as possible. Sampling the service and seeing the quality of the food was crucial to people’s longer-term retention.

We launched a whole series of tests, all designed to shorten the amount of time that it took to get people to their first order.

Every day, we checked in on the tests we’d run the previous day. We looked at what had worked and what didn’t. Doing this, we were able to bring the total time from landing-page-to-first-order down from an average of 45 minutes to 15 minutes within a few weeks.

Running a single test at a time or waiting too long in between tests would have made this take way too long to figure out. When you make daily change a part of your culture, you can sit down every morning and go through the tests you ran the day before and see what’s working and what’s not. Then, if something is working, find out why, and use that to add value to your product or develop a second test to go even deeper.

3. Your Values Are Your Guardrails

Testing is not the end-all-be-all of product development. Testing requires values, without which it will only give you short-term optimizations and not actually drive more value for your customers.

Say you program a robot to increase your product’s conversion rate by any means necessary. It starts spamming your customers with discount code emails and barraging them with push notifications. It covers your website in popovers. In the case of Farmstead, it might start working with a giant industrial food supplier so it could cut prices and decrease delivery times. It would probably succeed in the (extremely) short term, but in the long term it would ruin the customer experience.

The fact that some of these spammy sorts of tactics might work does not make them something that your business should be doing. Our customers rely on us to put food on their tables every week. We don’t send more than one email a day because we take that responsibility and people’s time seriously.

Your values give you constraints around the kind of messaging and marketing you can test. However, those constraints can be creative, too. One of our core values is saving people time and money on their grocery shopping. That means that when people are on our site, the shopping experience should feel frictionless, and easier than going to the grocery store and wandering through the aisles.

When we used session replay to analyze how people were actually navigating our site and checkout, we realized that we weren’t doing the best job at that. We built a new checkout flow based on some of the behavior we’d observed, and when we tested it we found that it increased our conversion rate by 20x.

Don’t be a robot—make sure your testing, like everything else, lives up to the values that you’ve enshrined as an organization and that your customers care about.

Test-Driven Product Management

Being test-driven and data-informed does not mean being uncreative in how you build product—to the contrary, effectively testing and using data requires that you have the creativity and risk threshold necessary to:

• test against and with your values as an organization,
• abandon all your preconceived customer and product notions, and
• move faster than feels comfortable.

Ultimately, testing means acknowledging that you, as the product manager, do not have all of the answers. You have intuition, and you have your experience, but your notions about what your customers want—they’re just notions. They’re not true or false until you put them to the test.