The six things you need from a great product data scientist and how to find them
Product Analytics is one of the most high leverage roles in a company that’s investing in software development. Having millions of touch points with your customers gives you a plethora of opportunities to learn from mistakes faster and build the conviction to take moonshots. However, most companies simply don’t have enough data literacy on their team to interpret the data available to them. This often leads to situations where product managers make decisions based on convenient anecdotes or become too paralyzed to take risks.
Product analysts can have huge impact in such teams by helping put together a vocabulary for how to understand user behavior and tie it to the business outcomes being pursued. They help create hypotheses and find insights for the team to build a better product and grow the business.
Below are the six things I look for in great product analysts. While some of these might seem pretty obvious, few people understand how to evaluate candidates for these qualities! Read on to learn how to tell the difference between a good and a great analyst.
1. Strong abstract analytical skills
The kind of analytical skills needed for Finance/Business Intelligence are extremely different from what’s needed in Product. To manage and understand the thousands of different interactions a user can have with your product, analysts need to be strong in the concepts of probability, causality, as well as algebra. Some questions you could ask in interviews would be:
- What’s a normal distribution? What would you think if you data was normally distributed? What if it it’s not?
- Suppose we changed a sign up process to include email token verification on Monday and 2 weeks later, someone notices that the conversion rate for email sign ups is now higher. How would you piece together what happened?
2. Great understanding of product and users
A product analyst needs to be able to gain a great understanding of a product very quickly. She needs to have strong intuition for what helps users get value out of a product and what doesn’t. Some of the questions you can ask to understand their product understanding would be:
- What do you think the critical event in our product A is?
- How do we know that users got value out of using our product?
- How do we know whether users are using the product often enough? What if we expected people to use this everyday but they only use it once a month?
3. Ability to set goals & define metrics smartly
Setting goals and defining product metrics is one of the most important things a product leader does in a company. It is critical to make sure the product analyst you are hiring can propagate that well. If a team’s goals are not set with clarity or can be gamed easily by building things that don’t improve UX or create value for the business, you are actively harming your team and your company. Your product analyst should be able to understand the ramifications of poorly defined metrics:
- What kind of metrics matter to product teams?
- How would you go about “improving” a product?
- Suppose we define our key product metric for the year as Week 3 retention. Is that a good definition? Why or why not?
- How do you track that a product initiative resulted in a sustainable improvement in their metric?
4. Brilliant hypotheses generation
Product analysts need to be prodigious machines when it comes to hypothesis generation. Whether it’s asking the question “Why did this happen?” or “What could we try” they need to work with product managers, engineers, customer support and everyone else in the company to make sure they are investigating the right areas. Product Analysts without an imagination will struggle to be an asset to the team:
- Our retention metrics went down last week. How would you investigate why?
- We found that users who have joined us in the last few months are not spending as much time in the app. How would you investigate why?
- We built something that’s a popular need but our users are not adopting it. Why do you think that’s happening?
5. Absolute mastery of telemetry and process
It’s essential that a product analyst knows exactly which user actions trigger events in your product’s event stream, what possible attributes each event might have and where are the knotty errors in tagging are (because you know they always exist!). This requires an inexhaustible amount of curiosity and working very closely with the engineering team to track and manage the vocabulary your customers use to communicate with you.
- Suppose you are looking at user purchases of a product and find that some of the revenue data doesn’t add up. How would you proceed?
- We give you access to our database. How would you get started on day 1?
6. Expertise in setting up high leverage tools for their team
Many companies only test candidates on their proficiency in SQL, Python or data engineering to stitch multiple sources together. I believe this is a mistake. The fact is, your product analyst cannot be the only person answering questions. You need them to set up a self-service platform like Amplitude that will help everyone from product, growth, design and engineering answer 90% of their regular questions directly. This allows your newly hired product data scientist to focus on complex, high leverage questions and challenges like predictive modeling of user growth and revenue outcomes, rather than becoming a bottleneck in your data team.
A great product analyst has all of these six powerful qualities that help teams build better products. They are hard to find and we hope this article will help you do so!
If you are curious to learn more about the right way to set up product analytics, check out our Playbook and explore the Amplitude Demo! If you want to talk about leadership in the age of dynamic product development, email me at sandhya@amplitude.com!
