At the Product Analytics Summit in Seattle this past March, we chatted with some of our customers about how they use Amplitude as part of their product development toolbox. Product managers and data scientists from Intuit, Under Armour, MINDBODY and Against Gravity shared with us the strategies they use to uncover their core metrics, figure out the types of insights they look for in their data and the role product analytics play in their product decision making.
Because each company is in a different stage of their product lifecycle—and a combination of enterprise and startups—their respective experiences can be applied to a wide range of digital products. Their individual approach to product development has worked for them, so we’re excited to share with you a few lessons that will help you boost your product development efforts.
1. User feedback opens the door to new opportunities
Feedback is one of the most powerful tools at your disposal. It helps you stay in tune with user needs and helps you make product development decisions that cater to those needs. If you’re consistently hearing the same feedback about a feature or how your product’s being used, stop for a second and explore how to use this feedback.
Back in the 80’s, Intuit released Quicken as a personal finance management tool. However, as small businesses began to crop up, users started to use Quicken to meet their business needs. Because Quicken wasn’t built for this, founders received lots of feedback from users explaining how the product wasn’t meeting a need it wasn’t built for.
Dave Marble, Senior Product Manager at Intuit, explains it this way, “It turns out there was a product in all of that [feedback] that was yet to be created.” Intuit recognized the value in the feedback and in 2002, launched Quickbooks — accounting software for small to medium sized businesses. Marble says, “It’s turned into our most profitable business today.” [Source]
The key takeaways here are:
- Use customer research to find out what users are trying to accomplish with your product. Ask users and track their paths within your product. Your research should go beyond just tracking how the product’s being used and uncover whether there are better ways to help users accomplish what they’re after.
- **Use power users as beta testers. **They’re more than willing to offer feedback so use them before launching something new within your product. Marble points out that, “If it isn’t a good fit for them, then it likely isn’t a good fit for anyone else.”
- **Don’t be afraid to say no to some users. **Since there are tiers of growth — upwards trends followed by momentary plateaus — use these plateaus as an opportunity to implement new features to move you up the growth curve. Launching new features during upwards movement isn’t the best time to launch something new. Power users don’t understand this and will likely leave. It’s OK because you’re satisfying the needs of all users, not just power users.
2. The obvious answer isn’t always the right answer
As a product manager, you know your users well and your product even better. Perhaps in the early days of your startup, it made sense to make product changes on the go because that worked for you. However, as your business grows and your user base expands, it gets harder to get away with on the fly changes. Don’t assume that what seems like an obvious solution is the best move. Instead, run beta tests to be sure.
MyFitnessPal, a calorie-counting and exercise tracking app from Under Armour, offers a premium version of the free product. Under Armour understands that power users are most likely to use the premium feature because this group wants to go beyond tracking general activity levels and track things like their daily carbohydrate and protein intake.
[Source]
The goal with the premium product was to show average users the added benefits by creating a premium onboarding flow that would get them to sign up for the premium service. The original CTA to upgrade to premium was in a scrolling menu but was lost when users scrolled down. The “obvious” solution was to put a fixed button at the bottom of the screen. Stu Kim-Brown, Director of Analytics at Under Armour, explained that,“It didn’t work. The number of people who got to the next step decreased by 7%.” It turned out that the new CTA looked like a banner ad.
The key takeaways here are:
- Beta test new updates before rolling them out. Just because they make sense internally doesn’t mean they make sense to users. Forgoing testing hinders your growth and hurts your product if it results in decreased uptake.
- **Get product designers on board with testing. **They’re a big part of delivering strong UX so make it standard that decisions aren’t rolled out without testing on a small but diverse group of users.
3. Change is necessary for progress
Just because a process or feature worked well in the past, doesn’t mean that it’ll be successful indefinitely. Marble says, “Often times folks get in their heads the way things should work and it’s tough to get yourself out of a mindset,” he continues, “But as data comes in as years and years go by, you have to be open to seeing different things happen.”
With Quickbooks, users use forms to enter their end users’ details like customer number and products they want to sell. This form had functioned the same way for a long time, but Intuit began to notice higher abandonment rates within it. Users were either taking longer to fill in the forms, or were leaving the forms before filling them out completely.
Marble explains that they performed timing analysis to troubleshoot the root of the issue. As a result, they were able to identify points of friction and abandonment with the form and improved it to boost follow-through conversion.
The key takeaways here are:
- Track how users use your product over time. If there are drops within certain features or processes, test different solutions to find the right one for your users.
- Don’t wait for users to tell you they want a change. Core users might not be consciously aware that they’d prefer a change because they’re comfortable with a particular process or feature. However, as UI trends change, update your product and users will be grateful for the boost in efficiency.
4. Don’t underestimate the power of users teaching other users
One surefire way to create a product that engages users and keeps them coming back is to create features with an entertaining element. As Nick Fajt, CEO of VR social app Against Gravity, puts it, “create actions that are delightful and worth teaching to others.”
For their part, Against Gravity launched new UI and UX features in the form of VR “gestures” to get users to connect with each other. For example, a virtual handshake was an invite to make friends.
Even though the gestures weren’t clearly addressed or introduced anywhere in the app, users who accidentally stumbled across them shared the knowledge with other users.
[Source] The gestures — like virtual handshakes and fist bumps — were fun, engaging and rewarding. Fajt explains that, “It acted as an ice breaker moment for a lot of new [users] in the app,” he continues, “It was a surprising thing to embrace these new paradigms because we worried that they would feel foreign to users and they wouldn’t figure it out. It ended up being a really big win for us.”
The key takeaways here are:
- As long as a feature is engaging, you don’t necessarily have to promote it. With the Against Gravity example, the gestures were hard to find on purpose, but users who found them enjoyed using them.
- Identify the core actions you want users to take and create fun ways for users to use them. With Against Gravity, users who gave virtual handshakes were linked as “friends” within the app.
5. A holistic approach to data will only get you so far
It doesn’t matter how well a product performs in one market, do research to confirm whether there’s product/market fit before launching in new markets. When it comes to data, Marble puts it this way, “Have sufficiently exclusive cohorts of people you’re trying to analyze instead of lumping all the data together.” Meaning, there are lots of ways to splice your data so figure out the best way to analyze your data.
For example, when it comes to geography, start with something as simple as using geography to uncover differences between countries or regions. Intuit created geographic behavioral cohorts segmented by country to understand how parts of their product would perform in Australia and the US. They found that research that backed up their US product, didn’t hold up in Australia despite the assumed similarities in population size and economics. The relationship Australians have with apps is very different from the American one. They don’t rely on apps to gather information as much as American users do.
The key takeaway here is:
- Don’t just look at data holistically, split it up to see what’s different in different markets. Start by creating behavioral cohorts and track each one. Marble explains that doing so might take your product in a different direction, which isn’t a bad thing. It just gives you the opportunity to cater your product to specific markets.
6. Uncover core metrics specific to your product
Metrics like CAC, LTV and churn are important to track, every business tracks them. But in addition to these, you have to refine your core metrics and find metrics that are specific to your business. Figure out what it is you need users to do consistently in order to help your product grow. Against Gravity uses invites as their core metric. Meaning, they track users who accessed their friends list to see where they were in the app. These users can then send an invite to these friends to either join them in a particular rec room or invite them to a new one.
[Source]
Fajt explains that to AG, “It was an unshakable single in the app that users were moving from a stranger to a friend. It was a meaningful social connection.” It’s a valuable metric for them because it not only tracked social connections, but it also showed that they were meaningful and not random connections.
The key takeaway here is:
- As you grow, refine what your core product metrics are. Continue to find user actions that’ll help you track how successful your product is. Start by figuring out what actions you want users to take within your app. Under Armour classifies their metric as “user success.” This means that the more information — for example, daily food intake — users upload to MyFitnessPal, the more data the app can access to provide the best user insights.
7. Data democracy promotes data-informed cultures
The data science team acts like a “safety net.” This means that they’re the hub everyone at the company goes to for data insights because they know the data better than anyone else. However, this dependence has its downfalls. Mainly, data science teams are usually small and unable manage multiple data requests at the same time. Turnaround time stretches out and they become a bottleneck in the system because they’re the only ones providing data analysis.
MINDBODY, a business management software for health and wellness companies, saw an opportunity in this. Before they began using Amplitude, there was very little data democracy at the company. Teagan Sorensen, a Product Data Scientist at MINDBODY explains that, “the data science team acted as gatekeepers to all things product data related.”
However, the turning point during the early days was establishing relationships with other product teams. This connectivity set them up for the spread of data democracy when they started using Amplitude.
By giving everyone access to a data analysis tool, less pressure was put on the data science team. However, they maintained their value in that teams still come to them to “double check numbers or identify issues with instrumentation tracking.” They’re no longer the sole source of insights. The key takeaway here is:
- Give everyone at the company access to data. They’ll be able to do quick analysis without the data science team which frees up data science to work on more complex analysis. Teams can make product development decisions that are backed up with data and most importantly, received in a timely manner. Trends change quickly so the quicker teams can act because of access to data, the better.
Hit the ground running
As you can see, there are many ways to approach product development. The key is to find strategies that work for you and adjust as needed. For instance, Against Gravity uses invites as its core metric. Because they’re still in the startup phase, as they grow they’ll have to look at new core metrics like—for example, referrals—as a new way to measure their success.
The lessons above provide perspectives from different types of companies that are each at different points in their product lifecycle.
Based on where your product is in the cycle, use these lessons to figure out new ways to approach your product development. If something doesn’t work initially, don’t be afraid to tweak and adjust until it’s a good fit for your product. What’s important to note with these companies is that while they all use analytics to make their products better, they have unique approaches that work for them.
