A couple months ago, Amplitude CEO Spenser Skates did an AMA with one of our favorite online communities GrowthHackers. GrowthHackers is a network of growth professionals looking to share their knowledge and collaborate on growth best practice; obviously we were excited to answer questions about user behavior analytics! Spenser also used the opportunity to share his personal experience as an entrepreneur, growing a SaaS business and his thoughts on analytics as a key driver of retention and growth.
If you missed out on the AMA, this recap highlights some of the really great questions we received!
On Data & Analytics
Q: Is analytics fundamentally reactive or proactive? That is, should you use your analytics to understand the results of your experiments, or should you use them to determine where you need to go and what you need to do?
At Amplitude, it’s about observational data on your the interactions your users are having with your application. You’re trying to understand “this is what has happened with my user base” more than anything else. That said, the reason you look at this data is so that you can use it to figure out what to do next. For example Facebook’s famous “7 friends” insight was that it took 7 friends to get a retained user- that observational. They took that and then figured out that adding features to encourage their users to add friends was where they needed to go.
Q: Is there a framework you can use to arrive at conclusions like Facebook’s “7 friends in 10 days”? How do you put the foundation for an observation like that in place when at the start you have no idea what you’re going to find?
The first thing I’d look at is whether or not an event was performed in the first day and how predictive that is of your retention a week later. Look at that across all of the events a user can do and that will give you great insight into what events are important (in order to do that you do need tracking of the events that could be important–I’d say track the top 20-50). That’s the most basic analysis I recommend starting with. Once you’ve done that then you can move onto more complex patterns of behavior.
Q: It always worries me when companies say they are “data-driven” because it sounds like they are chasing the numbers. Do you guys have any framework that you stick to which allows you to use data effectively, but still allows for your own experience, insight, and intuition?
If I could only pick one of between talking to my users and using analytics to see their behavior, I’d pick talking to users every single time, no matter what company I was at.
That sort of qualitative data is so valuable because you get insights around what people’s pains are and how they’re using your product that you’ll never get out of the data. That said, data is invaluable once you’re at scale and you need clarity because you have too much anecdotal data that points in different directions. Here’s how I think about it: generate hypotheses from qualitative experiences, then use data to test those hypotheses.
Q: What is the end goal or your vision for how far custom analytics can go in general and via Amplitude?
We are in the Stone Age of analytics compared to what analytics will be 10 years from now.
Right now you have to be an expert in where the data is stored, how to query on it, how much data there is, how to track it, when it’s available, and a million other things. In the future all of those things will be handled behind the scenes.
Analytics right now is like looking up information before search engines and the internet. If you wanted to learn something you’d have to look it up in an encyclopedia or worse, go to the library. Can you remember how many times you did that? I certainly didn’t. And then you’d feel bad about not putting in the work to learn more. Now Google, Wikipedia, and everything else makes you feel smart:
image source: XKCD
Analytics is the same way. Everyone says they should be more data driven and then feels badly they’re not. We recently did a user study to understand where people rated themselves on analytics expertise from novice, intermediate, to expert. Not a single person rated themselves as an expert–including a product manager at another analytics company. Eventually we’ll look back at today and can’t believe anyone put up with how clunky it was. What we’re doing at Amplitude is the next step in that direction
Career & Entrepreneurship
Q: What period of life & what during that period would you say you gained the most amount of knowledge?
I spent the year after I graduated from college trying to figure out whether I was good enough to start a company of my own. I read everything I could about starting a company and talked to as many entrepreneurs as I could. I applied to TechStars (see the gem below:)
The biggest thing I learned was 1) almost all successful startup stories have a point about a year to a year and a half in where things looked hopeless and they probably should have given up and 2) if you were willing to stick it out for at least 2 years you’d likely come out the other side with something successful. That convinced me that as long as I was willing to not doubt during that time then I could make it worthwhile.
Q: If someone new to the startup world receives a job offer and his company didn’t offer as much information as Amplitude does, how should he go about understanding and evaluating the equity part of the offer?
Ask about a specific scenario by saying “if the company sells tomorrow for X, what does that mean for this equity?” Sam Altman has a good way to do this here. You can also let them know that you expect it because companies like Amplitude/other competing offers you have are doing it.
User Behavior & Amplitude
Q: When you started Amplitude, what was the opportunity in the analytics space that you saw as untapped? There are clearly a lot of analytics companies and have been since the early days of the Internet. But I’ve heard from people who understand the space well that Amplitude is doing something special. It would be great to get some context on that initial vision and if that continues to be your main focus or you have shifted significantly as the market has evolved.
There were two big openings we saw when we started Amplitude. The first was that what growth teams were doing at large companies was 10x more sophisticated than what was out there at the time. Instead of continuing to improve on their analytics products, most analytics companies had pivoted to doing advertising or marketing products. The second was that there was about to be a massive shift to mobile and no one was really focused on the unique challenges of mobile. We’re still focused on pushing the boundary of what analytics can be and think there’s a long way to go.
Q: As someone who’s been trying to wrap their brain around existing solutions like KissMetrics and MixPanel, what would you point to as the key differentiators to make the switch to Amplitude? What are the key things anyone should be looking at to make an accurate comparison between all of these tools to decide what’s best?
There are two big ones. The first is what questions you’re trying to answer/what you’re looking out of an analytics solution, and the second is how much data you have. If you’re looking to understand user behavior deeply, like what behaviors lead to conversion or retention, then that’s something only Amplitude can do. If you’re looking to get an “all in one” analytics/marketing/A/B testing platform than Mixpanel would be a better choice. The second is how much data you have. If you don’t have many users (less than 10k monthly actives) than Amplitude/Mixpanel/Kissmetrics are the same. Once you get beyond that and scale becomes an issue then that’s where Amplitude can perform about 10x better. I’m biased on that one but I encourage you should look at each pricing page to see how they compare when it comes to data at scale.
Q: I read about your changes to the traditional equity compensation model and I was wondering how that has affected Amplitude’s growth? Was it harder in your early growth stages, or did it present difficulties as you matured, or has it been smooth sailing the whole way?
It hasn’t presented any difficulties at all so far. It means we’re slightly more generous with equity and we have to keep in mind that people will be able to hold onto it for a long time without exercising. I’d definitely recommend it as it’s a way to standout during recruiting. I know a few people who have interviewed here have mentioned it as a positive.
Q: What’s your take on the statistic that says something like 68% of companies struggle with their analytics because they don’t understand the key drivers of their business? It was a surprise to me, but maybe not to you. How can analytics help companies figure out what they should be measuring, or can it?
We call this coming up with your “magic threshold” metric to understand what is ultimately driving your business. We have a blog post that explains it fully here.
The idea is you want to figure out what outcome you care about for your users (for most businesses this is retention). Once you’ve done that, see what the differences are between users who retain and users who don’t. Is it a particular pattern of behavior? Is it what they’re buying? Is it where they’re coming from? Slicing your retained vs churned users along all these behaviors and attributes will allow you to understand what the key drivers of retention and ultimately your business are.
Q: What is Amplitude’s biggest growth challenge today and why?
The biggest challenge is having people who are evaluating other solutions know we’re a possibility too. Once someone engages with us they “get it” pretty quickly and we’re good at turning them into a successful long term user of analytics. The hardest part is to make sure we’re in the conversation in the first place.
