Meet the Team: Eric Metelka, Experimentation Director

How do you truly own an impact? For Eric Metelka, you have to prove it. Eric’s done just that as a scientific-minded product leader at companies like G2, Cameo, Eppo, and now at Amplitude as Director of Product Management for Experimentation.

Read on for his takes on the wisdom you can get from experimentation, the courage to search for proof over assumptions, and the power that generative AI could soon bring to A/B testing.

What first got you excited about product experimentation? What do you love about it?

I love that experimentation is based on data. I’ve always been data-driven. My background is in math and economics. My first job out of college involved analyzing large sets of numbers, and so I love that A/B testing builds upon being quantitative.

Frankly, I really got pumped about experimentation when I had my first couple of winners. When I saw that the results were positive and proved that our investments were working. It was awesome to both see that impact and own that impact.

What do you think about the intersection of AI and A/B testing? How is it helping?

First of all, I don’t want to paper over the ways that machine learning has intersected with the A/B testing world with multi-armed bandits and contextual bandits. Contextual bandits in particular allow the system to show the optimal variant for a given user’s profile, and that has been around for years now.

For Gen AI specifically, it’s early days for AI and A/B testing, but there’s a very bright future ahead. There’s a lot of low-hanging fruit just in terms of using AI to help generate hypotheses, which is the most common use case we’re seeing around AI right now. Another one that we'll see become pretty prevalent soon is helping to summarize results and provide guidance on setting up an A/B test correctly with agents.

When we consider the frontier of AI with A/B testing, that’s when we get into ideas such as how synthetic data can influence what gets built or how likely a feature is to be received. That’s something that will take some time, but we’ll see how it progresses.

What should scientific-minded teams be wary of when using AI?

Like anything, we need to be checking that AI is setting things up in the right way for us. For example, are the hypotheses that are being generated any good? We still have to use our judgment when using it.

When we think about this frontier that’s coming sometime in the future, where people are beginning to ask, “Can we use synthetic data?”, we will need to see the research. We need to see the proof. I’d be wary of anybody telling me they’ve built a better mousetrap for A/B testing without some data-backed proof or research behind it.

If variant A is the work day, let’s talk variant B. What’s something that helps you recharge and reset outside of work?

Recharging for me means reading a good book. I love books in the epic fantasy and sci-fi genres, ideally on a beach somewhere. I also love a good video game. I play on my Switch a good amount, which is the ultimate dad console. Particularly action-adventure and RPG games like The Legend of Zelda.

Hanging out with friends and family is also great. It could be just hanging out on the stoops in our neighborhood in Chicago having a good time, it could be seeing a good movie, it could be a good meal, it could be going to a music venue seeing a good show. Summertime Chi is the best.

What sealed the deal for you with this position at Amplitude? What are you looking forward to here?

I’ve used Amplitude for a number of years now, and I love the UX of Amplitude. I love that I, as a PM, can use it so easily to self-serve any answer and get it quickly. It feels like a superpower.

What really sold me on Amplitude was a vision of how we can help PMs and marketers use experimentation in the same way. The fact that Amplitude has experimentation and can easily connect metrics and analytics to experiments is really powerful. Plus, it has a great UX built for the jobs PMs and marketers have to do. No other experimentation vendor has that rich analytics capability all in one platform.