Mastering Quantitative Research is Product Management

You have to be creative. Then the science is really just measuring whether the bets that you're making are right. We're only as good as our bets.

Suhail Doshi, CEO of Mixpanel, shares best practices for using in-product analytics to make better product decisions, how to balance vision with data-driven decision making.

As the discipline of product management has advanced in recent years, so too have the strategies, tactics and tools for conducting product research. Product managers have the potential to access more data to make decisions than ever before.

One critical tool that modern product managers need to have in their tool kit is quantitative research. In other words, using in-product analytics to analyze user behavior and find opportunities to improve the experience.

Since Suhail is the CEO Mixpanel, one of the most advanced in-product analytics solutions on the market, he was the perfect guest to ask about the latest best practices.

He began by sharing his perspective on how to balance data with intuition when making decisions. Intuition helps you come up with hypotheses about how something could work, and data helps you measure whether the experiments you’re running are working, Suhail says. He warns listeners of the downsides failing to find balance between the two. If you over-optimize for hitting KPIs, you might create a poor user experience. If you over-rely on intuition, your success is dependent on luck.

I think it’s kind of like a science experiment. It’s kind of like when we were in sixth grade and we learn the science methodology. The first component is just having a hypothesis. The second component is testing it. The third component is measuring to see what the results are.

Suhail then provides several strategies for making better decisions using in-product analytics. I particularly liked his strategy of using a paired metric to measure the success of experiments. Measuring success based on two metrics, instead of just one, hinders you from hitting one metric at the expense of another. This may not be apparent in the short term, but that’s because “Not all of your tests, not all of your changes are things that have short term ramifications, either you have to keep monitoring them, even six months down the road.”

Suhail also acknowledges that data doesn’t always speak perfect truth:

The reality is that if you only use data, you’ll sort of get stuck in this local maxima, this place where you’re kind of optimizing and optimizing, but you’re only eking out incremental benefits. There’s probably a point where spending as much time and energy on making the UI easier to use or better looking has diminishing returns. There might be a better place to spend your time. So I think data can help you figure out when that’s happening, but it can’t really invent the creative things, or it can’t tell you the creative things that you need to do.

When it comes to actually conducting research, Suhail encourages listeners to simply ship features and analyze the results when there isn’t much downside risk:

I think with most decisions, if they’re not critical, [and] there’s some critical decisions where getting to 90% accuracy is very important, [but] for situations that aren’t, usually it’s better to just try and see what happens, to move quickly.

He also stresses the importance of  setting goals and targets and of making a bet in terms of your own sense of what you think will happen. Suhail recommends betting with your teams: It can just be your pride, it doesn’t have to be money or anything, but he says it’s good to have a clear bet before you go into it.

You’ll learn a lot from this episode about analytics, experimentation, and data-driven decision making.

Here are the highlights:

Suhail Doshi

CEO of Mixpanel

Suhail's Recommended Resources on Mastering Quantitative Research

What Suhail is Reading and Listening To Right Now

Suhail's Favorite New Product Outside of Work

Put our guest's advice into action.

Alpha is the fastest way to learn about your future customers.

Learn More

Subscribe to receive the latest episodes and exclusive resources

Related Episodes


Becoming Data-Driven is Product Management

Sam Nordstrom, Product Manager at Intuit, shares how Intuit transformed into a more customer-centric company and how teams can adopt data-driven approaches.

Listen to Episode

Revenue is Product Management

Jeff Morris, Director of Product Management - Revenue at Tinder, shares his approach to identifying monetization opportunities, launching new products, and measuring success.

Listen to Episode

High-Impact Experimentation is Product Management

Holly Hester-Reilly, Product Consultant and former Product Owner at Shutterstock, shares how she runs experiments to test critical assumptions and make better decisions throughout the product lifecycle.

Listen to Episode

Check out our new report on the top 20 product leaders of the year

View Report