LISTEN NOW

High-Impact Experimentation is Product Management

We don’t need to test everything. We need to test the things that are risky,”

Holly Hester-Reilly, Product Consultant and former Product Owner at Shutterstock, shares how she runs experiments to test critical assumptions and make better decisions throughout the product lifecycle.


Experimentation brings a scientific approach to understanding users’ needs and building products that they will find valuable. It gives product managers the ability to make decisions based on data as opposed to relying exclusively on intuition.

However, there are many common misconceptions about how to run an experiment effectively. When an experiment isn’t run properly, it can lead to poor decisions and lackluster products.

I was excited to interview Holly to learn exactly how to run experiments that lead to better product decisions — an approach she refers to as “high-impact experimentation.”

High-impact experimentation prioritizes and answers the most important questions teams have, in order to reduce the risk of failure, and build a better product.

Holly shares the exact steps she takes to run high-impact experiments, and shares real examples from her experience working with Shutterstock, Toys “R” Us and Weight Watchers.

She begins by providing a framework for determining what assumptions to test. Holly encourages product managers to ask themselves “what would the cost be if you got this wrong?” before leaving an assumption untested. “We don’t need to test everything. We need to test the things that are risky,” she says.

Holly then shares a number of ways to test these critical assumptions — including analyzing existing user data to running surveys. She goes on to discuss how to analyze the results of an experiment — including how to know when you need to dig deeper and when you’re ready to make a decisions, and how to share the results with stakeholders to get alignment on key decisions.

Here are the highlights:

Holly Hester-Reilly

Product Consultant and former Product Owner at Shutterstock

Holly's Recommended Resources on High-Impact Experimentation

What Holly is Reading and Listening To Right Now

Holly's Favorite New Product Outside of Work

Subscribe to receive the latest episodes and exclusive resources

Related Episodes

6

Experimentation is Product Management

Tristan Kromer shares the differences between qualitative and quantitative user feedback, how to effectively formulate hypotheses, and the role of statistical significance in interpreting results.

Listen to Episode
90

Cutting Through Red Tape is Product Management

Thor Ernstsson, CEO of Alpha, shares the greatest obstacles to building great products and how product managers can start practicing experimentation inside a large organization.

Listen to Episode
57

Customer Development is Product Management

Cindy Alvarez, Director of User Experience at Yammer (a Microsoft company), shares how she advocated for Lean methodologies at her organization and how to gain meaningful insights from customer interviews.

Listen to Episode
16

Prototyping is Product Management

Josh Wexler, Director of Solutions at Originate, shares how prototypes can serve as a valuable method for communicating internally and externally, how to gain insights from user testing, and the common mistakes he sees product teams make.

Listen to Episode