Concept Testing: 6 Lessons From Product Leaders

Building products that users want is the crux of every product manager’s job. Product managers are always coming up with new product and feature concept ideas to meet the needs of their users. There’s almost never a shortage of ideas. The challenge is determining which ideas users actually need.

concept test

When you choose to build a new product or feature, you are making an investment. You allocate time and money to build it. If that concept turns out not to be valuable to users, that investment will not have a positive ROI. In addition, when you choose to invest in one feature, by nature, you are choosing not to invest in another feature that may be more valuable to users.

It’s critical for product managers decide which concepts are worth building and which are not. Fortunately, the guests of This is Product Management have shared their best strategies for testing concepts before building them. The next time you’re testing an idea for a new product or feature concept, apply these lessons!

1. Get into a testing mindset

Before getting started with concept testing, it’s critical to have the right frame of mind. Testing concepts before building them is still a relatively new practice and requires a new way of thinking about product management. Concept testing has become far more prevalent as product managers have experienced the consequences of deploying a “build and see” approach that entails investing big budgets into concepts without knowing if there’s market demand. All too often, product managers would find out that customers won’t use a product or feature until after it’s been developed and marketed.

Concept testing helps product managers determine what products and features customers want before building them. This means that instead of having an idea and then building it, you have an idea, then test it to learn if it’s worth building.

Jon Stross, Co-Founder of Greenhouse, has successfully built new products in several industries without any prior domain expertise. When I interviewed him, he shared what’s contributed to his success in doing so. Jon emphasized the benefits of not having pre-conceived notions about what his customers need and being empathetic to their needs before building a product. Having empathy and a desire to learn through testing can help you gain meaningful customer insights and build better products.

2. Focus on users’ ultimate goals

In today’s world, it’s common for new technologies to render legacy products obsolete. You’ve probably heard about what happened to Blockbuster and Kodak. In this environment, it’s not enough to simply make incremental improvements on a product or solve small problems for users.

That’s what inspired Karen Dillon, Contributing Editor of Harvard Business Review and co-author of Competing Against Luck, to develop a framework with Clayton Christensen called “Jobs to Be Done.”

When I interviewed Karen, she said that customers make choices about what products to use based on what they’re trying to accomplish. In her perspective, customers “hire” products to help them make progress towards their ultimate goals. Therefore, it’s critical for product managers to discover the root of their customers’ problems and and find the best ways to help them achieve their goals.

Aakriti Agarwal, Deployments and Growth at Blend, shared how she puts this philosophy into practice. Aakriti focuses on testing ideas that will truly help her clients’ achieve their business goals, not just what they say they want. “We should build on things that are very high leverage and actually will move the business forward for the client and for our company rather than building features that only give incremental impact,” she said. To do so, Aakriti studies how her highest performing clients work, and when faced with a feature request from a client, she asks them for the consequences of not getting the feature.

3. Determine what needs to be tested

Testing product and feature concepts before building them reduces the risk of wasting time and money on something users don’t need, and helps product managers learn what users do need. But there are only so many hours in the day for running experiments.

Holly Hester-Reilly, a product consultant and former Product Owner at Shutterstock, shared her framework for determining what concepts needs to be tested in the first place. Holly plots each test her team is considering running on a four-quadrant diagram. The diagram maps “the negative impact if the concept isn’t viable” on the Y-axis, and “the likelihood that the concept isn’t viable” on the X-axis.

concept testing

Holly then tests the concepts in the upper right-hand quadrant – concepts that have a high likelihood of not being viable and would have a major negative impact if they aren’t viable. This framework helps her get answers to her most critical product questions.

4. Use the right testing strategy for the job

Once you’ve identified the most important concepts to test, you need to determine how to test them. Fortunately, there’s a strategy for answering just about any product question you’d want to answer.

In episode 63, Brent Tworetzky, Executive Vice President of Product at XO Group, shared his framework for deciding what testing strategy to employ. He divides his testing into two categories:

For user intent testing, Brent typically utilizes surveys, interviews, and competitive benchmarking. For user behavior testing, he typically utilizes prototyping, product analytics, and A-B testing. Conducting surveys and interviews and building prototypes have been the most common strategies our guests have discussed for testing ideas.

5. Run user surveys and interviews

User interviews and surveys are great for validating that users have a given problem and getting initial feedback on solution ideas. However, the answers you get from surveys and interviews are only as valuable as the questions you ask.

In episode 95, Noopur Bakshi, Senior Product Manager at Adobe, shared her best advice for gaining meaningful insights through interviews and surveys. Don’t go into interviews seeking to validate a hypothesis. Remain open-minded. Ask open-ended questions about users’ current needs and behaviors. Introduce your concept ideas and make qualitative assessments about your users’ interest.

6. Design a prototype to run an experiment

Once you’ve validated that users have the problem, it’s time to start testing the solution. Prototyping is a great strategy for testing and iterating on solution ideas. Prototyping enables you to provide users with a visual representation of the concept idea without having to spend all the engineering hours that would be required to actually build it. This enables you to gain insights that are more specific to the product than the insights you gain from surveys or interviews.

Prototyping is ideal for testing value propositions, onboarding flows, and feature sets. A prototype can be as simple as a static landing page that displays the value propositions of the product or as complex as a multi-page mobile onboarding experience. Here’s an example of a prototype we designed to test a concept for a chatbot for patient engagement:

prototype test

In episode 16, Josh Wexler recommends testing prototypes with users in a one-on-one setting. Conducting moderated sessions allows product managers to ask questions and get immediate feedback as users are interacting with the prototype.

Getting started with concept testing

The guests of This is Product Management have shared some amazing ideas for testing new product and feature concepts. Focus on being empathetic and learning about your users. Discover your users’ ultimate goals, then determine your riskiest assumptions, and use surveys, interviews, and prototypes to test them. Use what you learn to build a great product or determine what you need to test next.

Concept testing can save product managers from wasting time and money on products that users don’t want, but it can be hard to get started, especially if your company isn’t used to working this way.

Thor Ernstsson, CEO of Alpha, recommends starting small. He encourages product managers to start small, with even a few interviews, and not to worry about getting it perfect. You’ll learn as you go and start to get it embedded in your workflow. The insights you gain from these initial tests will also help you prove the value of testing to key stakeholders in your organization. Before you know it, you’ll be testing concepts throughout the product lifecycle and building the best products on the market.

At Alpha, we’ve created a platform that makes concept testing as simple as asking a question. Tell us what you want to test, and our platform sources your target users, builds prototypes, conducts interviews and surveys, and analyzes the results. Request a demo here.

Put our guest's advice into action.

Alpha is the fastest way to learn about your future customers.

Learn More

Subscribe to receive the latest episodes and exclusive resources

Related Episodes


Asking the Right Questions is Product Management

Noopur Bakshi, Senior Product Manager at Adobe, shares the most effective user interview questions and what she does before each interview to ensure that she gains meaningful insights.

Listen to Episode

Jobs to Be Done is Product Management

Karen Dillon, Contributing Editor of Harvard Business Review and co-author of Competing Against Luck, dissects the most common problems with corporate innovation efforts and shares how understanding customers' job to be done can help product teams make better decisions.

Listen to Episode

Cutting Through Red Tape is Product Management

Thor Ernstsson, CEO of Alpha, shares the greatest obstacles to building great products and how product managers can start practicing experimentation inside a large organization.

Listen to Episode

High-Impact Experimentation is Product Management

Holly Hester-Reilly, Product Consultant and former Product Owner at Shutterstock, shares how she runs experiments to test critical assumptions and make better decisions throughout the product lifecycle.

Listen to Episode

Check out our new report on the top 20 product leaders of the year

View Report