To be successful in the digital age, product teams need to deliver more value to customers faster than anyone else in the market. Doing so involves testing a number of assumptions, from strategic ones such as “what market should we enter?” to more tactical ones, such as “what features are most important to users?” Getting answers to these questions is often the difference between success and failure.
Fortunately, product teams today have a number of methodologies and tools at their disposal. There’s a major evolution happening in the way teams make decisions. In the past, teams relied exclusively on large, high-fidelity, and time consuming market research studies at the beginning of the product lifecycle. Now, the most effective product teams today, conduct research throughout the product life cycle and continuously iterate based on what they learn.
According to Gregg Archibald, Partner at Gen2 Advisors and GreenBook, there are two broad types of market research studies: qualitative and quantitative. It’s important to use the right type of study to gain the insights that you need and to understand what each can and cannot achieve. Beyond that, it’s important to understand how to combine them with experimentation in a product management context in order to build solutions that customers love.
Qualitative and Quantitative Research Goals and Use Cases
Qualitative and quantitative research can help teams achieve different goals and are designed for different use cases.
Qualitative Market Research
Qualitative research provides insights into user pain points and perceptions. It helps decision makers gain an understanding of users’ underlying needs, motivations, and opinions.
Here are some examples of insights you can glean from qualitative research:
- “Parents make car purchasing decisions based on ‘ride height’ because they want to be good parents”
- “Millennials find existing fitness tracking applications to be too complicated”
- “Chief Marketing Officers believe their teams lack the ability to create visual content for social media”
- “Baby boomers experience a sense of uncertainty as they reach their 60s”
Qualitative market research WON’T help you decide:
- If your product is the best solution for users’ problems
- How you should position your product
- The size of the market opportunity
- What segments of the market have the most significant need
- What feature set will achieve the best conversion rate
- What user onboarding flow provides the best user experience
Quantitative Market Research
Quantitative research quantifies problems. It’s used to measure industry trends, market sizes, and attitudes of large populations.
Here are some examples of insights you can glean from qualitative research:
- 15% of parents own an SUV
- 63% of millennials are willing to try products that provide nutrition advice
- 23% of Chief Marketing Officers plan to invest in social media marketing tools within the next year
- Baby boomers plan to spend 15% more on entertainment and leisure after they reach age 60
Quantitative market research WON’T help you decide:
- What products and features to build
- Why users behave or think the way they do
- How to position your product to your target market
Qualitative vs Quantitative Research Studies and Methodologies
Broadly speaking, qualitative research identifies problems and explains the “why,” while quantitative research provides measurements across broader markets and customer segments. How are they conducted in practice?
Qualitative research often entails more in-depth interactions with a smaller sample size of users. The most common qualitative research methodologies include:
- Focus groups
- Surveys that include open-ended questions
- One-on-one interviews
The output of these studies comes in the form of written responses, videos, and/or notes taken by the researcher. The decision maker then looks for patterns in the responses, cross references it with other research studies, fills in the gaps with their intuition, and draws conclusions.
Quantitative research generally provides less in-depth insights on larger populations of users.
It takes longer and costs more, but provides meaningful data on important strategic questions. Quantitative research is most commonly conducted through large scale surveys with multiple choice or scaled rating questions.
The output of these studies may include charts, graphs and figures. While quantitative research requires more upfront planning than qualitative research, it does not require as much analysis because the numbers are more tangible.
Experimentation and Market Research
Qualitative and quantitative market research are important tools for product teams. Market research enlightens the organization on strategic questions such as market size and go-to-market strategy. However, modern teams also need data to inform shorter-term iterations. Experimentation introduces data to the daily decisions with which teams often struggle. Designed specifically for continuous iterations, experimentation informs teams with directional insight into distinct user preferences and behaviors.
Experimentation is different than traditional market research in two important ways.
First, it’s iterative. Experimentation provides multiple opportunities to run tests as compared to doing one large study that may or may not yield the data that you need. This enables the ability to run tests throughout the product lifecycle and iterate as consumer preferences evolve.
Second, the insights that you glean from experimentation are directional. Experimentation increases confidence for decisions that need to be made within shorter time frames. By doing so, teams can quickly identify overlooked risk and opportunities for improvement.
Experimentation provides data where there previously wasn’t any at all. Many product decisions are currently made without any data whatsoever, or are informed by what someone at a nearby Starbucks says. This data is not of sufficient quality or quantity.
Experimentation provides data for important questions such as:
- Does “value proposition a” resonates more than “value proposition b”?
- Will this new feature idea improve user experience?
- What checkout flow will deliver better conversion rates?
Experimentation is done using techniques such as:
- Rapid prototyping
- Moderated and unmoderated usability testing
- A-B split testing value propositions or feature sets
Such experiments enable you to test solutions and measure users behavior with the depth of qualitative research but with greater scale. Experimentation provides insights on critical product decisions and allows teams to iterate quickly. There’s less of a need for planning, which means you get more shots on goal, and faster iteration.
The challenge, however, is that despite how experimentation brings product managers closer to consumers, it’s also a new way of working. Here’s how Leonard Murphy, Partner at Gen2 Advisors and GreenBook, described this challenge, and why it’s so important for product research teams to overcome it:
Product [managers] need to be close to the consumers, that’s ultimately what their job is – to give the consumers what they will buy. So, anything that helps that process I think is inherently a good thing. Now, systemically it doesn’t fit necessarily in a way of lot of organizations have been structured in the past, but in an agile nimble type of organizational structure that’s increasingly being adopted then that ability to create tools that give the power to product managers directly is primarily a good thing. It’s different, it has its dangers, but anything that gets folks closer to the consumer experience so they can deliver a better product or a better solution for the consumer is ultimately a win win for all engaged.”
As the marketplace changes faster than ever before, and agile development becomes more prevalent, it’s critical for product teams to have the ability to make decisions quickly. Experimentation and market research are critical tools in any fast-moving organization’s tool kit.