Still picking up the number of interviews for JTBD with a magic ball? Then we go to you

Quick intro

My name is Mitya Zhuravlev, in the Prequel team I am engaged in user and market research. At Prequel, we work to enable anyone to express themselves creatively, and right now our focus is on visual filters and effects for photo and video content. As a researcher, I help the product team:

minimize the risk of implementing wrong decisions (or competent decisions – wrong),
analyze new markets and introduce products to them,
track trends in different market segments and changing user preferences.

After talking with Pavel Shishkin, co-founder of the Career Workshop and ex-analyst from Yandex, I wrote this article in which I described a working framework for counting the required number of interviews if you need to understand if there is a segment for which you are creating a product.

It is known that in order to obtain reliable conclusions, a qualitative study must be supported by a quantitative one, but this step is often omitted for a variety of reasons. Often, teams will roughly estimate the TAM-SAM-SOM of the market they plan to enter and then immediately move on to the JTBD framework to see if they can make future users complete their large and small jobs faster/more conveniently/cheaper than competitors.

It is great if, at the same time, the teams accurately select a representative sample of the intended target audience, the motivation to purchase the future product of which is confirmed. In practice, many people use magic numbers for the required number of interviews, such as 12-15-16, and if the majority of interviewees meet the requirements, then the task is considered successful, which from a statistical point of view does not mean anything.

The search for a segment can also be complicated by the fact that traditional desk-research or even expert advice does not bring proper results, especially if you want to create a niche product.

If you don’t want to take a radical position “everything that is not quantified does not make sense”, but you also don’t want to run to create products without looking at the available statistical verification methods, then the framework described below is for you.

Framework on fingers

The statistical hypothesis formulation framework can be broken down into 4 simple steps:

Decide on a business problem
Form the most accurate portrait of your core user,
Formulate a business problem statistically (statistics will help if your question-hypothesis can be answered “yes” or “no”),
Make a direct calculation of the number of interviews per https://www.evanmiller.org/

We need to avoid the following trap: if I talk to 16 people from my wide potential target audience (a large market for millions of people) and 8 of them are valid, then this is success. Because in this case, you could be lucky to come across these 8 people, and in a big market valued in millions of people, there really is no need/pain. Or vice versa, only 1 out of 16 will confirm the need for you, which also does not mean that there will be no demand for the product in a large market.

The first key idea of the framework is to take the core of your future target audience in a large market and understand what characterizes it, making a portrait of future users as accurately as possible. And if in this small core of the target audience, you confirm that 50% have a need, then small samples of high-quality interviews will allow you to draw conclusions.

In our case, we have already successfully worked in a large market for processing photo and video content, were looking at a new, more business segment for us and wanted to know if there was a sub-segment with a specific need in it.

The question in our case was: “Is there a subsegment in a large segment such that /__insert your__/?”

Guide to action

Next, we will describe in more detail our actions from paragraph 4 of our framework.

Let’s follow the link https://www.evanmiller.org/ab-testing/sample-size.html

At this step, we are trying to understand how the sub-segment we need is distributed throughout the market. In the Minimum Detectable Effect, we enter 50%, based on the assumption that our segment has a strong concentration of people with the desired job (that is, there is a sub-segment that we need). And in the Baseline Conversion Rate we indicate 10%, assuming that there are quite a few people who will confirm the need both in the entire market and in our segment.

Instead of 10%, a more accurate estimate may be used, which may be lower and therefore require fewer interviews. However, it makes no sense to resort to high accuracy in practice due to the large number of assumptions.

An interpretation in terms of the null and alternative hypotheses could be as follows:

H0 – people with the right job are distributed evenly throughout the market and there are just as few of them in our segment (10%) as in the entire market

H1 people with the right job in our sub-segment are significantly more (50%) than the average in the rest of the market

If you do not increase the power of the criterion * and do not change the level of significance, then you can stop at 7 respondents. To improve the accuracy of your conclusions, work with the power of the criterion.

Then we go to tab https://www.evanmiller.org/ab-testing/chi-squared.html

And we enter the results of the previous step: using only one Sample, we enter the number of confirmed respondents in successes, and the total number of interviews conducted in trials.

The result of our work is the confidence interval of the last step. If, say, 7 out of 13 respondents are confirmed by us, then at a significance level of 95%, we can say that the real share of people in the segment who are ready to buy a product ranges from 29.1% to 76.8%. Thus, we can make a decision based on the lower bound of the confidence interval: is such a minimum percentage of the purchasing power of the segment satisfied with us? With such a share of those ready to buy, knowing the approximate size of the market, will we be able to build a positive unit economy?

What else is worth thinking about?

Due to the complexity of selecting a representative sample for interviews, the question is often not about the accuracy of obtaining the minimum of the above-mentioned interval, but about the principled existence of the segment in principle. Usually a spread of 3 times within the confidence interval is not bad, that is, if you have 0-2 out of 15, then it probably does not exist, and if 5 or more, then it probably exists.

In order for you not to spend rounds in vain, it is better to do a serious analysis before the first round and form an evidence-based portrait of your segment.

Also, if the product exists on the market, then you can also go from competitors’ users who either subscribed or otherwise confirm payment for using the product.

However, you need to understand that if you start with a selection of competitors, then you run the risk of not finding the target audience for which you want to make a product. So, they can use competitors to cover another need that you are not developing a product for, that is, you risk missing the existing segment with pain.

case

Perhaps a typical example of such a mistake is the case when Coca-Cola, based on massive research, including blind tastings of its product and Pepsi, decided to change the classic recipe and made its drink sweeter. Consumers were furious and soon the original recipe was re-introduced to the market and positioned as Classic Coca-Cola. However, we are not sure that this was not just part of the marketing team’s brilliant plan.

The second key idea of the framework: we take people with confirmed pain (they buy competitors) and study why they buy, or we come up with what a qualifying criterion can be, select people for it and calculate whether the segment exists or not.

These approaches can be used alternately, gradually increasing the accuracy of your knowledge of the market and segment.

Conclusion

In fact, alternately using the described ideas of the framework, we will carry out the following check: we found 15 people according to competitors, and if we randomly recruit from the market, will the found respondents also be ready to buy or not? We will probably have fewer confirmed answers, and then we can look at the difference in the sample by competitor and by market – how do they differ in order to identify the criterion of those who say they are confirmed and those who are not.