How SberMarket tests simple and complex product hypotheses on real cases

Hello everyone! I am Stas Dergunov, Head of Product at SberMarket. My portfolio includes three areas: pickup from the store and in the car, purchases in offline stores using the ScanPay service, and new none-food verticals (pharmacies, household appliances, and electronics).

In this article I will tell you how we test simple and complex hypotheses within the company
based on data, which allows you to support 1.6 times growth year on year.

What is the product approach in SberMarket?

This is the creation of value that benefits the business and users. It is based on a data-driven approach, where we focus on what users need, and not just on the opinion of experts within the company. To ensure regularity of value delivery, we use discovery/delivery cycles based on agile/scrum.

Today we’ll talk about the concept of simple and complex hypotheses and working with them.

First about the terms

The level of complexity comes from the number of abstractions that we want to explore. The higher their value, the more complex the hypothesis.

Simple hypothesis formulated in a classic framework “If …, then …”.

For example, we have a hypothesis:

If you add the functionality of choosing between delivery and pickup to the main screen of your mobile application, this will reduce the number of user errors
during the ordering stages.

We see that here in one place there is one change.

Complex hypothesis — are several levels of abstraction that consist of interconnected hypotheses, which require a series of studies to test.

Example:

If we give old customers a new shopping experience in offline stores that will help them save time and money, then they will start using it instead of regular checkouts.

We see that there are several hypotheses here at once in different places, at the intersection of offline and online.

We conduct research in three cases:

1. If we want test the hypothesiswhether the new solution improves user experience and product metrics compared to the current version.

2. If we plan find out how well the solution meets your needs users. For example, if such a function did not exist before, but it may be useful.

3. If we have there is no solution, but there is a request for inspiration or searching for clients’ needs and problems.

Understanding what hypothesis is on the table, we can use different studies to reduce the risks of unsuccessful decisions. This also helps to conduct studies cheaper, for example, without involving developers for A/B testing. I will tell you about the methods of verification later.

Methods for testing simple hypotheses

1. Unmoderated tests in quantitative research. Example – first click. You can use respondents on crowdsourcing platformsgive them a task and look at the heat maps of interaction with the interface. You can also use SberMarket respondents, filtering, for example, by order frequency/specificity, etc. We have recently begun to use our database much more often.
   In such a study, we see not only where the subject clicked, but also for how long.

2. Moderated tests, also known as in-depth usability interviews or UX researchWe have a hypothesis in the form of an interface and a task – to check whether the solution is clear to the client.

Unmoderated test – using store pickup as an example

Task: add to the mobile application the ability to choose the method of receiving an order (delivery or pickup from the selected store). We wanted to understand in what
Of the three options, visitors will find the pick-up button on the main screen faster.

Second task: understand where people click to arrange pickup if they are given several options.

To do this, we launched an unmoderated online survey of SberMarket's customer base.

Part 1: First Click Test

In the experiment, one respondent tested only one version of the home page.

The task sounded like this:

Imagine that you need to order groceries and pick them up.
from the shop. Where on the screen will you click to do this?

What conclusions did you draw?

In the category “Food”:

1. The “new 1” option works best – there, a larger share of people undoubtedly go to the “switch”: delivery or pickup.

2. The current version also works well, but compared to the “new 1”, people’s clicks are a little more “spread out” across the page, although they still do it quickly.

3. The “new 2” option is the worst: very heterogeneous cliques, people need to search and think – they waste the most time.

In the category “Alcohol”:

1. In general, they cope faster than with food

2. The “pickup”, “switch”, etc. buttons are no longer so important – the majority
   in any case, it “goes” through the “Alcohol” job.

Part 2: Subjective assessment

Exercise:

Below you will see what different people have said about how they find it most convenient to place orders for pickup through the app. Choose the statement
which you agree with and which personally seems closest to you.

1. I want to go into the app, select the “pickup” method, and then select an available store and fill out the basket.

2. I want to first assemble a basket of the goods I need, and then choose whether I need pick-up or delivery.

3. It is important for me to first select a specific store or point (understand the address and operating hours), and then assemble a basket for pickup in this particular store.

4. Other.

Above are the results we obtained. Based on them, two paths were outlined.

1. Use the “new 1” option (if our main goal is to build a more unambiguous scenario for the client).

2. Leave the current option, since it is also convenient and suitable, but slightly less specific (we see more “clicks to other places on the page).

Moderated test – using the example of UX research for a pickup service

Task: integrate products and retailers that only work into the customer experience
for pickup. This will increase the assortment for the buyer, and for the retailer
receive orders without delivery. We aim at the main points of contact with the client in search and search results.

Research method: moderated UX testing with interview elements. We tested a prototype with a self-pickup scenario in search. The average duration of one test is 40 minutes. A total of 6 UX tests were conducted.

Audience tested:

• Active SberMarket users make orders at least several times
  once a month.

• They actively use self-pickup – at least several times a month.

• Age of respondents: from 20 to 55 years

• Gender distribution: 6 women, 2 men.

What recommendations were formulated based on the study:

1. Work out the relevance of filters and their name.

2. Provide the ability to view the distance to the nearest store.

3. Simplify the work with choosing an address and building a route.

Methods for testing complex hypotheses – using the example of the ScanPay service

Above I talked about methods for testing simple hypotheses. It makes sense to move on to them when it is clear that the product is needed, and it remains to understand how to make it better, faster and more convenient.

I'll tell you how we looked for points of multiple growth for business using the example of launching a new ScanPay service in SberMarket.

Such cases require complex research of hypotheses using different methods (one of them is the first click test). The goal is to quickly confirm or refute them and make a pivot.

What is ScanPay?

This is a function inside the SberMarket application that allows you to scan products
in the store and pay without queues in the application, using the loyalty programs SberSpasibo and the store (details on link).

#1 Analyze the market and strategies

Be sure to dive into the company’s strategy to understand its development plans
and goals for the next few years. After conducting the analysis, we will see the current markets,
where the company plays and where it is not yet represented.

For example: we have determined that the Grocery market consists of two segments: online and offline. In online (e-grocery), SberMarket is the market leader at 165 billion rubles in 2023,
but SberMarket has no offline presence.

Grade food market in 2023 — 47.4 trillion rubles (year-on-year growth)
by 6.4% per year). We conclude that there is a huge offline audience market and it is time to conduct research into the needs of the segment. We are taking this hypothesis into work.

#2 Studying clients

To begin with, we conduct qualitative research, the purpose of which is to understand the needs
and customer problems in offline stores. In addition, we find out how clients are currently solving these problems; for this we use interviews.

Example of questions during an interview:

• How are you solving the problem now?

• What problems are there? In what situations and how often do you encounter them?

• How annoying are these problems?

• How do you cope with problems?

#3 Test the hypothesis quantitatively

The next step is to digitize and prioritize, using quantitative research, what exactly needs to be focused on among all the needs.
and problems. Objective: to identify the criticality of problems and needs. Method: survey.

Based on the results of these two studies, we found an interesting segment – omnichannel clients (make a purchase in the online and offline channel). They sometimes buy products online in SberMarket, and sometimes in offline stores.

We decided to focus on this audience and formulated a hypothesis:

If we give SberMarket customers a new service for offline purchases,
then we will increase the number of purchases and the average bill from such clients due to omni-experience,
We will also be able to attract customers to the new service faster and cheaper.

We structure our segment and build a tree of customer tasks according to the job-to-be-done methodology with the customer’s path along offline stages: before the store, in the store, after the store.
We took as a basis the qualitative research that we conducted earlier.

#4 Looking for a USP

In parallel, we conducted an analysis of competitors in the Russian Federation and abroad. According to the decisions
for segment:

This helps to develop an awareness of competitors and their decisions.

Based on the needs and problems, we create USP or value proposition options for our segment. Here's what we got:

• the ability to track the cost of individual items and the entire basket online;

• payment in a mobile application without queues;

• use of loyalty programs and SberSpasibo, and from the retailer, where you can write off and credit virtual rubles.

#4 Testing concepts

Now we need to understand whether the client is interested and ready to try our solution?

Validation tool – fake door This is a quantitative method for demand verification.
and collecting an audience that will submit a request to gain access to the service.

In other words, we share with the client a product that has supposedly already been created,
and look at his reaction.

For example, we sent a team of promoters to the stores, who interacted
with potential clients, offering them a product. Their task was to install the SberMarket application for new clients, and for old clients to activate the ScanPay mode on the main screen.

This is how we tested customer value concepts, where we got the information we needed:

• verified USP and communication with clients;

• validated the steps in the funnel. For example, we realized that the interaction
  with promoters (at some point while in the store, the client found out
  about the product) directly affects the willingness to use the service.

As a result, this allowed us to make adjustments to expectations from the product and its positioning.

How simple hypotheses fit into complex ones

Now let's look at an example when a simple hypothesis fits into a complex one through the tools discussed above.

For example, our goal is to test the visibility of a new option for the entry point to the ScanPay service compared to the current option.

Research method: unmoderated test on the Useberry platform, quantitative research. Target audience: SberMarket users with different experience of using the service. Number of respondents: 100 people for each design option, non-overlapping audiences.

Advantages of the method:

• The ability to quantitatively confirm how well the proposed solution meets the user's needs.

• If there are several interface solutions, evaluate the potential of each before the A/B test:
  This may be relevant when there are several entry points to a section.

• Thanks to the first click method, it is possible to find new solutions that were not previously obvious
  when developing a product.

On the old main: Only a third of respondents first clicked on the “Scan Pay | Scan” bar at the bottom of the screen. The majority of users (61%) associated entering the service with choosing a store where they planned to make a purchase.
This made it clear that it was necessary to implement an additional entry point at the main retailer.

On the new main page: almost half of respondents (46%) correctly considered the entry point
into the service, while the percentage of erroneous selection of other sections of the screen has decreased significantly compared to the current version. The choice of service on the retailer’s page was 31%.

Let me summarize: we have gone from searching for a market and studying the company’s strategy, working with clients in the market and finding problems and needs, focusing on omnichannel customers, studying the experience of competitors, and applying a quantitative method to confirm the hypothesis. This is how we look for opportunities for clients in products that clients have not yet realized.

conclusions

I told how we work with simple and complex hypotheses. I showed how simple hypotheses fit into complex ones and work together. I told about the areas of applicability of methods and limitations.

What else I would like to note: it is important in every study consult with researchersbecause there are no universal solutions, and the key to success is the right combination of approaches. Therefore, in my opinion, the key task of a product manager is increase your visual acuity and learn to use a different set of tools.

I thank Natasha Gureeva, a researcher from my team, for her help in preparing the article.

Product&data team of SberMarket manages social networks with news and announcements. If you want to know what's under the hood of high-load e-commerce, follow us
V Telegram and on YouTube. And also listen podcast “For tech and these” from our it managers.