Do you need a product on your ML team? Opinion from inside
Five years ago, I moved from regular product management to a team with data scientists. And my work process has changed a lot.
As it was before: I determined the user’s needs, then came to the development team with a ready-made task and design layouts. And then she took the working product to send it to an A/B test.
Things work differently in ML. The team is involved already at the research stage, immersed in business goals and the technical formulation of the problem. It is research that takes up the lion’s share of data scientists’ time, and only after that does development begin.
Well, or it doesn't start. Or development begins, but not at all of the idea that was at the beginning.
I'm Sasha Purgina, I lead the development of data-driven products at Lamoda Tech. In this article, I will tell you, using Lamoda as an example, why developing ML products is difficult and risky. And I will give examples of mistakes when a good product in a team can increase the chances of success with certain knowledge and skills.
Don't expect a silver bullet, but there should be a couple of interesting thoughts!
What if there is no product?
The idea of any product based on machine learning must be discussed with technical specialists from the very beginning. Only they know whether there is enough data and capacity for training, how much time it will take for research and other stages, whether we can use open source solutions, and so on.
Some ML teams don’t even think about product. All the same, when setting a task, you will have to immediately discuss it with the team lead or with the entire team. Then why do we need an additional layer in the form of a manager? His role in such situations is usually taken by the team lead or technical manager.
This has its advantages:
hypotheses are formulated immediately in a language understandable to the team,
requests from stakeholders come directly, without additional filters.
The team is happy! But there are also disadvantages:
The team lead can be overloaded with technical issues and working with people, and this does not allow them to fully immerse themselves in the details of the business.
When there are many requests from stakeholders, it becomes more difficult to process and prioritize them. Not the most valuable ideas may be used.
The tasks that a product usually performs in a team do not disappear. Someone must communicate with users and business customers, look for insights, formulate goals, evaluate the cost and value of the solution for the business, and prioritize the backlog.
It is possible to sit on two chairs, but only in a relatively narrow domain area of responsibility with a small number of business customers. If the product becomes a little larger, then it acquires a huge number of nuances and details that it is important not to forget about.
If you do not correctly distribute areas of responsibility in the team, then something important will definitely be forgotten. And this dramatically increases the risk of losing money and time without getting the effect of the product.
Errors in working with ML products and how a product can reduce their number
In my experience, there are a number of cases when an idea “didn’t take off.” Moreover, the model worked and did exactly what was asked of it, but it was not able to benefit the business. There were situations of a different nature: the project got stuck at the research stage, there was not enough data, or rolling out the model into production turned out to be impossible.
At each of these stages, a good product can help the ML team get the job done. Or at least explain the failure to stakeholders and draw the right conclusions from it. I’ll look at examples from the history of Lamoda Tech.
At the discovery stage
One day marketing came to us with a problem: if a user does not visit the site for a long time, then most likely he will not return and will churn. Colleagues already had an idea for a solution: start identifying such users and additionally communicate with them through marketing channels.
The team has developed an algorithm that perfectly predicts users who have churn.
Colleagues were pleased: here it is, the list of people who left us! But then problems began: marketing is trying to contact them, but people have already deleted the applications and do not read the letters. There is no expected effect.
What did we do wrong? It was necessary to predict not those who will not return, but recently active users who are thinking about leaving. You can still save them by offering something useful.
How can product help develop the right solution?
Share business insights. More context is better! At the start and beyond, it is very important to immerse the team in all the details of the problem and options for solving it, to discuss what, from the point of view of processes, business, and user, will happen after implementing the solution and what target metrics we want to influence. The product can dive into the context itself or bring the team together with business representatives and help set up a regular, open dialogue.
Reformulate the business request and product hypothesis into the format of ML problem formulation. There is a gap between the business request “we need to automate/optimize/personalize everything” to specific ML products that needs to be comprehensively worked out product-wise. Clarify from what data the dataset is formed, what models can be used to solve the problem, how ML metrics will be related to product and business metrics.
Evaluate existing data. If there is no data, then the product can set a task for collecting or marking it by assessors, and if there is data and the business value is confirmed, then connect the team for a more detailed dive.
Backlog prioritization
When a user does not find a product in the required size on Lamoda, he can subscribe to have it in stock. It happens that a product appears in a warehouse in a couple of days, and then it makes sense to wait for it. And it happens that it never appears again – if the collection is sold out and there will be no replenishment.
We wanted to use warehouse and delivery data to predict how long a user should wait and whether they should wait at all. We approached it sequentially: predicted business value, analyzed available data, collected a dataset, and selected an algorithm. But a solution could not be developed due to the lack of up-to-date data for forecasting with sufficient accuracy and completeness.
As a result, they decided to pause the project, deciding that they first needed to start collecting all the data necessary for forecasting. Later we will probably return to this idea, but at the moment we were not able to solve the problem.
In ML teams there is always a risk that the time and effort spent will not lead to the desired effect. And that's okay. The product’s task at the prioritization stage is to assess and explain these risks to stakeholders and balance the backlog so that as a result the team brings benefits to the business.
How a good product builds a process:
Regularly delivers a prioritized product backlog for the team's work. The higher the skill and the greater the depth of understanding of ML, the more noticeable the difference in the hypotheses proposed by the product.
Conducts an assessment of the economics and risks of an ML project. Even without the uncertainty that ML projects are often associated with, it is difficult to make predictions about the effects of certain features. In ML projects, this complexity increases several times, and in the Confidence part of the RICE methodology, additional factors and risks have to be included.
Balances the backlog between research, product and technical tasks. Product features can be assessed according to RICE; it is also customary to allocate fixed percentages for technical debt/development. But with research tasks everything is not so simple. If you evaluate all the ideas in the backlog simply by RICE, then research tasks that are important for future development with an unclear effect will never make it to the quarter. The product needs to find a balance and always have a balanced portfolio of products and tasks: some are clear and needed to obtain effects in the short term, others may at first be more vague in terms of implementation, but with a high potential impact on the future of the business.
Development and launch into production
One of the recent problems we encountered at Lamoda is the personalization of product recommendations. We wanted to personalize the user experience right during the session on the site: if we show more relevant products at the moment, this will increase product metrics. We decided to develop a second-level model that would rank recommendations taking into account the user’s current context.
But there are millions of users in the product, and there are also millions of viewed products. There were too many options for ranking; they did not fit into the allocated capacity of the product recommendation servers.
Lack of power prevented us from launching the A/B test on time and required serious architectural improvements.
Therefore, at the stage of development and launch into production, the product must:
Think through the design of experiments taking into account the load and nuances of the models. Particular difficulties here can be caused by tests with highly loaded services, with user segmentation or with flexible pricing for different groups. And also all tests related to the physical world: for example, the development of predictive models to optimize the operation of a warehouse and delivery.
Make a choice among several solution options and track research deadlines. Yes, research takes up most of the work. And it is important to approach an ML problem iteratively and not give up experimenting if your solution is really needed by users. But at some point it’s time to stop, right? The product must see this line and convey its vision to the team.
Evaluate intermediate results of models. For example, test a new ranking algorithm on yourself or attract a whole group of respondents to understand before launching an experiment how well a personal selection meets user expectations. And also be able to give early corrective feedback to direct the team in the right direction.
Make sure that documentation is compiled with the entire history of research and experiments, which led to the current production solution. This will help both with the development of functionality (so as not to repeat yourself in testing hypotheses and technical solutions), and with support (when something breaks and the metrics go down, the product needs to have a product description on hand to help the team find the cause and fix the problem) .
Working with a team and processes is a separate task
At Lamoda Tech, we have introduced a comfortable process of knowledge exchange between data scientists. Demos, meetups, brainstorms, and quarterly results meetings are held regularly for all teams.
Team leads of ML teams contribute exchange of insights found in the data: patterns that are interesting for the catalog can ultimately be useful for recommendations, search, and flexible pricing. And thanks to common approaches, we conduct research faster and compare better implementation ideas.
Products share quarterly results and plans. They also initiate a search for unexpected ideas on how existing technologies can help in product development.
This is a great example when two roles (product and team lead) complement each other in their work. This broadens horizons and energizes teams. There is an understanding that in a neighboring product, to solve a technically similar problem, they tried such and such algorithms, what worked and what didn’t.
Product benefits in numbers
When we are talking about a complex product, with a large number of stakeholders, dependencies and technical challenges, it is worth having a product that will build an effective work process, minimize the likelihood of errors and be responsible for the business result.
If the product helps to additionally bring at least 1-2 profitable ML products to production per year and thereby make millions of users happy with more convenient functionality, then investing in +1 employee will more than pay off for the company. The goal of the product at Lamoda Tech is to take care of the happiness of users and help build a growing, profitable business.
So what is the ideal product in an ML team?
Expectations from a manager working with an ML team include all the classic requirements for a product. In addition, it is important:
Understand and follow the development of technologies that underlie ML products. Help the team choose the optimal solutions in terms of implementation cost, implementation time, risks and potential effect.
Be able to manage outcome uncertainty and stakeholder expectations. Explain the probabilistic nature of ML results and understand at what stage it is time to make a decision to terminate the project.
Know the ML metrics of your products and the criteria by which you can evaluate the acceptable level of accuracy and completeness of model predictions. For example, when recommending size, it may be more important to provide accurate predictions, whereas when recommending related products, it may be more useful to provide maximum coverage.
Explain how models work for users and stakeholders (where possible and necessary).
Scale ML functionality: Expand the product's algorithm coverage and enable the collection of additional data sets to improve the quality of models.
And most importantly: to be a visionary, to promote innovation within the company and outside it. Try new approaches to solve age-old business problems, even if no one has done this before.
If such a product works with you or if this is you, I congratulate you!
If the necessary competencies are still lacking, the support of data scientists is certainly important here: ask, offer help, dive into technical details and especially process features. At first, data scientists from the recommendations and search teams at Lamoda Tech helped me a lot: they explained the details of the algorithms, the nuances of the processes and suggested where they needed product help in the first place.
On the one hand, there is a lot of new things, additional to the usual product management, on the other hand, with a good product base and the help of the team, you can restore the missing knowledge in six months. The expectations that I listed in the article can become a navigator for the product during the initial stages of learning and immersion in the topic.
I wish you to assemble a team of professionals in which everyone can rely on everyone, and regularly achieve great business results together!
