How to manage machine learning and data science projects
Machine learning and data science projects are difficult to manage because projects are often exploratory in nature and it is difficult to predict how long it will take to complete. It often starts with one idea and then spills over into a new direction when the proposed method doesn’t work, or if assumptions about the data turn out to be wrong.
Model building is also a lengthy process (compared to working in software and analytics), and it is not uncommon for a data scientist to fall into a rabbit hole and spend months on a project with no clear idea of progress. Another difference from standard software development practices is that model building is usually done by just one person, and this doesn’t quite fit into traditional team workflows like Kanban and Scrum.
I’ve spent quite a lot of time studying existing workflows (mostly in Jira) in terms of suitability for machine learning and data science project management, but to no avail. Most of the information is geared towards software development and focuses on Agile methodologies. Discussing this issue with colleagues and friends, I could not find anything that would be adapted for machine learning and data science. I noticed that some colleagues are trying to adapt their workflow to standard engineering practice, while in other cases, they are not trying to manage projects at all. The latter is especially problematic because projects that take too long and overshoot a subject area are more likely to fail.
Since I was unable to find a suitable solution, I decided to develop my own working framework for managing machine learning and data science projects. This process can be implemented in Jira and allows me to easily track the status of projects, keep records, and also keep the domain from bloating, avoiding building overly complex models. Our researchers have a workflow that helps them build the model, which increases their success on the project. I have been using this system for several years now and my team and I are very happy with it.
The stages of a machine learning project are clearly defined
Whether you are building a complex computer vision algorithm using deep learning techniques, a ranking learning model using LightGBM, or even simple linear regression, the process of building a machine learning model has well-defined steps. The diagram shows the steps of building a model from initial research to analyzing the results of A / B testing. Note that each stage has a deliverable (deliverable or milestone). This is the stage in the process when you discuss the result with the team or stakeholders.
1) Research
Research is the initial phase of a project. It includes negotiations with stakeholders to understand the goals and expectations of the project, negotiations with business analysts who help us find out what data is available to us and where it can be obtained. You can improve your intuitive understanding of the problem by formulating some initial queries and exploring the data.
It is assumed that at this stage the researcher is mainly engaged in studying the literature and choosing a methodology for solving the problem. This also includes the collective search for creative solutions with colleagues, if any. Sometimes, choosing a methodology also requires studying existing ready-made solutions and tools, and creating primitive prototypes using Jupyter notebook.
Final result: The result of this phase is a detailed implementation plan for the project, broken down into subsequent phases (e.g. data exploration, modeling, commercialization, and results analysis) and the associated estimated level of effort required (measured in weeks). It is also necessary to define the methodology and data to be used.
The plan is also shown to stakeholders for their feedback.
2) Data Exploration
This is the traditional step of exploring the data using Pandas and Jupiter notebook (or sometimes Tableau) in order to get some kind of conclusions about the data being used. Typical analyzes include counting the number of rows in the data, creating histograms for various aggregates of functions, trend plots over time, and multiple distribution plots. Researchers will also shape queries that will form the core of their ETL model.
Final result: a detailed report on data exploration with a Jupiter notebook, with graphs and comments that provide some insight into the data being used. The report will be shared with the rest of the team and project stakeholders.
3) Modeling
This is actually the whole essence of the project. At this point, researchers begin building their models using our internal framework. This includes creating an ETL system, constructing features, and preparing a model. This phase also includes building basic models and a comprehensive assessment of the final solution.
End results: The results of this stage are:
-
Model prototype
-
Report in Jupyter notebook with extensive model evaluation
A report on the results will be presented to the project team and stakeholders.
4) Commercial implementation (Productization)
At this stage, the code of the final version is implemented. Common tasks at this stage include adding comments to all functions and ensuring that the code is formatted correctly according to Python standards and team convention. The code is provided with reporting metrics such as the number of lines fetched, the number of lines in the output, the multi-metric prediction error, and feature importance when possible. Finally, the code is reviewed by one data scientist and one engineer.
Sometimes the commercial implementation process leads to constant interaction with the platform developers. This is especially true for real-time models where runtime performance is critical. It may also be that the memory requirements for the code are too high, leading to problems during pipelining. Engineers can cancel the release and require memory reduction to train the model.
Final result: the result of this stage is the code committed to the master branch, which is ready for deployment by the platform development team.
5) A / B Testing
Most models go through the A / B testing stage. With its help, specialists and stakeholders determine the details of the test: the testing period, percentage of traffic, testing control, interpretation of results, etc. During the testing period, team members will be involved in other projects, but they will still need to pay attention to testing.
6) Results Analysis
It is one of the tasks of each researcher to compile a detailed analysis of the results of their own model. At this stage, they will analyze the results metrics in different ways to understand what is really happening. In particular, if the test fails, we need to dig into the results to figure out what went wrong.
End results:
-
Detailed results report in Jupyter notebook.
2. A hypothesis as to why things did not go as expected (if possible)
The final report will be presented to the project team and stakeholders.
Working with Jira
While this framework may look great in theory, in reality the above steps are rarely strictly sequential. For example, there is often a swing from data mining to modeling and back to data mining. Also, this process doesn’t fit into the existing Jira framework, so how do you put it into practice?
It’s actually pretty simple. We use a Kanban board in Jira and horizontal columns (swimlanes, one per team member) with a few custom fields and changes. The following guidelines define the essence of this process:
-
For each project, a new Epic ticket is created, and the work is divided into tasks (Tasks).
-
Each task is marked with a Phase – a custom field in Jira for choosing one of the 6 stages listed above. (Note that there can be multiple tasks in one step.)
-
Tasks cannot last longer than 1 week (5 days). This forces team members to break down their work into smaller (but still quite significant) chunks so that progress can be tracked with minimal overhead.
-
There can be only one ticket in operation at a time. This ensures that we always see the status of the project.
-
The stages are not always sequential, and you can move between them as new tasks form.
Conclusion
Machine learning and data science project management doesn’t have to be difficult. At first, I spent about 30 minutes a day monitoring this process, but once the team got used to it, my time was reduced to 15 minutes a week! I know at what stage each project is at any given time, how long it took, and I can quickly identify problems in order to intervene and help my team if necessary. My data scientists have a clear framework for building a model, and they have become much more effective at it.
I hope that all this will be as useful for you as it is for me.
The translation of the article was prepared on the eve of the start of the course “Industrial ML on Big Data”…
We invite everyone to visit free demo lesson within the framework of which we will talk about why reproducibility of the result is extremely important in ML projects and how to achieve it.
