Regulations don’t work. Or how we invented a production framework for an IT team of 3000+ people

Hello everyone. It so happened that this article has two authors at once, we work at X5 Technologies and are responsible for building the production processes of IT solutions. Given the scale and complexity of our organization, this is not a trivial task. We thought that perhaps our experience will be useful to someone outside of X5, and this is how this article appeared.

It’s Complicated

It all started with the fact that in September 2020, the X5 Group merged several of its technology divisions into X5 Technologies – a dedicated structure that is responsible for digital development for all companies in the group.

The structure turned out to be large (3000+ people) and complex, combining in itself:

  • Tough guys responsible for the development of critical and high-loaded core systems from the IT department.

  • Agile teams developing products based on our Big Data from the Big Data Directorate, who are used to working in a very rapidly changing environment.

  • The guys from IT “Pyaterochka” and “Perekrestok”, who have always been as close to business as possible and have always lived in a paradigm – I see the goal, I see no obstacles.

We also got:

  • A complex and varied technology stack.

  • A large number of tools in which teams conduct their work (a huge customized Jira installation with hundreds of projects and dozens of integrations, a lightweight and flexible Youtrack that is practically not integrated with anything, several Git repositories, several Sonar installations, various CI / CD tools, etc.). etc.).

  • Different approach to working with these tools and creating artifacts.

  • Scattered and not always up-to-date knowledge in the form of hundreds of wiki pages on Confluence.

What problems do we solve

It is quite difficult for employees to navigate this environment, especially if you are a beginner. But even for an experienced employee, when moving to a new team, everything can start from scratch, because completely different tools and rules can be used in it.

For teams, creating and configuring all the necessary environment for work (Jira, Git Lab, Sonar, Confluence, Allure) takes a lot of time, which can affect the speed of development. And the complexity of finding the information you need can lead to the fact that sometimes you have to reinvent the wheel, making functionality that was previously implemented in another solution.

It is important for management that the production process is transparent, manageable and efficient, supporting the growth of the business and our volume of changes.

Regulations don’t work

In any case, this is so for us.

After all, we are faced with the task of finding a balance, where, on the one hand, is the organization’s desire to build a control system for the work of thousands of IT workers, and on the other hand, the teams are striving to be as independent as possible, but at the same time receive tools from the company for convenient work.

Regulations can be written (usually in dry clerical language), several thousand people can be notified of their introduction (10% of employees will read them, even less understand what they really want from them now), set KPIs to managers and then audit them with some frequency their implementation. But such a model will work only on paper, without solving real problems, creating the illusion of control in the organization and a feeling of encroachment on freedom for teams.

We decided to go a little differently.

Production framework

Now let’s try to tell you what we mean by this concept.

In X5, we mainly conduct development by dedicated teams (be it a product, project or IT system), and the teams are staffed with employees of certain competencies: java developer, business analyst, solution architect, delivery manager, data analysts, etc. other. In our structure, there is such a concept as a competence center – a unit that is responsible for recruiting, professional development and assigning employees with a certain competence to teams.

Together with the leaders of the competence centers, we began to solve our problem.

1. Tools

Every day, our employees use a variety of tools and services in their work: Jira, Git, Sonar Cube, Confluence and DevOps services. It’s long to list everything, but there are basic ones where most of the employees work.

The decision was that the tools could become the reasonable constraint that would allow setting some minimum boundaries for teams. Of course, some people like (conditional) Trello better, but sorry, we have a different corporate tracker. The same tools will create a framework to keep the sand from scattering outside the sandbox.

2. Artifacts

Various kinds of artifacts are created in these instruments every day. This could be a commit in Git, an issue in Jira, a test case in Allure, and so on.

Having set certain rules for creating such artifacts within each competency, we set a second reasonable level of restrictions that are understandable for teams.

3. Rules

Well, when your commit in Git is tied to an issue in Jira, and a test case in Allure is tied to the acceptance criterion of a user story in Confluence, somewhere a middle manager cries (with happiness, of course). Everyone understands that artifacts are logically related to each other, but not all of these connections are indicated in the course of their work.

It is in this connection that the last link of our model lies: the artifact tool – the interconnection of artifacts. This scheme is understandable and close to our employees.

This model formed the basis of our production framework:

  • Collected all the proven practices of working with tools, requirements for the format of artifacts and their relationship.

  • We grouped them according to production stages: business analysis, development, testing, etc.

This information was placed in a convenient and understandable form on the internal portal framework.x5.ru, where any employee can quickly find the information he needs.

Digital footprint

Now we have a set of basic rules that everyone understands and is available on a beautiful portal. How now to involve all participants in the development process and use the knowledge that we have collected on a separate portal? How to understand that everyone is using them in their work?

I need to call the guys from XSolla… We already know how to collect data from our
production tools (we call this the digital footprint) in our cluster
Big Data.

And by setting rules in the framework of the framework in the form of systems, artifacts and their interconnections, we form a digital model of the production process. By analyzing the digital footprint, the team and organization can understand whether the framework rules are being followed or not. And if not, then you can dive into the problematic and understand what went wrong.

Lego or framework as a service for teams

So, we already have clear rules of work for teams, there is a mechanism for tracking their execution through a digital footprint. How does the team work with this? How to use it in your daily work?

Now we are developing a service that should answer these questions.

1. Automation or building a constructor

As we said above, creating a working environment for a team is not a trivial task. And when we introduce additional rules, things get even more complicated. But this is if the environment is created as before, through several requests, which are executed by people responsible for supporting different tools.

When there is a fixed set of tools and clear rules for working with them within each of the production stages, the automation of creating an environment becomes a fairly understandable task.

This is the kind of service we are creating and are now piloting. The team assembles the production process for themselves and receives a customized environment in working tools.

2. Monitoring

Everything is clear enough here. At the team level, we show how it corresponds to the framework for each stage of production, what are the deviations and what needs to be done to eliminate these deviations.

So far, this is MVP, but further – more)

What’s next?

In addition to introducing a production framework to our many teams, we are thinking about the following things:

  1. Library of typical solutions. We have long wanted to systematize our code base and give teams the opportunity to save time by using ready-made libraries for solving typical tasks (monitoring, authorization, etc.).

  2. Metrics. We are already actively moving in this direction, but since the teams, as a rule, work in different tools and according to different rules, the date set leaves much to be desired. The implementation of the framework solves this problem and makes it possible to collect various metrics of the production process both in the context of teams and in an aggregated form by production stages and centers of competence.

  3. Gamification. Framework rules, metrics, typical solutions. All this can be wrapped in game mechanics and make the production process more interesting for employees and teams. We will try.

We hope it was interesting and not very boring. We are ready to answer your questions in the comments, and if the topic comes up, continue in more detail in the following articles.

Similar Posts

Leave a Reply