From function to culture: how DevOps evolved in a large company
We have already talked on Habré about the digital products of the Mail – registered emails, shipping for business, electronic stamps… Those who use Mail may have noticed that we have begun to release more and more applications and services and have learned how to do it promptly. Now the time to market for new products in Mail is only 3 months, and releases are released every week.
We decided to look back and remember how a large company implemented DevOps practices to quickly respond to new requests.
Make the application a year, and roll back
Let’s start with the fact that, historically, the Mail business is offline. Digital products began to appear in our country relatively recently. Therefore, the development of the first projects for the Post was completely outsourced to external contractors. This caused difficulties: the work was carried out according to the waterflow method (waterfall) and took a lot of time, the term of product development could take a year or more. And when it happened that the contractor changed before the launch of the project, then we found ourselves, if not at the beginning, then in the middle of the path, and the company had no experience and expertise left at all.
To speed up, the Department of Technology Development was created in the Post – it was supposed to help systematize and improve work on digital products and develop its own expertise. Then the process of the birth of DevOps in Mail began – the first centralized tools were purchased that allowed to build and control processes, implement DevOps practices. For example, using GitLab helped us manage the code and its changes, and now we see what happens to it every day. TeamCity made it possible to build and test products automatically. As soon as changes appeared in GitLab, TeamCity would add and run them through the system. And Maven made it possible to see the same build standard in all products so as not to deal with a bunch of different systems. In the diagram below, you can see a more complete list of tools that we selected and implemented in order to automate the standards that appeared then in the company.
This is how the process we built for launching new products in Mail looked like
Thanks to the innovations, we were able to put the development process in order and reduce the release time by 4 times. If earlier external developers showed only the final result at the appointed time, now they regularly demonstrated and agreed on intermediate results. Partners started working on our tools and our process.
And on this rather classic scheme, it was possible to work for a long time, some companies generally live with it to this day, and feel great. But the logistics market is growing rapidly, and in order to keep up with demand, the Post decided to create its own team, which was supposed to accelerate digitalization. So in 2018, a special branch appeared in the Post – Postal Technologies, which has now become a separate legal entity. Pochtatech was faced with the task of speeding up development, reducing the time to market, gaining and retaining expertise within the company.
The origin of culture from within
If earlier it could take a year or more from an idea to a product launch, now our goal was to spend no more than 3 months to create an MVP with which you can enter the market. In order to switch to our own development and start accumulating full-fledged expertise, we began to assemble our own development team. With the arrival of new people, fresh ideas and approaches began to appear. Teams themselves began to implement new tools in order to reduce development time and gradually began to share their results with others, telling how they accelerated, how they got rid of waiting for build queues, and so on.
Thanks to the implementation of DevOps processes, we have well accelerated the product release process from idea to first user (MVP):
2017 | 2018 | 2019 |
365 days | 180 days | > 90 days |
How did we accelerate? They just gave the developers the opportunity not to be distracted from their main work. When the code is ready, it is automatically assembled, tested and a decision is made on delivery to production. Although the last stage in some products also works automatically.
We started using a product approach – now at Pochtatech, each team is responsible for their product inside and out. The project has its own development team, dedicated DevOps engineers, and some projects have their own support. The process is built in such a way that the development team is the same as the support team, which knows not only how the code is written, but also how it is assembled, tested, delivered to environments, works in production, the same team provides product SLA.
Products created using DevOps continue to thrive after launch, even if you have to transfer specialists between projects or completely change the team. Competent version control and stand management allows you to maintain and maintain expertise within the company and ensure reliable and stable growth, while limiting the amount of manual work.
Case study: how DevOps accelerated server delivery
The main task of the DevOps specialist at Mail is to find technical solutions to existing challenges and problems from the operational and development side. For example, last year we realized that due to the use of the virtualization platform, we spend a lot of time ordering new server capacities. To get a server, you need to wait from one day to a week, and after receiving it, allow time for a functional check. As a result, 2-3 weeks passed from the moment of the request until the server was put into operation.
Then we decided to develop in two directions: deploy our own cloud platform and use Kubernetes. Now everything works like this – we have ordered ten powerful enough machines, and on them we create environments as we need. The service ceases to be tied to a specific server – Kubernetes manages everything.
After the introduction of Kubernetes on some products, the process started working much faster. Now, when a business has a new task, then after discussion it is added to the bug tracker, a branch is created in GitLab (where development for the task will be carried out) and immediately an environment appears in Kubernetes where edits only for this task are stored. Here they will be tested separately from the rest and will not interfere with anyone, reports will be generated immediately. When a task is closed, it is merged into the main integration branch where tests are run.
At the beginning of the story, we showed what the structure of interaction with third-party development looked like. Now we turn to the processes and tools depicted in this diagram:
As DevOps has evolved, many of our tools have moved into Mail standards. This is how Confluence and GitLab became commonplace throughout the company. Along with GitLab, GitLab CI support was added and now all our new products use it.
Also, thanks to the introduction of new products and practices, we:
We built a configuration management system.
We have built a continuous integration system – assembly and deployment.
We have formed our approaches to how products should get into all environments – dev, test, stage, prod.
We deployed logging systems and a code quality control system
Added systems for working with documentation, bugs, test cases.
Infrastructure as Code (IaC) has begun to be promoted.
DevOps has infiltrated the company from everyday practice, without a request from the “top”. This chaotic generation from within did not go smoothly. Everyone who has implemented their tools and approaches has done it in their own way. A person could establish some kind of process, and after he left, no one knew what and how should work. The consequences of non-systemic implementation now have to be dealt with. In order to reduce all processes to a single approach, in 2020, a DevOps community was formed at Mail. The goal of the community is to make the culture develop in a systemic way. To do this, we began to build a structure and hire people specifically for DevOps tasks. That is, the most interesting thing is happening right now – the main approaches and standards are being built.
To culture through roles
So far, DevOps in Mail is more of a role in which certain tasks are laid: maintenance and automation of releases, automation of setting up monitoring and analysis of logs, participation in improving the product for a microservice architecture, proposals for bringing the product to a cloud-ready / native state. The word “role” in relation to DevOps may sound strange and strange, but the way a company develops imposes a non-standard attitude to the approach.
We understand that DevOps is not the responsibility of one person, but of the entire team, so now our main goal is to make DevOps become a culture in the company. So that everyone who participates in the creation of a product realizes that he must fully fit into the DevOps processes, and therefore be adapted for automation.
Now our DevOps have the ability to be directly responsible for the product, and for this we have a whole set of tools: systems for collecting logs, collecting metrics, monitoring, building, automatic configuration management and version control. DevOps can solve any problem both from the server side and from the code side. The code does not lie somewhere far behind access for three requests. If you need logs or metrics, they are open to all projects. When there is not enough data, you can implement an additional system to get it. The main goal is for the product to work, the quality grows and the development proceeds faster. Freed from routine, DevOps specialists are engaged in one of the most interesting tasks – ensuring the stable operation of the product under high load and frequent releases.
