Front without a release engineer, or How I stopped being afraid and fell in love with deployment

Hi, I am a front-runner, and in ten years of development in enterprises, startups and small companies, for the first time I deploy my code myself and am responsible for its operation, and not just for the design and development of the service. How I got to this and why I am not going to stop, in this article.

In our company, independent deployment is part of the workflow of many developers. This practice is typical, rather, for startups and is rare in medium and large companies. Why? It seems to me that the decision to completely transfer the deployment to dedicated specialists who play the role of release engineers occurs for quite understandable reasons:

• Separation of responsibility – in some cases, the responsibility for the deployment turns out to be high.

• The need for additional skills – usually you need to have a good knowledge of the platform through which the deployment takes place, or at least understand scripts like Capistrano.

• Complexity of infrastructure – when deploying, you have to take into account a lot of supporting services, such as mesh networks, queues, workers, databases, etc.

How it happens with us

Uchi.ru created its own tool for rolling out services and setting up stages called “Shaman” – in fact, a custom interface for HashiCorp Nomad. Several years ago, he helped move away from executing scripts on local machines, automate the deployment process, and allowed developers to create test configurations as quickly as possible. My colleagues from the infrastructure team wrote about this in a separate article…

A bit of history

From the very beginning, all development for us was carried out within a single monolithic repository, so “Shaman” knew how to assemble and roll it out, while doing an excellent job with its task. But the company grew, and along with it, markets were mastered, new projects were launched, adding cross-logical connections to an already complex code.

But in advance, development approaches evolve with us. Therefore, at some point, having eaten up the pain of 200 people committing to the monolith, we slowly began to bite off small semantically related pieces from it and take them out to services. Over time, more and more services spun off, and they naturally need to somehow communicate with each other – the complexity of setting up applications has exponentially increased.

May the savior come

And here “Shaman” really came to the rescue, because setting up such bundles manually correctly and predictably is a rather difficult task. He abstracted away the complexity of managing new machines with a lot of parameters, laying out services on them through docker containers, configuring and registering the endpoints of these services in the orchestrator – in general, a bunch of things that every developer’s brain would be problematic to grasp, and, in all honesty, not necessary. Instead, I see a harsh, minimalist, but nevertheless fairly straightforward interface, freeing me from the need to look into the eyes of the abyss of low-level administration.

What if you need to bring up a copy of the environment to test the second release? Not to create the whole bundle entirely from scratch? And if it is slightly modified? For this, in “Shaman” it is possible to create a configuration template, in which the whole scattering of machines, services and endpoints is described using YAML. As a result, if your team maintains their template, it becomes a trivial task to raise a new instance of a sprawling bundle of services.

And my colleagues seem to agree with me:

“For me personally, the deployment is not difficult. And given that sometimes I have to do more than ten test rolls a day, as well as deployments of completely new applications, I am very glad that I do not need to connect someone from the infrastructure department every time. We have a lot of applications, and the possibility of self-roll-out greatly simplifies life and speeds up work ”, – Roman Litvinov, ROR developer.

The scheme of communication “Shaman” with the code

At the moment, almost all of our GitHub repositories are built using GitHub Actions, as a result of which the docker image is pushed into a shared private docker rep. This workflow can be triggered both automatically (push to the release branch) or by hand if you want to test some immediate fix. Then the “Shaman” picks up a fresh image and rolls it out onto the stage using a button. Well, not beauty, huh?

And since this process is a little more than completely in charge of the developers, we have the opportunity to simplify it. For example, I am still frustrated by the need to first go to GitHub to see the build status, and then to the “Shaman” – to click on the cherished green button to roll out the image. After a minor shake-up of colleagues from the infrastructure, it turned out that the latter provides an API, the handle of which can be pulled from Github Actions with the address for deployment and the ID of the image for deployment. This means that you can deploy the code completely automatically!

When something goes wrong …

It is clear that after rolling out the code to the product, you need to make sure that it behaves as intended. And here we have a little worse with automation, but nevertheless everything is under control. For monitoring, we use the Sentry service, which notifies both new individual errors and changes in the general background. Well, no one canceled food dashboards with metrics. And “Shaman”, in turn, quickly and easily allows you to roll back the back-end version of the front-line application in case of serious problems.

So does a developer need to deploy?

My answer is yes, if there is such a system that makes life easier. Because your service is your fortress. And you need to be responsible for it at all stages, including in the context of rollouts. Yes, it requires more knowledge, but our operations department did a cool abstraction that hides most of the low-level processes.

Such a scheme allows not to accumulate the completed tasks into large releases that are tested for a long time, but to roll them out as soon as they are ready. More control and less chance of error.

Well, it’s a matter of time: deploying yourself is much faster than giving the release to dedicated engineers. But, of course, with the assumptions:

• Services come and go, respectively, the service bundle template must be kept up-to-date, otherwise raising new test benches will turn into a pain.

• It is worth keeping in mind that we are working with an in-house solution with all the consequences: it can break. Therefore, here you need to be open enough and help colleagues fix the problem, and not scold the solution.

• This approach requires a high level of development of procedures and processes to maintain the quality and stability of the service at a high rate of rollouts, as well as some courage from key stakeholders.

How does deployment work in your company? Do you find our one-button approach optimal, or do you like other options for rolling out new services and updates? Share your opinion in the comments.