Hello everyone! My name is Timur Gilmullin, I am the deputy head of the department of technologies and development processes at Positive Technologies, I am engaged in automation and implement the ideas of DevOps. Today I’ll tell you how I got into the profession, how in our department we see the career of a DevOps engineer, what a competency map is, and how it helps to ensure employee growth.
This article is not an attempt to describe the ideal career path of a DevOps engineer. Our goal is to share experience and tell how it works with us. There are specialized companies (,) and communities () that make a significant contribution to the development of DevOps ideas in Russia, and we have selected a stack of technologies and techniques suitable for us.
About how we developed DevOps ideas in our department and in the company, you can read on Habré:
- Mission accomplishable: how to develop DevOps in a company with many projects (2016-09-21)
- Personal experience: what our Continuous Integration system looks like (2016-10-26)
- Automation of development processes: how we at Positive Technologies implemented the ideas of DevOps (2017-12-05)
- Chaos management: putting things in order with a technological map (2019-12-17)
And now, let’s go!
Why do we need DevOps
Each company has its own specifics of the automation department. As a rule, those tasks that are difficult or too costly to carry out at the level of several separate departments are transferred to a centralized group or automation department. In our company global goal the implementation of DevOps ideas is to ensure a consistent reduction in the cost of production and product support in quantitative terms (man-hours or machine-hours, CPU, RAM).
Based on this goal, stands out role automation department at the level of the whole company – this minimizes the cost of performing typical serial tasks at all stages of production.
How it works
Functional responsibilities Automation departments are also highly dependent on the specifics of a particular company. Our company is a developer and vendor of information security solutions, and the automation department is responsible for the production conveyor – from assembling individual components of products to sending them for testing and delivering updates to the customer’s infrastructure. We help automate the main development processes in teams and ensure the performance of our systems (continuous integration and continuous delivery (CI / CD)).
Our work is divided into several functional areas. The direction of continuous integration supports assembly and test pipelines for developers and testers. The direction of infrastructure is engaged in the operation and optimization of the use of all the “iron” resources of the department, as well as the automation of the deployment of virtual machines and their environment (infrastructure as code). The direction of monitoring provides control of the daily availability of services. We also provide monitoring as a service for developers (monitoring as a service).
The workflow direction provides teams with tools for managing development and testing processes, analyzing the state of the code and obtaining project analytics. And finally, webdev provides publishing releases on corporate update servers (GUS and FLUS), as well as licensing products using the License Lab service.
To support the production pipeline, we set up and maintain many different support services for developers. Stories about some of them can be heard on our old meetings: Op! DevOps! 2016 and Op! DevOps! 2017. We also develop internal automation tools, including:
The experience of our engineers
The guys from our automation department (for simplicity, informally called the DevOps department) have a completely different background: there are former testers, programmers, engineers and IT administrators. I am a diploma teacher in mathematics and computer science. As it turned out, thanks to the diversity of experience, we were able to cope with the tasks of all our areas.
There is not a single engineer in our department who could single-handedly solve all problems in all areas. But together we, as an administrative unit, are thereby SRE (just not a site, a services reliability engineers, since we support services for developers), which HR specialists in the person of one engineer unsuccessfully seek. We believe that a wide variety of product tasks of the company can be solved only as part of a team of diversified engineers.
We have a great many technical cases for implementing automation. But the main thing is to explain to people why they need this automation. The easiest way is when the technical task comes from business, that is, when people who bring money to the company have a clear understanding: what, how and when to do or optimize. I suggest you look at our automation cases: “Personal experience: what our Continuous Integration system looks like.”
About a career in our DevOps department
I would like to say that everything was ready and planned in advance, but this is not so. At first, we had nothing in our plans for growth and development. In 2014, when I moved to the department of technology and development processes, product development lived in the spirit of a startup: everything needs to be done here and now, long-term plans were not accepted, there were a lot of “legacy”. There were four engineers then, and we had many technical tasks: we urgently needed to do CI, scale the assembly line, before that, of course, create this line, and also build interaction with our internal customers – programmers from the development departments.
However, over time, the processes improved, the department grew. Together with him, there was a growing understanding of what our engineers wanted to know: what are their development prospects as specialists, what can the department offer for new engineers? The first thing that logically follows from this is that we will need a growth table for the positions and categories of engineers. As the saying goes: When society has no color differentiation of pants, then there is no purpose! And when there is no goal …
We made everything as simple as possible. The internal organizational structure of our department consists of three leadership positions (department head, deputy leader and group leaders) and four executive positions (junior, regular, senior and leading software engineers), which in turn are divided into three categories each. The human resources department often uses a similar job title, but without division into categories. For our department, the categories helped to ensure a smoother and more gradual growth of employees as their areas of responsibility and workload increased.
For each category, we developed documents with job descriptions, where the functions and roles of employees were prescribed, and also the areas of responsibility of employees for services and areas of work were indicated.
But since we, in addition to engineering, also love to program, and none of us likes to read boring documents, here too we simplified our life a little. We wrote each instruction not in Word, but in the form of plain text formatted with a special markup language Markdown. At the same time, its readability by a person in any editor is not lost. After that, we saved these texts in our GitLab repository. GitLab itself can very beautifully display various document formats, including Markdown draws so that it can not be distinguished from Word. And the standard Git client has many additional features, for example, it can show diff (difference) between two documents.
This greatly simplifies the life of an engineer who wants to find out what formal requirements he needs to follow in order to move to the next step (category) of personal growth in our department. To find out the difference between the formal requirements of the two job descriptions, you need to perform a few simple steps: download the repository with job descriptions from our GitLab, find documents, execute the Git command in the console to display a comparison of the two files. For example, you can see the difference between a senior engineer of the 2nd and 1st categories as follows:
git --no-pager diff --no-index
In the console, which all software engineers are used to, the minus sign and red color stand out for requirements that have changed or have been deleted, and the added lines stand out with a plus sign and green color: each line is plus one new requirement.
Yes, we are a little techie maniacs, but then it seemed to us a great solution:
DevOps Engineer Competency Map
As our department developed and the number of product conveyors supported by us increased, we realized that for each of the areas of work that we are engaged in, it will not be possible to describe a unique role and find a suitable engineer in the market. We have our own specifics of work and, for example, we do not need a mega-skilled programmer software developer for solving CI problems, but nevertheless, a CI engineer must be able to program small modules and scripts in Python at an acceptable level. Likewise, we do not need a mega-qualified Linux administrator, but we need a person with sufficient knowledge of the Debian or Ubuntu OS so that he can install the GitLab CI build agents, and even better, automate these works through SaltStack, Ansible or other tools.
So what should a DevOps engineer working in our department be able to do? To do this, you first need to decide what knowledge, skills and abilities are (abbreviated ZUN) in the general sense.
- Knowledge – these are the main laws of the subject area, allowing a person to solve specific production, scientific and other problems, that is, facts, concepts, judgments, images, relationships, estimates, rules, algorithms, heuristics, as well as decision-making strategies in this area. Knowledge is also information elements connected with each other and with the outside world.
- Abilities – they mean a method of performing an action mastered by a person, provided with a certain body of knowledge. The ability is expressed in the ability to consciously apply knowledge in practice.
- Skills – These are automated human actions that are developed in the process of consciously performing certain work operations. The fact that this action has become a skill means that as a result of exercises a person acquired the ability to carry out work operations without making this fulfillment his conscious goal.
If we define the ZUN more specifically in relation to the products developed in the company, then we will get a general list of competencies. Without mastering these competencies, the engineer will not be able to work qualitatively in our department. The list turned out to be very long, because it takes into account the specifics of work immediately in all areas, technologies and products.
Thus, we came to the need to describe and classify all our requirements for the engineer in a tabular form, and we had “DevOps Engineers Competency Map 1.0“. It looks like this:
The table is divided into four large sections:
- Description of the general competencies of the employees of our DevOps-department, necessary for the successful solution of everyday tasks.
- Knowledge is the specific, product-oriented knowledge of DevOps engineers.
- Skills – the ability to put knowledge into practice to solve our product problems; ability to work with the tools and technologies used by us.
- Skills – professional knowledge of our tools and technologies; studied and brought to automatism actions in solving typical tasks that do not require special efforts to perform them.
In the cells of the table, qualitative assessments are indicated: at what level should an engineer approximately have one or another competency. Unfortunately, I can’t imagine the full version of the table here, but this is not necessary, since each company and each department must create its own similar table, taking into account the specifics of the work.
In 2018, my colleagues and I developed and created the so-called technological map of the production process (read the article on the Habr “Chaos management: we put things in order using the technological map”). Thanks to her, we have come close to creating a vector of growth and development of the competencies of DevOps engineers depending on the product, the technologies used in it, and the stages of the product conveyor.
Such a map describes all the stages and steps that make up the technological conveyor for the production of our products. The main thing, from a product point of view, is to understand how many engineers, what qualifications and with which competencies we need to ensure the continuous operation of this conveyor. Better yet, plan and secure people so that, despite the support of critical services, they can, for example, go on vacation calmly, knowing that there is someone with cross-competencies in the department and who can replace them.
All together, this should lead us to the “DevOps Engineers 2.0 Competency Map”, which will clearly describe the ZUN depending on the product and on the necessary competencies for the automation work in this product. That is, some binding of the stages on the technological map to the map of engineers’ competencies should appear. In the next article I will try to tell you what came of it.
Author: Timur Gilmullin – deputy. Head of Technology and Development Processes (DevOps), Positive Technologies.