Introduction to Dev Platform
Internal Development platforms (Internal Development Platform, IDP), as one of the artifacts of the implementation of the platform approach (Platform Engineering), are becoming an important growth point for many companies engaged in development: they help to unify the stack, reduce the non-target load on teams, and reduce time-to-market. Therefore, many companies are starting to move towards creating their own IDPs.
We continue our series of articles on Platform Engineering and Internal Development Platform (IDP). In first And second In the materials of the series we have already considered the basic theory, got acquainted with the typical IDP architecture and implementation options. And in this article we will focus on the review of Dev Platform, developed by the VK Cloud team.
A bit of context
Platform Engineering is a methodology for organizing development teams and tools around them that allows developers to offload unnecessary non-core workload, thereby increasing their productivity in the context of delivering business value.
Internal Development Platform (IDP) is an internal development platform, which is one of the options for implementing a platform approach. Classically, an IDP is a portal that, in a “single window” format, gives developers access to tools, resources, built pipelines, stand templates, and other pre-configured components and solutions.
About Dev Platform
Building a platform is quite a complex task. This is due to the fact that the product must include many services to cover the entire development pipeline: from a task tracker and code storage to managing the infrastructure where the application will be deployed.
At the stage of the Dev Platform’s inception, we considered two options for implementing the solution:
in the form of a set of products that are integrated with each other (example: the Atlassian stack);
in the form of a large all-in-one combine that includes all services at once and does not involve the delivery of services separately (for example, Azure DevOps or Gitlab).
The first option certainly has a number of advantages – the development of individual services can easily be separated into a separate direction and has its own sales track. For example, Task Tracker can be developed separately and sold independently of other components.
But our research showed that the best Developer Experience was in solutions built as All-in-one. Moreover, the degree of integration of services with each other in solutions of this type is usually higher. Therefore, in order for development teams to get maximum opportunities with our product, we chose the path of creating a single solution that would include all the necessary services “out of the box”.
From concept to implementation
After defining the product boundaries and choosing a delivery option, we had to determine what exactly should become the “central core” for teams to work with the resources they need within the product: a project, a repository, or a specific service. The Developer Experience of platform users also depends on this.
For example, in GitHub all processes are built around the repository. That is, the repo-first approach is implemented. This is convenient for open-source development, when work is carried out with one or several publicly available repositories. For example, this is an excellent solution for the inner-source approach when working with code. It is obvious here – such a focus makes it difficult to saturate the platform with new services and their resources, since the repository will be in first place, even despite the fact that it is only one of the team's tools.
Gitlab has a different approach. Repo-first also prevails here, but the repository is integrated into the project, which in turn is the central point of work with resources: environments, code, CI/CD, tasks. That is, the project can be saturated with additional services and resources. For example, if a new service appears, it will simply become available in the project. This is how the project-first approach is implemented.
But this implementation in gitlab has a drawback – one project always corresponds to one repository. That is, there is an implicit dependency of the project on the repository.
This drawback is eliminated in Azure Devops. So, here the project, as in gitlab, is the central place of the team's work, but there is no rigid binding of the project to the repository. That is, the repository is one of the resources, and there can be several of them. This is convenient if one project where the team works needs several repositories – for example, for DevOps, QA, Front, Back.
One way or another, in github, gitlab and Azure DevOps projects and repositories are wrapped into a certain hierarchy:
in Azure DevOps and Github, these are organizations;
in Gitlab these are groups (organizations will also be added soon).
This hierarchy is very convenient for organizing multitenancy (relevant for SaaS and some on-prem customers) and working in one product of several divisions of one holding.
Having weighed all the pros and cons of existing solutions, we settled on a project-first approach for implementing Dev Platform without a rigid binding to a specific resource (like Azure DevOps). This will allow us to scale services in the product without changing the user experience and resource hierarchy. To implement multitenancy and provide flexibility to large companies, we are currently working on organizations into which projects will be integrated.
Thus, within the Dev Platform we implement the All-in-One approach with an organizational-project-oriented model. This allows our users to work with such unambiguous and clear concepts as: organization, project, service and resource. On the one hand, this makes it possible to support the organizational structure of our clients in the product, and on the other hand, it will help to organize the work of each individual development team in a single window mode.
Dev Platform Composition
We provide Dev Platform users with access to services such as:
task tracker;
git repositories;
CI/CD;
infrastructure deployment services;
code quality management system.
It is worth mentioning here that we do not implement all services ourselves – this is difficult, irrational and limits users. Instead, we focus on two points:
we allow the platform to be integrated into any surrounding landscape;
We implement the platform in such a way that the development and onboarding of new services is easier for end users.
For example, at the first stage, for integration with other products, we are actively developing a system of web hooks and a public API of the product. In the future, as development proceeds, a system of plugins will be implemented, which will allow us to significantly expand the capabilities of the platform.
Dev Platform Delivery Options
An on-premise version of the Dev Platform delivery is already available — the solution can be installed in the Customer's infrastructure (on top of the VK private cloud). In the near future, we will also provide the ability to deploy in the VK Public Cloud, as a single-tenant installation. The plans for 2025 include the implementation of a SaaS version of the product.
At the same time, as we have already mentioned in previous articles (Here And Here), the greatest positive effect is provided by the ecosystem. That is, the maximum potential of Dev Platform is possible in conjunction with VK Cloud (both public and private clouds). Moreover, on the basis of the VK Cloud platform, it is possible not only to deploy infrastructure for IDP, but also to build a catalog of services. Using the entire stack of tools available on the cloud platform and deploying Dev Platform on top of VK Cloud allows you to build a fully ecosystem solution and get more value due to deep integration with cloud infrastructure management mechanisms. Moreover, in this case, for the developer to receive the necessary resources, no additional approvals, complex integrations, or long waits are needed.
Also, in the future, it will be possible to work with Dev Platform without being tied to the VK Cloud – on the infrastructure or platform that the user has.
Dev Platform as an IDP
When building an IDP, Dev Platform can act as a ready-made “core” with pre-configured, integrated and compatible services. Moreover, such integration of Dev Platform into IDP can be implemented with minimal effort on the part of the user – the solution is supplied as a “box” that is installed using an installer.
The implementation of Dev Platform primarily closes greenfield projects that involve developing a new solution or initiative. Objectively, it is much easier for teams to start using the platform to develop a new product.
But with proper preparation, brownfield projects can also be transferred to IDP, i.e. existing systems that have been developed for a long time by internal teams or external contractors. The experience of our internal teams has shown that this is possible, although not always easy.
At the same time, it is important to understand that installing Dev Platform as-is, unfortunately, will not lead to a restructuring of the organization and an explosive growth of the development culture. Following Team Topologiesto get the maximum benefit from implementing IDP in a company, areas of responsibility should be divided. Thus, the following are needed:
product teams (functional teams) that develop products;
the platform team that administers the Platform;
an enabling team that helps other participants in the process get involved in the new realities as quickly and effectively as possible.
What's “under the hood”
“Under the hood” of our platform are both common open source tools and services developed by VK. For example:
the platform core as an orchestrator of solution components;
self-service portal as a single window for development team members;
environments – a component that manages resources in VK Cloud;
GitLab is used as VCD and CI/CD;
The artifact repository is represented by Nexus Repository;
the environment deployment service is represented by our own development, which deploys infrastructure to VK Cloud based on Terraform files;
ALM system tasks are performed by a partner solution (optional: Devprom ALM or TeamStorm);
SonarQube is available for code quality analysis.
At the same time, the Dev Platform is mostly based on open-source components. The choice in favor of open-source is conscious and provides a number of important advantages.
Low entry threshold. Many companies work with open-source tools, so they can switch to Dev Platform almost seamlessly – in most cases, you won’t have to laboriously and painfully rebuild the pipeline from scratch, abandon custom implementations and scripts, or master new solutions.
Reducing vendor lock-in risks. By providing open source services to users, we eliminate situations in which a tool may become unavailable due to a decision by the developer company. This allows us to build stable development platforms.
But we are not satisfied with the capabilities of open-source “in its pure form”, and we expand them within our product. For example, we have implemented Gitlab and Nexus HA, and we also plan to add in the near future:
multiple approvers in code review;
merge request approval settings;
prohibiting merge without receiving approval from all reviewers;
merge request approval rules;
merge rules;
code owners;
remote repository pull mirroring.
At the same time, we are responsible for updating all components so that users have access to all the requested functions and updates.
Features of our implementation
Dev Platform provides a portal that is a single entry point with SSO (single sign-on) support for fast, convenient and, most importantly, secure access to tools.
Within the Dev Platform, all tools are already integrated with each other, are ready to work almost immediately and are supplied as a single installer. At the same time, the VK Cloud team is responsible for updating and administering the Dev Platform components, removing the burden from users. The configuration of all components remains on the user side, which allows for flexible adaptation of the Dev Platform to the specific tasks of the company.
The Dev Platform portal allows for centralized role model management to distribute access rights to tools, functions, and work environments. This allows for logical division by teams, projects, and even organizations (which is especially important when outsourcing specialists). What is configured via the portal cannot be changed at the level of a separate tool, eliminating the risk of unauthorized changes to access rights.
Together with the platform, we supply a set of various practices, pre-configured Terraform and CI/CD templates that will speed up the assembly and deployment of applications on the VK Cloud infrastructure. For sharing developments, we currently use a repository based on GitLab.
The entry threshold to working with Dev Platform is minimal. Firstly, together with the platform we supply a set of methodological recommendations with guides on deploying solutions, setting up, examples of best practices and other materials. Secondly, we offer training and consulting services in terms of setting up tools, restructuring processes, auditing, support, implementing Application Security components, building an end-to-end development process and other tasks.
Dev Platform allows you to centrally manage your infrastructure. The solution also makes it possible to manage development not only within projects, but also within organizations.
Dev Platform has a tool for integrating external systems to easily connect new solutions via API and web hooks. This predetermines the possibility of further expansion of the Dev Platform stack and the functionality of the platform as a whole.
In Dev Platform, secure development is implemented through integration with solutions from our partners, who are leaders in the field of AppSec. When working with the platform, you can choose one of the prepared templates for scanning the pipeline for errors and vulnerabilities.
Dev Platform Scenarios
Dev Platform is an all-in-one product, so you can work with it in different ways.
Building an IDP. A basic scenario that involves using Dev Platform as a basis for building an IDP. In this scenario, the user can quickly deploy a ready-made set of pre-configured and integrated tools from the installer, reducing the cost of resources and time for preparing the IDP.
Setting up your own development process. In many enterprise companies, compliance is important in terms of security and IT infrastructure. Dev Platform allows you to limit the functions of tools at the portal level and set up templates for working with them (for example, in terms of storage, CI/CD, deployment, and more) in order to unify the pipeline, increase control over development cycles, and eliminate the misuse of resources.
Ensuring safe development. Nowadays, many companies prioritize secure development and integration with DevSecOps tools. Therefore, Dev Platform, together with the largest companies in the field of information security, has implemented integrations with AppSec tools. This allows you to completely close the end-to-end cycle, taking into account the requirements of the security software development lifecycle (SSDLC).
Separation into different development contours. Many companies implement separation by contours — for example, Dev and Prod. At the same time, the resulting entities are maximally decoupled from each other — as a rule, users have different access to each of the segments. For example, the Prod contour is more trusted, and a number of checks must be passed for code or artifact to get into it. If you deploy environments using Dev Platform, due to the maximum identity of tools and environments, there are no compatibility issues when transferring code to prod. Similarly, an additional instance of Dev Platform can be deployed to create a test contour of out-source development, closely connected to the main contour.
Working with contractors and remote offices. It is also possible to ensure the connection of contractors to the Dev Platform by dividing into organizations. This approach allows for providing differentiated access, including to development teams belonging to different business units and even departments. This way, you can create conditions for isolated work and information protection, but provide a single development environment to deliver value to end users as quickly as possible.
How it all works
For clarity, let's briefly examine the possibilities of working with Dev Platform for two categories of users:
project manager (or any other person in the company responsible for launching development and who has the authority to allocate resources);
developer.
Project manager
In Dev Platform, a project manager has the ability to create a new project or work with an existing one.
In addition, from a single window he can:
create and configure the necessary components;
manage users and/or user groups (integrated with AD);
grant access (for example, to a DevOps engineer to set up stands) and more.
Here it is necessary to make several remarks immediately.
The cloud project must be added in advance. This is done by the specialist responsible for virtual resources, or by DevOps himself, to whom the credits were transferred. Addition occurs through the service connection mechanism.
When deploying an environment, DevOps uses a terraform script to create the necessary resources in the selected cloud project.
The file with variables is needed so that developers can independently adjust the VM content and their number to their needs without the help of DevOps. At the same time, this option can be disabled to avoid potential misuse of resources.
Note: We will consider the implementation of interaction between our platform and VK Cloud in a separate article.
Developer
In most projects, developers have limited capabilities compared to managers, for example. A similar approach can be implemented in Dev Platform. For example, you can prohibit creating repositories from scratch to prevent modification of existing pipelines.
At the same time, the developer usually has full access to the tools and functions that he needs for work. What is especially important is that such access is provided within the framework of a “single window” – there is no need to search and launch dozens of programs in the background.
What's next?
Currently, Dev Platform is primarily an orchestrator of development tools. This allows us to collect all the necessary metrics from the tools connected to us and provide analytics so that users can track the effectiveness of delivering the value of their development and respond to incidents in a timely manner.
At the same time, we are currently continuing to work on moving from a tool orchestrator to a development process orchestrator, which will include a set of gates. With their help, it will be possible to standardize development and minimize the impact of the human factor (I remind you that disputes in the comments regarding the limitation of the process developer's role are not only allowed, but also welcomed).
In addition, our priority for the medium term is to increase technological independence and localization of the tools included in the platform stack. To this end, we plan a gradual transition to our own solutions and partner services developed in the Russian Federation.
We are already working closely with Dev Platform to solve internal tasks, including when developing Dev Platform — we close a significant part of the development pipeline with its help. At the same time, in the future, we plan to transfer development not only within VK Tech, but also, possibly, the entire VK to Dev Platform. What will come of this — we will tell in the following articles.
