The bank outsource its projects to many contractors. “Outsiders” write the code, then transmit the results in a not very convenient form. Specifically, the process looked like this: they transferred a project that passed functional tests from them, and then was tested already within the banking perimeter for integration, load, and so on. Often it was discovered that tests were feyl. Then everything went back to the external developer. As you can guess, this meant a long time to fix errors.
The bank decided that it is possible and necessary to drag the entire pipeline to itself “under the wing” from commits to release. So that everything is consistent and under the control of the teams responsible for the product in the bank. That is, as if the external contractor simply worked somewhere in the next room of the office. On the corporate stack. This is an ordinary devopa.
Where did Sec come from? Bank security has set high demands on how an external contractor can work in the network segment, who has access, how and who works with the code. It’s just that IB didn’t yet know that when contractors work outside, few banking standards are respected. And here in a couple of days everyone needs to start observing them.
One simple revelation that the contractor has full access to the product code has already turned their world upside down.
Which bank made practical conclusions from this situation
There was a lot of controversy about the fact that everything is being done in the sphere wrong. The developers said that security is only occupied with attempts to hinder development, and they, as watchmen, try to prohibit without thinking. In turn, security guards hesitated between choosing between points of view: “developers create vulnerabilities in our circuit” and “developers do not create vulnerabilities, but they themselves are.” The debate would go on for a long time, if not for the new market requirements and the emergence of the DevSecOps paradigm. It was possible to explain that this very automation of processes taking into account the requirements of information security “out of the box” will help everyone to be satisfied. In the sense that the rules are written immediately and do not change during the game (IS will not prohibit something unexpectedly), and developers keep IS informed about everything that happens (IS does not encounter something suddenly). Each team is responsible for ultimate safety, and not some abstract older brothers.
- Since external employees already have access to the code and a number of internal systems, you can probably remove from the documents the requirement “development should be carried out entirely on the bank’s infrastructure”.
- On the other hand, it is necessary to strengthen control over what is happening.
- A compromise was the creation of cross-functional teams, when employees work closely with outside people. In this case, you need to make the team work on tools on the bank’s servers. From the beginning to the end.
That is, contractors can be allowed, but you need to make them separate segments. So that they do not drag outside some kind of infection into the bank’s infrastructure and that they don’t see more than necessary. Well, so that their actions are logged. DLP for protection against “sinks”, all this was applied.
In principle, all banks come to this sooner or later. Then we went off the beaten track and agreed on the requirements for such environments where “outsiders” work. There appeared a maximum set of access control tools, vulnerability checkers, antivirus analysis on circuits, assemblies and tests. This is what they called DevSecOps.
It suddenly became clear that if before this DevSecOps banking security had no control over what was happening on the side of the developer, then in the new paradigm, security is controlled in the same way as ordinary events on the infrastructure. Only now there are alerts on assemblies, library control, and so on.
It remains only to transfer the team to a new model. Well, create an infrastructure. But these are little things, this is how to draw an owl. Actually, we helped with the infrastructure, and at this time the development processes changed.
What was changing
They decided to implement it in small steps, because they realized that many processes would fall apart, and many “outsiders” might not be able to withstand the new working conditions under the supervision of everyone.
First, we created cross-functional teams and learned how to organize projects taking into account new requirements. In terms of organizational discussion, what processes. It turned out the assembly pipeline with all the responsible.
- CI: Git, Jenkins, Maven, Roslyn, Gradle, jUnit, Jira, MF Fortify, CA Harvest, GitlabCI.
- CD: Ansible, Puppet, TeamCity, Gitlab TFS, Liquidbase.
- Test: Sonarqube, SoapUI, jMeter, Selenium: MF Fortify, Performance Center, MF UFT, Atascama.
- Presentation (reporting, communication): Grafana, Kibana, Jira, Confluence, RocketChat.
- Operations (maintenance, management): Ansible, Zabbix, Prometheus, Elastic + Logstash, MF Service Manager, Jira, Confluence, MS Project.
Selected a stack:
- Knowledge Base – Atlassian Confluence;
- Task Tracker – Atlassian Jira;
- Repository of artifacts – “Nexus”;
- Continuous Integration System – “Gitlab CI”;
- Continuous analysis system – “SonarQube”;
- Application Security Analysis System – “Micro Focus Fortify”;
- Communication system – “GitLab Mattermost”;
- Configuration Management System – “Ansible”;
- Monitoring system – “ELK”, “TICK Stack” (“InfluxData”).
They began to create a team that would be ready to drag contractors in. The realization has come that there are several important things:
- Everything should be unified, at least in code transfer. Because how many contractors there were – so many different development processes with features. It was necessary to put everyone in about one, but with options.
- There are many contractors, and manual creation of infrastructure is not suitable. Any new task should start very quickly – that is, the instance should be deployed almost instantly, so that developers receive sets of solutions for managing their pipeline.
To take the first step, you had to understand what was being done. And we had to determine how to proceed to this. We started by helping to draw the architecture of the target solution, both in infrastructure and in CI / CD automation. Then we began to assemble this conveyor. We needed one infrastructure, the same for everyone, where the same conveyors would run. We offered options with settlements, the bank thought, then made a decision on what and on what means it will be built.
Further creation of the circuit – software installation, configuration. Development of scripts for deployment infrastructure and management. Next is the transition to pipeline support.
We decided to run everything on the pilot. Interestingly, during the piloting, a certain stack appeared in the bank for the first time. Among other things, a domestic vendor of one of the solutions for an early launch was proposed for the pilot’s volume. Safety got to know him during the piloting, and this left an unforgettable experience. When they decided to switch, fortunately, the infrastructure layer was replaced by a nutanix solution that was already in the bank before. And before that it was for VDI, and we reused it for infrastructure services. On small volumes, it did not fit into the economy, but on large volumes it became an excellent environment for development and testing.
The rest of the stack is more or less familiar to everyone. Automation tools in the Ansible part were used, and security guards worked closely with them. The Atlassinovsky stack was used by the bank before the project. Fortinet Security Tools – It was proposed by the security guards themselves. The testing frame was created by the bank, no questions asked. Questions were caused by the repository system, I had to get used to it.
Contractors were given a new stack. They gave time to rewrite under GitlabCI, and to migrate Jira to the bank segment and so on.
Step by step
Stage 1. At first, we used the solution of a domestic vendor, the product was switched to a new created DSO network segment. The platform was selected for delivery times, scalability, and full automation capabilities. Conducted tests:
- Possibility of flexible and fully automated management of the virtualization platform infrastructure (network, disk subsystem, computing resources subsystem).
- Automation of life cycle management of virtual machines (templating, snapshots, backups).
After the installation and basic configuration of the platform, it was used as a point of placement of the second stage subsystems (DSO tools, retail system development outlines). The necessary sets of pipelines were created – creating, deleting, modifying, backing up virtual machines. These pipelines were used as the first stage of the deployment process.
Result – the equipment provided does not meet the bank’s requirements for performance and fault tolerance. DIT Bank decided to create a complex based on PAC Nutanix.
Stage 2. We took the stack that was defined and wrote scripts for the automated installation and post-configuration for all subsystems, so that everything would be transferred from the pilot to the target circuit as quickly as possible. All systems were deployed in a fault-tolerant configuration (where this feature is not limited to the vendor’s licensed policies), and are connected to the subsystems for collecting metrics and events. IB analyzed the compliance with its requirements and gave a green light.
Stage 3. Migration of all subsystems and their settings to the new PAC. The infrastructure automation scripts were rewritten, and the DSO subsystem migration was performed in a fully automated mode. The IS development paths were recreated by development team pipelines.
Stage 4. Automation of application software installation. These tasks were set by team leaders of new teams.
Stage 5. Exploitation.
Development teams asked for maximum flexibility in working with the circuit, and the requirement for remote access from personal laptops was put at the very beginning of the project. The bank already had remote access, but it was not suitable for developers. The fact is, the scheme used a user connection to a secure VDI. This was suitable for those who had enough mail and an office package at the workplace. Developers would need heavy clients, high-performance, with lots of resources. And of course, they had to be static, since the loss of a user session for those who work with VStudio (for example) or another SDK is unacceptable. Organizing a large number of thick static VDIs for all development teams greatly increased the cost of the existing VDI solution.
We decided to work out remote access directly to the resources of the development segment. Jira, Wiki, Gitlab, Nexus, assembly and testing stands, virtual infrastructure. Security guards have set requirements that access can be organized only subject to the following:
- Using the technologies already available in the bank.
- The infrastructure should not use existing domain controllers that store records of productive accounts / objects.
- Access should be limited only to those resources required by a particular team (so that a team of one product cannot access the resources of another team).
- Maximum control over RBAC in systems.
As a result, a separate domain was created for this segment. This domain housed all the resources of the development segment, both user credentials and infrastructure. The life cycle of records in this domain is managed using the existing IdM in the bank.
Directly remote access was organized on the basis of existing bank equipment. Access control was divided into AD groups, which corresponded to the rules in the contexts (one product group = one rule group).
Manage VM Templates
The speed of creating the assembly and testing circuit is one of the main KPIs set by the head of the development unit, because the speed of preparation of the environment directly affects the overall execution time of the pipeline. Two options for preparing basic VM images were considered. The first is the minimum image size, default for all products / systems, the maximum compliance with the bank policies for the settings. The second is a basic image containing the installed heavyweight software / software, the installation time of which could greatly affect the speed of the pipeline.
Infrastructure and security requirements were also taken into account during development – maintaining the images up to date (patches, etc.), integration with SIEM, security settings according to the standards adopted by the bank.
As a result, it was decided to use minimal images in order to minimize the cost of maintaining them up to date. It is much easier to update the base OS than to install patches in each image for new versions of software / software.
Based on the results, a list was compiled from the minimum required set of operating systems, whose update is carried out by the operating team, and the scripts from the pipeline are fully responsible for updating the software / software, and if necessary, changing the version of the installed software is sufficient to transfer the desired tag to the pipeline. Yes, this requires the devops’a product team to have more complex deployment scenarios, but it significantly reduces the operating time for supporting basic images, which could fall on servicing more than a hundred basic VM images.
Access to the Internet
Another stumbling block with banking security was access from the development environment to Internet resources. Moreover, this access can be divided into two categories:
- Infrastructure Access.
- Access to developers.
Infrastructure access was organized by proxying external repositories by Nexus. That is, direct access from virtual machines was not provided. This made it possible to compromise with IS, which was categorically against providing any access to the outside world from the development segment.
Internet access for developers was required for obvious reasons (stackoverflow). And although all the teams, as mentioned above, had remote access to the circuit, it is not always convenient when you cannot do ctrl + v from the developer’s workplace in the bank in the IDE.
It was agreed with IS that initially, at the testing stage, access will be provided through a bank proxy based on a white-list. At the end of the project – access will be transferred to the black-list. Huge access tables were prepared, which indicated the main resources and repositories that needed access at the start of the project. Coordination of these accesses took a decent amount of time, which allowed us to insist on the fastest transition to black lists.
The project ended a little less than a year ago. Oddly enough, but all the contractors on time switched to the new stack and no one left because of the new automation. IB is in no hurry to share positive reviews, but he does not complain, from which we can conclude that they like. Conflicts subsided, because IS again feels control, but at the same time does not interfere with the development processes. The teams got more responsibility, and in general, the attitude towards information security has become better. The bank understood that the transition to DevSecOps was almost inevitable, and in my opinion did it most gently and correctly.
Alexander Shubin, systems architect.