To begin with, in general about the project. We have built a secure development process in a large trading company in which the IT department has a huge staff and is divided into many areas that are minimally correlated with each other. Conventionally, these areas can be divided into 3 main groups. The first, a very large group, is cash register software, which is written mainly in Java (90% of projects). The second, most extensive group of systems in terms of the amount of code is SAP applications. And finally, the third block was a “hodgepodge” of portals and mobile applications: various external sites for the company’s clients, mobile applications for these sites, as well as internal resources – mobile applications and web portals for the retailer’s staff.
The client of the project – the information security department – formulated the general task in a rather standard way for all three groups: “we want to have fewer vulnerabilities and secure development of all systems created within the company”. But in practice, in each specific department, everything looked very different from that of other colleagues, because at each step of implementing secure development, we had to make a million different compromises. Some nuances helped to build the process, while others, on the contrary, interfered. In the end, we still managed to create a more or less general approach for most projects.
We formulated this approach as simply as possible: the most relevant code for all developers is scanned. If we speak in terms of Gitflow, and all project groups, with the exception of SAP, had branches of development in Gitflow, the main development branch is scanned on a schedule.
But, as always, there are exceptions to any rule: the general approach could not be applied everywhere “as is” for a number of reasons. First, our tool (code analyzer) has several limitations due to the fact that we want to be able, if necessary, to do the most in-depth analysis of some programming languages. So, in the case of Java, the analysis bytecode is much deeper than that of the source code. Accordingly, scanning Java projects required preliminary assembly of bytecode and only then sending it for analysis. In the case of C ++, Objective C and iOS applications, the analyzer was built into the process at the build stage. We also had to take into account the various individual requirements from the developers of all projects. Below is how we built the process for portals and mobile applications.
Portals and mobile applications
It seems that all these applications are combined into one logical group, but in reality they were a terrible mess. There were more than 120 portals (!). The company is very large, with many business, administrative and technical departments, and from time to time each of them decides that he needs his own portal and mobile application. This portal and application are created, used for some time, and then safely abandoned. As a result, at the initial stage, we had to conduct an inventory for the customer, since even the developers of these applications did not have a single list of code bases. For example, to manage the repositories in this group, the developers used two GitLabs with different administrators. In addition, among portals and mobile applications, a significant part of the projects was implemented using external development. Therefore, when the release time approached, contractors often transferred the source codes of the new version to the company almost on a USB flash drive. As a result, the company had a zoo of various applications and a complete mess in their code. We had to make a list of all projects, find all those responsible for them – technical owners, team leaders, and then agree with the main customer – the information security department, which of them we will analyze.
As a result, we chose production systems and supported software for analysis, and did not touch the archiving systems at all. A number of internal applications were considered non-critical, since they could not cause any financial damage to the company, and were not selected for analysis. For example, a management system for packers within one warehouse or loaders. There is nothing vulnerable for external clients of the company in them, and their hacking by some of the internal employees will only cause minor internal inconveniences to a number of departments.
The information security service has formulated the introduction of code analysis for vulnerabilities as a priority task for this software group, and the developers – to build a convenient verification process integrated into development cycles.
Integration according to the standard scheme
GitLab of two different versions was used as a version control system in the group of portals and mobile applications.
Setting up integration with GitLab
Not all applications used CI / CD, and where it was not, we had to insist on using it. Because if you want to truly automate the process of checking the code for vulnerabilities (and not just manually upload a link for analysis) so that the system itself downloads it to the repository and itself gives the results to the necessary specialists, then you cannot do without installing runners. Runners in this case are agents that automatically contact version control systems, download the source code and send it to Solar appScreener for analysis.
The developers of the Portal and Mobile Application Group wanted to organize secure development as a semi-automated process so that the code was scanned for vulnerabilities without any involvement on their part. For the security officer to verify the results of the analysis for vulnerabilities and assign tasks to developers in Jira, if he considered the vulnerabilities to be critical, or send them to the developers for clarification. The developers would decide whether to urgently fix the vulnerability or not. And if necessary, they would plan in which release they can include the fixes.
Jira was mainly used as a bug tracker, into which the code analyzer automatically supplied information about the vulnerabilities found.
Jira integration setup
In rare cases, team leads looked at the crawl results themselves and started tasks in Jira manually.
Create issue in Jira
We also registered such cases in the regulations as a separate feature. In some projects, in general, all fixes were discussed in Slack or Telegram, and tasks were set in real time.
As a result, the process of secure development after the implementation of Solar appScreener began to look like this: the portals are checked daily for changes in the code of the main development branch. If the main, most relevant branch has not been updated within 24 hours, then nothing happens. If it has been updated, then this branch is sent for analysis to the corresponding project for this repository. The repository in GitLab corresponded to a specific project in the code analyzer, and it was in this project that the main branch was scanned. After that, the security officer reviewed the analysis results, verified them, and started tasks for corrections in Jira.
Analysis results and vulnerability fix tasks created in Jira
We started to fix vulnerabilities, as a rule, from critical ones, which needed to be eliminated urgently. When such vulnerabilities ended, the team proceeded to fix new errors found in the code. And already at the third stage, for example, within the framework of closing some technical debt, the old remaining vulnerabilities were also eliminated.
Non-standard as standard
This, at first glance, not so complicated process had two serious limitations. First, to analyze Android applications (i.e. written in Java), we needed an assembly. And secondly, iOS needed macOS machines on which our agent would be installed and there would be an environment that would allow us to build applications. We dealt with Android applications quite simply: we wrote our parts into the scripts already available to the developers, which were also launched according to the schedule. Our parts of the scripts pre-launched the build of the project in the widest configuration, which was sent to Solar appScreener for analysis. To check iOS applications, we installed our MacOS agent on the Mac machine, which assembled the code and also sent the code to the analyzer for scanning via GitLab CI. Then, as in the case with other types of software, the security officer reviewed the analysis results, verified them and started tasks for fixing in Jira.
We also referred to portals and mobile applications as any projects written in Java – we collected and analyzed them in a similar way.
In those projects where there was no CI / CD, which was a prerequisite for us, we simply said: “Guys, if you want to analyze, collect it manually and load it into the scanner yourself. If you don’t have Java or JVM-like languages - Scala, Kotlin and others, you can simply upload the code to the repository from the link, and everything will be fine. “
Complexity of the project
As you can see from the above, in this application stack, the main problem was the lack of CI / CD in many projects. Developers often did builds by hand. We started integrating our analyzer with Sharepoint portals in C #. Now C # has more or less switched to Linux systems, although not quite full-fledged. And when the project was in full swing, this language was still working on Windows, and we had to install an agent on Windows for GitLab. This was a real challenge as our specialists are used to using Linux commands. Special solutions were needed, for example, in some cases it was necessary to specify the full path to the exe file, in some it was not, something had to be escaped, etc. And after the implementation of the integration with Sharepoint, the team of the mobile application project in PHP said that they also do not have a runner and they want to use C #. I had to repeat the operations for them.
As a result, despite such a heterogeneous park of technologies, teams and processes, we managed to group the main cases of this case into several pipelines, automate their execution, where appropriate, and implement them. In our example, we were able to make sure that:
- The solution we are implementing is old enough to be flexible enough to build DevSecOps processes in radically different deployment environments. Flexibility is achieved due to a large set of built-in and custom integrations, without which labor costs for implementation would increase significantly or make it impossible;
- setting up the necessary automation and subsequent analysis of the results do not require an immense amount of labor even with a huge scope of work. Coordination and construction of the implemented processes and their full automation are possible through the efforts of a small expert group of 3-4 people;
- Implementing automated code validation tools and DevSecOps practices reveals shortcomings in current DevOps processes and becomes a reason for customizing, improving, unifying and regulating them. Ultimately, we get a win-win situation for all participants in the process, from ordinary developers to top managers of engineering and information security departments.
As a reminder, this is the first part of a series of articles on building a secure development process in a large retailer. In the next post, we will reveal the details of the implementation of this project in the application group of the SAP family.
Have you had your own experience in implementing similar projects? We will be glad if you share with us your cases of implementing secure development practices in the comments!
Author: Ivan Staroselsky, Head of Information Systems Operation and Automation Department