Implementing a Scanning Service Based on OWASP ZAP
To protect organizations' digital assets, it is important to quickly identify and fix vulnerabilities. Scanning tools automate this process, allowing you to efficiently find weaknesses in systems and applications.
Hi! My name is Nikita, I work in information security at RuStore. Today I will tell you how we created our vulnerability scanning service based on OWASP ZAP, what difficulties we encountered and what approaches we used to solve them.
What needed to be done
Our team faced the task of organizing dynamic scanning for each release. With the growing number of releases, the issue of automation and optimization of this process became especially relevant. We needed a tool capable of performing dynamic analysis of applications, with flexible configuration of scanning rules and analysis goals.
Why OWASP ZAP
We chose OWASP ZAP, an open source software that provides all the necessary tools for deep dynamic analysis. This solution met our specific requirements and was superior to others for two key reasons:
adaptability to the specifics of the project;
powerful API that makes it easy to add new features, rules and plugins.
Integration service
As a first step in developing the scanning system, we created an integration service that allows flexible customization of OWASP ZAP for each new release. A service was created in Python that can perform the following tasks:
collecting information about services and releases;
Configuring scanning parameters via the OWASP ZAP API based on JSON configuration.
Example of input configuration:
"scan host": [
{
"targetProto": "https",
"targetHost": "apps.rustore.ru",
"header token": [
{
"User-Token": "***"
}
],
"enabled": "true",
"swagger scope": {
"enabled": "False",
"url": "https://backapi.rustore.ru/v3/api-docs"
},
"scan config": {
"scanner": {
"enabled": "true",
"exclusion": [
{
"value": ".*logout.*",
"enabled": "true"
}
]
},
"spider": {
"enabled": "true",
"ajax spider": "true",
"exclusion": [
{
"value": "https://console.rustore.ru",
"enabled": "true"
}
]
}
}
}
]To configure scanning parameters, we have developed a JSON file structure that includes the following features:
Connecting modules and scanning rules: allows you to flexibly configure the required scanning types and rules for each scanning target.
Rules for inclusion and exclusion of addresses: allow you to determine which URLs should be included in scanning and which ones should be excluded.
Setting up work with web sockets: Provides options to control interaction with web sockets during scanning, which is important for testing the actual behavior of an application under active data exchange.
Specific authorization for each service: allows you to set authorization data for access to protected parts of the application, which is critical for checking vulnerabilities in protected segments.
Managing report formats: Includes settings for generating scan results reports, making it easier to analyze and present test results.
Additionally, OWASP ZAP is deployed on a separate node in our infrastructure, and a new clean instance is launched for each scan. This ensures session isolation and clean testing, which is critical to obtaining accurate and objective results.
As a result, the service has several modules, divided according to the logic of operation:
Core
The core module includes setting up a ZAP session, setting up authentication, setting up a scan context, scanning policies with exceptions, working with web sockets, and launching other modules.
Scanning and results processing module
The module implements the configuration and addition of scanning rules, plugin configuration. To create it, we researched and tested many public plugins for OWASP ZAP from the internal store and external sources, such as GitHub and GitLab. We assessed the configuration capabilities and performance of each plugin, choosing the most suitable for our technology stack. And now we are constantly enriching the rules and vulnerability search mechanisms to improve the quality of scanning.
Problems and their solutions
After implementing this functionality, we encountered the problem of the volume of operations and the speed of their processing. It was decided to sort the checks by the percentage of false positive triggers. The idea is to select classes of triggers without false positives and immediately send them to work without additional analysis. A report with such triggers is generated separately, and it can be immediately sent to the development team.
Another problem turned out to be internal ZAP mechanism working with a false positive that refused to work as we needed. So we implemented our own module that tracked task statuses from scan to scan.
The list of triggers previously marked as false positives is downloaded from S3 storage as a separate file, then parsed by the script, and the false positive results relevant to this scan are loaded into the AlertFilter method, a built-in ZAP API method.
The essence of a false positive alert contains information about the key parameters of the sent request and potential vulnerability:
ruleId — identifier of the executed ZAP rule;
url – request URL;
Method — the HTTP method used for the request;
Parameter — the name of the vulnerable parameter;
attack — payload sent by ZAP in the request parameter;
CWEId — CWE identifier of the vulnerability found.
Example of a false positive alert entity:
{
"ruleId": "43",
"url": "https://***.rustore.ru/applicationData/10?pageSize=21&pageNumber=0",
"Method": "GET",
"Parameter": "pageSize",
"attack": "10",
"Instances": "541",
"CWEId": "33"
}
Information collection module
The idea of creating an information collection module arose from the need to minimize the impact on the processes of our product teams. The module was needed to define a specific scanning area for a release or task, since a full scan takes too much time and does not meet our TTM requirements within the SDLC processes.
Our task was to collect as much data as possible to accurately describe the scan area for the release. An important requirement was to automate data collection and translate it into a format understandable to the service. Therefore, we abandoned human-written descriptions of tasks and services in free form. The ideal solution was to track all changes to API specifications, any changes to endpoints and parameters. We wrote the functionality of parsing API specifications and storing several previous states to prevent errors associated with accidental or erroneous changes to Swagger.
Another source of data on changes are merge requests into release branches. This data allows you to find out in advance which services are being prepared for release and scan them. Data from this module is transferred to the CORE module to run a scan with a specific scope.
Example of application output when checking Swagger changes:
Services:
https://backapi.rustore.ru
https://api.rustore.ru
https://app-selection.*.rustore.ru
https://dev.rustore.ru
There are updates at the following stands:
https://backapi.rustore.ru
https://api.rustore.ruReport generation module
At this stage, we generate several reports and send them to different systems. Two types of reports are created:
Full scan: The report includes all vulnerabilities found, excluding false positives. It is saved in S3 storage along with the ZAP session so that information security engineers can get a complete picture of the service and the scan.
Report for developers: This report contains only those rules that have a very low percentage of false positives or no false positives at all.
Both reports are presented in easy-to-read HTML format. We also save copies of them in JSON format for ease of automated processing and subsequent uploading to vulnerability management systems.
The scanning results are also sent to the internal vulnerability management system — VK Security Gate. This VK tool is designed for automated scanning of source codes using static analysis tools (SAST), searching for vulnerable dependencies (SCA), and for processing the received alarms. Security Gate allows you to more conveniently process findings and track their statuses for each service in any development environment.
In addition, the release approval process has been automated. Now the statuses of all findings in Security Gate are checked, and a release cannot be approved if there are unprocessed issues or vulnerabilities with a certain criticality among them.
What's the bottom line?
We have created a universal service for integrating and configuring OWASP ZAP with a modular system and optimized the process of detecting and fixing vulnerabilities.
The process from publishing a new build to confirming a bug fix includes the following steps:
Creating a build: Assembling a microservice and deploying it to dev stands.
Import specifications: The OpenAPI specification is loaded into ZAP via the API.
Launching the information collection module: The module tracks changes in microservices according to API specifications on all stands and changes in relay branches.
Setup and scanning: ZAP sets up the testing context and runs an active scan over the imported scan area.
Processing results: Automatically exclude false positives and generate reports.
Submitting reports: Reports are sent to VK Teams and Security Gate.
Vulnerability Handling: An information security engineer confirms or rejects vulnerabilities (e.g. false positives) and monitors bug fixes.
Further support of the process: monitoring the bug fixing by the information security engineer until the problem is closed, the release is approved and the secure code is deployed to production.
Currently this service operates in two modes:
Scan for changes mode: Performed on dev and stage environments to identify vulnerabilities in new versions of services.
Full scheduled scan: It is performed on stage and prod environments to check interservice interactions and prod state. All scanning results are processed according to the scheme described above.
This solution simplified the process of tracking and fixing vulnerabilities and reduced the time it took to analyze changes to services.
Future plans
We continue to expand the functionality of our service to provide even deeper and more accurate analysis of application security.
In the nearest plans:
