CI Security
Usually, when a conversation about DevSecOps begins, everyone immediately remembers the need to integrate SAST, DAST, SCA tools into the pipeline, conduct various tests for application security and regularly scan the solution running in a production environment.
But at the same time, we should not forget about the need to ensure the security of the CI/CD pipeline itself. In today's article, we will consider ensuring the security of a pipeline built on Jenkins. But first, let's consider a brief description of this solution, what Jenkins is and what it is used for.
What kind of Jenkins is this?
Jenkins is one of the most well-known tools for creating automation pipelines and integrating them with other tools involved in the CI/CD process. It has an active community that has contributed thousands of plugins to extend the core functionality of the solution, which is the main reason why Jenkins is an industry standard for creating build, test, and deployment processes.
Jenkins can connect and automate the SDLC (Software Development Life Cycle) processes of many developers across their cloud and on-premises infrastructure using its many plugins. However, without proper monitoring, Jenkins can also become a weak point for your organization. In addition to obvious supply chain threats like the SolarWinds hack, a compromised Jenkins system can threaten the entire production environment and become a starting point for malware injection and leakage of sensitive data.
Renewal as a basis
As boring and trivial as it may sound, the security of any application starts with the security of the operating system on which everything is deployed. Keeping a vulnerable version of the OS is extremely risky, as attackers can hack your server using publicly available exploits. Therefore, it is important to install all the latest updates for the operating system. However, patching the OS is not enough.
We also need to update our application software and while updating the core version is usually done manually, updating plugins is simple and can be done with a few clicks in the Jenkins user interface.
Operating system protection
Most Jenkins server deployments will likely be running on Linux, so general best practices for securing Linux should be followed to prevent server intrusion, such as:
First of all, security at the network level. Open only the ports that are necessary. Set up packet filtering so that only access to permitted ports is opened.
No sudo. Make sure that the Jenkins user cannot elevate its permissions using the sudo command. You can do this by editing the /etc/sudoers file. Run Jenkins as a non-administrator user. Simplify the permissions on the JENKINS_HOME directory for the appropriate Jenkins user only.
Authentication
Authentication in Jenkins can be achieved through various methods. However, the best practice for securing Jenkins is to not use the built-in methods, but instead use a centralized third-party provider for authentication, such as GitLab, Github, LDAP, SAML, Google. Using these methods, certain rules can be applied to passwords, such as password complexity, which helps prevent attackers from accessing the server.
But if you still want to use local Jenkins users as a temporary solution, we recommend disabling the “Allow users to register” option and managing registered users manually. This will allow you to control which users appear in the system.
Authorization
Jenkins provides various built-in authentication methods, such as “Anyone can do anything”, “Legacy mode” or “Logged-in users can do anything”. We strongly recommend not to use these built-in methods, but to use plugins for more advanced authentication methods.
The most well-known plugins for this purpose are Matrix Authorization Strategy and Role-based Authorization Strategy, which provide greater flexibility in implementing the principle of least privilege by defining privileges for authenticated users or specific users. In addition, you can also define privileges for each project or assign created roles to each user.
Safe Controller
Pipelines running on the Controller node have direct access to the Jenkins file system, configured workspace, encrypted secrets, and additional resources. Of course, in training and test configurations we often deploy everything on a single server, but in a production environment this is unacceptable. It is important to understand that running builds on the Controller node is extremely dangerous and should be disabled.
To make the appropriate settings, select “Manage Jenkins” → ”Manage Nodes and Clouds” → ”Built-In Node” → ”Configure” → set “Number of executors” to 0.
Let's limit incoming connections
The default interaction between an agent and a controller is accomplished by agents initiating TCP connections to the controller over a dedicated port. This method is called an incoming agent.
If you do not use this method, the incoming port on the controller node should be completely disabled. If your environment requires incoming connections, it is important to ensure that these connections are encrypted. Jenkins offers a setting called “TCP Agent Incoming Protocol/4 (TLS Encryption)” that should be enabled to allow incoming connections over the secure TLS protocol.
These settings can be found under “Manage Jenkins” → “Configure Global Security” → ״Agents״.
Also, if the SSH server plugin is installed, make sure SSH is disabled by going to “Manage Jenkins” → “Configure Global Security” → “SSH Server” → “Disable”.
Restricting agent access rights
By default, build pipelines will run with the permissions of the internal SYSTEM user. This gives builds the ability to run code on any node, create and delete jobs, start and cancel other builds, and more. However, running builds with these permissions can cause serious security issues if, for example, Jenkins runs malicious build pipelines from an SCM platform that the Jenkins administrator is not monitoring. Here's an example of a build log with default privileges, with the permissions listed on the second line:
Using the Authorize Project plugin, you can configure which user, and therefore what permissions, will run the build. A rule of thumb would be to set the least privileges for each project. Here is an example of how the build log, after changing its permissions in “Manage Jenkins” → “Configure Global Security” → “Access Control for Builds”, will have the same rights as the user who ran the build.
Launch of container agents
There are many methods and guides for running agents on different nodes – physical machine, virtual machine, container, Kubernetes cluster and more. From a security perspective, we want to minimize the impact of a compromised build on other builds or on the entire system. Therefore, we prefer that each build environment is created as a new container from scratch.
For example, you can create a container image with all the necessary dependencies and deploy the job to a remote docker service. This ensures that each build is run in a separate, clean container.
You can also use the Kubernetes plugin to provision agents dynamically, thus creating a complete temporary environment that depends solely on the SCM system. Below is an example of such a configuration in a Jenkinsfile:
pipeline {
agent {
kubernetes {
yaml '''
apiVersion: v1
kind: Pod
metadata:
labels:
some-label: some-label-value
spec:
containers:
- name: maven
image: maven:alpine
command:
- cat
tty: true
'''
}
}
stages {
stage('Run maven') {
steps {
container('maven') {
sh 'mvn -version'
}
}
}
}
}Safe HTML
With the default settings, Jenkins uses text-based rendering, which means that all descriptions are treated as text and all HTML tags are escaped to protect against cross-site scripting. You can also configure HTML rendering using the OWASP Markup Formatter, which implements a subset of HTML without risky tags like
