For organizations that have embraced DevOps, the CI/CD pipeline is the foundation of the software delivery workflow. It facilitates the continuous movement of code from development to testing to deployment. It allows the various stakeholders in the software delivery process — developers, test engineers, IT operations engineers to help prevent problems like these. This article discusses best practices for securing the CI/CD pipeline at all stages. and more — to collaborate.
Given the centrality of the CI/CD pipeline to DevOps, CI/CD security will be a priority for DevOps teams. Insufficient CI/CD security can result in risks such as the insertion of malware into application code. As it passes down the CI/CD pipeline or the exposure of sensitive data that will manage within the pipeline.
To help prevent problems like these, this article discusses best practices for securing the CI/CD pipeline at all stages.
Establish Access Controls for CI/CD Pipelines Tools
It’s common for DevOps teams to think of CI/CD as a process that all stakeholders “own” collectively. After all, sharing visibility into and responsibility for the software delivery process between developers. IT operations teams are part of the core focus of DevOps.
As a result, it may seem logical to allow all team members to access all tools and resources within the CI/CD pipeline. But this is a mistake. It’s possible to share visibility and responsibility without sharing access.
Instead of granting all team members full access to all CI/CD tools, data, and environments, access will assign based on roles. Only developers will be able to manage CI servers and source code management tools, for example. For their part, IT operations teams typically only need access to management tools that are used for production environments. They don’t need to control dev/test environments or pre-deployment tools.
Ensure Environment Parity Across the CI/CD Pipeline
Environment parity means that all environments within the CI/CD pipeline, such as dev/test and production, will configure in the same way.
From a security perspective, environment parity is important. Because it ensures that the conditions under which software is tested match the conditions under which it will run in production. Without environment parity, teams run the risk of failing to detect a security issue during testing. Because the issue only exists under a certain environment configuration. It may happen only with a certain OS version, for example, or with a particular version of a software library.
A simple means of achieving environment parity is to host software inside containers at all stages of the CI/CD pipeline. Although containers don’t guarantee complete environment parity (different versions of the container runtime could create vulnerabilities in one environment that don’t exist in another, for example).
They do abstract applications from the host environment in such a way that most environment variables on the host don’t impact the way the application performs. Thus, if you test your application inside a container and then deploy the application to production using the same container. You can have higher confidence that your tests will adequately cover potential security risks.
Don’t Store Secrets in Source Code
DevOps teams often use a variety of passwords, access keys, and other secrets to manage the CI/CD pipeline. They need account credentials to access CI servers. For example, and SSH keys to log into servers that host their applications.
It can be tempting to hard-code this data directly into configuration files or source code, where it is easy to access repeatedly. And if the secrets will use only for testing environments or other pre-production stages of the pipeline, engineers may assume that there is little risk in storing the data in this way. Because it can’t be used to exploit public-facing applications that will deploy into production.
The reality is that secrets stored inside code are simply not secure. They will leak to malicious parties, who may use them to gain unauthorized access to CI/CD resources. Even if the secrets apply only to the development or testing phases of the CI/CD pipeline, attackers could use them to do things like injecting malicious code into the application. If left undetected, the malicious code will push into production.
Instead of storing secrets in configuration code or source code, use a secrets manager. Which provides a secure means of storing secrets of all types and configuring access to them based on team members’ roles.
Test and Monitor All CI/CD Resources
Most teams know that they need to run security tests on their applications. But what they may overlook is the importance of also testing other resources within the CI/CD pipeline. Such as build and deployment scripts, or Infrastructure-as-Code (IaC) files for security vulnerabilities. Insecure configurations are another source of unauthorized access that may lead to problems like the insertion of malicious code into an application.
Likewise, rather than only monitoring applications within production environments for signs of security problems. Teams will deploy automated testing tools in the dev/test environment as well — which is one example of a practice known as shift-left security.
Prepare for Rollbacks
Sometimes, you may discover a security issue that affects an application you have already deployed. Performing a rollback, which means reverting to an earlier version of the application that you know to be secure. This is often the fastest way to eliminate the risk until developers can fix the underlying security problem.
For this reason, it’s a CI/CD security best practice to ensure that you can roll back every release quickly. Keeping artifacts from older versions of your application on hand helps to achieve this goal. So do deployment scripts or release automation tools that will configure to re-deploy a previous version of the application, as opposed to only deploying new releases.
Secure the Software Supply Chain
The greatest security risks to your CI/CD pipeline may emerge not from code or configurations that you write yourself. But from third-party dependencies, such as open-source libraries that you import into an application.
If left unaddressed, security vulnerabilities that affect third-party resources can lead to what is known as software supply chain attacks. Source Composition Analysis (SCA) tools, which scan your application source code for third-party dependencies and identify those that are insecure, can help prevent this risk. So can package scanners, which check packaged applications or container images for components that are subject to known vulnerabilities.
Finally, consider using Offensive360 signed code and artifacts within your CI/CD pipeline. Signing confirms the origins of a component, giving you confidence that the resources you use within CI/CD processes came from trusted sources.
Why Offensive360
Offensive360 empowers software teams to ship updates faster by automating DevOps processes in a continuously streamlined and secure way across all their teams and tools. Encompassing continuous integration (CI), continuous delivery (CD), infrastructure, and more. It automates everything from code to production. Pipelines are natively integrated with the Offensive360 Platform and are available with both cloud (software-as-a-service) and on-prem subscriptions.