Defending software build pipelines from malicious attack

Utilizing automated build pipelines, such as those implemented in CI/CD (Continuous Integration/Continuous Delivery), significantly enhances the security of software building and deployment.

The NCSC’s guidance on secure development and deployment elaborates on how automation can instill greater confidence in the reliability and consistency of security measures while producing comprehensive logs and audit data. As indicated in principle 6 of the guidance, it is essential for the pipeline to be well-protected, safeguarding each build against others within the pipeline. Compromised external systems could lead an attacker to perform lateral movement targeting your build pipeline, where the compromise of one build could cascade into more critical builds due to poor isolation.

Maintaining a clear chain of custody from source code to build artifact is vital, ensuring that all checks applied within the pipeline are intact and no alterations occur post-checks.

A well-defended pipeline does not eliminate risks from a potentially malicious build.

If a compromised build can affect others, it risks extending the compromise, stealing sensitive data, and complicating subsequent remediation efforts. Therefore, robust separation of builds through effective compute, network, and storage strategies is essential.

For instance, running each build in a dedicated virtual machine offers greater protection compared to builds sharing an OS kernel, which could easily interfere with one another. Similarly, if a build can communicate with others through network connections, it may exploit those links to launch attacks or extract information.

Moreover, if a build accesses data from others, such as source codes or build artifacts, it raises the potential for secret theft or unauthorized modifications.

Automated build pipelines excel at performing consistent security checks on software. However, the value of these checks is undermined if there is no assurance that they are enforced consistently.

It’s crucial to demonstrate that checks took place and that builds remain unaltered afterward.

The first step involves ensuring that all build information is secured during transit via protocols like TLS. This security should cover everything from coding repository access to the deployment of build artifacts.

Additionally, employing cryptographic checksums to log the pipeline’s processed data is essential. This includes checksums for the built source code (like a ‘commit hash’), hashes for resulting build artifacts, and any relevant configurations, tools, or environments utilized during the build process. A cryptographic signature on these hashes can significantly enhance the difficulty for an attacker attempting to manipulate builds undetected. For example, if malicious code is introduced, the checksum will only align with either the original fetched code or the code built, never both—thus aiding in problem detection.

Creating and upholding a secure build pipeline involves significant resources, work, and expertise. Many prevalent pipeline tools default to insecure architecture, leading to inadequate separation of builds and minimal defenses against malicious incursions. An effective managed cloud service for build pipelines is designed to initially guard against malicious clients, which simultaneously protects legitimate customers.

When it’s challenging for you to compromise your build pipeline, it becomes equally challenging for external attackers. Moreover, defenses protecting one customer’s builds are likely to also shield all your builds from one another. Responsibility for keeping the pipeline updated and secure amid evolving threats typically falls to the service provider, offering you enhanced security as a default. It is advisable to refer to the NCSC’s Cloud Security Guidance for evaluating the security of cloud build services, factoring in the overall process security.

Securing your build pipeline requires considerable resources, but the resulting protection against compromising attacks can be tremendously beneficial.

In summary:

• Adopt the same security principles for the pipeline as you would for other information systems while incorporating specific pipeline considerations such as diligent code reviews and careful management of pull requests.
• Implement strong isolation measures to keep builds secure from one another and ensure the safety of the build pipeline itself.
• Establish a rigorous chain of custody to facilitate the identification of suspicious activities and to create an authoritative audit trail.
• Utilizing a reputable managed service can greatly simplify these security measures.

Remember to leverage the NCSC’s Cloud Security Guidance to enhance your understanding of service security. Your build pipeline is pivotal in your system’s security framework, meriting careful attention.

Jamie H
Senior Security Researcher, NCSC