Defending software build pipelines from malicious attack

Automated build pipelines, commonly used in CI/CD (continuous integration and continuous delivery or deployment), provide a reliable method for securing software development and deployment.

The NCSC’s Secure Development and Deployment Guidance explains how automation enhances the consistency and reliability of security measures, while also generating detailed logs and audit trails. As outlined in Principle 6 of the Guidance, it is crucial for the pipeline to be robust and to safeguard each build against vulnerabilities from other builds. Attackers who compromise external systems may leverage lateral movement to gain access to your build pipeline. If any builds are compromised, inadequate isolation can result in more critical builds being affected.

It is also essential to maintain a clear  chain of custody from the source code to the build artifact, ensuring all checks within the pipeline are enforced, without alterations post-checks.

Even with a well-guarded pipeline, a malicious build can still present significant threats.

If a compromised build can influence others, it can serve as a vector for spreading malware, extracting secrets, or complicating future remediation efforts. Consequently, it’s vital to isolate builds through strong separation in computing, networking, and storage.

For instance, running each build on a single-use virtual machine minimizes the risk of one build being able to exploit shared resources (like CPUs), whereas builds sharing an OS kernel face greater potential for interference. Furthermore, network connectivity between builds can lead to attacks or information theft.

Lastly, if builds can access each other’s stored data (such as source code or build artifacts), this may lead to the theft of secrets or unauthorized modifications.

A major benefit of automated build pipelines is their consistent execution of security checks on the software they generate. However, if you cannot ensure these checks are indeed enforced, their value diminishes significantly.

Thus, demonstrating that checks have occurred and that builds are not altered afterward is crucial.

The initial step involves ensuring that build information remains protected throughout its transit, using protocols like TLS. This protection is important during code retrieval from the repository and when transferring build artifacts to the artifact repository for deployment.

Next, implementing cryptographic checksums is vital for tracking the data processed by the pipeline. This should encompass checksums for the source code (like ‘commit hash’), hashes of the resultant build artifacts, and any ancillary data including configurations, tools, and environments used for building artifacts. Enhanced security measures, such as applying cryptographic signatures over these hashes, significantly reduce an attacker’s ability to alter builds stealthily. For example, should an attacker inject malicious code into a build, the checksums would only align with either the built code or the fetched code, but not both, which simplifies detection.

Establishing and maintaining a secure build pipeline requires substantial effort, resources, and expertise. Many widely-used pipeline products come with insecure configurations, lacking adequate protections between builds. Choosing a reputable managed cloud service can significantly improve security, as these services are tailored to defend against potential threats from malicious actors, thereby safeguarding legitimate workloads.

The more challenging it is for you to compromise the build pipeline, the more difficult it is for attackers as well. Often, safeguards that protect consigned builds from each other also fortify your builds against vulnerabilities. Additionally, the service provider assumes responsibility for continually updating the pipeline and strengthening its defenses against emerging threats, yielding you superior security from the onset. The NCSC’s Cloud Security Guidance, including the 14 Cloud Security Principles, can aid in evaluating the security of cloud build services by considering end-to-end security throughout the entire process.

Securing your build pipeline demands considerable resources, but the reward is substantial—an attacker breaching your pipeline can lead to severe consequences.

To summarize:

• The pipeline should be secured using the same principles applied to other information systems, coupled with pipeline-specific approaches like careful code reviews and management of pull requests.
• Strong isolation measures are required to secure builds from one another and protect the overall build pipeline.
• Implementing a robust chain of custody will facilitate easier detection of suspicious activities and establish a credible audit trail.
• Exploiting a reliable managed service for your pipeline simplifies these processes immensely.

Remember to leverage the NCSC’s Cloud Security Guidance for a clearer understanding of service security. Your build pipeline is a cornerstone of your overall system security—allocate it the focus it rightfully demands.

Jamie H
Senior Security Researcher, NCSC