Defending software build pipelines from malicious attack

Automated build pipelines, such as those utilized in CI/CD (continuous integration/continuous delivery or deployment), provide a robust method for securing software development and deployment.

The NCSC’s Secure Development and Deployment Guidance explains how automation enhances the reliability and consistency of security measures, producing comprehensive log and audit data. As discussed in Principle 6 of the guidance, it is imperative that the pipeline is securely defended, safeguarding each build from others in the pipeline. In cases where an intruder breaches other systems, they may exploit lateral movement to compromise your build pipeline. If any builds are endangered, insufficient separation between builds could jeopardize more crucial ones.

It is also vital to maintain a clear chain of custody from the source code to the build artifact, ensuring that all pipeline checks are applied without alterations post-check.

Even with a well-defended pipeline, a malicious build can pose significant dangers.

If a compromised build manages to affect other builds, it could further spread the breach, steal sensitive information, or complicate remediation efforts. Hence, robust isolation of builds through dedicated compute, networking, and storage is essential.

For instance, executing each build in a temporary virtual machine reduces the likelihood of one build attacking another through shared hardware (like CPU), while builds sharing an operating system kernel present myriad risks for interference. Similarly, if builds can communicate over a network (including loopback devices), they could potentially attack each other or extract data.

Lastly, if a build gains access to data from other builds (such as their source code or build artifacts), it may steal secrets or modify those builds.

One of the primary advantages of automated build pipelines is their ability to conduct security checks on the software being built—reliably and consistently. However, if you cannot trust that the pipeline enforces these checks, their value diminishes.

It is imperative to demonstrate that the checks were executed and that modifications to the build do not occur post-check.

The initial step involves ensuring the protection of build information in transit, using secure protocols like TLS. This applies when fetching code from repositories as well as when transmitting build artifacts to the artifact repository and final environment.

Additionally, cryptographic checksums should capture the data managed by the pipeline. This encompasses checksums of the built source code (such as the ‘commit hash’), resulting build artifacts, and other critical data like configurations, tools, and environment used for building. If the pipeline applies a cryptographic signature on these hashes, it becomes difficult for an attacker to clandestinely manipulate the build. For example, if malicious code is injected into the build, the checksum for the code will either match the original code or the fetched code, but not both, facilitating problem detection.

Creating and maintaining a secure build pipeline entails substantial resources, work, and expertise. Many widely-used pipeline products inherently possess insecure architectures, presenting inadequate separation between builds and offering few defenses against malicious builds. A reliable managed cloud service tailored for build pipelines is specifically designed to guard against malicious activities from customers, thus protecting legitimate customer operations as well.

The more challenging it is for you to compromise the build pipeline, the more complex it is for external attackers. Moreover, the defenses that safeguard one customer’s builds from another can effectively shield your builds from each other as well. Furthermore, the onus of keeping the pipeline current and enhancing its security in light of emerging threats falls to the service provider, typically resulting in superior security by default. Leverage the NCSC’s Cloud Security Guidance (which includes the 14 Cloud Security Principles) to evaluate a cloud build service’s security comprehensively throughout the entire process.

Securing your build pipeline may require considerable resources, yet the payoff is significant as the ramifications of a compromised build pipeline can be immense.

In summary:

• The pipeline must be shielded using the same security principles applicable to other information systems, alongside pipeline-specific considerations like meticulous code reviews and judicious management of pull requests.
• Strong isolation techniques should be employed to prevent builds from affecting one another and to protect the pipeline itself.
• Implementing a strict chain of custody facilitates easier detection of suspicious activities while establishing an authoritative audit trail.
• Utilizing a well-managed service for the pipeline can significantly ease this process.

Remember to consult the NCSC’s Cloud Security Guidance to enhance your understanding of the service’s security measures. Your build pipeline is a cornerstone of your system security, deserving the diligence it warrants.

Jamie H
Senior Security Researcher, NCSC