This Shit is Hard: Factory

This Shit is Hard: Factory-scale toolchain management - View all articles Amelia Crate Staff Software Engineer Chainguard Amelia Crate Staff Software Engineer The Chainguard Factory builds a lot of software, and builds it very quickly after upstream cuts a release. The scale and speed of the Chainguard Factory mean that the humans who run the Factory can’t and don’t keep up with every single package build. Instead, we spend a lot of time designing traditional and agentic automation to manage these package builds for us. In this blog post, I will discuss Chainguard’s automation around toolchains, which are often high-human-touch areas, and how we manage them to deliver software updates quickly at scale. What is a toolchain, anyway? The definition of a toolchain is a bit fuzzier than one might think, but it is generally understood to refer to software that is used to build or develop other software. Chainguard wants to solve two problems when managing our toolchains. We need to build with the latest version of a toolchain to get the latest security and CVE fixes. We need to ensure our package builds are extremely reliable so we can rebuild apps quickly when they receive an update. Fundamentally, these are problems of version selection at build time. Applications need to use the latest working version of their toolchain, and toolchains occasionally need to rebuild all their users to apply fixes. However, the problem of toolchain management applies to many kinds of software, some commonly thought of as toolchains and some not. Selecting a toolchain To solve these problems, we version-stream our toolchains. For example, many packages are built with the llvm toolchain. This is a foundational toolchain and includes the clang compiler. Whenever we add a new application, we don’t just build with the latest version of llvm , whatever that happens to be. Instead, we split llvm into version streams: llvm-22 , llvm-21 , llvm-20 , and so on. New applications select the latest compatible toolchain version stream and use that for their package build. Over time, this grows stale as llvm adds new versions and the application adds support for new llvm versions. So whenever we add a new version of llvm , the Chainguard Factory finds applications using older version streams and attempts a build and test with our package, rebuilt using the latest version stream of llvm . If everything looks good, we update the package build to the latest version. But we have another problem! Statically compiled languages like Go link their standard library into every application and can’t be just compiled and forgotten. Whenever we have a new patch for a Go release, we need to rebuild all users with that patch to ensure they receive the latest standard library fixes. So we have two kinds of toolchain automation to solve our two separate problems. The first kind of automation ensures that a toolchain’s users use the most recent version stream they can, and the second ensures that a toolchain’s users are rebuilt with the latest minor version fixes from that stream. Most toolchains require the former, but usually only statically linked toolchains require the latter. Of course, one can see how this version selection automation might be useful for more than just classic toolchains like compilers. In fact, we also use this automation to rebuild nginx plugins whenever we update our nginx packages. You can see our work indicating to the Chainguard Factory which packages are toolchains and what kind of automation they need in Wolfi. llvm-22 , go-1.27 , and nginx-mainline all carry melange annotations indicating these options. Safe upgrades with less human effort Here’s a benefit of this version-streaming approach that may not be immediately obvious: because packages at Chainguard do not directly use the latest unversioned toolchain in their build dependencies, adding new major versions is very safe. In most distributions, upgrading to a new major version of a toolchain is a high-touch affair: humans must find and rebuild every package build depending on the toolchain, or at least verify that the major version upgrade did not break those package builds, before it can land. Instead, at Chainguard, we are constantly improving our functional package tests to confirm that the toolchain we’re using is within upstream’s support matrix and effectively exercises our packages. We’ve written before about the importance of testing at Chainguard, explaining that testing is critical to safely updating our toolchains with automation. With a focus on package tests and our version-streaming strategy, we can add a new version of the toolchain immediately and be confident it has no impact on package builds. It only begins to impact our package builds as we start the scan, rebuild, and test process to upgrade them to use this version. Of course, there are occasional breakages. We might discover a gap in our testing, or upstream might accidentally ship a breaking change in a compiler. Finding and fixing these breakages and continuously improving our test coverage so we don’t do it again are part of the pain of being a rolling-release distribution. We’ve written before about how the pain of many small upgrades is often preferable to the pain of the big lift of one large upgrade. Staying secure doesn’t just mean being secure at a point in time; it means continuously upgrading to the latest, safest software. Version streaming the un-version-streamable Some toolchains, of course, can’t have multiple copies in use alongside each other. glibc is very important, used by nearly everything, and is not version-streamable, as only one package can ship /lib64/ld-linux-x86-64.so.2 . So how can we achieve the same reliability and low-impact upgrades? We can rely on glibc’s ABI compatibility guarantees. Of course, we don’t just take glibc’s word for it. We generate a compatibility report comparing glibc’s public ABI against the public ABIs of prior versions and ensure that the new ABI meets our ABI compatibility threshold. Once we’ve addressed ABI compatibility at runtime, we can focus on eliminating build disruption caused by the new version. To solve this, we version-stream the headers and linker scripts that applications use to build against a glibc version. This is fundamentally the opposite and upside-down approach to build-time & run-time transition compared to other distributions. Runtime moves atomically, and build-time transition is decoupled into granular, atomic, and hermetic per-package upgrades. This removes the need to maintain runtime ABI stability by creating forks of glibc and trying to retrofit security and bug fixes onto an old codebase, which over time becomes ever more difficult. Using this strategy, applications can continue calling the older symbols, which we know are ABI compatible at runtime, until we’ve verified that the application can build and pass tests when rebuilt against the new version of glibc. Chainguard OS: a secure foundation Chainguard’s mission is to make all open source trustworthy. Chainguard OS, our purpose-built Linux operating system distribution designed for modern, cloud-native workloads, is the foundation of everything we do at Chainguard and enables our customers to develop securely. Our toolchains are the foundation of the foundation, enabling us to build that software securely. We’ve written extensively about how we work to harden our toolchains, from hardening glibc to enhancing our compiler hardening flags. Continuously upgrading and hardening our toolchains has allowed us to mitigate real vulnerabilities before they were ever disclosed. Interested in learning more about Chainguard OS and how it works? Download this white paper today. Share this article Related articles - engineering What it took to reach 1 billion build manifests - engineering This Shit is Hard: Taming the Thundering Herd - engineering This Shit is Hard: Getting AI to prove where a number came from - engineering This Shit is Hard: Patching a vulnerability that has no fix - engineering This Shit is Hard: How Chainguard is sandboxing Athena - engineering You can't trust what you can't see: How we keep an eye on a fleet of AI agents