Challenges with Reproducible Builds: scale and maintenance

Reproducible Builds (R-Bs) mitigate trust gaps in the build step by making builds perfectly deterministic: the output is always bit-for-bit identical. This allows software consumers to detect if they have been given an incorrect artifact. However, R-Bs bring challenges that are often underestimated.

Many software artifacts and dependencies are distributed pre-compiled. This lifts the burden of managing build toolchains from the integrating party, improves developer experiences by reducing compilation times, and is sometimes used to protect underlying IP. Shipping pre-compiled binaries is common for dependencies (e.g., PyPI and Maven) and packages (e.g., APT).

However, this opens a wide range of supply chain risks that are often overlooked but have become the target of recent attacks: trust in the build infrastructure itself. When downloading these “black box” artifacts, we place significant trust in the developer and all machines responsible for generating and publishing them. Examples include the SolarWinds incident and the LiteLLM compromise.

Reproducible Builds (R-Bs) mitigate some of these risks. R-Bs work by making the build process perfectly deterministic, which allows the integrating party (or the software consumer) to later confirm they were given a correct artifact. They can do so by downloading the same source code, executing the build, and verifying that they end up with the exact same artifact.

Setting up deterministic builds can be hard

The first challenge with making a build reproducible is identifying and eliminating all sources of non-determinism. Typical sources include:

• References to time or timestamps
• Parallel builds and linking ordering
• Heuristic compiler optimisations
• Environment variables (hostname, user, etc.)
• Order of inodes and file traversal
• …

Hunting all of these down requires careful control of the environment and an in-depth understanding of all tools involved. In many cases, it requires modifications to the build setup and source code. In extreme cases, we might even need to rewrite a portion of the program where existing tooling cannot easily support determinism (e.g., code generation).

However, even when two subsequent builds result in the same artifact, we cannot know for sure that we have succeeded. We might have missed one factor that just happens to be constant in our tests—a new joiner might find that a build that is deterministic for us yields a different result for them.

“

Often we won’t reliably know if we have really achieved a fully deterministic build.

”

This dilemma of not being able to guarantee correctness might remind you of Edsger W. Dijkstra’s famous quote: “Program testing can be used to show the presence of bugs, but never to show their absence!”

That said, there are programming ecosystems where this is easier. For instance, Go has featured great support for deterministic builds from the start, including for its own toolchain.

High maintenance tax and unexpected surprises

The often under-appreciated challenge with R-Bs is that the target is constantly moving. At any time, any (transitive) dependency might no longer be reproducible—either because it has been deprioritised or a regression was not spotted. Often, identifying which dependency or toolchain component caused the issue requires substantial debugging.

“

Any change, anywhere upstream along the dependency or toolchain, can introduce hard-to-debug regressions.

”

One particularly pressing matter is (ironically) supply chain security. Good hygiene requires frequent updates to existing dependencies—at the latest when they or their transitive dependencies have known vulnerabilities.

Applying these security updates can be tricky: while most projects now have good processes to deploy fixes, very few treat deterministic builds with the same priority. This can place the downstream integrating party in a predicament: how to apply a security fix timely without risking a break in the deterministic build.

The solutions for a dependency that needs to be updated, but where the update breaks determinism, are rarely enticing:

• Fix the new upstream version to restore R-B → does not scale to wide dependency trees.
• Vendor the dependency and patch out-of-trunk → only delays the problem.
• Replace the dependency with an alternative → potentially high engineering effort.

This makes managed reproducible build projects such as StageX interesting. However, they intentionally only offer a small collection of packages.

Who is actually verifying?

In a naïve R-B deployment model, the integrating party simply rebuilds the artifact to verify that the provided ones are correct. This provides little benefit over simply building everything from scratch, which carries downsides like long compilation times (looking at you, Rust).

A more practical approach is to rely on deterrence. Assuming there is an established trust relationship between the software provider and the integrating party, R-Bs are a critical commitment from the provider. The integrating party can rely on sampling builds to exert effective deterrence. However, in practice, it is difficult to ensure this covers a wide range of artifacts.

An alternative is to rely on distributed reproducible builders. This reduces the burden on the integrating party but provides no confidentiality for the source code, which can be a roadblock for enterprise software. Additionally, maintaining the commitment of independent builders with sufficient resources is challenging.

Attestable Builds: A scalable approach for transparency

The main benefit of Reproducible Builds is that they provide verifiable transparency for the build infrastructure. However, as we have seen, they can come with unexpectedly high costs.

“

We have designed Attestable Builds to provide verifiability benefits similar to Reproducible Builds without the fuss.

”

Attestable Builds rely on a hardware-based trust anchor to provide a cryptographic attestation of the build process. It allows the verifying party to check that the following properties bind together in a trustworthy manner:

• Exact source code snapshot
• Enforcement of hardware-based isolation (SLSA L3+)
• Exact build environment
• Digest of the output artifact

Since Attestable Builds (A-Bs) do not rely on deterministic build processes, there is no need for costly changes to the build setup. The isolated build environment can execute standard processes, such as Docker-based builds.

Similarly, A-Bs do not require ongoing maintenance, and updates to dependencies can be applied immediately. Each dependency can have its own A-B; as such, the certificates of transitive dependencies easily compound with the final artifact build.

Finally, with A-Bs we do not rely on trust, deterrence, or distributed builders. Every integrating party can instantly verify the truthful provenance of each artifact using a cryptographic attestation document.