Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Reproducible builds and build provenance

After this lesson: I can check whether a build is reproducible.

60 min lesson7 commandsSQL injection lab: attack a weak shop search, fix the query and prove the fix

The same source should give the same bytes

A reproducible build is one where the same source code, built with the same tools, gives exactly the same output, bit for bit. It matters because anyone can then rebuild a release and check that nobody changed it on the way.

In March 2024 a backdoor was found in versions 5.6.0 and 5.6.1 of xz-utils, a compression library in many Linux systems. Part of it sat in the published release archives and not in the code most people read.

By the end of this lesson

Note

I can check whether a build is reproducible.

Key words

WordWhat it meansExample
Reproducible builda build that gives the same bytes every time from the same source
Hasha short fingerprint of some bytes that changes if any byte changessha256sum
SOURCE_DATE_EPOCHa fixed time, in seconds since 1970, that a build uses instead of the clockSOURCE_DATE_EPOCH=1709283600
Provenancea record of what was built, from which source, by which build
Signatureproof, made with a key, that a file comes from someone and is unchanged
SBOMa software bill of materials: a list of every component in a product

The cells students fill in are left blank in this preview.

The one rule

Warning

The lab is a shop that is insecure on purpose. Work in your own private copy and nothing else. It has no way out to the internet. Never break into a system that is not yours without written permission. In the UK that is a crime under the Computer Misuse Act 1990.

Why a build that repeats itself is safer

A build turns source code into the file you ship: an archive, a program or a container image. Users download that file, not the source.

If two people build the same source and get the same hash, nobody slipped anything in between the source and the file. If the hashes differ, something changed, and you can look for it. A build that differs every time hides that signal.

Build the same two files twice

Your lab can make a tiny release with tar, which packs files into one archive, and sha256sum, which prints its hash. The next Run writes two small source files into your shop's container.

The Run after it packs them twice. Between the two builds it only changes the files' times, by one hour, as if the source were checked out later.

Write two source files

Command
docker exec app sh -c "mkdir -p /tmp/build/src && printf 'print(42)\n' > /tmp/build/src/main.py && printf 'shop 1.0\n' > /tmp/build/src/VERSION && cat /tmp/build/src/VERSION"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Pack them twice, an hour apart

Command
docker exec app sh -c "cd /tmp/build/src && touch -d '2024-03-01 09:00:00' main.py VERSION && tar -cf - main.py VERSION | sha256sum && touch -d '2024-03-01 10:00:00' main.py VERSION && tar -cf - main.py VERSION | sha256sum"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Read the times inside the archive

Command
docker exec app sh -c "cd /tmp/build/src && touch -d '2024-03-01 10:00:00' main.py VERSION && tar -cf - main.py VERSION | tar -tvf -"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

One hour changed the whole hash

The first Run prints shop 1.0. The second prints two hashes: one starts 860a6728 and the other starts 711f4dc5. The bytes of the two files never changed.

The third Run lists the archive. Each line shows the file's mode, its owner, its size and then its time, 2024-03-01 10:00:00. The archive stores that time, so a new time is a new archive.

A common mistake

Warning

Many people think the same source always builds the same file. It does not, unless every input is fixed: the times, the order, the paths and the tools. Treat a build as reproducible only after two separate builds give the same hash.

Worked example: what makes two builds differ

Source of differenceExampleUsual fix
TimestampsEach file keeps the time it was last changedSet every time to SOURCE_DATE_EPOCH
File orderThe folder lists files in a different orderSort the names before packing
Build pathThe archive stores /home/ana/shop or /tmp/shopPack relative names from inside the folder
LocaleSorting puts capitals first in one setting and not in anotherSort with LC_ALL=C
VersionsThe next install picks a newer libraryPin every version in a lock file

The cells students fill in are left blank in this preview.

Fix the time and the order

SOURCE_DATE_EPOCH is a shared convention: a build tool that sees it uses that time instead of the clock. Many compilers and packers read it already. Here the script sets every file's time to it with touch.

Order matters too. The next Run packs the same files at the same time, once in each order. Then you write a small build script that sorts the names first.

Same files, other order

Command
docker exec app sh -c "cd /tmp/build/src && touch -d '2024-03-01 10:00:00' main.py VERSION && tar -cf - main.py VERSION | sha256sum && tar -cf - VERSION main.py | sha256sum"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Write a build script that fixes both

Command
docker exec app sh -c "printf '%s\n' 'cd /tmp/build/src' 'touch -d @\$SOURCE_DATE_EPOCH *' 'tar -cf - \$(ls | LC_ALL=C sort) | sha256sum' > /tmp/build/build.sh && cat /tmp/build/build.sh"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Build with SOURCE_DATE_EPOCH

Command
docker exec -e SOURCE_DATE_EPOCH=1709283600 app sh /tmp/build/build.sh

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Touch every file, then build again

Command
docker exec -e SOURCE_DATE_EPOCH=1709283600 app sh -c "touch /tmp/build/src/main.py /tmp/build/src/VERSION && sh /tmp/build/build.sh"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Two builds, one hash

The order Run prints 711f4dc5 and then dcedcb05: same bytes, same times, other order. The script Run prints its three lines.

Both builds then print a hash starting 73367b62, although the second first touched every file to the current time. LC_ALL=C sort puts VERSION before main.py, because capitals sort first in that setting. That is why the locale is fixed too.

One byte is enough to change a hash

Code: python
import hashlib

# Change the 2 in second to a 3 and press Run.
first = b"print(42)\n"
second = b"print(42)\n"
print(hashlib.sha256(first).hexdigest()[:16])
print(hashlib.sha256(second).hexdigest()[:16])
print("same build:", hashlib.sha256(first).digest() == hashlib.sha256(second).digest())

Provenance says how a file was built

Reproducing a build is one check. Provenance is another: a record that says which source, which commit and which build made a file. A record signed by the build platform is called an attestation.

SLSA, Supply-chain Levels for Software Artifacts, is a public framework that grades how much you can trust that record. Its build track has three levels.

Worked example: the SLSA build levels

LevelWhat it asksWhat it helps stop
Build L1Provenance exists and says how the file was builtMistakes, and not knowing what a release came from
Build L2A hosted build platform makes and signs the provenanceChanges to a file or its record after the build
Build L3A hardened platform keeps builds apart and keeps the signing key from the build stepsTampering during the build, even by a build step

The cells students fill in are left blank in this preview.

Signatures and an SBOM complete the picture

A signature proves who published a file. Sigstore is a free public service for this. Its tool cosign signs a container image or a file. It uses a short-lived certificate tied to an account and records the signature in a public log. This lab has no copy of cosign, so you only read about it here.

An SBOM lists every component and version inside a product. When an advisory appears, you search the SBOMs instead of guessing.

About this lab and its source: SQL injection lab: attack a weak shop search, fix the query and prove the fix

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.