The same source should give the same bytes
A reproducible build is one where the same source code, built with the same tools, gives exactly the same output, bit for bit. It matters because anyone can then rebuild a release and check that nobody changed it on the way.
In March 2024 a backdoor was found in versions 5.6.0 and 5.6.1 of xz-utils, a compression library in many Linux systems. Part of it sat in the published release archives and not in the code most people read.
By the end of this lesson
Note
I can check whether a build is reproducible.
Key words
| Word | What it means | Example |
|---|---|---|
| Reproducible build | a build that gives the same bytes every time from the same source | |
| Hash | a short fingerprint of some bytes that changes if any byte changes | sha256sum |
| SOURCE_DATE_EPOCH | a fixed time, in seconds since 1970, that a build uses instead of the clock | SOURCE_DATE_EPOCH=1709283600 |
| Provenance | a record of what was built, from which source, by which build | |
| Signature | proof, made with a key, that a file comes from someone and is unchanged | |
| SBOM | a software bill of materials: a list of every component in a product |
The cells students fill in are left blank in this preview.
The one rule
Warning
The lab is a shop that is insecure on purpose. Work in your own private copy and nothing else. It has no way out to the internet. Never break into a system that is not yours without written permission. In the UK that is a crime under the Computer Misuse Act 1990.
Why a build that repeats itself is safer
A build turns source code into the file you ship: an archive, a program or a container image. Users download that file, not the source.
If two people build the same source and get the same hash, nobody slipped anything in between the source and the file. If the hashes differ, something changed, and you can look for it. A build that differs every time hides that signal.
Build the same two files twice
Your lab can make a tiny release with tar, which packs files into one archive, and sha256sum, which prints its hash. The next Run writes two small source files into your shop's container.
The Run after it packs them twice. Between the two builds it only changes the files' times, by one hour, as if the source were checked out later.
Write two source files
docker exec app sh -c "mkdir -p /tmp/build/src && printf 'print(42)\n' > /tmp/build/src/main.py && printf 'shop 1.0\n' > /tmp/build/src/VERSION && cat /tmp/build/src/VERSION"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Pack them twice, an hour apart
docker exec app sh -c "cd /tmp/build/src && touch -d '2024-03-01 09:00:00' main.py VERSION && tar -cf - main.py VERSION | sha256sum && touch -d '2024-03-01 10:00:00' main.py VERSION && tar -cf - main.py VERSION | sha256sum"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Read the times inside the archive
docker exec app sh -c "cd /tmp/build/src && touch -d '2024-03-01 10:00:00' main.py VERSION && tar -cf - main.py VERSION | tar -tvf -"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
One hour changed the whole hash
The first Run prints shop 1.0. The second prints two hashes: one starts 860a6728 and the other starts 711f4dc5. The bytes of the two files never changed.
The third Run lists the archive. Each line shows the file's mode, its owner, its size and then its time, 2024-03-01 10:00:00. The archive stores that time, so a new time is a new archive.
A common mistake
Warning
Many people think the same source always builds the same file. It does not, unless every input is fixed: the times, the order, the paths and the tools. Treat a build as reproducible only after two separate builds give the same hash.
Worked example: what makes two builds differ
| Source of difference | Example | Usual fix |
|---|---|---|
| Timestamps | Each file keeps the time it was last changed | Set every time to SOURCE_DATE_EPOCH |
| File order | The folder lists files in a different order | Sort the names before packing |
| Build path | The archive stores /home/ana/shop or /tmp/shop | Pack relative names from inside the folder |
| Locale | Sorting puts capitals first in one setting and not in another | Sort with LC_ALL=C |
| Versions | The next install picks a newer library | Pin every version in a lock file |
The cells students fill in are left blank in this preview.
Fix the time and the order
SOURCE_DATE_EPOCH is a shared convention: a build tool that sees it uses that time instead of the clock. Many compilers and packers read it already. Here the script sets every file's time to it with touch.
Order matters too. The next Run packs the same files at the same time, once in each order. Then you write a small build script that sorts the names first.
Same files, other order
docker exec app sh -c "cd /tmp/build/src && touch -d '2024-03-01 10:00:00' main.py VERSION && tar -cf - main.py VERSION | sha256sum && tar -cf - VERSION main.py | sha256sum"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Write a build script that fixes both
docker exec app sh -c "printf '%s\n' 'cd /tmp/build/src' 'touch -d @\$SOURCE_DATE_EPOCH *' 'tar -cf - \$(ls | LC_ALL=C sort) | sha256sum' > /tmp/build/build.sh && cat /tmp/build/build.sh"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Build with SOURCE_DATE_EPOCH
docker exec -e SOURCE_DATE_EPOCH=1709283600 app sh /tmp/build/build.shShown here, not run. Press Start lab to type it into your own lab’s terminal.
Touch every file, then build again
docker exec -e SOURCE_DATE_EPOCH=1709283600 app sh -c "touch /tmp/build/src/main.py /tmp/build/src/VERSION && sh /tmp/build/build.sh"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Two builds, one hash
The order Run prints 711f4dc5 and then dcedcb05: same bytes, same times, other order. The script Run prints its three lines.
Both builds then print a hash starting 73367b62, although the second first touched every file to the current time. LC_ALL=C sort puts VERSION before main.py, because capitals sort first in that setting. That is why the locale is fixed too.
One byte is enough to change a hash
import hashlib
# Change the 2 in second to a 3 and press Run.
first = b"print(42)\n"
second = b"print(42)\n"
print(hashlib.sha256(first).hexdigest()[:16])
print(hashlib.sha256(second).hexdigest()[:16])
print("same build:", hashlib.sha256(first).digest() == hashlib.sha256(second).digest())
Provenance says how a file was built
Reproducing a build is one check. Provenance is another: a record that says which source, which commit and which build made a file. A record signed by the build platform is called an attestation.
SLSA, Supply-chain Levels for Software Artifacts, is a public framework that grades how much you can trust that record. Its build track has three levels.
Worked example: the SLSA build levels
| Level | What it asks | What it helps stop |
|---|---|---|
| Build L1 | Provenance exists and says how the file was built | Mistakes, and not knowing what a release came from |
| Build L2 | A hosted build platform makes and signs the provenance | Changes to a file or its record after the build |
| Build L3 | A hardened platform keeps builds apart and keeps the signing key from the build steps | Tampering during the build, even by a build step |
The cells students fill in are left blank in this preview.
Signatures and an SBOM complete the picture
A signature proves who published a file. Sigstore is a free public service for this. Its tool cosign signs a container image or a file. It uses a short-lived certificate tied to an account and records the signature in a public log. This lab has no copy of cosign, so you only read about it here.
An SBOM lists every component and version inside a product. When an advisory appears, you search the SBOMs instead of guessing.