The broken change on Friday
A pipeline as code is a script or file that runs the same checks, in the same order, for every change to a project. It matters because it stops at the first check that fails, so a bad change never reaches the apply.
Somebody edits a file at four o'clock on Friday, pushes it and goes home. Nobody ran the checks, and the next apply fails halfway.
By the end of this lesson
Note
I can write a pipeline as stages that stop at the first failure.
Key words
| Word | What it means | Example |
|---|---|---|
| Pipeline | steps that run in a fixed order | |
| Stage | one step of a pipeline | |
| Exit code | the number a command returns, where 0 means success | echo $? |
| Gate | a stage that stops the pipeline when a check fails | |
| Plan file | a saved plan that apply carries out exactly | plan.bin |
The cells students fill in are left blank in this preview.
Every command ends with an exit code
When a command finishes it returns a number, its exit code. Zero means it worked. Any other number means it did not.
The shell keeps the last one in $?. A pipeline reads exit codes and nothing else, so a check that exits with a non-zero number is a check that stops the run.
Make a folder for the pipeline
mkdir -p ~/iac-demo/pipe && cd ~/iac-demo/pipeShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write a project with one resource
printf '%s\n' 'variable "shop" {' ' type = string' ' default = "Corner Shop"' '}' '' 'resource "terraform_data" "door" {' ' input = format("%s: open", var.shop)' '}' > main.tfShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Prepare the folder
tofu init -no-color | grep successfullyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write a second file with untidy spacing
printf '%s\n' 'resource "terraform_data" "sign" {' ' input = "Open"' '}' > sign.tfShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Check the layout and read the exit code
tofu fmt -check; echo "exit code $?"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A pipeline stops at the first failure
A script can run each stage and stop when one fails. Here one small function, stage, runs a command and exits if it returns anything but zero.
The stages go in the order you would run them by hand: format, validate, plan, gate, apply. A failed stage means the later ones never start.
Write the gate
printf '%s\n' '[.resource_changes[].change.actions[]] | index("delete") | not' > gate.jqShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pipeline script
printf '%s\n' '#!/bin/bash' 'stage() { echo "== stage: $1"; name=$1; shift; "$@" > /dev/null || { echo "stage $name failed, stopping"; exit 1; }; }' 'gate() { tofu show -json plan.bin | jq -e -f gate.jq; }' 'stage format tofu fmt -check -no-color' 'stage validate tofu validate -no-color' 'stage plan tofu plan -no-color -out=plan.bin' 'stage gate gate' 'stage apply tofu apply -no-color plan.bin' 'echo "pipeline passed"' > ci.shShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it while sign.tf is untidy
bash ci.sh; echo "pipeline exit $?"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Fix the layout and run it again
tofu fmt -no-color; bash ci.sh; echo "pipeline exit $?"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a green pipeline means the change is safe. It means every check you wrote passed. A change can be valid, planned and applied, and still be the wrong change. The gate and a person reading the plan are the safety.
A gate reads the plan before apply
The plan stage saved the plan in plan.bin, and the apply stage carries out that file and nothing else. So what the gate approved is what runs.
tofu show -json turns the plan into data. The gate filter says no when any resource is going to be deleted.
Delete a resource and run it
rm sign.tf; bash ci.sh; echo "pipeline exit $?"; tofu state listShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A hosted pipeline is a file
On a service that hosts pipelines, the stages are written in a file in the repository, and the service runs them on its own machine. A job can say needs to wait for another.
You cannot run one here, because there is no hosted runner. You can write one and read it.
Write a workflow file
mkdir -p .github/workflows && printf '%s\n' 'name: infra' 'on:' ' pull_request:' ' push:' ' branches: [main]' 'jobs:' ' check:' ' runs-on: ubuntu-latest' ' steps:' ' - uses: actions/checkout@v4' ' - run: tofu fmt -check' ' - run: tofu validate' ' - run: tofu plan -out=plan.bin' ' apply:' ' needs: check' " if: github.ref == 'refs/heads/main'" ' runs-on: ubuntu-latest' ' steps:' ' - run: tofu apply plan.bin' > .github/workflows/infra.ymlShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Find the order and the commands in it
grep -nE "^ (check|apply):|needs:|run:" .github/workflows/infra.ymlShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A pipeline run worked out in Python
# Run a pipeline over stages that each need earlier ones.
jobs = {"check": [], "plan": ["check"], "apply": ["plan"]}
failing = {"plan"}
state = {}
for name in ["check", "plan", "apply"]:
blocked = False
for need in jobs[name]:
if state[need] != "passed":
blocked = True
if blocked:
state[name] = "skipped"
elif name in failing:
state[name] = "failed"
else:
state[name] = "passed"
print(name, state[name])
# Empty the failing set and look again.