What an agent step in CI is
Continuous integration (CI) runs checks on every change pushed to a repository. A team can add a coding agent as one step, for example to fix typos in the docs.
It matters because nobody watches a pipeline step as it runs. So its limits, its tests and a person's approval are written down before it ever runs.
By the end of this lesson
Note
I can design a pipeline step for a coding agent with a person's approval before the merge.
Key words
| Word | What it means | Example |
|---|---|---|
| Pipeline | steps that run in order on every change | |
| Permission | something a step is allowed to do | write-branch |
| Least privilege | only the permissions a step needs | |
| Approval gate | a point where a person must say yes | |
| Log | a record of what the step did |
The cells students fill in are left blank in this preview.
An agent step gets the fewest permissions
A step can be allowed to read the code, write to a branch, write to the main branch or read the team's secrets. An agent step needs to read and to write a branch. It never needs the main branch or the secrets.
It also lists the folders it may change and a time limit. A star means "anything", which is no limit at all.
Worked example: permissions for an agent step
| Permission | Agent step | Why |
|---|---|---|
| read | Yes | It must read the code |
| write-branch | Yes | Its change goes on a branch of its own |
| write-main | No | Only a merge a person approved writes main |
| secrets | No | A secret in a prompt can leak |
| may change | docs/ only | The folders its task needs |
| time limit | 30 minutes or less | A stuck agent stops |
The cells students fill in are left blank in this preview.
Tests, a person, then the merge
After the agent step, the tests run again on its branch. Then a person reads the change and approves it, and only then does the merge run.
The step keeps a log: the task it was given, each command it ran and each file it changed. When something goes wrong a month later, the log says what happened.
A common mistake
Warning
Many people think an agent in CI is safe because it only opens a branch. A step that can read secrets or push to main can still do harm. Give it the fewest permissions, and put tests and a person before every merge.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pipeline checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const pipelines = {' ' \"docs bot\": [' ' { name: \"checkout\", kind: \"setup\", permissions: [\"read\"] },' ' { name: \"agent\", kind: \"agent\", permissions: [\"read\", \"write-branch\"], may_change: [\"docs/\"],' ' keeps_log: true, max_minutes: 20 },' ' { name: \"tests\", kind: \"test\", permissions: [\"read\"] },' ' { name: \"merge\", kind: \"merge\", permissions: [\"write-main\"], needs_approval: true },' ' ],' ' \"auto fixer\": [' ' { name: \"agent\", kind: \"agent\", permissions: [\"read\", \"write-main\", \"secrets\"], may_change: [\"*\"],' ' keeps_log: false, max_minutes: 90 },' ' { name: \"merge\", kind: \"merge\", permissions: [\"write-main\"], needs_approval: false },' ' ],' '};' > ci.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pipeline checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function stepProblems(s) {' ' const out = [];' ' if (s.permissions.indexOf(\"write-main\") >= 0) out.push(\"may push to main\");' ' if (s.permissions.indexOf(\"secrets\") >= 0) out.push(\"can read secrets\");' ' if (s.may_change.length === 0 || s.may_change.indexOf(\"*\") >= 0) out.push(\"may change anything\");' ' if (s.keeps_log === false) out.push(\"keeps no log\");' ' if (s.max_minutes > 30) out.push(\"runs too long\");' ' return out.map(function (p) { return s.name + \": \" + p; });' '}' >> ci.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pipeline checker, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function problems(steps) {' ' let out = [];' ' let agent = false;' ' let tested = false;' ' steps.forEach(function (s) {' ' if (s.kind === \"agent\") { out = out.concat(stepProblems(s)); agent = true; tested = false; }' ' if (s.kind === \"test\") tested = true;' ' if (s.kind === \"merge\" && agent && tested === false) out.push(s.name + \": no tests after the agent\");' ' if (s.kind === \"merge\" && agent && s.needs_approval === false) out.push(s.name + \": no approval\");' ' });' ' return out;' '}' >> ci.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pipeline checker, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' Object.keys(pipelines).forEach(function (name) {' ' const found = problems(pipelines[name]);' ' console.log(name + \": \" + (found.length === 0 ? \"safe\" : \"unsafe\"));' ' found.forEach(function (p) { console.log(\" \" + p); });' ' });' ' const q = \"In one short sentence, why should a person approve before an agent change is merged?\";' ' const sample = \"A person can see what the change means, which the tests and the agent cannot judge.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> ci.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the pipeline checker
docker exec -w /home/node/project agent node ci.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| docs bot: safe | Fewest permissions, tests, then an approval |
| auto fixer: unsafe | Seven problems listed under it |
| agent: may push to main | It can skip every person |
| merge: no tests after the agent | Nothing checks its change |
| merge: no approval | Nobody says yes before the merge |
| model said: ... | A sentence from the lab. A Recorded sample line if no model answers |
The cells students fill in are left blank in this preview.