How to review a coding agent's changes
A change set is every file a coding agent added, changed or deleted for one task. An agent can change twenty files in a minute, and reading them all with the same care does not scale.
A risk-first review finds the risky files and reads those first. The safe files are still read, but after the ones that could hide a problem.
By the end of this lesson
Note
I can sort a coding agent's change set by risk before I read it.
Key words
| Word | What it means |
|---|---|
| Change set | every file changed for one task |
| Scope creep | changes beyond what the task asked for |
| Dependency | a package the code needs to run |
| Risk flag | a sign that a change needs care |
| Weight | how many points a flag is worth |
The cells students fill in are left blank in this preview.
Five signs a change needs care
A file the task did not name is scope creep. A deleted test, or a test with fewer lines, can hide a broken feature. A changed config file can change how everything builds.
A very large new file is hard to read. A new dependency brings in code nobody on the team wrote. Each of these is a risk flag.
Worked example: flags and their weights
| Flag | When it is raised | Weight |
|---|---|---|
| test deleted | A test file is deleted | 5 |
| test shrank | A test file loses more lines than it gains | 3 |
| outside scope | The task did not name the file | 3 |
| config | package.json, package-lock.json or ci.yml | 2 |
| large | More than 200 lines added and removed | 1 |
The cells students fill in are left blank in this preview.
Score, sort, then read
Add up the weights of each file's flags to give it a score. Sort the files with the highest score first, and keep ties in the order they came.
Then read the top files in full, line by line. The low files still get read, but they no longer hide the risky ones. A deleted test sends the whole change back.
A common mistake
Warning
Many people think a review means reading every line with the same care. With 400 lines, the risky ten hide among the safe ones. Sort by risk first, then read every risky file in full.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk review script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// The task named two files. The agent handed back seven.' 'const allowed = [\"shop.js\", \"shop.test.js\"];' 'const CONFIG = [\"package.json\", \"package-lock.json\", \"ci.yml\"];' 'const WEIGHT = { \"test deleted\": 5, \"test shrank\": 3, \"outside scope\": 3, \"config\": 2, \"large\": 1 };' > risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk review script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const changes = [' ' { path: \"shop.js\", added: 12, removed: 2, kind: \"modified\" },' ' { path: \"shop.test.js\", added: 1, removed: 9, kind: \"modified\" },' ' { path: \"prices.json\", added: 1, removed: 1, kind: \"modified\" },' ' { path: \"old.test.js\", added: 0, removed: 40, kind: \"deleted\" },' ' { path: \"package.json\", added: 1, removed: 0, kind: \"modified\" },' ' { path: \"README.md\", added: 3, removed: 0, kind: \"modified\" },' ' { path: \"utils/format.js\", added: 230, removed: 0, kind: \"added\" },' '];' 'const before = [\"date-tools\"];' 'const after = [\"date-tools\", \"money-format\"];' >> risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk review script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function flags(c) {' ' const out = [];' ' const isTest = c.path.indexOf(\".test.\") >= 0;' ' if (isTest && c.kind === \"deleted\") out.push(\"test deleted\");' ' if (isTest && c.kind === \"modified\" && c.removed > c.added) out.push(\"test shrank\");' ' if (allowed.indexOf(c.path) < 0) out.push(\"outside scope\");' ' if (CONFIG.indexOf(c.path) >= 0) out.push(\"config\");' ' if (c.added + c.removed > 200) out.push(\"large\");' ' return out;' '}' 'function score(c) {' ' return flags(c).reduce(function (sum, f) { return sum + WEIGHT[f]; }, 0);' '}' >> risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk review script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const order = changes.slice().sort(function (a, b) { return score(b) - score(a); });' ' order.forEach(function (c) {' ' console.log(score(c) + \" \" + c.path + (flags(c).length ? \": \" + flags(c).join(\", \") : \"\"));' ' });' ' after.filter(function (d) { return before.indexOf(d) < 0; })' ' .forEach(function (d) { console.log(\"new dependency: \" + d); });' ' const gone = changes.some(function (c) { return flags(c).indexOf(\"test deleted\") >= 0; });' ' console.log(gone ? \"send back: a test was deleted\" : \"review in this order\");' ' const q = \"In one short sentence, why should a reviewer look at a deleted test file first?\";' ' const sample = \"A deleted test can hide a broken feature, because nothing checks it any more.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the risk review
docker exec -w /home/node/project agent node risk.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| 8 old.test.js: test deleted, outside scope | The riskiest file, read first |
| 5 package.json: outside scope, config | A config file the task never named |
| 4 utils/format.js: outside scope, large | A big new file nobody asked for |
| 3 shop.test.js: test shrank | The named test lost lines |
| 0 shop.js | The one file the task was about |
| new dependency: money-format | New code from outside the team |
| send back: a test was deleted | The decision for the whole change set |
The cells students fill in are left blank in this preview.