A checklist before an agent ships
A safety review checks an agent's design against a checklist before it is used. Each item is a question with a yes or no answer, so two reviewers get the same result.
That matters because the lessons in this module only help if they are all in place. A design with a good folder but no way to stop it is still unsafe.
By the end of this lesson
Note
I can review an agent's design against a checklist.
Key words
| Word | What it means |
|---|---|
| Checklist | a fixed list of yes or no questions |
| Design | what tools, data and limits an agent has |
| Limit | a cap on steps or on cost |
| Stop switch | a way for a person to halt the agent |
| Log | a record of what the agent did |
The cells students fill in are left blank in this preview.
Six areas, eight questions
The checklist covers six areas: tools, data, folders, limits, a way to stop, and a record. Each area has one or more yes or no questions.
A design passes only if every question passes. Report each failure with its area, so the fix is easy to find.
Worked example: three designs
| Design | Failures | Which |
|---|---|---|
| helper | 0 | None: it passes |
| builder | 4 | Has a shell tool, writes anywhere, no step limit, no log |
| scout | 5 | No tool list, secrets in the prompt, no limits, no stop |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think one strong control makes a design safe. It does not. Builder has an allowlist and a stop switch but writes anywhere. Safety is layers, and the weakest missing layer is what fails first.
Numbers need an edge
A limit is only a check if it has an edge. Here a run may take 20 steps at most. A design with 20 passes. A design with 21, or with no limit at all, fails.
Say where the edge is, test it on both sides, and treat a missing value as a failure.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the design checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const checks = [' ' [\"tools: no allowlist\", function (d) { return d.allowlist === false; }],' ' [\"tools: has shell\", function (d) { return d.tools.indexOf(\"shell\") >= 0; }],' ' [\"folders: writes are not confined\", function (d) { return d.canWrite && (d.root === \"\" || d.root === \"/\"); }],' ' [\"data: secrets in the prompt\", function (d) { return d.secrets; }],' ' [\"limits: no step limit\", function (d) { return d.maxSteps === null || d.maxSteps > 20; }],' ' [\"limits: no cost limit\", function (d) { return d.maxCost === null; }],' ' [\"stop: no way to stop\", function (d) { return d.stop === false; }],' ' [\"record: no log\", function (d) { return d.log === false; }],' '];' 'function failures(d) { return checks.filter(function (c) { return c[1](d); }).map(function (c) { return c[0]; }); }' > review.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the design checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const designs = [' ' { name: \"helper\", tools: [\"read\", \"write\"], allowlist: true, canWrite: true, root: \"work/\", secrets: false, maxSteps: 10, maxCost: 5, stop: true, log: true },' ' { name: \"builder\", tools: [\"read\", \"write\", \"shell\"], allowlist: true, canWrite: true, root: \"/\", secrets: false, maxSteps: null, maxCost: 20, stop: true, log: false },' ' { name: \"scout\", tools: [\"search\"], allowlist: false, canWrite: false, root: \"\", secrets: true, maxSteps: 50, maxCost: null, stop: false, log: true },' '];' >> review.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the design checker, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' designs.forEach(function (d) {' ' const f = failures(d);' ' console.log(d.name + \": \" + f.length + \" failures\" + (f.length > 0 ? \": \" + f.join(\"; \") : \"\"));' ' });' ' const q = \"In one short sentence, why must an agent have a way to be stopped?\";' ' const sample = \"If it goes wrong, a person needs to be able to halt it at once and see what it did.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> review.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the design checker
docker exec -w /home/node/project agent node review.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| helper: 0 failures | It passes every check |
| builder: 4 failures: tools: has shell; ... | It fails in four areas |
| scout: 5 failures: tools: no allowlist; ... | It fails the most |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.