Keeping a decision log for an AI feature
A decision log is a short list of the choices behind a feature, each with its options, its choice and its evidence. It matters because in six months nobody remembers why the team kept the small model.
Fernside Library's team has made five decisions in four weeks. A script now checks that each one points at a record that exists.
By the end of this lesson
Note
I can record the decisions behind a feature so others can follow them.
Key words
| Word | What it means | Example |
|---|---|---|
| Decision log | a dated list of choices and the reasons for them | |
| Option | one of the ways you could have gone | |
| Choice | the option you took | |
| Evidence | a record that shows why you chose | eval:v4 |
| Revisit | to look again when the evidence changes |
The cells students fill in are left blank in this preview.
A decision has options, a choice and evidence
An entry says what was decided, which options were open, which one was taken and what showed it was right. It has a date and an id.
The options matter most. A decision with one option was not a decision. The reader needs to see what you gave up.
Worked example: decision D4
| Field | Value |
|---|---|
| Decision | Cap attempts at 2 |
| Options | Cap at 2, or keep 5 |
| Choice | Cap at 2 |
| Evidence | event:cause_found and log:11 |
The cells students fill in are left blank in this preview.
Evidence is a pointer a script can check
Write evidence as an id: eval:v4 for a run, log:11 for an hour, event:cause_found for an event. A script can then check that each one exists.
It can also catch an entry with no evidence, a choice that is not among the options and an id used twice.
Add an entry, never rewrite one
Revisit a decision when the evidence changes. Do not edit the old entry. Add a new one that says what it replaces.
The log then shows what you believed and when, which is what a reader needs.
A common mistake
Warning
Many people think a decision log is paperwork to write afterwards. Written afterwards, it becomes a story that fits the result. Write each entry on the day, with the evidence you have that day.
Write the decision log, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const DECISIONS = [' ' { id: \"D1\", date: \"2026-03-02\", decision: \"Put the fact sheet in the prompt\", options: [\"in the prompt\", \"search first\"], choice: \"in the prompt\", evidence: [\"eval:v1\"] },' ' { id: \"D2\", date: \"2026-03-09\", decision: \"Forbid figures that are not on the sheet\", options: [\"add a rule\", \"do nothing\"], choice: \"add the rule\", evidence: [\"eval:v2\"] },' ' { id: \"D3\", date: \"2026-03-16\", decision: \"Keep the small model\", options: [\"small model\", \"bigger model\"], choice: \"small model\", evidence: [\"eval:v5\", \"cost:v5\"] },' > decisions.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the decision log, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' { id: \"D4\", date: \"2026-03-24\", decision: \"Cap attempts at 2\", options: [\"cap at 2\", \"keep 5\"], choice: \"cap at 2\", evidence: [\"event:cause_found\", \"log:11\"] },' ' { id: \"D5\", date: \"2026-03-25\", decision: \"Accept long answers about printing\", options: [\"accept\", \"shorten\"], choice: \"accept\", evidence: [] },' '];' 'fs.writeFileSync(\"decisions.json\", JSON.stringify(DECISIONS, null, 1));' >> decisions.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Create decisions.json
docker exec -w /home/node/project agent node decisions.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the log checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'function known() {' ' const ids = [];' ' [\"v1\", \"v2\", \"v3\", \"v4\", \"v5\"].forEach(function (v) { ids.push(\"eval:\" + v); });' ' [\"08\", \"09\", \"10\", \"11\", \"12\", \"13\", \"14\"].forEach(function (h) { ids.push(\"log:\" + h); });' ' [\"sheet_published\", \"alert_fired\", \"cause_found\", \"attempts_capped\", \"cost_normal\"].forEach(function (c) { ids.push(\"event:\" + c); });' ' return ids;' '}' > check-log.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the log checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function run() {' ' const entries = JSON.parse(fs.readFileSync(\"decisions.json\", \"utf8\"));' ' const ids = known();' ' const seen = [];' ' let bad = 0;' ' for (const e of entries) {' ' const problems = [];' ' if (e.evidence.length === 0) problems.push(\"no_evidence\");' ' else if (e.evidence.some(function (id) { return ids.includes(id) === false; })) problems.push(\"unknown_evidence\");' ' if (e.options.includes(e.choice) === false) problems.push(\"bad_choice\");' ' if (seen.includes(e.id)) problems.push(\"duplicate_id\");' ' seen.push(e.id);' ' if (problems.length > 0) { bad = bad + 1; console.log(e.id, problems.join(\" \")); }' ' }' ' console.log(entries.length + \" decisions, \" + bad + \" with problems\");' '}' 'module.exports = { run };' 'if (require.main === module) run();' >> check-log.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Check the log
docker exec -w /home/node/project agent node check-log.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the checker prints
| Line | What it says |
|---|---|
| D2 | bad_choice |
| D3 | unknown_evidence |
| D5 | no_evidence |
| Last line | 5 decisions, 3 with problems |
The cells students fill in are left blank in this preview.
Write the fix
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const entries = JSON.parse(fs.readFileSync(\"decisions.json\", \"utf8\"));' 'entries[1].choice = \"add a rule\"; // the choice is one of the options' 'entries[2].evidence = [\"eval:v5\"]; // cost:v5 is not a record' 'entries[4].evidence = [\"eval:v4\"]; // v4 still has the long answer, c9' 'fs.writeFileSync(\"decisions.json\", JSON.stringify(entries, null, 1));' 'require(\"./check-log\").run();' > fix-log.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Mend the log and check again
docker exec -w /home/node/project agent node fix-log.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.