Limits and cost of coding agents
A coding agent works in steps, and each step costs time and money. Without a budget and a stop rule, an agent can loop for an hour on a task you could do in ten minutes.
Every change it makes still needs your review, and that takes your time too. A few simple numbers show when a task is worth giving to an agent.
By the end of this lesson
Note
I can set a budget and a stop rule for a coding agent and decide whether a task is worth giving it.
Key words
| Word | What it means | Example |
|---|---|---|
| Step | one action of an agent: read, edit or run | |
| Budget | the most steps or money a run may use | |
| Stop rule | a test that ends a run early | |
| Patience | steps allowed with no progress | 3 |
| Net minutes | minutes saved after writing and reviewing |
The cells students fill in are left blank in this preview.
A budget and a stop rule
Give every run a budget: at most 10 steps and at most 40 pence. Also give it patience: if the number of passing tests has not risen for 3 steps, stop.
Check after every step, in a fixed order: done, over budget, no progress, out of steps. The first one that is true ends the run.
Worked example: the stuck run, step by step
| Step | Cost | Spent | Passing of 6 | Steps since a rise |
|---|---|---|---|---|
| 1 | 4 | 4 | 2 | 0 |
| 2 | 5 | 9 | 3 | 0 |
| 3 | 6 | 15 | 3 | 1 |
| 4 | 5 | 20 | 3 | 2 |
| 5 | 6 | 26 | 3 | 3, so stop: no progress |
The cells students fill in are left blank in this preview.
Time saved against time spent
An agent does not save the minutes a task takes by hand. You still spend minutes writing the task and reviewing the change. Net minutes are the minutes by hand, take away both.
Keep a task when the net is 0 or less. Keep it too when it is high risk, such as a payment check or deleting data, whatever the minutes say.
Worked example: four tasks
| Task | By hand | Spec plus review | Net | Decision |
|---|---|---|---|---|
| Rename a function in 12 files | 40 | 15 | 25 | Delegate |
| Add a docs page | 30 | 15 | 15 | Delegate |
| Fix a one-line typo | 2 | 5 | -3 | Keep |
| Change the payment check | 60 | 40 | 20 | Keep: high risk |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think an agent's time is free because it works while they do something else. Your own time goes on the task, the review and the fixes. Count those minutes before you delegate, and keep high-risk work whatever the count says.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const LIMIT = { steps: 10, pence: 40, patience: 3 };' 'const runs = {' ' \"steady\": { total: 6, cost: [3, 4, 3, 5, 4, 3], passing: [1, 2, 3, 4, 5, 6] },' ' \"stuck\": { total: 6, cost: [4, 5, 6, 5, 6, 7, 6, 5], passing: [2, 3, 3, 3, 3, 3, 3, 3] },' ' \"expensive\": { total: 8, cost: [9, 12, 10, 11, 9], passing: [1, 2, 3, 4, 5] },' '};' > budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function stop(r) {' ' let spent = 0;' ' let best = 0;' ' let rose = 0;' ' for (let i = 0; i < r.cost.length; i += 1) {' ' const step = i + 1;' ' spent += r.cost[i];' ' if (r.passing[i] > best) { best = r.passing[i]; rose = step; }' ' if (best === r.total) return \"done at step \" + step + \", \" + spent + \" pence\";' ' if (spent > LIMIT.pence) return \"over budget at step \" + step + \", \" + spent + \" pence\";' ' if (step - rose >= LIMIT.patience) return \"no progress at step \" + step + \", \" + spent + \" pence\";' ' if (step >= LIMIT.steps) return \"out of steps at step \" + step + \", \" + spent + \" pence\";' ' }' ' return \"record ended, \" + spent + \" pence\";' '}' >> budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const tasks = [' ' { name: \"rename a function in 12 files\", byHand: 40, spec: 5, review: 10, risk: \"low\" },' ' { name: \"add a docs page\", byHand: 30, spec: 5, review: 10, risk: \"low\" },' ' { name: \"fix a one-line typo\", byHand: 2, spec: 3, review: 2, risk: \"low\" },' ' { name: \"change the payment check\", byHand: 60, spec: 10, review: 30, risk: \"high\" },' '];' 'function decide(t) {' ' if (t.risk === \"high\") return \"keep: high risk\";' ' const net = t.byHand - t.spec - t.review;' ' return net > 0 ? \"delegate, saves \" + net + \" minutes\" : \"keep: saves nothing\";' '}' >> budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' Object.keys(runs).forEach(function (name) { console.log(name + \": \" + stop(runs[name])); });' ' tasks.forEach(function (t) { console.log(t.name + \": \" + decide(t)); });' ' const q = \"In one short sentence, why should a coding agent stop when it makes no progress?\";' ' const sample = \"A run that is not getting closer only spends more money and gives you more to review.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the budget script
docker exec -w /home/node/project agent node budget.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| steady: done at step 6, 22 pence | Every test passes inside the budget |
| stuck: no progress at step 5, 26 pence | Three steps with no rise |
| expensive: over budget at step 4, 42 pence | It passed 40 pence |
| rename a function in 12 files: delegate, saves 25 minutes | Worth giving to an agent |
| fix a one-line typo: keep: saves nothing | Faster by hand |
| change the payment check: keep: high risk | Kept whatever the minutes say |
| model said: ... | A sentence from the lab. A Recorded sample line if no model answers |
The cells students fill in are left blank in this preview.