Stop a feature spending too much
A budget for an LLM feature turns tokens into money and refuses a call that would go over a limit. It matters because cost grows with use, and one busy user or one loop can spend a month of budget in an afternoon.
A model is billed by the token, and nobody sees the bill until the end of the month. A limit in your own code is the only brake that works at once.
By the end of this lesson
Note
I can stop a feature spending more than it should, with a per-user limit, a daily limit and a clear message.
Key words
| Word | What it means |
|---|---|
| Token | the unit a model reads and writes, and is billed in |
| Price list | the cost of a million tokens, in and out |
| Per-user limit | the most one account may spend in a day |
| Daily limit | the most the whole feature may spend in a day |
| Ledger | the running record of what has been spent |
| Refusal | turning a call away, with a reason |
The cells students fill in are left blank in this preview.
Turn tokens into money
A model has two prices: one for tokens in, one for tokens out, usually per million. A call costs the input tokens times the first price plus the output tokens times the second.
Never use floating point for money. Count in a small whole unit. Here, euros per million tokens means each token costs that many micro-euros, so the sums are whole numbers. The prices in this lesson are invented.
Worked example: two calls
| Model | In | Out | Sum | Cost |
|---|---|---|---|---|
| small, price 1 and 4 | 900 | 300 | 900 x 1 + 300 x 4 | 2100 micro-euros |
| large, price 10 and 40 | 200 | 100 | 200 x 10 + 100 x 40 | 6000 micro-euros |
The cells students fill in are left blank in this preview.
Check a limit before you charge
Keep a ledger: the total for each user and the total for the day. Before a call is charged, check it. Refuse it if it takes the user over their limit, or the day over its limit. A total exactly at a limit is allowed.
Ask the user limit first, because it has the more helpful message. A refused call is not charged. And the refusal says why, in words a person can act on.
A common mistake
Warning
Many people think a limit is only for abuse, so they set it after launch. Costs also grow from a bug, such as a retry loop or a long prompt sent to every user. Set both limits before the first user arrives.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' '// An invented price list: euros per million tokens. So tokens times price is micro-euros.' 'const PRICE = { small: { in: 1, out: 4 }, large: { in: 10, out: 40 } };' 'const USER_LIMIT = 5000; // micro-euros for one user in a day' 'const DAY_LIMIT = 8000; // micro-euros for everyone in a day' 'const spent = { user: new Map(), day: 0 };' 'function cost(model, inTokens, outTokens) {' ' return inTokens * PRICE[model].in + outTokens * PRICE[model].out;' '}' > budget3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function check(user, price) {' ' const mine = spent.user.get(user) || 0;' ' if (mine + price > USER_LIMIT) return \"user\";' ' if (spent.day + price > DAY_LIMIT) return \"day\";' ' return \"ok\";' '}' 'function charge(user, price) {' ' spent.user.set(user, (spent.user.get(user) || 0) + price);' ' spent.day = spent.day + price;' '}' 'function message(why, user) {' ' if (why === \"user\") return \"Sorry \" + user + \", this would take you over your allowance for today. It resets at midnight.\";' ' return \"The assistant has reached its daily limit for everyone. Please try again tomorrow.\";' '}' >> budget3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function demo() {' ' const prompt = \"Say hello in five words.\";' ' const reply = await ask([{ role: \"user\", content: prompt }], \"Hello, how can I help?\", 20);' ' const inT = reply.inTokens === null ? Math.ceil(prompt.length / 4) : reply.inTokens;' ' const outT = reply.outTokens === null ? Math.ceil(reply.text.length / 4) : reply.outTokens;' ' const how = reply.inTokens === null ? \"estimated\" : \"counted\";' ' console.log(\"Your call: \" + inT + \" in, \" + outT + \" out (\" + how + \"), \" + cost(\"small\", inT, outT) + \" micro-euros\");' '}' >> budget3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const calls = [' ' { user: \"ana\", model: \"small\", in: 900, out: 300 },' ' { user: \"ben\", model: \"small\", in: 1000, out: 500 },' ' { user: \"ana\", model: \"large\", in: 200, out: 100 },' ' { user: \"cy\", model: \"large\", in: 300, out: 50 },' ' { user: \"cy\", model: \"small\", in: 500, out: 100 },' ' { user: \"dan\", model: \"large\", in: 200, out: 50 },' '];' >> budget3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget, part 5
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' await demo();' ' calls.forEach(function (c, at) {' ' const price = cost(c.model, c.in, c.out);' ' const why = check(c.user, price);' ' const label = \"#\" + (at + 1) + \" \" + c.user + \" \" + c.model + \" \" + price + \" micro-euros: \";' ' if (why === \"ok\") { charge(c.user, price); console.log(label + \"ok\"); }' ' else console.log(label + message(why, c.user));' ' });' ' console.log(\"Spent today: \" + spent.day + \" micro-euros = \" + (spent.day / 1000000).toFixed(6) + \" euros\");' '}' 'main();' >> budget3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the budget
docker exec -w /home/node/project agent node budget3.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| Your call: 42 in, 17 out (counted)... | Your own call. Counted when the reply carries usage, estimated when it does not. It differs each run |
| #1 ana small 2100 micro-euros: ok | Charged |
| #3 ana large 6000 micro-euros: Sorry ana... | Refused by the user limit, with a message |
| #4 cy large 5000 micro-euros: The assistant... | Refused by the daily limit |
| Spent today: 6000 micro-euros = 0.006000 euros | Only the charged calls count |
The cells students fill in are left blank in this preview.