How a context budget works
A context budget is the number of tokens you let each part of a prompt use. A model can only read so much at once, so something has to give when the parts add up to too much.
The module on cost, tracing and reliability budgets money and calls. This lesson budgets the window itself and goes further: which part goes first, and what you tell the user was left out.
By the end of this lesson
Note
I can decide what fits in a window and what goes first.
Key words
| Word | What it means | Example |
|---|---|---|
| Token | a small piece of text that a model counts | About one word |
| Window | the most tokens a model can read and write in one call | |
| Budget | the tokens one part of the prompt may use | |
| Priority | how important a part is. 0 is never dropped | |
| Trim | to cut a part down so it fits | |
| Reserve | room kept free for the reply |
The cells students fill in are left blank in this preview.
The reply needs room too
The window is shared by the prompt and the reply. If the prompt fills it, the reply is cut off or the call fails.
So first keep room for the reply. Then keep a small safety margin, because your count is only an estimate. What is left is the prompt budget.
Worked example: the prompt budget
| Step | Tokens | Why |
|---|---|---|
| Window | 4000 | The most the model can handle |
| Reserve for the reply | 500 | The reply needs space |
| Safety, 5 percent | 200 | 5 percent of 4000, in case the count is a little off |
| Prompt budget | 3300 | 4000 minus 500 minus 200 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think the window is only for the prompt. It also holds the reply. A prompt that fills the whole window leaves no room to answer.
Which part goes first?
Give every part a priority. The rules and the user question are priority 0: they are never dropped. If they do not fit, the prompt is broken and you must say so.
Then take the rest in order of importance. Keep a part if it fits. If it does not, drop it, but a smaller part later on may still fit.
Worked example: a budget of 3300
| Part | Tokens | Priority | Total so far | Result |
|---|---|---|---|---|
| Rules | 300 | 0 | 300 | Kept |
| Question | 100 | 0 | 400 | Kept |
| Document A | 900 | 1 | 1300 | Kept |
| Document B | 800 | 2 | 2100 | Kept |
| Document C | 700 | 2 | 2800 | Kept |
| History | 1400 | 3 | 4200 would go over | Dropped |
The cells students fill in are left blank in this preview.
Trim instead of dropping all
Dropping the history loses the whole conversation. A better move is to trim it: keep the newest messages that fit.
Walk from the newest message back. Stop at the first one that does not fit, because a gap in the middle of a conversation makes no sense.
Worked example: 500 tokens left
| Message | Tokens | Total | Result |
|---|---|---|---|
| Newest | 200 | 200 | Kept |
| One before | 250 | 450 | Kept |
| Two before | 300 | 750 would go over | Stop here |
| Oldest | 40 | Never reached | Dropped, even though it would fit |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function tokens(text) { return text.split(\" \").length; }' 'const parts = [' ' { name: \"rules\", tokens: 60, priority: 0 },' ' { name: \"question\", tokens: 25, priority: 0 },' ' { name: \"doc A\", tokens: 120, priority: 1 },' ' { name: \"doc B\", tokens: 90, priority: 2 },' ' { name: \"history\", tokens: 150, priority: 3 },' ' { name: \"doc C\", tokens: 80, priority: 2 },' '];' > budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function assemble(parts, budget) {' ' const order = parts.slice().sort(function (a, b) { return a.priority - b.priority; });' ' const kept = [];' ' let used = 0;' ' for (const p of order) {' ' if (p.priority === 0 || used + p.tokens <= budget) { kept.push(p.name); used = used + p.tokens; }' ' }' ' return { kept: kept, used: used };' '}' >> budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' for (const budget of [350, 600]) {' ' const r = assemble(parts, budget);' ' const names = parts.map(function (p) { return p.name; });' ' console.log(\"budget \" + budget + \": kept \" + names.filter(function (n) { return r.kept.includes(n); }).join(\", \"));' ' console.log(\" dropped \" + (names.filter(function (n) { return r.kept.includes(n) === false; }).join(\", \") || \"none\") + \", used \" + r.used);' ' }' >> budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const q = \"In one short sentence, what should a prompt lose first when it is too long?\";' ' const sample = \"Lose the oldest or least useful parts first, and never the rules or the question.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' ' console.log(\"my count \" + tokens(q) + \", model count \" + (reply.inTokens === null ? \"not reported\" : reply.inTokens));' '}' 'main();' >> budget.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the budget
docker exec -w /home/node/project agent node budget.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| budget 350: kept rules, question, doc A, doc B | The parts that fit, in their first order |
| dropped history, doc C, used 295 | What was left out and the tokens used |
| budget 600: kept ... and dropped none | With room, every part goes in |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
| my count 15, model count ... | My word count beside the model count, or not reported |
The cells students fill in are left blank in this preview.