When the conversation gets too long
Trimming a chat history means sending the model fewer messages so that the list fits inside its limit. It matters because a long chat grows with every turn, and every message costs time and credit.
After an hour with a travel chatbot, the first questions are no longer needed. Dropping them keeps the chat fast and cheap.
By the end of this lesson
Note
I can keep a conversation inside the model's limit.
Key words
| Word | What it means |
|---|---|
| Token | a small piece of text that a model counts |
| Context limit | the most tokens a model accepts in one call |
| Budget | the most tokens you choose to send |
| Trim | to drop old messages until the list fits |
| Summary | a short text that stands in for dropped messages |
The cells students fill in are left blank in this preview.
Count the tokens before you send
A model counts text in tokens, and it accepts only so many in one call. Real counts come from the model's own tokenizer. A rough estimate is one token for every four characters.
This lesson adds four for each message, for the role and the framing. Add up all the messages and compare the total with your budget.
Worked example: counting six messages
| Message | Characters | Tokens |
|---|---|---|
| system: You help with train travel. Be brief. | 37 | 14 |
| user: How long is the train from Leeds to York? | 41 | 15 |
| assistant: About 25 minutes. | 17 | 9 |
| user: And a ticket? | 13 | 8 |
| assistant: It depends on the time of day. | 30 | 12 |
| user: Is there a quiet carriage? | 26 | 11 |
| Total | 164 | 69 |
The cells students fill in are left blank in this preview.
Drop the oldest, keep the rules
To trim, remove the oldest messages first until the total fits the budget. Two messages stay whatever the budget says: the system message at the start, and the newest message at the end.
The system message holds the rules. The newest message is the question being asked now.
A summary stands in for dropped messages
Dropping loses facts. A summary keeps the main ones in a few words. Ask the model to summarise the dropped messages, then add the summary to the system message.
This costs one extra call each time you trim. For a short chat, plain dropping is often enough.
A common mistake
Warning
Many people think the oldest message is always the least important. It is often the system message with the rules, or the one fact the chat depends on. Keep the system message, and summarise before you drop anything that matters.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the trimming script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const BUDGET = 45; // a small budget so that trimming happens' '' 'function tokens(message) { return Math.ceil(message.content.length / 4) + 4; }' 'function total(list) {' ' return list.reduce(function (sum, m) { return sum + tokens(m); }, 0);' '}' > trim2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the trimming script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function trim(list, budget) {' ' const kept = list.slice();' ' const dropped = [];' ' while (total(kept) > budget && kept.length > 2) {' ' const at = kept[0].role === \"system\" ? 1 : 0;' ' dropped.push(kept.splice(at, 1)[0]);' ' }' ' return { kept: kept, dropped: dropped };' '}' >> trim2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the trimming script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const history = [' ' { role: \"system\", content: \"You help with train travel. Be brief.\" },' ' { role: \"user\", content: \"How long is the train from Leeds to York?\" },' ' { role: \"assistant\", content: \"About 25 minutes.\" },' ' { role: \"user\", content: \"And a ticket?\" },' ' { role: \"assistant\", content: \"It depends on the time of day.\" },' ' { role: \"user\", content: \"Is there a quiet carriage?\" },' '];' >> trim2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the trimming script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const result = trim(history, BUDGET);' ' console.log(\"Tokens before: \" + total(history) + \", after: \" + total(result.kept) + \", dropped: \" + result.dropped.length);' ' const old = result.dropped.map(function (m) { return m.role + \": \" + m.content; }).join(\" \");' ' // Recorded samples, used only when no model answers.' ' const first = \"The user asked how long the Leeds to York train takes and was told about 25 minutes.\";' ' const summary = await ask([{ role: \"user\", content: \"Summarise this in one short sentence: \" + old }], first, 50);' ' console.log(\"Summary:\", summary);' ' const prompt = result.kept.slice();' ' prompt[0] = { role: \"system\", content: prompt[0].content + \" Earlier in this chat: \" + summary };' ' console.log(\"Reply:\", await ask(prompt, \"Yes, most trains have a quiet carriage.\", 50));' '}' 'main();' >> trim2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Trim and summarise
docker exec -w /home/node/project agent node trim2.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| Tokens | before: 69, after: 45, dropped: 2 (always the same, nothing is guessed) |
| Summary | One sentence about the train time that was dropped |
| Reply | A short answer about a quiet carriage |
The cells students fill in are left blank in this preview.