Do not pay twice for one answer
Caching an LLM response means storing a reply and returning it for the same request later. It matters because a call costs money and seconds, and many requests repeat.
A help page asks the model to summarise the same policy for every visitor. Calling it a thousand times buys one answer a thousand times. One call and nine hundred and ninety-nine hits does not.
By the end of this lesson
Note
I can cache model answers safely and say exactly what makes a hit.
Key words
| Word | What it means |
|---|---|
| Cache | a store of answers you have already paid for |
| Cache key | the value that decides whether two requests match |
| Hit | a request answered from the cache |
| Miss | a request that has to call the model |
| Expiry | the time after which an entry is dropped |
| Personal data | anything that identifies one person |
The cells students fill in are left blank in this preview.
A key is built from the whole request
Two requests may share an answer only if the model would see the same thing. So the cache key covers everything that changes the reply. That is the model, every message with its role, and the parameters such as temperature and the reply limit.
Leave one out and you serve the wrong answer. A hash of that text keeps the key short.
Worked example: what changes the key
| Change | Same key? | Why |
|---|---|---|
| Same model, messages and parameters | Yes | The model sees the same request |
| One word different in a message | No | The messages are part of the key |
| Temperature 0 and then 1 | No | The replies may differ |
| Another model | No | Another model answers differently |
The cells students fill in are left blank in this preview.
Entries expire, and personal data stays out
An answer goes stale. Give each entry an expiry: a time to live in seconds. After that the entry is dropped and the next request is a miss.
Never cache what is personal. A reply built from one user's address or order could be handed to the next user with the same key. Check the request before the lookup, and skip the cache for it.
A common mistake
Warning
Many people think caching is always safe because the answer is only text. A cached reply that holds one person's details can be shown to another. Decide what may be cached before the first entry is stored.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const crypto = require(\"crypto\");' 'const { ask } = require(\"./model\");' '' 'let now = 0; // seconds, on a clock we move ourselves' 'const TTL = 60; // an entry lives for 60 seconds' 'const cache = new Map();' > cache2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// The key is built from the whole request: model, messages and parameters.' 'function keyOf(request) {' ' const whole = JSON.stringify([request.model, request.messages, request.temperature, request.max_tokens]);' ' return crypto.createHash(\"sha256\").update(whole).digest(\"hex\").slice(0, 12);' '}' '// Personal data is never cached. Here: an email address or a long number.' 'function personal(request) {' ' return /@|[0-9]{7,}/.test(JSON.stringify(request.messages));' '}' >> cache2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function answer(request, sample) {' ' if (personal(request)) return \"skipped, personal\";' ' const key = keyOf(request);' ' const found = cache.get(key);' ' if (found && found.expires > now) return \"hit\";' ' const reply = await ask(request.messages, sample, request.max_tokens);' ' cache.set(key, { text: reply.text, expires: now + TTL });' ' return found ? \"expired, miss\" : \"miss\";' '}' >> cache2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const tip = [{ role: \"user\", content: \"Give one tip about caching, in one sentence.\" }];' ' const base = { model: \"lab\", temperature: 0, max_tokens: 40, messages: tip };' ' const other = { ...base, temperature: 1 };' ' const secret = { ...base, messages: [{ role: \"user\", content: \"Mail [email protected] a tip.\" }] };' ' const sample = \"Cache only what is the same for everyone.\";' ' for (const at of [0, 30, 90]) {' ' now = at;' ' console.log(\"t=\" + at + \" s: \" + (await answer(base, sample)));' ' }' ' console.log(\"same key at another temperature: \" + (keyOf(base) === keyOf(other)));' ' console.log(\"personal request: \" + (await answer(secret, sample)));' '}' 'main();' >> cache2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the cache
docker exec -w /home/node/project agent node cache2.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
This is a stand-in
The lab has no cache server, so the cache is a Map in the script and the clock is a number the script sets. A real system uses a real cache service shared by every server.
The rules are the same: a key from the whole request, an expiry, and a check for personal data.
One possible output
| Line | What it shows |
|---|---|
| t=0 s: miss | The first request calls the model |
| t=30 s: hit | The same request, answered from the cache |
| t=90 s: expired, miss | The entry expired at 60 seconds, so the model is called again |
| same key at another temperature: false | A new parameter is a new key |
| personal request: skipped, personal | The cache was never touched |
The cells students fill in are left blank in this preview.