How to forecast what a feature costs
The cost of an AI feature is the number of calls you pay for times the price of one call. A cache cuts the first number, because a repeated question is answered from memory.
The module on cost, tracing and reliability shows how to make a cache key and when to expire it. This lesson goes further: it counts hits over a week of traffic and forecasts a month.
By the end of this lesson
Note
I can predict what a feature will cost and how much a cache saves.
Key words
| Word | What it means | Example |
|---|---|---|
| Cache | a store of answers you already paid for | |
| Hit | a request answered from the cache | |
| Miss | a request the cache cannot answer, so you pay | |
| Hit rate | the percent of requests that are hits | 50 |
| Expiry | how long a stored answer stays usable | |
| Forecast | a sum that predicts the future from the past |
The cells students fill in are left blank in this preview.
What one call costs
A call costs its input tokens times the input price, plus its output tokens times the output price. Output usually costs more.
To keep the sums in whole numbers, count in units. One unit is a millionth of a pound. The prices here are invented.
Worked example: one call
| Part | Tokens | Price per token | Units |
|---|---|---|---|
| Input | 120 | 3 | 360 |
| Output | 80 | 15 | 1200 |
| One call | 200 | 1560 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a cache saves the same amount for every feature. It saves only as much as the repeated questions allow. A feature where nobody asks the same thing twice saves almost nothing.
Replay traffic through a cache
To measure a hit rate, replay real or seeded requests. Number them from 0. A request is a hit if the same question was stored fewer than ttl requests ago.
A hit does not renew the stored answer. It expires on its own clock, so a popular question still gets a fresh answer once its time runs out.
Worked example: ttl of 3
| Request number | Question | Stored at | Result |
|---|---|---|---|
| 0 | a | Nothing | Miss, store at 0 |
| 1 | a | 0 | Hit, 1 is under 3 |
| 2 | b | Nothing | Miss, store at 2 |
| 3 | a | 0 | Miss, 3 is not under 3, store at 3 |
| 4 | a | 3 | Hit |
The cells students fill in are left blank in this preview.
Forecast a month from a week
Add up one week of requests. A month is about 30 days, so month requests are the week times 30, divided by 7.
You pay only for the misses. Multiply by 100 minus the hit rate, divide by 100, then multiply by the cost of one call.
Worked example: a week of 835 requests
| Step | Sum | Result |
|---|---|---|
| Week total | 120 + 135 + 150 + 160 + 170 + 60 + 40 | 835 |
| Month requests | 835 times 30 divided by 7 | 3578 |
| Paid calls at 50 percent | 3578 times 50 divided by 100 | 1789 |
| Cost in units | 1789 times 1560 | 2790840 |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function next(seed) { return (seed * 75 + 74) % 65537; }' 'const table = [0, 0, 0, 1, 1, 2, 3, 4, 5, 6];' 'function traffic(seed, count) {' ' const out = [];' ' for (let i = 0; i < count; i++) { seed = next(seed); out.push(\"q\" + table[seed % 10]); }' ' return out;' '}' > cache.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function replay(requests, ttl) {' ' const stored = {};' ' let hits = 0;' ' requests.forEach(function (key, i) {' ' if (key in stored && i - stored[key] < ttl) hits = hits + 1;' ' else stored[key] = i;' ' });' ' return hits;' '}' 'const week = [120, 135, 150, 160, 170, 60, 40];' 'const perCall = 120 * 3 + 80 * 15;' >> cache.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const requests = traffic(7, 40);' ' console.log(\"first 8: \" + requests.slice(0, 8).join(\" \"));' ' for (const ttl of [5, 10, 20]) {' ' const hits = replay(requests, ttl);' ' const percent = Math.floor(hits * 100 / 40);' ' const month = Math.floor(week.reduce(function (a, b) { return a + b; }, 0) * 30 / 7);' ' const paid = Math.floor(month * (100 - percent) / 100);' ' console.log(\"ttl \" + ttl + \": \" + hits + \" hits, \" + percent + \" percent, month pays for \" + paid + \" calls, \" + paid * perCall + \" units\");' ' }' >> cache.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cache script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const reply = await ask([{ role: \"user\", content: \"In one short sentence, why can a cache make a feature cheaper?\" }], \"A repeated question is answered from memory, so no new call is paid for.\", 60);' ' console.log(\"model said: \" + reply.text);' ' console.log(\"output tokens: \" + (reply.outTokens === null ? \"not reported\" : reply.outTokens));' '}' 'main();' >> cache.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the cache replay
docker exec -w /home/node/project agent node cache.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| first 8: q6 q6 q0 q2 q0 q0 q5 q0 | The seeded traffic, the same for everyone |
| ttl 5: 16 hits, 40 percent | A short expiry gives fewer hits |
| ttl 20: 27 hits, 67 percent | A longer expiry gives more hits and a smaller bill |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
| output tokens: ... | The count the reply reports, or not reported |
The cells students fill in are left blank in this preview.