Cheap per message, dear per problem
A shop adds a chat feature. The team proudly says each message costs almost nothing. The manager asks a different question: how much does it cost to solve one customer problem?
The two numbers can be far apart. Showing that a feature is worth its cost means picking the right one.
By the end of this lesson
Note
I can show whether an AI feature is worth what it costs.
Key words
| Word | What it means |
|---|---|
| Cost per message | total cost divided by every message |
| Cost per resolved task | total cost divided by tasks that were solved |
| Resolved | the person got what they needed |
| Baseline | how the job was done before the feature |
| Hidden cost | a cost that is real but left out |
The cells students fill in are left blank in this preview.
Two costs that are not the same
Add up everything the feature cost. Divide by the number of messages and you get a small number. Divide by the number of problems solved and you get the number that matters.
Unsolved chats still cost money. Module 69 shows how to count tokens and price a call. Here we use those totals.
Compare with how it was done before
A cost means little alone. Compare it with the baseline: how much did it cost, and how long did it take, before the feature?
Count the hidden costs, such as the person who takes over. A feature that looks cheaper can lose once they are counted.
A common mistake
Warning
Many people think the model price is the whole cost. It is only part. A person who takes over and a chat that is not solved both cost money too. Always divide the whole cost by the tasks that were actually solved.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cost checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const PRICE = { inCost: 300, outCost: 1500, human: 2000000 };' 'const RUNS = [' ' { id: \"r1\", tokensIn: 1000, tokensOut: 200, resolved: true, handed: false },' ' { id: \"r2\", tokensIn: 2000, tokensOut: 500, resolved: true, handed: true },' ' { id: \"r3\", tokensIn: 1500, tokensOut: 100, resolved: false, handed: false },' ' { id: \"r4\", tokensIn: 500, tokensOut: 50, resolved: true, handed: false },' '];' > worth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cost checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function callCost(r) {' ' return r.tokensIn * PRICE.inCost + r.tokensOut * PRICE.outCost;' '}' 'function total() {' ' let sum = 0;' ' for (const r of RUNS) sum += callCost(r) + (r.handed ? PRICE.human : 0);' ' return sum;' '}' >> worth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cost checker, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const resolved = RUNS.filter(function (r) { return r.resolved; }).length;' ' console.log(\"total \" + total() + \" micro-cents\");' ' console.log(\"per message \" + Math.floor(total() / RUNS.length));' ' console.log(\"per resolved \" + Math.floor(total() / resolved));' ' console.log(\"resolved \" + resolved + \" of \" + RUNS.length);' ' const msg = \"Write one short line saying a cost per message can look low while a cost per solved problem is high.\";' ' const reply = await ask([{ role: \"user\", content: msg }], \"A low cost per message can hide the cost of problems left unsolved.\", 30);' ' console.log(\"verdict: \" + reply.text);' '}' 'main();' >> worth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the cost checker
docker exec -w /home/node/project agent node worth.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| total 4775000 micro-cents | Four calls plus one person |
| per message 1193750 | Looks low |
| per resolved 1591666 | Higher, because r3 was not solved |
| resolved 3 of 4 | The number we divide by |
The cells students fill in are left blank in this preview.