Own the machine or pay by the call
Self-hosting a model means running it on machines you look after. Renting means paying someone else for each call. Each has a cost that rises in a different way.
That matters because the cheaper choice changes with how many calls you make. A sum tells you where the line is, and a list tells you what the sum forgot.
By the end of this lesson
Note
I can decide whether to run a model myself.
Key words
| Word | What it means |
|---|---|
| Self-host | to run the model on your own machines |
| Rent | to pay someone else for each call |
| Break-even | the volume where two costs are equal |
| Capacity | the calls one machine can do in a month |
| Outage | a time when the service is down |
The cells students fill in are left blank in this preview.
Two costs that rise differently
Renting costs the same for every call, so it rises in a straight line. Hosting is a step cost: each machine holds a fixed number of calls, and you pay for a whole machine at a time. You also pay for the people who look after it.
The break-even is the volume where the two are equal. Above it, hosting is cheaper.
Invented prices, pounds a month
| Calls a month | Rent | Host |
|---|---|---|
| 300,000 | 900 | 2,400 |
| 1,000,000 | 3,000 | 3,300 |
| 1,200,000 | 3,600 | 3,300 |
| 2,000,000 | 6,000 | 5,100 |
The cells students fill in are left blank in this preview.
What the sum leaves out
A price sum is not a decision. It leaves out the outage when the machine fails at night. It leaves out the time to update the software and the machine that sits idle. It also leaves out the cost of a mistake with private data.
It also leaves out the other side: some data may not leave your building, and then renting is not allowed at any price.
A common mistake
Warning
Many people think that once they own the machine, each call is free. The machine, the people and the outages are still paid for every month. Work out the break-even with the steps and the people in, then list what is still left out.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cost sum, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const price = 3;' 'const machine = 900;' 'const capacity = 600000;' 'const people = 1500;' 'function rent(calls) {' ' return Math.floor(calls * price / 1000);' '}' 'function host(calls) {' ' return Math.ceil(calls / capacity) * machine + people;' '}' > host.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cost sum, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"calls a month | rent | host | cheaper\");' ' for (const calls of [300000, 600000, 1000000, 1200000, 2000000]) {' ' const pick = host(calls) < rent(calls) ? \"host\" : \"rent\";' ' console.log(calls + \" | \" + rent(calls) + \" | \" + host(calls) + \" | \" + pick);' ' }' ' console.log(\"simple break-even: \" + Math.ceil((machine + people) * 1000 / price) + \" calls\");' >> host.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the cost sum, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const q = \"In one short sentence, name one cost of running your own model that a price per call leaves out.\";' ' const sample = \"Someone has to be on call when the machine fails, and their time is not in the price per call.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> host.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the cost sum
docker exec -w /home/node/project agent node host.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| 300000 | 900 | 2400 | rent | Low volume: rent wins |
| 1000000 | 3000 | 3300 | rent | The second machine is bought, so host jumps |
| 1200000 | 3600 | 3300 | host | Now host wins |
| simple break-even: 800000 calls | Too early: it ignores the steps |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.