The biggest model is not always best
Choosing a model size means matching the smallest model that does a job well enough to the job itself. A small model on a narrow job can match a big one for a fraction of the price.
That matters because the price of every call adds up, and so does the wait. You find the right size by testing, never by guessing.
By the end of this lesson
Note
I can pick a model size for a job from a test and a price.
Key words
| Word | What it means |
|---|---|
| Narrow job | one small, clear task |
| Test set | fixed cases used to score a model |
| Pass rate | the share of cases a model gets right |
| Price per call | what each request costs |
| Good enough | a pass rate the job can accept |
The cells students fill in are left blank in this preview.
Test each size on the same cases
Write a test set: a fixed list of real cases for your job, such as ten support tickets to sort. Run each model size over the same cases and record pass or fail for each.
The pass rate is the share it gets right. Module 68 has more on test sets. This lesson uses recorded results, so nothing is run now.
Recorded results on six tickets
| Size | Right out of 6 | Pass rate | Price per 1000 calls |
|---|---|---|---|
| small | 4 | 66.7% | 2 |
| medium | 5 | 83.3% | 9 |
| large | 6 | 100% | 40 |
The cells students fill in are left blank in this preview.
Look at the cases a small model fails
A pass rate hides which cases fail. The small model above fails tickets 5 and 6. Read them. If they are rare in real life, small may still be fine.
Then work out the price per right answer: the price divided by the share that passes. A cheap model that is often wrong can cost more per right answer than a dearer one.
A common mistake
Warning
Many people think the biggest model is the safe choice. It is the dearest, and it is slower. If a smaller size passes your own test set, it is good enough. Choose the cheapest size that passes, and test again when the job changes.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the size chooser, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const recorded = {' ' small: { cost: 2, passed: [1, 1, 1, 1, 0, 0] },' ' medium: { cost: 9, passed: [1, 1, 1, 1, 1, 0] },' ' large: { cost: 40, passed: [1, 1, 1, 1, 1, 1] },' '};' 'function wins(m) {' ' return m.passed.reduce(function (a, b) { return a + b; }, 0);' '}' 'function rate(m) {' ' return wins(m) * 100 / m.passed.length;' '}' > choose.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the size chooser, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function perCorrect(m) {' ' return m.cost * m.passed.length / wins(m);' '}' 'function choose(need) {' ' let best = \"none\";' ' let bestCost = Infinity;' ' for (const name of Object.keys(recorded)) {' ' const m = recorded[name];' ' if (rate(m) >= need && m.cost < bestCost) { best = name; bestCost = m.cost; }' ' }' ' return best;' '}' >> choose.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the size chooser, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"Recorded results on 6 tickets (not run now):\");' ' for (const name of Object.keys(recorded)) {' ' const m = recorded[name];' ' console.log(name + \": \" + rate(m).toFixed(1) + \"% right, price \" + m.cost + \", price per right answer \" + perCorrect(m).toFixed(2));' ' }' ' console.log(\"need 80%: \" + choose(80) + \" | need 95%: \" + choose(95) + \" | need 100%: \" + choose(100));' ' const q = \"In one short sentence, why test a small model on your own cases before choosing it?\";' ' const sample = \"A small model may do well on a narrow job, and only your own cases show where it fails.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> choose.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the size chooser
docker exec -w /home/node/project agent node choose.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| small: 66.7% right, price 2, price per right answer 3.00 | Cheap, and not as dear per right answer as it looks |
| large: 100.0% right, price 40, ... 40.00 | Always right, dearest |
| need 80%: medium | need 95%: large | The need decides the size |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.