The biggest model is not always best
Choosing a language model means matching a model to a job. The factors are cost, speed, how much text it can read, quality on your own examples, and where it runs.
It matters because the choice repeats on every call. A model ten times dearer than you need, called a million times, is a large bill for no gain.
By the end of this lesson
Note
I can choose a model for a job and say why.
Key words
| Word | What it means |
|---|---|
| Context window | the most text a model can read in one call |
| Latency | how long you wait for a reply |
| Price per token | what each token costs, often counted per million |
| Hosted | run by a provider, reached over the internet |
| Local | run on your own machine |
The cells students fill in are left blank in this preview.
Five things to weigh
Start from the job. A job has a size, such as how much text goes in and how much comes out. It has a delay a person will accept and a budget.
Then ask five questions of each model, as the table shows. A model that fails one of them is out, however good it is at the rest.
The five questions
| Factor | The question to ask |
|---|---|
| Cost | What do the tokens of this job cost? |
| Speed | Is the reply quick enough for a person? |
| Size | Does it read all the text of the job? |
| Quality | Is it good enough on my own examples? |
| Place | May the text leave my machine? |
The cells students fill in are left blank in this preview.
Time the lab model on two jobs
docker exec -w /home/node/project agent node -e 'const lab = require("./lab.js"); const asks = [ { name: "short", max: 10, text: "Name one fruit.", sample: "An apple." }, { name: "longer", max: 100, text: "Describe a rainy day in three sentences.", sample: "Rain taps on the window. The street is quiet and grey." }, ]; (async () => { for (const one of asks) { const started = Date.now(); const got = await lab.ask({ max_tokens: one.max, messages: [{ role: "user", content: one.text }] }, one.sample); const ms = Date.now() - started; if (got.label === "Recorded sample") { console.log(one.name + ": Recorded sample, " + got.text.length + " characters, no timing for a sample."); continue; } const used = got.data.usage ? got.data.usage.completion_tokens + " tokens written" : "no token count in the reply"; console.log(one.name + ": Model answer, " + ms + " ms, " + got.text.length + " characters, " + used + "."); } })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Work out the cost before you call
A price is quoted for every million tokens, one price for tokens in and a higher one for tokens out. The cost of a call is the input tokens times the input price, plus the output tokens times the output price.
Count the tokens of a typical job, work out one call, then multiply by the calls you expect. The price list used here is invented, so only the method is real.
Price one job on three invented models
docker exec -w /home/node/project agent node -e '/* An invented price list. The prices are units for every million tokens. */ const models = [ { name: "Small", inPrice: 20, outPrice: 80, delayMs: 600, context: 8000 }, { name: "Medium", inPrice: 100, outPrice: 400, delayMs: 1500, context: 32000 }, { name: "Large", inPrice: 500, outPrice: 2000, delayMs: 4000, context: 128000 }, ]; const job = { inTokens: 3000, outTokens: 300, maxDelayMs: 2000, budget: 0.5 }; console.log("The job: " + job.inTokens + " tokens in, " + job.outTokens + " tokens out, an answer within " + job.maxDelayMs + " ms."); for (const model of models) { const cost = (job.inTokens * model.inPrice + job.outTokens * model.outPrice) / 1000000; const fits = job.inTokens + job.outTokens <= model.context && model.delayMs <= job.maxDelayMs && cost <= job.budget; console.log(model.name.padEnd(7) + "cost " + cost.toFixed(3) + " units, delay " + model.delayMs + " ms, " + (fits ? "fits" : "does not fit")); }'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: one job, three models
| Model | Cost per call | Delay | Fits? |
|---|---|---|---|
| Small | 0.084 units | 600 ms | Yes |
| Medium | 0.420 units | 1500 ms | Yes |
| Large | 2.100 units | 4000 ms | No, too slow and too dear |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think the largest model is always best. It is usually the slowest and the dearest, and a small one may do a simple job just as well. Test each model on your own examples, then take the cheapest one that passes.
Where a model runs changes the rules
A hosted model is run by a provider. You send text over the internet and pay by the token. A local model runs on your own machine, so the text stays with you, but you need the hardware and it may be slower.
A job with private text may have to stay local. Then the rule is: cheapest, among those that are allowed.
Going further: price your own job
Change the job in the costs script to one you would build. Say how many tokens go in and come out, and how long a person will wait. Run it.
Write which model you would pick and which factor decided it. Say what you would test next before you trusted the choice.