Send each request to the cheapest fit
Routing models means choosing, for each request, the cheapest model that is good enough. It matters because the dearest model costs many times the smallest, and most requests do not need it.
A support tool labels tickets, answers short questions and drafts long replies. Labelling does not need the model that drafts. Sending every request to one big model pays for power that most of them never use.
By the end of this lesson
Note
I can route each request to the cheapest model that is good enough and check the choice against recorded cases.
Key words
| Word | What it means |
|---|---|
| Router | the rule that picks a model for a request |
| Tier | a group of models by size and price |
| Good enough | right often enough for that task |
| Recorded case | a request with the tier a person chose |
| Disagreement | a case where the router and the person differ |
| Wrapper | the code round a model call that adds the rules |
The cells students fill in are left blank in this preview.
A router is a simple rule
Start with a rule you can read. Short text or a labelling task goes to the small tier. Long text or a reasoning task goes to the large tier. The rest goes to medium.
A rule on size and task is cheap, instant and easy to explain. You can also ask a small model to choose, but that adds a call to every request, so try the rule first.
Worked example: the rule on four requests
| Request | Task | Size | Tier |
|---|---|---|---|
| Label a ticket | classify | 150 characters | small |
| Summarise a page | summarise | 1500 characters | medium |
| Plan a migration | reason | 900 characters | large |
| Summarise a long report | summarise | 5000 characters | large |
The cells students fill in are left blank in this preview.
Check the rule against cases
A router saves money and may cost quality, so test it. Keep recorded cases: a request, its size and task, and the tier a person says it needs. Run the router over them. List the disagreements and total the cost.
A disagreement is not always a bug in the rule. Read each one. Then change the rule, or change the case.
A common mistake
Warning
Many people think the cheapest model is always the best saving. A cheap model that gives a poor answer costs a retry, a complaint or a lost user. Route down only where recorded cases show the answer is still good enough.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the router check, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' '// Invented prices: euros per million tokens, so tokens times price is micro-euros.' 'const PRICE = { small: { in: 1, out: 4 }, medium: { in: 3, out: 12 }, large: { in: 10, out: 40 } };' 'function route(request) {' ' if (request.task === \"classify\" || request.chars <= 200) return \"small\";' ' if (request.task === \"reason\" || request.chars > 2000) return \"large\";' ' return \"medium\";' '}' 'function cost(tier, c) { return c.in * PRICE[tier].in + c.out * PRICE[tier].out; }' > route6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the router check, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Recorded cases: what each request is, and the tier a reviewer says it needs.' 'const CASES = [' ' { id: 1, task: \"classify\", chars: 150, in: 60, out: 5, expected: \"small\" },' ' { id: 2, task: \"qa\", chars: 180, in: 70, out: 40, expected: \"small\" },' ' { id: 3, task: \"summarise\", chars: 1500, in: 500, out: 150, expected: \"medium\" },' ' { id: 4, task: \"reason\", chars: 900, in: 400, out: 600, expected: \"large\" },' '];' >> route6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the router check, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const MORE = [' ' { id: 5, task: \"summarise\", chars: 5000, in: 1400, out: 300, expected: \"large\" },' ' { id: 6, task: \"classify\", chars: 3000, in: 800, out: 5, expected: \"small\" },' ' { id: 7, task: \"qa\", chars: 700, in: 250, out: 120, expected: \"medium\" },' ' { id: 8, task: \"summarise\", chars: 190, in: 80, out: 30, expected: \"medium\" },' '];' >> route6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the router check, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const all = CASES.concat(MORE);' ' let total = 0, dear = 0, wrong = [];' ' for (const c of all) {' ' const tier = route(c);' ' total = total + cost(tier, c);' ' dear = dear + cost(\"large\", c);' ' if (tier === c.expected) console.log(\"#\" + c.id + \" \" + c.task + \": \" + tier + \" ok\");' ' else { wrong.push(c.id); console.log(\"#\" + c.id + \" \" + c.task + \": \" + tier + \", expected \" + c.expected); }' ' }' ' console.log(\"Disagreements: \" + wrong.length + \" of \" + all.length + \", cases \" + wrong.join(\", \"));' ' console.log(\"Routed cost: \" + total + \". Every request on large: \" + dear + \". Saved: \" + (dear - total));' >> route6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the router check, part 5
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const reply = await ask([{ role: \"user\", content: \"Label as billing, bug or other: I was charged twice.\" }], \"billing\", 10);' ' console.log(\"Routed to \" + route(all[0]) + \", reply: \" + reply.text.slice(0, 40));' '}' 'main();' >> route6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the router check
docker exec -w /home/node/project agent node route6.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The lab has one model
In the lab one model answers whichever tier the rule picks, and the platform chooses it. So the Run shows the choice and the cost, and the one live call only shows that a routed request is answered.
The prices and the eight cases are invented and recorded, so every student prints the same lines.
One possible output
| Line | What it shows |
|---|---|
| #1 classify: small ok | The router and the person agree |
| #8 summarise: small, expected medium | A disagreement to read and decide on |
| Disagreements: 1 of 8, cases 8 | The count and the case numbers |
| Routed cost: 60820. Every request on large: 85600. Saved: 24780 | The total in micro-euros |
| Routed to small, reply: ... | One live call. The words differ each run |
The cells students fill in are left blank in this preview.