Better at one thing, worse at another
A tuned model scored higher on the job you trained it for. Is it better? Not yet. It may have got worse at something else, or cost twice as much to run.
Judging a tuned model means comparing it with the baseline on the same cases, in every area that matters.
By the end of this lesson
Note
I can say whether a tuned model is better than the baseline.
Key words
| Word | What it means |
|---|---|
| Gain | tuned results minus baseline results |
| Margin | the smallest gain that counts as real |
| Regression | a case the tuned version now gets wrong |
| Area | a kind of case, such as tone or maths |
| Recorded | saved results, not made live |
The cells students fill in are left blank in this preview.
Compare on the same cases
Run both versions over one fixed set of cases that neither saw in training. Count what each gets right. The difference is the gain.
A gain of one case out of eight is too small to trust. Decide the margin you need before you look, so the result cannot talk you round.
Worked example: eight cases, three areas
| Area | Baseline right | Tuned right | Change |
|---|---|---|---|
| tickets | 2 of 4 | 3 of 4 | +1 |
| tone | 1 of 2 | 2 of 2 | +1 |
| maths | 2 of 2 | 1 of 2 | -1 |
| all | 5 of 8 | 6 of 8 | +1 |
The cells students fill in are left blank in this preview.
A better answer can cost more
A tuned model may answer in more words, or need a larger plan to run. Work out the cost rise as a percent of the baseline cost.
Set the most you will accept first. A small gain that doubles the bill is usually not worth shipping.
A common mistake
Warning
Many people think a higher score on the trained task proves the model is better. It only proves it is better there. Check the other areas, count the cases that got worse, and look at the cost before you decide.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// Recorded: 1 means right. The same eight cases for both versions.' 'const cases = [' ' { id: \"c1\", area: \"tickets\", base: 0, tuned: 1 }, { id: \"c2\", area: \"tickets\", base: 0, tuned: 1 },' ' { id: \"c3\", area: \"tickets\", base: 1, tuned: 1 }, { id: \"c4\", area: \"tickets\", base: 1, tuned: 0 },' ' { id: \"c5\", area: \"tone\", base: 0, tuned: 1 }, { id: \"c6\", area: \"tone\", base: 1, tuned: 1 },' ' { id: \"c7\", area: \"maths\", base: 1, tuned: 1 }, { id: \"c8\", area: \"maths\", base: 1, tuned: 0 },' '];' > adapt5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function total(which) {' ' let n = 0;' ' for (const c of cases) n = n + c[which];' ' return n;' '}' 'const worse = cases.filter(function (c) { return c.base === 1 && c.tuned === 0; });' 'const change = {};' 'for (const c of cases) change[c.area] = (change[c.area] || 0) + c.tuned - c.base;' >> adapt5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"base \" + total(\"base\") + \" of 8, tuned \" + total(\"tuned\") + \" of 8\");' ' console.log(\"got worse: \" + worse.map(function (c) { return c.id; }).join(\", \"));' ' console.log(\"change by area: \" + JSON.stringify(change));' ' const q = \"In one sentence, why check areas the tuning was not aimed at?\";' ' const sample = \"Tuning for one job can quietly make other jobs worse, and only a check on those shows it.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 40);' ' console.log(\"model: \" + reply.text);' '}' 'main();' >> adapt5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the comparison
docker exec -w /home/node/project agent node adapt5.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the comparison printed
Both sets of results are recorded and written into the script. The Run counts each version, lists the cases that got worse and shows the change in each area.
The last line is one short model call, or a Recorded sample.