Why a model must not change by itself
Pinning a model version means naming the exact version a feature uses. A name that always means the newest one is not a pin.
It matters because a feature that worked on Monday can behave differently on Tuesday with no change from you. A pin, and a check of every config, stop the change nobody chose.
By the end of this lesson
Note
I can stop a model changing under a feature.
Key words
| Word | What it means | Example |
|---|---|---|
| Pin | to name one exact version | |
| Alias | a name that points at a version that can move | chat-latest |
| Config | the file that says which model a feature uses | |
| Deprecation notice | a warning that a version will stop working | |
| Retirement day | the day a version stops answering |
The cells students fill in are left blank in this preview.
An alias can move, a version cannot
A config names the model a feature calls. If it says chat-latest, the supplier decides what that means. One day it points at a new version.
If it says chat-v3, it means that version only. The feature changes when you edit the config and not before.
Worked example: five names
| Name in the config | Pinned? | Why |
|---|---|---|
| chat-v3 | Yes | Ends in v and a whole number |
| chat-v10 | Yes | Ten is a whole number |
| chat-latest | No | Latest is a word, not a number |
| chat | No | No version at all |
| chat-v2b | No | The 2b is not a whole number |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a pin is enough for ever. A pinned version still ends: the supplier retires it on a set day. A pin gives you the time to move on your own terms, and only if you act before the retirement day.
A notice gives you a day to move
A deprecation notice names a version and its retirement day. Days left is that day minus today.
Choose a warning window, such as 30 days. Inside it, a feature is retiring: plan the move now. On or after the day, it is retired and already failing.
Worked example: today is day 100
| Version | Retirement day | Days left | Status with a 30 day window |
|---|---|---|---|
| chat-v1 | 90 | -10 | Retired |
| chat-v2 | 125 | 25 | Retiring |
| chat-v4 | 300 | 200 | ok |
| chat-v5 | no notice | none | ok |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pin checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const today = 100;' 'const configs = [' ' { feature: \"summary\", model: \"chat-v3\" },' ' { feature: \"search\", model: \"chat-latest\" },' ' { feature: \"triage\", model: \"chat-v2\" },' ' { feature: \"tags\", model: \"chat-v1\" },' ' { feature: \"draft\", model: \"chat\" },' '];' 'const notices = { \"chat-v2\": 125, \"chat-v1\": 90 };' > pin.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pin checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function pinned(model) {' ' const tail = model.split(\"-\").pop();' ' const rest = tail.slice(1);' ' return tail.length > 1 && tail[0] === \"v\" && String(Number(rest)) === rest;' '}' 'function status(c) {' ' if (pinned(c.model) === false) return \"UNPINNED\";' ' const end = notices[c.model];' ' if (end === undefined) return \"ok\";' ' if (end <= today) return \"RETIRED\";' ' return end - today <= 30 ? \"RETIRING in \" + (end - today) + \" days\" : \"ok\";' '}' >> pin.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the pin checker, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' let bad = 0;' ' configs.forEach(function (c) {' ' const s = status(c);' ' if (s > \"ok\" || s < \"ok\") bad = bad + 1;' ' console.log(c.feature + \" \" + c.model + \": \" + s);' ' });' ' console.log(\"need work: \" + bad + \" of \" + configs.length);' ' const q = \"In one short sentence, why pin a model to a version?\";' ' const sample = \"A pinned version cannot change under the feature, so you decide when to move.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> pin.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the pin checker
docker exec -w /home/node/project agent node pin.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the checker prints
| Line | What it shows |
|---|---|
| summary chat-v3: ok | Pinned and no notice |
| search chat-latest: UNPINNED | An alias, so it can move |
| triage chat-v2: RETIRING in 25 days | Pinned, but the end is near |
| tags chat-v1: RETIRED | Past its retirement day |
| draft chat: UNPINNED | No version at all |
| need work: 4 of 5 | Counted by rules, not by a model |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.