A tuned model can go stale
Model drift is a model getting worse over time with nobody changing it. The world moves on. New products, new slang and new kinds of ticket arrive, and the model was tuned on last year.
If nobody watches the score, the first sign is an unhappy customer.
By the end of this lesson
Note
I can notice when a tuned model has gone stale.
Key words
| Word | What it means |
|---|---|
| Drift | a model getting worse as the world changes |
| Monitoring | checking a score again and again |
| Trigger | a sign that says act now |
| Schedule | a fixed date to act, whatever happened |
| Retraining | tuning again on newer examples |
The cells students fill in are left blank in this preview.
Watch a score every week
Keep a small set of fresh cases and score the model on them every week. Record the numbers. The first weeks after a release give you a baseline.
One bad week can be chance. Ask for two weeks in a row below your alert line before you say the score has fallen.
Worked example: weekly scores
| Weeks | Scores | What it shows |
|---|---|---|
| 1 to 3 | 88, 90, 89 | Baseline 89 (the average, rounded down) |
| 4 to 6 | 88, 87, 86 | Small slip, above the alert line of 84 |
| 7 and 8 | 80, 79 | Two weeks below 84: the score has fallen |
| 9 to 12 | 78, 79, 77, 76 | It stays down |
The cells students fill in are left blank in this preview.
Look at the inputs for the cause
When the score falls, read the new inputs. Do they hold words, products or topics the training data never had? List the words that are new.
That tells you what examples to add. A long list of new words is a strong sign that the inputs have changed.
A trigger or a schedule
A schedule retrains on a fixed date, such as every fourth week. It is simple, but it can retrain when nothing changed, or too late when something did.
A trigger retrains when the score falls. It acts on evidence. Many teams use the trigger first and keep the schedule as a back-stop.
A common mistake
Warning
Many people think a tuned model stays good once it passes its tests. The tests only describe the day they were run. Keep scoring fresh cases, and retrain and compare again when the score falls.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the drift check, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// Recorded weekly scores for a ticket sorter, in percent. Week 1 is the first.' 'const scores = [88, 90, 89, 88, 87, 86, 80, 79, 78, 79, 77, 76];' 'const THRESHOLD = 5;' > adapt6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the drift check, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function baseline(list) {' ' return Math.floor((list[0] + list[1] + list[2]) / 3);' '}' 'function findDrop(list, drop) {' ' const cut = baseline(list) - drop;' ' for (let i = 3; i < list.length - 1; i++) {' ' if (list[i] < cut && list[i + 1] < cut) return i + 1;' ' }' ' return 0;' '}' >> adapt6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the drift check, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function newWords(oldTexts, newTexts) {' ' const seen = oldTexts.join(\" \").toLowerCase().split(\" \");' ' const out = [];' ' for (const w of newTexts.join(\" \").toLowerCase().split(\" \")) {' ' if (seen.includes(w) === false && out.includes(w) === false) out.push(w);' ' }' ' return out;' '}' >> adapt6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the drift check, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"baseline \" + baseline(scores) + \"%, alert below \" + (baseline(scores) - THRESHOLD) + \"%\");' ' console.log(\"score fell in week \" + findDrop(scores, THRESHOLD));' ' console.log(\"new words: \" + newWords([\"where is my parcel\"], [\"where is my drone parcel\", \"locker code\"]).join(\" \"));' ' const q = \"In one sentence, why can a model get worse when nobody has changed it?\";' ' const sample = \"The world and the inputs change, so a model tuned on old inputs fits new ones less well.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 40);' ' console.log(\"model: \" + reply.text);' '}' 'main();' >> adapt6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the drift check
docker exec -w /home/node/project agent node adapt6.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the drift check printed
The twelve scores are recorded and written into the script. The check works out the baseline, finds the week the score fell and lists the words that are new in a few invented messages.
The last line is one short model call, or a Recorded sample.