Treat a prompt like code you ship
A prompt changes what a feature does, so it needs the care code gets. That means a file, a version, a review, a gradual rollout and a way back. It matters because a one-word edit can lift one case and break another, and nobody sees it without a test.
A team edits a prompt in a dashboard on Friday afternoon. Monday brings complaints. Nobody can say what changed, or put it back.
By the end of this lesson
Note
I can version, review and roll out a prompt like code.
Key words
| Word | What it means | Example |
|---|---|---|
| Prompt version | a named, saved state of a prompt | v2 |
| Eval case | one input with a pass or fail rule | |
| Regression | a case that passed before and fails now | |
| Staged rollout | serving a change to a share of users | |
| Bucket | a number from 0 to 99 for one user |
The cells students fill in are left blank in this preview.
A prompt lives in a versioned file
Keep each prompt in a file in the repository, with a name and a version. A change is then a commit, a review and a diff. Code refers to a prompt by name and version, never by a pasted string.
The version is written into every log record. When an answer is bad, you can say which prompt made it.
Judge a change against recorded eval results
Run the same eval cases on the old and the new prompt. Compare the pass rate and list the regressions: cases that passed and now fail. Two prompts with the same pass rate can still differ in which cases pass.
Mark some cases as must pass, such as a refusal that protects a user. A change that fails one does not ship, whatever the rate.
Worked example: v1 against v2
| Case | v1 | v2 | Note |
|---|---|---|---|
| t1 (must) | pass | pass | Kept |
| t2 (must) | pass | pass | Kept |
| t3 | fail | pass | Fixed |
| t4 | pass | pass | Kept |
| t5 | pass | fail | Regression: a person must look |
The cells students fill in are left blank in this preview.
Roll out by a deterministic bucket
Do not switch every user at once. Turn a user id into a bucket from 0 to 99 with a fixed function. A user is served the new prompt when the bucket is below the rollout percent.
The same user always gets the same version, and raising the percent only adds users. Never use a random number per request: one person would see both versions.
Keep a way back
Keep the old version in the repository and the rollout percent in one setting. To go back, set the percent to zero. No deploy, no edit, no argument.
Raise the percent in steps and watch the same measures at each step: pass rate on live samples, cost and complaints.
A common mistake
Warning
Many people think a higher pass rate means a safe change. It can hide a swap: one case fixed, another broken. Read the list of regressions and check the cases that must pass, not only the rate.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt release, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const { ask } = require(\"./model\");' '// A prompt lives in a file with a version, so a change is a diff.' 'const file = { name: \"summary\", version: \"v2\", text: \"Summarise the ticket in one sentence. Name the product.\" };' 'fs.writeFileSync(\"prompt-summary.json\", JSON.stringify(file, null, 1));' '// Recorded eval results: did each case pass? Invented for the lesson.' 'const oldRun = { t1: true, t2: true, t3: false, t4: true, t5: true };' 'const newRun = { t1: true, t2: true, t3: true, t4: true, t5: false };' 'const MUST = [\"t1\", \"t2\"]; // cases that may never fail' > prompts.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt release, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function rate(run) {' ' const names = Object.keys(run);' ' const passed = names.filter(function (n) { return run[n]; });' ' return Math.floor(passed.length * 100 / names.length);' '}' 'function reasons(a, b) {' ' const out = [];' ' if (rate(b) < rate(a)) out.push(\"lower pass rate\");' ' MUST.forEach(function (n) { if (b[n] === false) out.push(\"must fail: \" + n); });' ' return out;' '}' 'const slipped = Object.keys(oldRun).filter(function (n) { return oldRun[n] && newRun[n] === false; });' 'console.log(\"old\", rate(oldRun), \"new\", rate(newRun));' 'console.log(\"may ship:\", reasons(oldRun, newRun).length === 0, \"slipped:\", slipped.join(\",\"));' >> prompts.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt release, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function bucket(user) {' ' let h = 0;' ' for (const c of user) h = (h * 31 + c.charCodeAt(0)) % 1000003;' ' return h % 100;' '}' 'const rollout = { prompt: \"v2\", fallback: \"v1\", percent: 20 };' 'function version(user) { return bucket(user) < rollout.percent ? rollout.prompt : rollout.fallback; }' '[\"ben\", \"eli\", \"ida\", \"ana\", \"dara\", \"fay\"].forEach(function (u) { console.log(u, bucket(u), version(u)); });' 'rollout.percent = 0; // the way back is one number' 'console.log(\"after rollback:\", version(\"ben\"), version(\"eli\"));' >> prompts.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt release, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const text = file.text + \" Ticket: The Orbit phone will not charge since Monday.\";' ' const sample = \"The Orbit phone has not charged since Monday.\";' ' console.log(\"Live call with v2:\", await ask([{ role: \"user\", content: text }], sample, 40));' '}' 'main();' >> prompts.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Judge and roll out a prompt
docker exec -w /home/node/project agent node prompts.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What is recorded and what is real
The eval results are recorded and invented. A real team runs its own cases and stores the results beside the prompt version. The bucket function and the rollout setting are the real technique.
The Run writes prompt-summary.json. Open it and read the version.
One possible output
| Line | What it shows |
|---|---|
| old 80 new 80 | The pass rate is the same |
| may ship: true slipped: t5 | The rule allows it, and a person still reads t5 |
| ben 19 v2 and the other users | Each user has a fixed bucket and version |
| after rollback: v1 v1 | Percent zero sends everybody to v1 |
The cells students fill in are left blank in this preview.