Test a prompt against fixed cases
A prompt test runs a fixed set of cases and applies a rule to each reply. The rule is plain: a word must appear, another must not, the reply must be short.
Without a test, a changed prompt that fixes the case in front of you may quietly break two others you did not look at.
By the end of this lesson
Note
I can say whether a changed prompt is better.
Key words
| Word | What it means |
|---|---|
| Case | one input you test with |
| Rule | a plain check on a reply |
| Baseline | the prompt you compare against |
| Regression | a case that passed and now fails |
| Pass | a reply that meets its rule |
| Recorded | a reply saved so a test can repeat |
The cells students fill in are left blank in this preview.
What is a case and a rule?
A case is one input, such as a passenger asking for a refund. A rule is a check you can run in code on the reply. "Contains the word refund" is a rule. "Sounds friendly" is not, because code cannot check it.
Keep a small fixed set of cases. Run every prompt against the same set, so the only thing that changed is the prompt.
Prompt A and prompt B on three cases
| Case | Prompt A | Prompt B |
|---|---|---|
| refund | pass | fail (says sorry) |
| delay | fail | pass |
| thanks | pass | fail (mentions delay) |
| Total | 2 of 3 | 1 of 3 |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt tester, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const cases = [' ' { name: \"refund\", must: [\"refund\"], not: [\"sorry\"], max: 25 },' ' { name: \"delay\", must: [\"delay\"], not: [], max: 25 },' ' { name: \"thanks\", must: [\"thank\"], not: [\"delay\"], max: 25 },' '];' 'const A = [\"We will refund you within five days.\", \"The bus is late, we apologise.\", \"Thank you for your message.\"];' 'const B = [\"Sorry, we will refund you within five days.\", \"There is a delay of ten minutes.\", \"Thank you. Sorry for the delay.\"];' > test5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt tester, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function passes(reply, c) {' ' const low = reply.toLowerCase();' ' const long = reply.split(\" \").length > c.max;' ' const has = c.must.every(function (m) { return low.includes(m); });' ' const bad = c.not.some(function (n) { return low.includes(n); });' ' return has && bad === false && long === false;' '}' 'function run(replies) {' ' return replies.map(function (r, i) { return passes(r, cases[i]); });' '}' >> test5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the prompt tester, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const ra = run(A);' ' const rb = run(B);' ' const names = function (flags, other) {' ' return cases.filter(function (c, i) { return flags[i] && other[i] === false; }).map(function (c) { return c.name; });' ' };' ' console.log(\"A passes \" + ra.filter(Boolean).length + \" of 3, B passes \" + rb.filter(Boolean).length + \" of 3\");' ' console.log(\"helped: \" + names(rb, ra).join(\", \") + \" | hurt: \" + names(ra, rb).join(\", \"));' ' const ask1 = \"Reply to a passenger who wants a refund, in under 25 words.\";' ' const reply = await ask([{ role: \"user\", content: ask1 }], A[0], 50);' ' console.log(\"live reply passes the refund rule: \" + passes(reply.text, cases[0]));' '}' 'main();' >> test5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run and compare two prompts
docker exec -w /home/node/project agent node test5.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
The first line says A passes 2 of 3 and B passes 1 of 3. The second says B helped the delay case and hurt refund and thanks. So you keep A.
Those two lines use recorded replies, so every student sees them. The last line checks one live reply and may be true or false. It shows a real reply can fail a rule. With no model it prints a Recorded sample first.
A common mistake
Warning
Many people think a prompt is better if it fixed the case that annoyed them. One case is one sample. Run the whole set and count what helped and what hurt before you decide.