The first version of an AI feature
The first version of an AI feature is the simplest one you can measure: one prompt, one model and a small set of test questions. It matters because you cannot improve what you cannot score.
Fernside Library now has a brief. The team builds a help assistant that answers from a nine-line fact sheet, then scores it on ten questions.
By the end of this lesson
Note
I can build the simplest version that can be measured.
Key words
| Word | What it means | Example |
|---|---|---|
| Fact sheet | the short list of facts the assistant may use | |
| Test case | one question with the answer it must hold | c4 |
| Eval | a set of test cases and a rule that marks each | |
| Check | the rule that marks one answer pass or fail | |
| Baseline | the first score, the one later versions beat |
The cells students fill in are left blank in this preview.
One prompt, one model, one sheet
The simplest version has four parts. A system prompt with a rule. A fact sheet put inside it. One model. One check on the answer.
It has no search, no memory and no second model. Each missing part is a part that cannot fail yet. You add one only when a score says you need it.
Worked example: version 1
| Part | In the first version |
|---|---|
| Prompt | Answer in one short sentence |
| Fact sheet | Nine facts, pasted into the prompt |
| Model | One, chosen by the platform |
| Check | The right figure, no other figure, 40 words |
The cells students fill in are left blank in this preview.
An eval is test cases and a rule
A test case is a question and the figure the answer must hold. The check marks each answer with the same rule every time.
The check here finds four kinds of failure. A refusal. An invented figure that is not on the sheet. A vague answer without the right figure. A long answer over 40 words.
Start with ten cases
Ten cases is small on purpose. You can read every answer, and you will see why each failed.
The cases come from what people ask, not from what is easy. When a real failure appears later, you add it as case 11. The eval grows from evidence.
A common mistake
Warning
Many people think a good demo is an eval. A demo is one case you chose. An eval is many cases you did not choose, marked by the same rule every time. A demo shows what can work. An eval shows how often it does.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the eval, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const SHEET = \"Loan 3 weeks. Renew 2 times. Fine 20 pence a day. Open 9 to 17. Limit 8 items. New card 5 pounds. Room 4 hours. Print 10 pence a page. Under 18 joins free.\";' 'const LIMIT = 40; // the most words an answer may have' 'const SHEET_NUMBERS = SHEET.match(/[0-9]+/g);' 'const RAW = [' ' [\"c1\", \"How long can I keep a book?\", \"3\", \"You can keep a book for 3 weeks.\", 8, false],' ' [\"c2\", \"Can I renew a book?\", \"2\", \"Yes, you can renew a book 2 times.\", 8, false],' ' [\"c3\", \"What is the fine for a late book?\", \"20\", \"The fine is 50 pence a day.\", 7, false],' ' [\"c4\", \"When do you open?\", \"9\", \"We open at 9 every day.\", 6, false],' ' [\"c5\", \"When do you close?\", \"17\", \"We close in the evening.\", 5, false],' > eval.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the eval, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' [\"c6\", \"How many books can I borrow?\", \"8\", \"You can borrow 8 items at once.\", 7, false],' ' [\"c7\", \"How much is a new card?\", \"5\", \"A new card costs 7 pounds.\", 6, false],' ' [\"c8\", \"Can I book a study room for a day?\", \"4\", \"I cannot help with that one. Please ask the desk.\", 10, true],' ' [\"c9\", \"How much does printing cost?\", \"10\", \"Printing costs 10 pence a page. Black and white and colour cost the same. You pay at the desk by card or with coins. The printers stand by the east window, next to the large tables, and the scanners there are free to use for everybody who holds a library card or a day pass.\", 55, false],' ' [\"c10\", \"Can children join?\", \"18\", \"Anyone under 18 gets a free card.\", 7, false],' '];' 'const CASES = RAW.map(function (r) {' ' return { id: r[0], q: r[1], expect: r[2], reply: r[3], length: r[4], refused: r[5] };' '});' >> eval.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the eval, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function kind(c) {' ' if (c.refused) return \"refusal\";' ' const found = c.reply.match(/[0-9]+/g) || [];' ' for (const n of found) {' ' if (SHEET_NUMBERS.includes(n) === false) return \"invented\";' ' }' ' if (found.includes(c.expect) === false) return \"vague\";' ' if (c.length > LIMIT) return \"long\";' ' return \"ok\";' '}' 'const counts = {};' 'for (const c of CASES) {' ' const k = kind(c);' ' counts[k] = (counts[k] || 0) + 1;' ' console.log(c.id, k);' '}' 'console.log(\"Recorded run v1:\", counts.ok, \"of\", CASES.length, \"passed\");' 'console.log(JSON.stringify(counts));' 'require(\"fs\").writeFileSync(\"eval-v1.json\", JSON.stringify(CASES, null, 1));' >> eval.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the eval, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function live() {' ' const system = \"Answer in one short sentence, using only this sheet: \" + SHEET;' ' const messages = [{ role: \"system\", content: system }, { role: \"user\", content: \"When do you open?\" }];' ' const reply = await ask(messages, \"We open at 9 every day.\", 40);' ' const found = reply.match(/[0-9]+/g) || [];' ' console.log(\"Live question:\", reply);' ' console.log(\"Holds the figure 9:\", found.includes(\"9\"), \"(not part of the score)\");' '}' 'live();' >> eval.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Score version 1
docker exec -w /home/node/project agent node eval.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it says |
|---|---|
| c1 to c10 | One kind each: ok, invented, vague, refusal or long |
| Recorded run v1 | 5 of 10 passed |
| JSON line | 5 ok, 2 invented, 1 vague, 1 refusal, 1 long |
| Live question | One short call, never part of the score |
The cells students fill in are left blank in this preview.