A thumbs down is a free test case
A feedback loop turns what users say about answers into cases you test every change against. It matters because users find the questions you never thought of, and a thumbs down with a correction is the best test case there is.
It also holds personal words, so you must decide who may see it and how long it is kept.
By the end of this lesson
Note
I can turn what users say about answers into better cases.
Key words
| Word | What it means |
|---|---|
| Feedback | a rating or a correction a user gives an answer |
| Correction | the answer the user says it should have been |
| Case | a question with the answer you expect |
| Retention | how long a record is kept before it is deleted |
| Access | who is allowed to read a record |
The cells students fill in are left blank in this preview.
A correction becomes a case
Not every piece of feedback is a case. A thumbs down with no correction says something is wrong but not what is right. A thumbs up has nothing to fix.
Keep the question and the corrected answer. Leave out who said it. A case that holds a user id or an email address is personal data you now have to protect.
Worked example: three pieces of feedback
| Rating | Correction | Becomes a case? |
|---|---|---|
| down | At 6 pm on Sundays. | Yes: the question and that answer |
| down | (none) | No: nothing to expect |
| up | (none) | No: nothing to fix |
The cells students fill in are left blank in this preview.
Decide who sees it and how long
Raw feedback is for the few people who improve the feature. The cases made from it can go to the whole team, as long as no question names a person.
Set a retention age and delete raw feedback past it. In the Run the age is a number in each record, so everyone sees the same result. A real system compares dates.
A common mistake
Warning
Many people think that more feedback kept for longer is always better. Raw feedback holds what users typed, which may be personal. Keep the case, delete the rest on a schedule, and write down how long.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the feedback loop, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' 'const KEEP_DAYS = 30; // feedback older than this is deleted' 'const feedback = [' ' { id: \"f1\", rating: \"down\", question: \"When does the shop close?\", fix: \"At 6 pm on Sundays.\", user: \"u17\", age: 3 },' ' { id: \"f2\", rating: \"up\", question: \"Where is my order?\", fix: \"\", user: \"u22\", age: 5 },' ' { id: \"f3\", rating: \"down\", question: \"Can I return shoes?\", fix: \"\", user: \"u31\", age: 12 },' ' { id: \"f4\", rating: \"down\", question: \"Do you ship to Wales?\", fix: \"Yes, in 3 days.\", user: \"u40\", age: 45 },' '];' > feedback.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the feedback loop, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// A case holds the question and the expected answer, never the user.' 'function toCase(f) {' ' if (f.rating === \"down\" && f.fix.length > 0) return { id: f.id, question: f.question, expected: f.fix };' ' return null;' '}' >> feedback.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the feedback loop, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' feedback.forEach(function (f) {' ' if (f.age > KEEP_DAYS) {' ' console.log(f.id + \": older than \" + KEEP_DAYS + \" days, deleted\");' ' return;' ' }' ' const c = toCase(f);' ' console.log(f.id + \": \" + (c ? \"new case: \" + c.question + \" -> \" + c.expected : \"no case\"));' ' });' ' const q = \"In one sentence, what should a test of this case check? Question: When does the shop close? Expected: At 6 pm on Sundays.\";' ' const r = await ask([{ role: \"user\", content: q }], \"It should check that the answer says 6 pm and that it mentions Sundays.\", 60);' ' console.log(\"One possible check: \" + r.text);' '}' 'main();' >> feedback.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the feedback loop
docker exec -w /home/node/project agent node feedback.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| f1: new case: When does the shop close? -> ... | A recent down with a correction |
| f2: no case and f3: no case | A thumbs up, and a down with no correction |
| f4: older than 30 days, deleted | The retention rule applied |
| One possible check: ... | Your own call. It differs each run, or says Recorded sample |
The cells students fill in are left blank in this preview.