Thumbs that nobody presses
A chat feature shows a thumbs up and a thumbs down under every answer. After a month, one person in fifty has pressed anything. The team has no idea what goes wrong.
Feedback is only useful if people give it. That depends on how little it costs them.
By the end of this lesson
Note
I can collect feedback that people will actually give.
Key words
| Word | What it means | Example |
|---|---|---|
| Feedback | what a person tells you about an answer | |
| Reason | why a person was not happy | too long |
| Response rate | the share of people who answer | |
| Prompt | the question that asks for feedback | |
| Backlog | the list of fixes waiting to be done |
The cells students fill in are left blank in this preview.
Make the first answer one tap
Ask for a vote first. It takes one tap. Only after a thumbs down, offer a short list of reasons to tap: wrong, too long, not what I asked.
A free text box on its own gets almost nothing. A list of reasons gets a reason from many more people, and it can be counted.
Count it, then act on the biggest
Count the reasons from thumbs down. Fix the biggest first. Tell people when you did, because people who see their feedback used give more of it.
Do not ask every time. A rule such as "ask after every fifth answer" keeps the cost low.
A common mistake
Warning
Many people think more feedback is always better, so they ask on every answer. People then stop answering. Ask rarely and keep it to one tap. Judge the design by its response rate: the share of people who answered.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the feedback counter, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const VOTES = [' ' { vote: \"up\", reason: \"\" },' ' { vote: \"down\", reason: \"wrong\" },' ' { vote: \"down\", reason: \"too long\" },' ' { vote: \"up\", reason: \"\" },' ' { vote: \"down\", reason: \"wrong\" },' ' { vote: \"down\", reason: \"\" },' ' { vote: \"up\", reason: \"\" },' ' { vote: \"down\", reason: \"wrong\" },' ' { vote: \"down\", reason: \"too long\" },' ' { vote: \"up\", reason: \"\" },' '];' > feedback.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the feedback counter, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function countReasons(votes) {' ' const counts = {};' ' for (const v of votes) {' ' if (v.vote === \"down\" && v.reason.length > 0) {' ' counts[v.reason] = (counts[v.reason] || 0) + 1;' ' }' ' }' ' return counts;' '}' >> feedback.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the feedback counter, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const counts = countReasons(VOTES);' ' const downs = VOTES.filter(function (v) { return v.vote === \"down\"; }).length;' ' const named = Object.values(counts).reduce(function (a, b) { return a + b; }, 0);' ' console.log(\"votes \" + VOTES.length + \" down \" + downs + \" with a reason \" + named);' ' console.log(\"response rate \" + Math.floor((named * 100) / downs) + \" percent\");' ' console.log(\"reasons \" + JSON.stringify(counts));' ' const msg = \"Write one short question asking why an answer was not helpful.\";' ' const reply = await ask([{ role: \"user\", content: msg }], \"What was wrong with this answer?\", 25);' ' console.log(\"question: \" + reply.text);' '}' 'main();' >> feedback.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the feedback counter
docker exec -w /home/node/project agent node feedback.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| votes 10 down 6 with a reason 5 | Six thumbs down, five gave a reason |
| response rate 83 percent | 5 of 6 people gave one, rounded down |
| reasons {"wrong":3,"too long":2} | Wrong is the biggest, so fix it first |
The cells students fill in are left blank in this preview.